Skip to content

Instantly share code, notes, and snippets.

@andrewberls
Created August 23, 2014 04:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save andrewberls/797cf76e3fe33101bf14 to your computer and use it in GitHub Desktop.
Save andrewberls/797cf76e3fe33101bf14 to your computer and use it in GitHub Desktop.
Aug 23 00:46:04 app-3 kernel: [958008.758126] Killed process 3599 (ruby) total-vm:12240780kB, anon-rss:402668kB, file-rss:0kB
Aug 23 00:46:04 app-3 kernel: [958008.761313] in:imklog invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 23 00:46:04 app-3 kernel: [958008.761316] in:imklog cpuset=/ mems_allowed=0
Aug 23 00:46:04 app-3 kernel: [958008.761319] CPU: 0 PID: 418 Comm: in:imklog Tainted: GF 3.13.0-24-generic #46-Ubuntu
Aug 23 00:46:04 app-3 kernel: [958008.761320] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Aug 23 00:46:04 app-3 kernel: [958008.761321] 0000000000000000 ffff880079eb9980 ffffffff81715a64 ffff88007f90dfc0
Aug 23 00:46:04 app-3 kernel: [958008.761325] ffff880079eb9a08 ffffffff8171039f ffffffff81067876 ffff880079eb99e0
Aug 23 00:46:04 app-3 kernel: [958008.761327] ffffffff810c74dc 0000000000000000 ffff88007fffae28 0000000000000000
Aug 23 00:46:04 app-3 kernel: [958008.761330] Call Trace:
Aug 23 00:46:04 app-3 kernel: [958008.761334] [<ffffffff81715a64>] dump_stack+0x45/0x56
Aug 23 00:46:04 app-3 kernel: [958008.761338] [<ffffffff8171039f>] dump_header+0x7f/0x1f1
Aug 23 00:46:04 app-3 kernel: [958008.761341] [<ffffffff81067876>] ? put_online_cpus+0x56/0x80
Aug 23 00:46:04 app-3 kernel: [958008.761343] [<ffffffff810c74dc>] ? rcu_oom_notify+0xcc/0xf0
Aug 23 00:46:04 app-3 kernel: [958008.761346] [<ffffffff8115197e>] oom_kill_process+0x1ce/0x330
Aug 23 00:46:04 app-3 kernel: [958008.761348] [<ffffffff812d0135>] ? security_capable_noaudit+0x15/0x20
Aug 23 00:46:04 app-3 kernel: [958008.761351] [<ffffffff811520b4>] out_of_memory+0x414/0x450
Aug 23 00:46:04 app-3 kernel: [958008.761354] [<ffffffff81158223>] __alloc_pages_nodemask+0x983/0xa20
Aug 23 00:46:04 app-3 kernel: [958008.761357] [<ffffffff81195413>] alloc_pages_current+0xa3/0x160
Aug 23 00:46:04 app-3 kernel: [958008.761361] [<ffffffff8114e577>] __page_cache_alloc+0x97/0xc0
Aug 23 00:46:04 app-3 kernel: [958008.761364] [<ffffffff8114ff85>] filemap_fault+0x185/0x410
Aug 23 00:46:04 app-3 kernel: [958008.761367] [<ffffffff811744df>] __do_fault+0x6f/0x530
Aug 23 00:46:04 app-3 kernel: [958008.761369] [<ffffffff81178682>] handle_mm_fault+0x492/0xf10
Aug 23 00:46:04 app-3 kernel: [958008.761394] [<ffffffff810bb75c>] ? print_time.part.8+0x6c/0x90
Aug 23 00:46:04 app-3 kernel: [958008.761398] [<ffffffff810bb7ef>] ? print_prefix+0x6f/0xb0
Aug 23 00:46:04 app-3 kernel: [958008.761400] [<ffffffff817219a4>] __do_page_fault+0x184/0x560
Aug 23 00:46:04 app-3 kernel: [958008.761403] [<ffffffff8101b7d9>] ? sched_clock+0x9/0x10
Aug 23 00:46:04 app-3 kernel: [958008.761406] [<ffffffff8109d13d>] ? sched_clock_local+0x1d/0x80
Aug 23 00:46:04 app-3 kernel: [958008.761409] [<ffffffff811112fc>] ? acct_account_cputime+0x1c/0x20
Aug 23 00:46:04 app-3 kernel: [958008.761411] [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
Aug 23 00:46:04 app-3 kernel: [958008.761413] [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
Aug 23 00:46:04 app-3 kernel: [958008.761415] [<ffffffff81721d9a>] do_page_fault+0x1a/0x70
Aug 23 00:46:04 app-3 kernel: [958008.761418] [<ffffffff81721419>] do_async_page_fault+0x29/0xe0
Aug 23 00:46:04 app-3 kernel: [958008.761420] [<ffffffff8171e238>] async_page_fault+0x28/0x30
Aug 23 00:46:04 app-3 kernel: [958008.761422] Mem-Info:
Aug 23 00:46:04 app-3 kernel: [958008.761424] Node 0 DMA per-cpu:
Aug 23 00:46:04 app-3 kernel: [958008.761426] CPU 0: hi: 0, btch: 1 usd: 0
Aug 23 00:46:04 app-3 kernel: [958008.761427] CPU 1: hi: 0, btch: 1 usd: 0
Aug 23 00:46:04 app-3 kernel: [958008.761428] Node 0 DMA32 per-cpu:
Aug 23 00:46:04 app-3 kernel: [958008.761429] CPU 0: hi: 186, btch: 31 usd: 30
Aug 23 00:46:04 app-3 kernel: [958008.761431] CPU 1: hi: 186, btch: 31 usd: 59
Aug 23 00:46:04 app-3 kernel: [958008.761434] active_anon:477575 inactive_anon:1806 isolated_anon:2
Aug 23 00:46:04 app-3 kernel: [958008.761434] active_file:0 inactive_file:0 isolated_file:0
Aug 23 00:46:04 app-3 kernel: [958008.761434] unevictable:0 dirty:0 writeback:0 unstable:0
Aug 23 00:46:04 app-3 kernel: [958008.761434] free:13226 slab_reclaimable:2658 slab_unreclaimable:5962
Aug 23 00:46:04 app-3 kernel: [958008.761434] mapped:4629 shmem:5307 pagetables:4910 bounce:0
Aug 23 00:46:04 app-3 kernel: [958008.761434] free_cma:0
Aug 23 00:46:04 app-3 kernel: [958008.761437] Node 0 DMA free:8268kB min:348kB low:432kB high:520kB active_anon:7100kB inactive_anon:84kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:140kB shmem:116kB slab_reclaimable:12kB slab_unreclaimable:144kB kernel_stack:0kB pagetables:60kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 23 00:46:04 app-3 kernel: [958008.761441] lowmem_reserve[]: 0 1983 1983 1983
Aug 23 00:46:04 app-3 kernel: [958008.761443] Node 0 DMA32 free:44636kB min:44704kB low:55880kB high:67056kB active_anon:1903200kB inactive_anon:7140kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):8kB isolated(file):0kB present:2080760kB managed:2034124kB mlocked:0kB dirty:0kB writeback:0kB mapped:18376kB shmem:21112kB slab_reclaimable:10620kB slab_unreclaimable:23704kB kernel_stack:4440kB pagetables:19580kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:176 all_unreclaimable? yes
Aug 23 00:46:04 app-3 kernel: [958008.761458] lowmem_reserve[]: 0 0 0 0
Aug 23 00:46:04 app-3 kernel: [958008.761461] Node 0 DMA: 13*4kB (UEM) 0*8kB 2*16kB (EM) 7*32kB (UEM) 3*64kB (UEM) 1*128kB (E) 2*256kB (EM) 2*512kB (EM) 2*1024kB (UE) 2*2048kB (ER) 0*4096kB = 8308kB
Aug 23 00:46:04 app-3 kernel: [958008.761472] Node 0 DMA32: 654*4kB (UEM) 596*8kB (UEM) 389*16kB (UEM) 230*32kB (UE) 99*64kB (UE) 62*128kB (UEM) 21*256kB (UEM) 4*512kB (E) 0*1024kB 1*2048kB (R) 0*4096kB = 44712kB
Aug 23 00:46:04 app-3 kernel: [958008.761483] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 23 00:46:04 app-3 kernel: [958008.761484] 5314 total pagecache pages
Aug 23 00:46:04 app-3 kernel: [958008.761486] 0 pages in swap cache
Aug 23 00:46:04 app-3 kernel: [958008.761487] Swap cache stats: add 0, delete 0, find 0/0
Aug 23 00:46:04 app-3 kernel: [958008.761488] Free swap = 0kB
Aug 23 00:46:04 app-3 kernel: [958008.761489] Total swap = 0kB
Aug 23 00:46:04 app-3 kernel: [958008.761490] 524188 pages RAM
Aug 23 00:46:04 app-3 kernel: [958008.761491] 0 pages HighMem/MovableOnly
Aug 23 00:46:04 app-3 kernel: [958008.761492] 11659 pages reserved
Aug 23 00:46:04 app-3 kernel: [958008.761493] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Aug 23 00:46:04 app-3 kernel: [958008.761497] [ 285] 0 285 4968 150 13 0 0 upstart-udev-br
Aug 23 00:46:04 app-3 kernel: [958008.761500] [ 291] 0 291 12834 171 27 0 -1000 systemd-udevd
Aug 23 00:46:04 app-3 kernel: [958008.761501] [ 341] 102 341 9810 114 23 0 0 dbus-daemon
Aug 23 00:46:04 app-3 kernel: [958008.761503] [ 369] 0 369 10863 93 27 0 0 systemd-logind
Aug 23 00:46:04 app-3 kernel: [958008.761505] [ 410] 101 410 63961 1373 30 0 0 rsyslogd
Aug 23 00:46:04 app-3 kernel: [958008.761507] [ 699] 0 699 3984 219 12 0 0 upstart-file-br
Aug 23 00:46:04 app-3 kernel: [958008.761509] [ 708] 0 708 3881 118 13 0 0 upstart-socket-
Aug 23 00:46:04 app-3 kernel: [958008.761510] [ 775] 0 775 3955 39 13 0 0 getty
Aug 23 00:46:04 app-3 kernel: [958008.761512] [ 779] 0 779 3955 39 13 0 0 getty
Aug 23 00:46:04 app-3 kernel: [958008.761514] [ 785] 0 785 3955 39 13 0 0 getty
Aug 23 00:46:04 app-3 kernel: [958008.761515] [ 786] 0 786 3955 40 13 0 0 getty
Aug 23 00:46:04 app-3 kernel: [958008.761517] [ 788] 0 788 3955 39 13 0 0 getty
Aug 23 00:46:04 app-3 kernel: [958008.761519] [ 807] 0 807 15341 169 34 0 -1000 sshd
Aug 23 00:46:04 app-3 kernel: [958008.761520] [ 814] 0 814 5914 56 17 0 0 cron
Aug 23 00:46:04 app-3 kernel: [958008.761522] [ 815] 0 815 4785 43 12 0 0 atd
Aug 23 00:46:04 app-3 kernel: [958008.761524] [ 826] 0 826 1092 37 8 0 0 acpid
Aug 23 00:46:04 app-3 kernel: [958008.761525] [ 907] 106 907 61929 2915 60 0 -900 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761527] [ 909] 106 909 61965 2479 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761529] [ 910] 106 910 61929 557 52 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761531] [ 911] 106 911 61929 355 51 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761532] [ 912] 106 912 62145 535 54 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761534] [ 913] 106 913 25881 367 49 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761536] [ 1027] 0 1027 3955 41 12 0 0 getty
Aug 23 00:46:04 app-3 kernel: [958008.761538] [ 1850] 1000 1850 10530 1813 18 0 0 redis-server
Aug 23 00:46:04 app-3 kernel: [958008.761539] [ 2839] 0 2839 22227 303 41 0 0 nginx
Aug 23 00:46:04 app-3 kernel: [958008.761541] [ 2840] 65534 2840 22485 552 41 0 0 nginx
Aug 23 00:46:04 app-3 kernel: [958008.761543] [ 2841] 65534 2841 22456 489 41 0 0 nginx
Aug 23 00:46:04 app-3 kernel: [958008.761544] [ 2843] 65534 2843 22482 533 41 0 0 nginx
Aug 23 00:46:04 app-3 kernel: [958008.761546] [ 2844] 65534 2844 22483 536 41 0 0 nginx
Aug 23 00:46:04 app-3 kernel: [958008.761548] [19194] 1000 19194 69423 21930 135 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761550] [19214] 1000 19214 98772 33499 158 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761551] [19217] 1000 19217 104385 39332 168 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761562] [19220] 1000 19220 109427 44364 179 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761565] [19282] 106 19282 62438 2191 63 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761566] [19283] 106 19283 62421 2215 63 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761568] [19284] 106 19284 62418 2138 63 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761570] [32309] 0 32309 26444 259 55 0 0 sshd
Aug 23 00:46:04 app-3 kernel: [958008.761572] [32357] 0 32357 5636 495 16 0 0 bash
Aug 23 00:46:04 app-3 kernel: [958008.761573] [ 327] 0 327 1809 23 9 0 0 tail
Aug 23 00:46:04 app-3 kernel: [958008.761575] [ 357] 0 357 26444 274 55 0 0 sshd
Aug 23 00:46:04 app-3 kernel: [958008.761577] [ 416] 0 416 5636 495 16 0 0 bash
Aug 23 00:46:04 app-3 kernel: [958008.761579] [ 955] 0 955 16238 116 36 0 0 sudo
Aug 23 00:46:04 app-3 kernel: [958008.761580] [ 956] 0 956 16133 113 37 0 0 su
Aug 23 00:46:04 app-3 kernel: [958008.761582] [ 957] 1000 957 5662 521 16 0 0 bash
Aug 23 00:46:04 app-3 kernel: [958008.761584] [ 1113] 1000 1113 1809 28 10 0 0 tail
Aug 23 00:46:04 app-3 kernel: [958008.761585] [ 1217] 0 1217 26408 249 56 0 0 sshd
Aug 23 00:46:04 app-3 kernel: [958008.761587] [ 1268] 0 1268 5636 494 17 0 0 bash
Aug 23 00:46:04 app-3 kernel: [958008.761589] [ 1308] 0 1308 16238 115 35 0 0 sudo
Aug 23 00:46:04 app-3 kernel: [958008.761590] [ 1309] 0 1309 16133 114 37 0 0 su
Aug 23 00:46:04 app-3 kernel: [958008.761592] [ 1310] 1000 1310 5660 521 16 0 0 bash
Aug 23 00:46:04 app-3 kernel: [958008.761594] [ 1467] 1000 1467 444168 49060 457 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761596] [ 2219] 106 2219 62418 2171 63 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761597] [ 2232] 106 2232 62399 1095 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761599] [ 2236] 106 2236 62400 1106 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761601] [ 2240] 106 2240 62409 2143 63 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761602] [ 2244] 106 2244 62395 1066 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761604] [ 2247] 106 2247 62395 1052 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761606] [ 2252] 106 2252 62395 1066 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761607] [ 2257] 106 2257 62395 1066 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761609] [ 2259] 106 2259 62395 1065 59 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761611] [ 2263] 106 2263 62409 2145 63 0 0 postgres
Aug 23 00:46:04 app-3 kernel: [958008.761612] [ 2990] 1000 2990 3052024 71227 404 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761614] [ 3028] 1000 3028 375171 31711 347 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761616] [ 3344] 1000 3344 3064418 61620 398 0 0 ruby
Aug 23 00:46:04 app-3 kernel: [958008.761618] [ 3601] 1000 3599 3060195 101301 470 0 0 ruby-timer-thr
Aug 23 00:46:04 app-3 kernel: [958008.761620] [ 3816] 1000 3816 68705 22084 133 0 0 ruby
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment