Описание проблемы
Есть небольшой кластер из восьми машин для запуска квантово-химических расчётов. На главной ноде (Ubuntu 12.04) расшарена директория по NFS, на остальных семи (Ubuntu 12.04 Server) она примонтирована с параметром «rw». Ноды в кластере объединены двумя гигабитными сетками (одна для NFS, другая иногда используется для MPI). Входные данные для расчётов читаются из расшаренной директории, результаты и временные файлы пишутся туда же. Пару недель назад появилась проблема: главная нода виснет, если не ошибаюсь, из-за переполнения оперативной памяти. От момента запуска расчётов до «зависания» проходит около 12 часов. Помогите, пожалуйста, определить причину. Буду благодарен любым советам.
На данный момент
На главной ноде занято 1.5 Гб оперативной памяти из 8 Гб, и этот объём продолжает расти.
Load AVG на главной ноде:
6.00 5.98 5.96
Load AVG на остальных:
4.89 4.94 4.95
3.93 3.92 3.93
5.00 4.99 4.95
2.07 1.96 1.84
1.92 1.92 1.99
1.94 1.95 1.95
3.91 3.86 3.82
При этом на главной ноде в htop у всех процессов CPU% по нулям. В нормальном режиме LoadAVG должно быть 0.00 на главной и 4.00 на всех остальных.
Фрагмент syslog главной ноды
Mar 6 08:17:02 hulk CRON[14918]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Mar 6 08:52:16 hulk kernel: [59290.135675] gdbus invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 Mar 6 08:52:16 hulk kernel: [59290.135678] gdbus cpuset=/ mems_allowed=0 Mar 6 08:52:16 hulk kernel: [59290.135681] Pid: 2042, comm: gdbus Tainted: P C O 3.2.0-38-generic #61-Ubuntu Mar 6 08:52:16 hulk kernel: [59290.135682] Call Trace: Mar 6 08:52:16 hulk kernel: [59290.135688] [<ffffffff810c0f4d>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Mar 6 08:52:16 hulk kernel: [59290.135692] [<ffffffff8111bf61>] dump_header+0x91/0xe0 Mar 6 08:52:16 hulk kernel: [59290.135694] [<ffffffff8111c2e5>] oom_kill_process+0x85/0xb0 Mar 6 08:52:16 hulk kernel: [59290.135696] [<ffffffff8111c68a>] out_of_memory+0xfa/0x220 Mar 6 08:52:16 hulk kernel: [59290.135699] [<ffffffff8112208c>] __alloc_pages_nodemask+0x8dc/0x8f0 Mar 6 08:52:16 hulk kernel: [59290.135702] [<ffffffff8115aa63>] alloc_pages_vma+0xb3/0x190 Mar 6 08:52:16 hulk kernel: [59290.135705] [<ffffffff8114fbc2>] read_swap_cache_async+0xf2/0x160 Mar 6 08:52:16 hulk kernel: [59290.135708] [<ffffffff8115368e>] ? valid_swaphandles+0x10e/0x130 Mar 6 08:52:16 hulk kernel: [59290.135710] [<ffffffff8114fcb6>] swapin_readahead+0x86/0xb0 Mar 6 08:52:16 hulk kernel: [59290.135713] [<ffffffff8113d1b5>] do_swap_page.isra.51+0x115/0x630 Mar 6 08:52:16 hulk kernel: [59290.135717] [<ffffffff8104595c>] ? ptep_set_access_flags+0x6c/0x70 Mar 6 08:52:16 hulk kernel: [59290.135720] [<ffffffff8113ec71>] handle_pte_fault+0x1a1/0x200 Mar 6 08:52:16 hulk kernel: [59290.135722] [<ffffffff8113fe39>] handle_mm_fault+0x269/0x370 Mar 6 08:52:16 hulk kernel: [59290.135726] [<ffffffff81660e90>] do_page_fault+0x150/0x520 Mar 6 08:52:16 hulk kernel: [59290.135729] [<ffffffff81013878>] ? __switch_to+0x288/0x360 Mar 6 08:52:16 hulk kernel: [59290.135732] [<ffffffff81056f8d>] ? set_next_entity+0xad/0xd0 Mar 6 08:52:16 hulk kernel: [59290.135734] [<ffffffff8105712a>] ? finish_task_switch+0x4a/0xf0 Mar 6 08:52:16 hulk kernel: [59290.135737] [<ffffffff8165ae3c>] ? __schedule+0x3cc/0x6f0 Mar 6 08:52:16 hulk kernel: [59290.135739] [<ffffffff8165daf5>] page_fault+0x25/0x30 Mar 6 08:52:16 hulk kernel: [59290.135741] Mem-Info: Mar 6 08:52:16 hulk kernel: [59290.135742] Node 0 DMA per-cpu: Mar 6 08:52:16 hulk kernel: [59290.135744] CPU 0: hi: 0, btch: 1 usd: 0 Mar 6 08:52:16 hulk kernel: [59290.135745] CPU 1: hi: 0, btch: 1 usd: 0 Mar 6 08:52:16 hulk kernel: [59290.135747] CPU 2: hi: 0, btch: 1 usd: 0 Mar 6 08:52:16 hulk kernel: [59290.135748] CPU 3: hi: 0, btch: 1 usd: 0 Mar 6 08:52:16 hulk kernel: [59290.135749] Node 0 DMA32 per-cpu: Mar 6 08:52:16 hulk kernel: [59290.135751] CPU 0: hi: 186, btch: 31 usd: 0 Mar 6 08:52:16 hulk kernel: [59290.135752] CPU 1: hi: 186, btch: 31 usd: 0 Mar 6 08:52:16 hulk kernel: [59290.135753] CPU 2: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: imklog 5.8.6, log source = /proc/kmsg started. Mar 6 08:58:54 hulk rsyslogd: [origin software=«rsyslogd» swVersion=«5.8.6» x-pid=«15080» x-info="http://www.rsyslog.com«] start Mar 6 08:58:54 hulk rsyslogd: rsyslogd's groupid changed to 103 Mar 6 08:58:54 hulk rsyslogd: rsyslogd's userid changed to 101 Mar 6 08:58:53 hulk rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ] Mar 6 08:58:54 hulk kernel: :2864kB pagetables:21508kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:30896 all_unreclaimable? yes Mar 6 08:58:54 hulk kernel: [59297.079461] lowmem_reserve[]: 0 0 0 0 Mar 6 08:58:54 hulk kernel: [59297.079463] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB Mar 6 08:58:54 hulk kernel: [59297.079470] Node 0 DMA32: 221*4kB 193*8kB 70*16kB 41*32kB 26*64kB 20*128kB 16*256kB 7*512kB 1*1024kB 1*2048kB 6*4096kB = 44412kB Mar 6 08:58:54 hulk kernel: [59297.079476] Node 0 Normal: 548*4kB 561*8kB 2032*16kB 6*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 41112kB Mar 6 08:58:54 hulk kernel: [59297.079482] 3760 total pagecache pages Mar 6 08:58:54 hulk kernel: [59297.079483] 1653 pages in swap cache Mar 6 08:58:54 hulk kernel: [59297.079484] Swap cache stats: add 121568, delete 119915, find 24688/35524 Mar 6 08:58:54 hulk kernel: [59297.079486] Free swap = 7617844kB Mar 6 08:58:54 hulk kernel: [59297.079487] Total swap = 7812092kB Mar 6 08:58:54 hulk kernel: [59297.095577] 2097136 pages RAM Mar 6 08:58:54 hulk kernel: [59297.095579] 97046 pages reserved Mar 6 08:58:54 hulk kernel: [59297.095581] 5865 pages shared Mar 6 08:58:54 hulk kernel: [59297.095583] 1971086 pages non-shared Mar 6 08:58:54 hulk kernel: [59297.095585] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Mar 6 08:58:54 hulk kernel: [59297.095594] [ 364] 0 364 5604 1 2 -17 -1000 udevd Mar 6 08:58:54 hulk kernel: [59297.095598] [ 666] 0 666 5463 1 0 -17 -1000 udevd Mar 6 08:58:54 hulk kernel: [59297.095602] [ 667] 0 667 5463 1 1 -17 -1000 udevd Mar 6 08:58:54 hulk kernel: [59297.095608] [ 976] 0 976 12489 1 0 -17 -1000 sshd Mar 6 08:58:54 hulk kernel: [59297.095611] [ 988] 0 988 6385 1 1 0 0 rpc.idmapd Mar 6 08:58:54 hulk kernel: [59297.095615] [ 990] 101 990 62368 261 0 0 0 rsyslogd Mar 6 08:58:54 hulk kernel: [59297.095619] [ 992] 102 992 6273 246 0 0 0 dbus-daemon Mar 6 08:58:54 hulk kernel: [59297.095623] [ 1011] 0 1011 19760 0 0 0 0 modem-manager Mar 6 08:58:54 hulk kernel: [59297.095627] [ 1031] 0 1031 26045 0 0 0 0 cupsd Mar 6 08:58:54 hulk kernel: [59297.095630] [ 1033] 0 1033 5297 0 1 0 0 bluetoothd Mar 6 08:58:54 hulk kernel: [59297.095634] [ 1048] 107 1048 8100 88 0 0 0 avahi-daemon Mar 6 08:58:54 hulk kernel: [59297.095638] [ 1051] 107 1051 8045 0 1 0 0 avahi-daemon Mar 6 08:58:54 hulk kernel: [59297.095642] [ 1061] 0 1061 59470 42 0 0 0 NetworkManager Mar 6 08:58:54 hulk kernel: [59297.095645] [ 1077] 0 1077 48901 0 0 0 0 polkitd Mar 6 08:58:54 hulk kernel: [59297.095649] [ 1109] 0 1109 4996 1 1 0 0 getty Mar 6 08:58:54 hulk kernel: [59297.095653] [ 1116] 0 1116 4996 1 3 0 0 getty Mar 6 08:58:54 hulk kernel: [59297.095656] [ 1129] 0 1129 4996 1 1 0 0 getty Mar 6 08:58:54 hulk kernel: [59297.095660] [ 1130] 0 1130 4996 1 3 0 0 getty Mar 6 08:58:54 hulk kernel: [59297.095664] [ 1132] 0 1132 4996 1 0 0 0 getty Mar 6 08:58:54 hulk kernel: [59297.095667] [ 1141] 0 1141 1115 0 1 0 0 acpid Mar 6 08:58:54 hulk kernel: [59297.095670] [ 1145] 0 1145 4227 0 0 0 0 atd Mar 6 08:58:54 hulk kernel: [59297.095674] [ 1146] 0 1146 4778 1 0 0 0 cron Mar 6 08:58:54 hulk kernel: [59297.095678] [ 1160] 0 1160 67666 0 0 0 0 lightdm Mar 6 08:58:54 hulk kernel: [59297.095682] [ 1161] 0 1161 3995 83 0 0 0 irqbalance Mar 6 08:58:54 hulk kernel: [59297.095686] [ 1178] 0 1178 76530 8448 0 0 0 Xorg Mar 6 08:58:54 hulk kernel: [59297.095690] [ 1180] 105 1180 50548 0 1 0 0 whoopsie Mar 6 08:58:54 hulk kernel: [59297.095693] [ 1182] 116 1182 119322 0 0 0 0 apt-cacher-ng Mar 6 08:58:54 hulk kernel: [59297.095699] [ 1242] 0 1242 7080 76 0 0 0 rpc.mountd Mar 6 08:58:54 hulk kernel: [59297.095703] [ 1254] 0 1254 1816 1 2 0 0 dhclient Mar 6 08:58:54 hulk kernel: [59297.095707] [ 1273] 0 1273 25913 36 0 0 0 winbindd Mar 6 08:58:54 hulk kernel: [59297.095711] [ 1289] 0 1289 25913 53 0 0 0 winbindd Mar 6 08:58:54 hulk kernel: [59297.095715] [ 1514] 0 1514 4996 1 1 0 0 getty Mar 6 08:58:54 hulk kernel: [59297.095719] [ 1599] 0 1599 40255 0 1 0 0 lightdm Mar 6 08:58:54 hulk kernel: [59297.095723] [ 1602] 0 1602 30383 40 0 0 0 accounts-daemon Mar 6 08:58:54 hulk kernel: [59297.095727] [ 1607] 65534 1607 8255 0 0 0 0 dnsmasq Mar 6 08:58:54 hulk kernel: [59297.095731] [ 1666] 0 1666 522908 0 0 0 0 console-kit-dae Mar 6 08:58:54 hulk kernel: [59297.095735] [ 1741] 1001 1741 98694 280 1 0 0 gnome-session Mar 6 08:58:54 hulk kernel: [59297.095743] [ 1779] 1001 1779 3123 0 0 0 0 ssh-agent Mar 6 08:58:54 hulk kernel: [59297.095746] [ 1782] 1001 1782 6639 0 0 0 0 dbus-launch Mar 6 08:58:54 hulk kernel: [59297.095749] [ 1783] 1001 1783 6761 480 0 0 0 dbus-daemon Mar 6 08:58:54 hulk kernel: [59297.095752] [ 1792] 1001 1792 110314 0 0 0 0 gnome-keyring-d Mar 6 08:58:54 hulk kernel: [59297.095755] [ 1797] 1001 1797 178326 200 0 0 0 gnome-settings- Mar 6 08:58:54 hulk kernel: [59297.095758] [ 1803] 0 1803 54975 0 0 0 0 upowerd Mar 6 08:58:54 hulk kernel: [59297.095761] [ 1953] 1001 1953 13095 33 0 0 0 gvfsd Mar 6 08:58:54 hulk kernel: [59297.095764] [ 1971] 1001 1971 68361 0 1 0 0 gvfs-fuse-daemo Mar 6 08:58:54 hulk kernel: [59297.095767] [ 1980] 103 1980 125015 0 1 0 0 colord Mar 6 08:58:54 hulk kernel: [59297.095770] [ 1991] 1001 1991 109171 0 3 0 0 pulseaudio Mar 6 08:58:54 hulk kernel: [59297.095773] [ 1993] 111 1993 42218 8 1 0 0 rtkit-daemon Mar 6 08:58:54 hulk kernel: [59297.095777] [ 1999] 1001 1999 14431 107 0 0 0 gconfd-2 Mar 6 08:58:54 hulk kernel: [59297.095780] [ 2003] 1001 2003 23988 0 1 0 0 gconf-helper Mar 6 08:58:54 hulk kernel: [59297.095783] [ 2006] 1001 2006 113761 0 2 0 0 gnome-fallback- Mar 6 08:58:54 hulk kernel: [59297.095787] [ 2007] 1001 2007 76910 0 1 0 0 polkit-gnome-au Mar 6 08:58:54 hulk kernel: [59297.095790] [ 2010] 1001 2010 224302 0 0 0 0 nautilus Mar 6 08:58:54 hulk kernel: [59297.095793] [ 2016] 1001 2016 17574 147 2 0 0 gvfs-gdu-volume Mar 6 08:58:54 hulk kernel: [59297.095796] [ 2017] 1001 2017 125000 0 0 0 0 nm-applet Mar 6 08:58:54 hulk kernel: [59297.095799] [ 2019] 0 2019 48360 159 2 0 0 udisks-daemon Mar 6 08:58:54 hulk kernel: [59297.095802] [ 2020] 1001 2020 108251 0 0 0 0 bluetooth-apple Mar 6 08:58:54 hulk kernel: [59297.095805] [ 2023] 0 2023 11379 0 3 0 0 udisks-daemon Mar 6 08:58:54 hulk kernel: [59297.095809] [ 2026] 1001 2026 15084 0 3 0 0 gvfs-gphoto2-vo Mar 6 08:58:54 hulk kernel: [59297.095812] [ 2032] 1001 2032 35519 0 1 0 0 gvfs-afc-volume Mar 6 08:58:54 hulk kernel: [59297.095815] [ 2040] 1001 2040 14218 130 0 0 0 gvfsd-trash Mar 6 08:58:54 hulk kernel: [59297.095818] [ 2045] 1001 2045 13102 0 2 0 0 gvfsd-burn Mar 6 08:58:54 hulk kernel: [59297.095821] [ 2052] 1001 2052 98124 0 3 0 0 bamfdaemon Mar 6 08:58:54 hulk kernel: [59297.095825] [ 2060] 1001 2060 11523 0 2 0 0 gvfsd-metadata Mar 6 08:58:54 hulk kernel: [59297.095828] [ 2071] 1001 2071 79936 66 3 0 0 gtk-window-deco Mar 6 08:58:54 hulk kernel: [59297.095831] [ 2081] 1001 2081 132467 294 0 0 0 unity-panel-ser Mar 6 08:58:54 hulk kernel: [59297.095834] [ 2083] 1001 2083 107742 0 0 0 0 hud-service Mar 6 08:58:54 hulk kernel: [59297.095838] [ 2093] 1001 2093 91992 33 0 0 0 indicator-messa Mar 6 08:58:54 hulk kernel: [59297.095841] [ 2095] 1001 2095 150567 0 1 0 0 indicator-sessi Mar 6 08:58:54 hulk kernel: [59297.095844] [ 2097] 1001 2097 139319 0 0 0 0 indicator-datet Mar 6 08:58:54 hulk kernel: [59297.095847] [ 2099] 1001 2099 122321 0 0 0 0 indicator-print Mar 6 08:58:54 hulk kernel: [59297.095850] [ 2101] 1001 2101 132430 0 0 0 0 indicator-sound Mar 6 08:58:54 hulk kernel: [59297.095853] [ 2103] 1001 2103 86951 0 0 0 0 indicator-appli Mar 6 08:58:54 hulk kernel: [59297.095857] [ 2147] 1001 2147 11964 18 0 0 0 geoclue-master Mar 6 08:58:54 hulk kernel: [59297.095860] [ 2149] 1001 2149 40170 15 1 0 0 ubuntu-geoip-pr Mar 6 08:58:54 hulk kernel: [59297.095863] [ 2151] 1001 2151 82946 99 1 0 0 gdu-notificatio Mar 6 08:58:54 hulk kernel: [59297.095866] [ 2156] 1001 2156 107354 0 0 0 0 telepathy-indic Mar 6 08:58:54 hulk kernel: [59297.095869] [ 2160] 1001 2160 80349 0 0 0 0 mission-control Mar 6 08:58:54 hulk kernel: [59297.095872] [ 2165] 1001 2165 100042 0 0 0 0 goa-daemon Mar 6 08:58:54 hulk kernel: [59297.095875] [ 2403] 1001 2403 77492 10 2 0 0 gnome-screensav Mar 6 08:58:54 hulk kernel: [59297.095879] [ 2404] 1001 2404 104364 40 0 0 0 zeitgeist-datah Mar 6 08:58:54 hulk kernel: [59297.095882] [ 2412] 1001 2412 87030 0 0 0 0 zeitgeist-daemo Mar 6 08:58:54 hulk kernel: [59297.095885] [ 2418] 1001 2418 59212 0 1 0 0 zeitgeist-fts Mar 6 08:58:54 hulk kernel: [59297.095888] [ 2426] 1001 2426 2845 0 1 0 0 cat Mar 6 08:58:54 hulk kernel: [59297.095891] [ 2811] 1001 2811 120268 145 0 0 0 update-notifier Mar 6 08:58:54 hulk kernel: [59297.095894] [ 3058] 1001 3058 103780 34 0 0 0 unity-applicati Mar 6 08:58:54 hulk kernel: [59297.095898] [ 3060] 1001 3060 219118 0 1 0 0 unity-files-dae Mar 6 08:58:54 hulk kernel: [59297.095901] [ 3062] 1001 3062 184568 0 0 0 0 unity-music-dae Mar 6 08:58:54 hulk kernel: [59297.095903] [ 3064] 1001 3064 91830 17 3 0 0 unity-lens-vide Mar 6 08:58:54 hulk kernel: [59297.095905] [ 3109] 1001 3109 136450 4 0 0 0 unity-scope-vid Mar 6 08:58:54 hulk kernel: [59297.095907] [ 3111] 1001 3111 146603 0 3 0 0 unity-musicstor Mar 6 08:58:54 hulk kernel: [59297.095909] [ 3151] 1001 3151 72371 0 1 0 0 deja-dup-monito Mar 6 08:58:54 hulk kernel: [59297.095911] [11034] 0 11034 24634 0 0 0 0 sshd Mar 6 08:58:54 hulk kernel: [59297.095913] [11164] 1001 11164 24634 0 1 0 0 sshd Mar 6 08:58:54 hulk kernel: [59297.095915] [11165] 1001 11165 7763 1 1 0 0 bash Mar 6 08:58:54 hulk kernel: [59297.095917] [11522] 1001 11522 10362 0 0 0 0 ssh Mar 6 08:58:54 hulk kernel: [59297.095919] [11781] 1001 11781 10362 0 0 0 0 ssh Mar 6 08:58:54 hulk kernel: [59297.095921] [11784] 1001 11784 10362 0 0 0 0 ssh Mar 6 08:58:54 hulk kernel: [59297.095923] [13416] 1001 13416 65522 0 3 0 0 dconf-service Mar 6 08:58:54 hulk kernel: [59297.095925] [13449] 1001 13449 131164 60 0 0 0 gnome-terminal Mar 6 08:58:54 hulk kernel: [59297.095931] [13457] 1001 13457 3697 0 0 0 0 gnome-pty-helpe Mar 6 08:58:54 hulk kernel: [59297.095933] [13458] 1001 13458 6882 1 0 0 0 bash Mar 6 08:58:54 hulk kernel: [59297.095935] [14044] 1001 14044 10363 0 1 0 0 ssh Mar 6 08:58:54 hulk kernel: [59297.095938] [14825] 7 14825 13036 0 0 0 0 dbus Mar 6 08:58:54 hulk kernel: [59297.095940] Out of memory: Kill process 988 (rpc.idmapd) score 1 or sacrifice child Mar 6 08:58:54 hulk kernel: [59297.095943] Killed process 988 (rpc.idmapd) total-vm:25540kB, anon-rss:0kB, file-rss:4kB Mar 6 08:58:54 hulk kernel: [59297.113938] indicator-appli invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 Mar 6 08:58:54 hulk kernel: [59297.113941] indicator-appli cpuset=/ mems_allowed=0 Mar 6 08:58:54 hulk kernel: [59297.113943] Pid: 2103, comm: indicator-appli Tainted: P C O 3.2.0-38-generic #61-Ubuntu Mar 6 08:58:54 hulk kernel: [59297.113945] Call Trace: Mar 6 08:58:54 hulk kernel: [59297.114016] [<ffffffff810c0f4d>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Mar 6 08:58:54 hulk kernel: [59297.114019] [<ffffffff8111bf61>] dump_header+0x91/0xe0 Mar 6 08:58:54 hulk kernel: [59297.114021] [<ffffffff8111c2e5>] oom_kill_process+0x85/0xb0 Mar 6 08:58:54 hulk kernel: [59297.114024] [<ffffffff8111c68a>] out_of_memory+0xfa/0x220 Mar 6 08:58:54 hulk kernel: [59297.114027] [<ffffffff8112208c>] __alloc_pages_nodemask+0x8dc/0x8f0 Mar 6 08:58:54 hulk kernel: [59297.114030] [<ffffffff8115aa63>] alloc_pages_vma+0xb3/0x190 Mar 6 08:58:54 hulk kernel: [59297.114033] [<ffffffff8114fbc2>] read_swap_cache_async+0xf2/0x160 Mar 6 08:58:54 hulk kernel: [59297.114035] [<ffffffff8115368e>] ? valid_swaphandles+0x10e/0x130 Mar 6 08:58:54 hulk kernel: [59297.114038] [<ffffffff8114fcb6>] swapin_readahead+0x86/0xb0 Mar 6 08:58:54 hulk kernel: [59297.114041] [<ffffffff8113d1b5>] do_swap_page.isra.51+0x115/0x630 Mar 6 08:58:54 hulk kernel: [59297.114044] [<ffffffff811184ea>] ? unlock_page+0x2a/0x40 Mar 6 08:58:54 hulk kernel: [59297.114046] [<ffffffff8113b659>] ? __do_fault+0x439/0x550 Mar 6 08:58:54 hulk kernel: [59297.114048] [<ffffffff8113ec71>] handle_pte_fault+0x1a1/0x200 Mar 6 08:58:54 hulk kernel: [59297.114050] [<ffffffff8113fe39>] handle_mm_fault+0x269/0x370 Mar 6 08:58:54 hulk kernel: [59297.114054] [<ffffffff81660e90>] do_page_fault+0x150/0x520 Mar 6 08:58:54 hulk kernel: [59297.114058] [<ffffffff81060600>] ? try_to_wake_up+0x200/0x200 Mar 6 08:58:54 hulk kernel: [59297.114061] [<ffffffff811c01ac>] ? eventfd_read+0x3c/0x70 Mar 6 08:58:54 hulk kernel: [59297.114064] [<ffffffff811796b0>] ? vfs_read+0xb0/0x180 Mar 6 08:58:54 hulk kernel: [59297.114067] [<ffffffff8165daf5>] page_fault+0x25/0x30 Mar 6 08:58:54 hulk kernel: [59297.114068] Mem-Info: Mar 6 08:58:54 hulk kernel: [59297.114069] Node 0 DMA per-cpu: Mar 6 08:58:54 hulk kernel: [59297.114071] CPU 0: hi: 0, btch: 1 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114072] CPU 1: hi: 0, btch: 1 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114074] CPU 2: hi: 0, btch: 1 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114075] CPU 3: hi: 0, btch: 1 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114076] Node 0 DMA32 per-cpu: Mar 6 08:58:54 hulk kernel: [59297.114078] CPU 0: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114079] CPU 1: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114080] CPU 2: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114082] CPU 3: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114083] Node 0 Normal per-cpu: Mar 6 08:58:54 hulk kernel: [59297.114084] CPU 0: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114086] CPU 1: hi: 186, btch: 31 usd: 2 Mar 6 08:58:54 hulk kernel: [59297.114087] CPU 2: hi: 186, btch: 31 usd: 0 Mar 6 08:58:54 hulk kernel: [59297.114088] CPU 3: hi: 186, btch: 31 usd: 85 Mar 6 08:58:54 hulk kernel: [59297.114092] active_anon:2114 inactive_anon:428 isolated_anon:32 Mar 6 08:58:54 hulk kernel: [59297.114092] active_file:961 inactive_file:1009 isolated_file:0 Mar 6 08:58:54 hulk kernel: [59297.114093] unevictable:0 dirty:3 writeback:1453 unstable:0 Mar 6 08:58:54 hulk kernel: [59297.114094] free:25203 slab_reclaimable:5014 slab_unreclaimable:1924106 Mar 6 08:58:54 hulk kernel: [59297.114095] mapped:8980 shmem:0 pagetables:5377 bounce:0 Mar 6 08:58:54 hulk kernel: [59297.114096] Node 0 DMA free:15880kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 6 08:58:54 hulk kernel: [59297.114103] lowmem_reserve[]: 0 3182 7970 7970 Mar 6 08:58:54 hulk kernel: [59297.114106] Node 0 DMA32 free:44412kB min:26928kB low:33660kB high:40392kB active_anon:448kB inactive_anon:264kB active_file:0kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3258388kB mlocked:0kB dirty:0kB writeback:264kB mapped:44kB shmem:0kB slab_reclaimable:220kB slab_unreclaimable:3194928kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3421 all_unreclaimable? yes Mar 6 08:58:54 hulk kernel: [59297.114113] lowmem_reserve[]: 0 0 4788 4788 Mar 6 08:58:54 hulk kernel: [59297.114115] Node 0 Normal free:40520kB min:40520kB low:50648kB high:60780kB active_anon:8008kB inactive_anon:1448kB active_file:3844kB inactive_file:4028kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:4902912kB mlocked:0kB dirty:12kB writeback:5548kB mapped:35876kB shmem:0kB slab_reclaimable:19836kB slab_unreclaimable:4501480kB kernel_stack:2864kB pagetables:21508kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:33439 all_unreclaimable? yes
