LINUX.ORG.RU
Форум — Admin  

зависает сервер.

 ,


0

1

Есть сервер: nat, dhcp, dns. Linux version 2.6.32-358.23.2.el6.i686, 12G ОЗУ 4 сетевые карты 2 в мир подключены, 2 в локалку: Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) В пиках наргузка примерно 1,5-1,7 gbit и вот в єти моменты пиковые и иногда при меншей нагрузке 1-1,2 gbit, виснит сервер с вот таким вот логом в dmesg 

 swapper: page allocation failure. order:0, mode:0x120
Pid: 0, comm: swapper Not tainted 2.6.32-358.23.2.el6.i686 #1
Call Trace:
 [<c04f524e>] ? __alloc_pages_nodemask+0x64e/0x810
 [<c07d08ee>] ? ip_finish_output+0x11e/0x290
 [<f7ef83e6>] ? igb_alloc_rx_buffers+0xe6/0x1b0 [igb]
 [<f7efc419>] ? igb_poll+0xc29/0x1280 [igb]
 [<c040f180>] ? nommu_sync_single_for_device+0x0/0x10
 [<c04be1d9>] ? force_quiescent_state+0x139/0x180
 [<c07a296e>] ? net_rx_action+0xde/0x280
 [<c045fcca>] ? __do_softirq+0x8a/0x1a0
 [<c04bd501>] ? move_native_irq+0x11/0x50
ipcadcnt: page allocation failure. order:0, mode:0x120
 [<c045fe1d>] ? do_softirq+0x3d/0x50
 [<c045ff75>] ? irq_exit+0x65/0x70
Pid: 13086, comm: ipcadcnt Not tainted 2.6.32-358.23.2.el6.i686 #1
Call Trace:
 [<c040b030>] ? do_IRQ+0x50/0xc0
 [<c04f524e>] ? __alloc_pages_nodemask+0x64e/0x810
 [<c0409f10>] ? common_interrupt+0x30/0x38
 [<c07d08ee>] ? ip_finish_output+0x11e/0x290
 [<c045007b>] ? sched_create_group+0x28b/0x3c0
 [<c06516cf>] ? intel_idle+0xaf/0x140
 [<f7ef83e6>] ? igb_alloc_rx_buffers+0xe6/0x1b0 [igb]
 [<c07a04ff>] ? netif_receive_skb+0x3f/0x50
 [<c0771182>] ? cpuidle_idle_call+0x72/0x100
 [<c0408884>] ? cpu_idle+0x94/0xd0
 [<c07a05cf>] ? napi_skb_finish+0x2f/0x40
 [<c0844609>] ? start_secondary+0x2af/0x2f4
 [<f7efc419>] ? igb_poll+0xc29/0x1280 [igb]
 [<c047e563>] ? hrtimer_forward+0x163/0x1b0
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 184
CPU    1: hi:  186, btch:  31 usd: 176
CPU    2: hi:  186, btch:  31 usd: 156
CPU    3: hi:  186, btch:  31 usd: 153
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd: 145
CPU    1: hi:  186, btch:  31 usd:  90
CPU    2: hi:  186, btch:  31 usd:  63
CPU    3: hi:  186, btch:  31 usd:  45
active_anon:29089 inactive_anon:37283 isolated_anon:0
 active_file:731990 inactive_file:948094 isolated_file:0
 unevictable:3141 dirty:49 writeback:0 unstable:0
 free:1200282 slab_reclaimable:25039 slab_unreclaimable:78667
 mapped:11567 shmem:130 pagetables:826 bounce:0
 [<c07a296e>] ? net_rx_action+0xde/0x280
DMA free:3480kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:1640kB inactive_file:584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15740kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:732kB slab_unreclaimable:1012kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 863 12142 12142
Normal free:1356kB min:3724kB low:4652kB high:5584kB active_anon:0kB inactive_anon:0kB active_file:123524kB inactive_file:30688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:883912kB mlocked:0kB dirty:160kB writeback:0kB mapped:28700kB shmem:0kB slab_reclaimable:99424kB slab_unreclaimable:313656kB kernel_stack:1336kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 90236 90236
HighMem free:4796292kB min:512kB low:12680kB high:24852kB active_anon:116356kB inactive_anon:149132kB active_file:2802796kB inactive_file:3761104kB unevictable:12564kB isolated(anon):0kB isolated(file):0kB present:11550248kB mlocked:12564kB dirty:36kB writeback:0kB mapped:17568kB shmem:520kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:3304kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 55*4kB 150*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3468kB
Normal: 1*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1356kB
HighMem: 73*4kB  [<c045fcca>] ? __do_softirq+0x8a/0x1a0
 [<c04bd501>] ? move_native_irq+0x11/0x50
28*8kB 22*16kB 4935*32kB 11823*64kB 2359*128kB 1798*256kB 1015*512kB 376*1024kB 99*2048kB 491*4096kB = 4796292kB
1680709 total pagecache pages
0 pages in swap cache
 [<c045fe1d>] ? do_softirq+0x3d/0x50
 [<c045ff75>] ? irq_exit+0x65/0x70
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 7101432kB
Total swap = 7101432kB
 [<c040b030>] ? do_IRQ+0x50/0xc0
 [<c045ff45>] ? irq_exit+0x35/0x70
 [<c042be03>] ? smp_apic_timer_interrupt+0x53/0x90
 [<c0409f10>] ? common_interrupt+0x30/0x38
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd: 184
CPU    1: hi:  186, btch:  31 usd: 176
CPU    2: hi:  186, btch:  31 usd: 156
CPU    3: hi:  186, btch:  31 usd: 153
HighMem per-cpu:
CPU    0: hi:  186, btch:  31 usd: 145
CPU    1: hi:  186, btch:  31 usd:  90
CPU    2: hi:  186, btch:  31 usd:  63
CPU    3: hi:  186, btch:  31 usd:  45
active_anon:29089 inactive_anon:37283 isolated_anon:0
 active_file:731990 inactive_file:948094 isolated_file:0
 unevictable:3141 dirty:49 writeback:0 unstable:0
 free:1200282 slab_reclaimable:25039 slab_unreclaimable:78667
 mapped:11567 shmem:130 pagetables:826 bounce:0

Лог длинный, выложил только начало.

При нормальной работе выделение памяти в пределах 800-900 Мбайт, при доступных 12G

вывод ifconfig

сетевые в локалку: eth0 Link encap:Ethernet HWaddr A0:36:9F:44:41:08 UP BROADCAST RUNNING MTU:1536 Metric:1 RX packets:2195281065 errors:0 dropped:7068 overruns:7068 frame:0 TX packets:467173249 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3057071664 (2.8 GiB) TX bytes:2520377657 (2.3 GiB)

eth1 Link encap:Ethernet HWaddr A0:36:9F:44:41:09 UP BROADCAST RUNNING MTU:1536 Metric:1 RX packets:2215465272 errors:0 dropped:7920 overruns:7920 frame:0 TX packets:4089101922 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3131298491 (2.9 GiB) TX bytes:3278288107 (3.0 GiB)

сетевы в мир: eth2 Link encap:Ethernet HWaddr A0:36:9F:44:41:0A UP BROADCAST RUNNING NOARP SLAVE MTU:1500 Metric:1 RX packets:2309848037 errors:0 dropped:109463 overruns:109463 frame:0 TX packets:798742 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:550255592 (524.7 MiB) TX bytes:2885243726 (2.6 GiB)

eth3 Link encap:Ethernet HWaddr A0:36:9F:44:41:0A UP BROADCAST RUNNING NOARP SLAVE MTU:1500 Metric:1 RX packets:2305575965 errors:0 dropped:109652 overruns:109652 frame:0 TX packets:782435 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2473224177 (2.3 GiB) TX bytes:2876119319 (2.6 GiB)

на сетевых стоящих в мир много пакетов overruns. Зависание сервера происходит по-разному: от простого отказа обслуживать dhcp и dns до завершения ядром процессов вплоть до sshd



Последнее исправление: forella (всего исправлений: 1)
← Как отработает fail2ban.
Ошибка при сборке nginx →

Попробуйте увеличить этот параметр в sysctl - vm.min_free_kbytes.

zuzzas ★
()
Ответ на: комментарий от itn 
           CPU0       CPU1       CPU2       CPU3
  0:        122          0          0          0   IO-APIC-edge      timer
  1:          2          0          0          0   IO-APIC-edge      i8042
  3:          2          0          0          0   IO-APIC-edge
  4:          2          0          0          0   IO-APIC-edge
  8:          1          0          0          0   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 10:          2          0          0          0   IO-APIC-edge
 12:          4          0          0          0   IO-APIC-edge      i8042
 16:         60          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
 23:         28          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2
 24:       1108          0          0          0  HPET_MSI-edge      hpet2
 25:          0          0          0          0  HPET_MSI-edge      hpet3
 26:          0          0          0          0  HPET_MSI-edge      hpet4
 27:          0          0          0          0  HPET_MSI-edge      hpet5
 29:   50550177          0          0          0   PCI-MSI-edge      ahci
 30:          2          0          0          0   PCI-MSI-edge      eth0
 31:     199554          0 2536007818          0   PCI-MSI-edge      eth0-TxRx-0
 32: 2581437354          0          0          0   PCI-MSI-edge      eth0-TxRx-1
 33: 2164130490          0          0          0   PCI-MSI-edge      eth0-TxRx-2
 34:          2          0          0          0   PCI-MSI-edge      eth1
 35:      58242          0 1816612342          0   PCI-MSI-edge      eth1-TxRx-0
 36: 1802849843          0          0          0   PCI-MSI-edge      eth1-TxRx-1
 37: 2001727351          0          0          0   PCI-MSI-edge      eth1-TxRx-2
 38:          1          0          0          0   PCI-MSI-edge      eth2
 39:     271433          0          0 1695392760   PCI-MSI-edge      eth2-TxRx-0
 40:     307158          0          0 1991584116   PCI-MSI-edge      eth2-TxRx-1
 41:     295885 2459488265          0          0   PCI-MSI-edge      eth2-TxRx-2
 42:          1          0          0          0   PCI-MSI-edge      eth3
 43:     271981          0          0 1695474523   PCI-MSI-edge      eth3-TxRx-0
 44:     306993          0          0 1991952755   PCI-MSI-edge      eth3-TxRx-1
 45:     295764 2460570811          0          0   PCI-MSI-edge      eth3-TxRx-2
NMI:          0          0          0          0   Non-maskable interrupts
LOC: 1457797771 2169151909 1028811307 1980686574   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:    4709212    5596439    4247463    3942286   Rescheduling interrupts
CAL:       5456       5492       5251       5917   Function call interrupts
TLB:     475618    1163651    1591689    1879985   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:       1109       1109       1109       1109   Machine check polls
ERR:          0
MIS:          0
forella
() автор топика
Вы не можете добавлять комментарии в эту тему. Тема перемещена в архив.
← Как отработает fail2ban.
Admin
Ошибка при сборке nginx →