ixgbe 2x6cores CPU + HT system interrupts

0

1

Всем привет, запустил на x520 сетевушке с 2мя процами 6 ядерными и включенным гипертрейдингом трафик около 3 гигабит. И заметил странную вещь (по ядрам раскидал прерывания). А в топе следующее

%Cpu0  :  0.0 us,  1.1 sy,  0.0 ni, 92.6 id,  0.0 wa,  0.0 hi,  6.4 si,  0.0 st
%Cpu1  :  1.3 us,  0.0 sy,  0.0 ni, 86.1 id,  0.0 wa,  0.0 hi, 12.7 si,  0.0 st
%Cpu2  :  1.1 us,  1.1 sy,  0.0 ni, 92.5 id,  0.0 wa,  0.0 hi,  5.4 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni, 90.6 id,  0.0 wa,  0.0 hi,  9.4 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni, 91.3 id,  0.0 wa,  0.0 hi,  8.7 si,  0.0 st
%Cpu6  :  0.0 us,  1.1 sy,  0.0 ni, 88.8 id,  0.0 wa,  0.0 hi, 10.1 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni, 93.5 id,  0.0 wa,  0.0 hi,  6.5 si,  0.0 st
%Cpu8  :  0.0 us,  1.2 sy,  0.0 ni, 91.8 id,  0.0 wa,  0.0 hi,  7.1 si,  0.0 st
%Cpu9  :  1.2 us,  1.2 sy,  0.0 ni, 93.0 id,  0.0 wa,  0.0 hi,  4.7 si,  0.0 st
%Cpu10 :  1.1 us,  1.1 sy,  0.0 ni, 89.7 id,  0.0 wa,  1.1 hi,  6.9 si,  0.0 st
%Cpu11 :  2.4 us,  0.0 sy,  0.0 ni, 87.1 id,  0.0 wa,  0.0 hi, 10.6 si,  0.0 st
%Cpu12 :  1.3 us,  0.0 sy,  0.0 ni, 78.5 id,  0.0 wa,  0.0 hi, 20.3 si,  0.0 st
%Cpu13 :  1.2 us,  0.0 sy,  0.0 ni, 88.0 id,  0.0 wa,  1.2 hi,  9.6 si,  0.0 st
%Cpu14 :  0.0 us,  1.1 sy,  0.0 ni, 89.7 id,  0.0 wa,  0.0 hi,  9.2 si,  0.0 st
%Cpu15 :  0.0 us,  0.0 sy,  0.0 ni, 94.4 id,  0.0 wa,  0.0 hi,  5.6 si,  0.0 st
%Cpu16 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu17 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu18 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu19 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu20 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu21 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu22 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu23 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

Проблема вся в том, что по 8 последним ядрам нет нагрузки по прерываниям. Очереди драйвер создал 24 на одну сетевку и 24 на другую (TxRx). То есть размазывает прерывания по 16 ядрам, по остальным не хочет. Это специфика драйвера ixgbe такая?

по прерываниям такая картинка

eth1

  98:  451691993          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-0
  99:          1  445016073          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-1
 100:          1          0  446014472          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-2
 101:          1          0          0  444088129          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-3
 102:          1          0          0          0  478161044          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-4
 103:          1          0          0          0          0  483059259          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-5
 104:          1          0          0          0          0          0  475629251          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-6
 105:          1          0          0          0          0          0          0  474842510          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-7
 106:          1          0          0          0          0          0          0          0  469005169          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-8
 107:          1          0          0          0          0          0          0          0          0  470758432          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-9
 108:          1          0          0          0          0          0          0          0          0          0  459667617          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-10
 109:          1          0          0          0          0          0          0          0          0          0          0  467170424          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-11
 110:          1          0          0          0          0          0          0          0          0          0          0          0  450319404          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-12
 111:          1          0          0          0          0          0          0          0          0          0          0          0          0  452297356          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-13
 112:          1          0          0          0          0          0          0          0          0          0          0          0          0          0  447373716          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-14
 113:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0  451893552          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-15
 114:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     382535          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-16
 115:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     168467          0          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-17
 116:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     336076          0          0          0          0          0   PCI-MSI-edge      eth1-TxRx-18
 117:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0    1293952          0          0          0          0   PCI-MSI-edge      eth1-TxRx-19
 118:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     541888          0          0          0   PCI-MSI-edge      eth1-TxRx-20
 119:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     447686          0          0   PCI-MSI-edge      eth1-TxRx-21
 120:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     442839          0   PCI-MSI-edge      eth1-TxRx-22
 121:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     574538   PCI-MSI-edge      eth1-TxRx-23

eth0

  65:  431321605          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-0
  66:          3  419828534          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-1
  67:          3          0  423198727          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-2
  68:          3          0          0  424799395          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-3
  69:          3          0          0          0  452225991          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-4
  70:          3          0          0          0          0  466992092          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-5
  71:          3          0          0          0          0          0  452022478          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-6
  72:          3          0          0          0          0          0          0  443537105          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-7
  73:          3          0          0          0          0          0          0          0  442787960          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-8
  74:          3          0          0          0          0          0          0          0          0  444841677          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-9
  75:          3          0          0          0          0          0          0          0          0          0  445165476          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-10
  76:          3          0          0          0          0          0          0          0          0          0          0  450890583          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-11
  77:          3          0          0          0          0          0          0          0          0          0          0          0  433584179          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-12
  78:          3          0          0          0          0          0          0          0          0          0          0          0          0  427230045          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-13
  79:          3          0          0          0          0          0          0          0          0          0          0          0          0          0  422875469          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-14
  80:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0  429782736          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-15
  81:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66036          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-16
  82:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66010          0          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-17
  83:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66052          0          0          0          0          0   PCI-MSI-edge      eth0-TxRx-18
  84:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      68424          0          0          0          0   PCI-MSI-edge      eth0-TxRx-19
  85:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66085          0          0          0   PCI-MSI-edge      eth0-TxRx-20
  86:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66149          0          0   PCI-MSI-edge      eth0-TxRx-21
  87:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66026          0   PCI-MSI-edge      eth0-TxRx-22
  88:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0      66048   PCI-MSI-edge      eth0-TxRx-23

Сам драйвер загружается так, версия последняя 3.22.3

[    7.216155] ixgbe: Direct Cache Access (DCA) set to 0
[    7.216157] ixgbe: 0000:04:00.0: ixgbe_check_options: DCA is disabled
[    7.216159] ixgbe: Invalid Receive-Side Scaling (RSS) specified (24),  using default.
[    7.216161] ixgbe: Enabled/Disable FCoE offload Disabled
[    7.216163] ixgbe: 0000:04:00.0: ixgbe_check_options: FCoE Offload feature disabled
[    7.216164] ixgbe: LRO - Large Receive Offload Disabled
[    7.216165] ixgbe: allow_unsupported_sfp Enabled
[    7.381646] ixgbe 0000:04:00.0: irq 65 for MSI/MSI-X
[    7.381657] ixgbe 0000:04:00.0: irq 66 for MSI/MSI-X
[    7.381665] ixgbe 0000:04:00.0: irq 67 for MSI/MSI-X
[    7.381674] ixgbe 0000:04:00.0: irq 68 for MSI/MSI-X
[    7.381682] ixgbe 0000:04:00.0: irq 69 for MSI/MSI-X
[    7.381690] ixgbe 0000:04:00.0: irq 70 for MSI/MSI-X
[    7.381698] ixgbe 0000:04:00.0: irq 71 for MSI/MSI-X
[    7.381706] ixgbe 0000:04:00.0: irq 72 for MSI/MSI-X
[    7.381714] ixgbe 0000:04:00.0: irq 73 for MSI/MSI-X
[    7.381722] ixgbe 0000:04:00.0: irq 74 for MSI/MSI-X
[    7.381730] ixgbe 0000:04:00.0: irq 75 for MSI/MSI-X
[    7.381738] ixgbe 0000:04:00.0: irq 76 for MSI/MSI-X
[    7.381746] ixgbe 0000:04:00.0: irq 77 for MSI/MSI-X
[    7.381755] ixgbe 0000:04:00.0: irq 78 for MSI/MSI-X
[    7.381762] ixgbe 0000:04:00.0: irq 79 for MSI/MSI-X
[    7.381770] ixgbe 0000:04:00.0: irq 80 for MSI/MSI-X
[    7.381778] ixgbe 0000:04:00.0: irq 81 for MSI/MSI-X
[    7.381786] ixgbe 0000:04:00.0: irq 82 for MSI/MSI-X
[    7.381794] ixgbe 0000:04:00.0: irq 83 for MSI/MSI-X
[    7.381802] ixgbe 0000:04:00.0: irq 84 for MSI/MSI-X
[    7.381810] ixgbe 0000:04:00.0: irq 85 for MSI/MSI-X
[    7.381817] ixgbe 0000:04:00.0: irq 86 for MSI/MSI-X
[    7.381825] ixgbe 0000:04:00.0: irq 87 for MSI/MSI-X
[    7.381833] ixgbe 0000:04:00.0: irq 88 for MSI/MSI-X
[    7.381842] ixgbe 0000:04:00.0: irq 89 for MSI/MSI-X
[    7.384085] ixgbe 0000:04:00.0: PCI Express bandwidth of 32GT/s available
[    7.384089] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    7.384177] ixgbe 0000:04:00.0 eth0: MAC: 2, PHY: 9, SFP+: 3, PBA No: E68793-006
[    7.384179] ixgbe 0000:04:00.0: 90:e2:ba:5b:f2:dc
[    7.384182] ixgbe 0000:04:00.0 eth0: Enabled Features: RxQ: 24 TxQ: 24 FdirHash 
[    7.384214] ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
[    7.384452] ixgbe: Direct Cache Access (DCA) set to 0
[    7.384454] ixgbe: 0000:04:00.1: ixgbe_check_options: DCA is disabled
[    7.384456] ixgbe: Invalid Receive-Side Scaling (RSS) specified (24),  using default.
[    7.384459] ixgbe: Enabled/Disable FCoE offload Disabled
[    7.384460] ixgbe: 0000:04:00.1: ixgbe_check_options: FCoE Offload feature disabled
[    7.384461] ixgbe: LRO - Large Receive Offload Disabled
[    7.384463] ixgbe: allow_unsupported_sfp Enabled
[    7.548426] ixgbe 0000:04:00.1: irq 98 for MSI/MSI-X
[    7.548436] ixgbe 0000:04:00.1: irq 99 for MSI/MSI-X
[    7.548444] ixgbe 0000:04:00.1: irq 100 for MSI/MSI-X
[    7.548452] ixgbe 0000:04:00.1: irq 101 for MSI/MSI-X
[    7.548460] ixgbe 0000:04:00.1: irq 102 for MSI/MSI-X
[    7.548469] ixgbe 0000:04:00.1: irq 103 for MSI/MSI-X
[    7.548477] ixgbe 0000:04:00.1: irq 104 for MSI/MSI-X
[    7.548485] ixgbe 0000:04:00.1: irq 105 for MSI/MSI-X
[    7.548493] ixgbe 0000:04:00.1: irq 106 for MSI/MSI-X
[    7.548501] ixgbe 0000:04:00.1: irq 107 for MSI/MSI-X
[    7.548509] ixgbe 0000:04:00.1: irq 108 for MSI/MSI-X
[    7.548517] ixgbe 0000:04:00.1: irq 109 for MSI/MSI-X
[    7.548525] ixgbe 0000:04:00.1: irq 110 for MSI/MSI-X
[    7.548533] ixgbe 0000:04:00.1: irq 111 for MSI/MSI-X
[    7.548541] ixgbe 0000:04:00.1: irq 112 for MSI/MSI-X
[    7.548549] ixgbe 0000:04:00.1: irq 113 for MSI/MSI-X
[    7.548559] ixgbe 0000:04:00.1: irq 114 for MSI/MSI-X
[    7.548567] ixgbe 0000:04:00.1: irq 115 for MSI/MSI-X
[    7.548575] ixgbe 0000:04:00.1: irq 116 for MSI/MSI-X
[    7.548582] ixgbe 0000:04:00.1: irq 117 for MSI/MSI-X
[    7.548591] ixgbe 0000:04:00.1: irq 118 for MSI/MSI-X
[    7.548598] ixgbe 0000:04:00.1: irq 119 for MSI/MSI-X
[    7.548606] ixgbe 0000:04:00.1: irq 120 for MSI/MSI-X
[    7.548615] ixgbe 0000:04:00.1: irq 121 for MSI/MSI-X
[    7.548623] ixgbe 0000:04:00.1: irq 122 for MSI/MSI-X
[    7.550841] ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s available
[    7.550845] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    7.550931] ixgbe 0000:04:00.1 eth1: MAC: 2, PHY: 9, SFP+: 4, PBA No: E68793-006
[    7.550934] ixgbe 0000:04:00.1: 90:e2:ba:5b:f2:dd
[    7.550936] ixgbe 0000:04:00.1 eth1: Enabled Features: RxQ: 24 TxQ: 24 FdirHash 
[    7.550966] ixgbe 0000:04:00.1 eth1: Intel(R) 10 Gigabit Network Connection
[    9.087569] ixgbe 0000:04:00.0: registered PHC device on eth0
[    9.260937] ixgbe 0000:04:00.0 eth0: detected SFP+: 3
[    9.393484] ixgbe 0000:04:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[   10.437546] ixgbe 0000:04:00.0 eth0: detected SFP+: 3
[   10.576836] ixgbe 0000:04:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[   12.525256] ixgbe 0000:04:00.1: registered PHC device on eth1
[   12.690187] ixgbe 0000:04:00.1 eth1: detected SFP+: 4
[   13.704555] ixgbe 0000:04:00.1 eth1: detected SFP+: 4
[   13.943641] ixgbe 0000:04:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: None

Ссылка

Invalid Receive-Side Scaling (RSS) specified (24), using default.

В документации указано, что RSS может быть от 0 до 16, по умолчанию не более числа процессоров или 16. Возможно, именно по этому только 16 процессоров обрабатывают прерывания.

mky ★★★★★
(10.12.14 10:35:31 MSK)

Ответ на: комментарий от mky 10.12.14 10:35:31 MSK

прикольно, а как быть если вообще 4х процессорная мамка с 32 ядрами без HT к примеру? накидывать каждую сетевку на проц по прерываниям?

init_ ★★★
(10.12.14 11:23:27 MSK) автор топика

Ответ на: комментарий от init_ 10.12.14 11:23:27 MSK

Обработка прерываний не столь длительное действие по сравнению с дальнейшей обработкой пакета.

Обработку принимаемых пакетов можно распаралелить по ядрам с помощью RPS/RFS.

vel ★★★★★
(10.12.14 14:55:02 MSK)

Ответ на: комментарий от vel 10.12.14 14:55:02 MSK

а можно поподробней?

init_ ★★★
(10.12.14 15:40:15 MSK) автор топика

Ответ на: комментарий от init_ 10.12.14 15:40:15 MSK

Documentation/networking/scaling.txt

Это помогает если SoftIRQ кушает больше чем IRQ.

vel ★★★★★
(10.12.14 16:07:36 MSK)

Похожие темы