Всем бобра!
2е ноды Proxmox 6 1 сан нас Freenas 11.2 Везде стоят карты qlogic 2462/2432, в общем как по мануалу.. Но, если соеденить напрямую, т.е. оптическим кабелем 2 адаптера 2462 у ноды и фринаса - всё здорово, все работает без ошибок. Но, как только в схему добавляется свитч - в логах ошибки доступа и через некоторое время фринас вовсе отваливается от обоих нод.
dmesg с ноды
[10399.613062] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10399.629085] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=000000001182c89e.
[10399.631165] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=000000001182c89e.
[10399.631360] sd 3:0:0:0: Power-on or device reset occurred
[10418.756463] buffer_io_error: 100 callbacks suppressed
[10418.756465] Buffer I/O error on dev dm-5, logical block 982125, async page read
[10430.331069] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10430.347063] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=00000000324876d8.
[10430.349109] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=00000000324876d8.
[10430.349275] sd 3:0:0:0: Power-on or device reset occurred
[10448.943917] Buffer I/O error on dev dm-5, logical block 1222915, async page read
[10478.772637] Buffer I/O error on dev dm-5, logical block 1134009, async page read
[10478.773202] Buffer I/O error on dev dm-5, logical block 1134009, async page read
[10481.527827] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10481.543838] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=00000000324876d8.
[10481.545939] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=00000000324876d8.
[10481.546109] sd 3:0:0:0: Power-on or device reset occurred
[10509.418071] Buffer I/O error on dev dm-5, logical block 1056498, lost async page write
[10509.418589] Buffer I/O error on dev dm-5, logical block 2075239, lost async page write
[10509.418942] Buffer I/O error on dev dm-5, logical block 2081138, lost async page write
[10509.419259] Buffer I/O error on dev dm-5, logical block 2531743, lost async page write
[10509.419566] Buffer I/O error on dev dm-5, logical block 2531744, lost async page write
[10509.419867] Buffer I/O error on dev dm-5, logical block 2531745, lost async page write
[10509.420162] Buffer I/O error on dev dm-5, logical block 2531746, lost async page write
[10509.420449] Buffer I/O error on dev dm-5, logical block 2531747, lost async page write
[10509.420727] Buffer I/O error on dev dm-5, logical block 2531748, lost async page write
[10509.421005] Buffer I/O error on dev dm-5, logical block 2531749, lost async page write
[10512.245892] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10512.265876] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=000000009b57018c.
[10512.267929] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=000000009b57018c.
[10512.268096] sd 3:0:0:0: Power-on or device reset occurred
[10538.928372] buffer_io_error: 9 callbacks suppressed
[10538.928374] Buffer I/O error on dev dm-5, logical block 990326, async page read
[10542.964022] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10542.980040] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=00000000c0ab3bb9.
[10542.982086] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=00000000c0ab3bb9.
[10542.982249] sd 3:0:0:0: Power-on or device reset occurred
[10552.009159] sd 3:0:0:0: Power-on or device reset occurred
[10568.756425] Buffer I/O error on dev dm-5, logical block 474872, async page read
[10598.928246] Buffer I/O error on dev dm-5, logical block 972252, async page read
[10604.400398] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10604.416401] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=00000000aa543d93.
[10604.418441] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=00000000aa543d93.
[10604.418612] sd 3:0:0:0: Power-on or device reset occurred
[10628.693954] Buffer I/O error on dev dm-5, logical block 425659, async page read
[10628.694514] Buffer I/O error on dev dm-5, logical block 425659, async page read
[10658.772125] Buffer I/O error on dev dm-5, logical block 990105, async page read
[10658.772640] Buffer I/O error on dev dm-5, logical block 990105, async page read
[10718.756594] Buffer I/O error on dev dm-5, logical block 963335, async page read
[10719.581941] Buffer I/O error on dev dm-5, logical block 2102270, lost async page write
[10735.465056] qla2xxx [0000:05:00.0]-801c:3: Abort command issued nexus=3:0:0 — 0 2003.
[10735.493059] qla2xxx [0000:05:00.0]-8009:3: DEVICE RESET ISSUED nexus=3:0:0 cmd=000000008cdfa8b1.
[10735.495100] qla2xxx [0000:05:00.0]-800e:3: DEVICE RESET SUCCEEDED nexus:3:0:0 cmd=000000008cdfa8b1.
[10735.495270] sd 3:0:0:0: Power-on or device reset occurred
[10748.725213] Buffer I/O error on dev dm-5, logical block 1056047, async page read
dmesg freenas
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[115228] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[115bc8] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[11976c] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119798] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[1197f0] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[11981c] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119848] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119874] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[1198f8] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119be4] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119c10] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119c68] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119cec] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119d18] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[119ed0] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[121abc] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[115ac0] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[115b44] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[114d84] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[116abc] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[1299c0] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[129b4c] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[1224e0] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[12271c] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[124c68] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[118f00] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[118f2c] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Firmware timed out on command
isp0: isp_handle_platform_ctio: CTIO7[120cd0] seq 0 nc 1 sts 0xb flg 0x1 sns 0 resid 0 MID
isp0: Polled Mailbox Command (0x69) Timeout (100000us) (isp_fw_state:7199)
isp0: Mailbox Command 'GET FW STATE' failed (TIMEOUT)
isp0: mailbox 0x4000 with no waiters
switchshow
switchName: core01
switchType: 34.0
switchState: Online
switchMode: Native
switchRole: Principal
switchDomain: 1
switchId: fffc01
switchWwn: 10:00:00:05:1e:07:d5:ff
zoning: ON (conf_node_01_02_f_nas)
switchBeacon: OFF
Area Port Media Speed State Proto
=====================================
0 0 id N4 Online F-Port 21:00:00:1b:32:85:60:fd
1 1 id N4 Online F-Port 21:00:00:1b:32:83:90:72
2 2 id N4 Online F-Port 21:00:00:1b:32:85:02:93
3 3 — N4 No_Module
4 4 — N4 No_Module
5 5 — N4 No_Module
6 6 — N4 No_Module
7 7 — N4 No_Module
8 8 — N4 No_Module
9 9 — N4 No_Module
10 10 — N4 No_Module
11 11 — N4 No_Module
12 12 — N4 No_Module
13 13 — N4 No_Module
14 14 — N4 No_Module
15 15 — N4 No_Module
cfgshow
Defined configuration:
cfg: conf_node_01_02_f_nas
z_node_01; z_node_02
zone: z_node_01
node_01; f_nas
zone: z_node_02
node_02; f_nas
alias: f_nas 21:00:00:1b:32:85:60:fd
alias: node_01 21:00:00:1b:32:83:90:72
alias: node_02 21:00:00:1b:32:85:02:93
Effective configuration:
cfg: conf_node_01_02_f_nas
zone: z_node_01
21:00:00:1b:32:83:90:72
21:00:00:1b:32:85:60:fd
zone: z_node_02
21:00:00:1b:32:85:02:93
21:00:00:1b:32:85:60:fd
