Приветствую, ЛОР!
Есть банальная задача: Соединить два centos7 сервера ipip-туннелем и смаршрутизировать через него локальные сети друг друга, которые висят на bridge интерфейсах. Что-то пошло не так на этапе маршрутизации, и если с host1 с адреса туннеля доступны адреса br1 и br2 на host2, то с адресов br1 и br2 на host1 - уже нет. Под катом инфа сети и диагностика.
Карта сети (или что должно быть в итоге)
#
# host1 config
#
#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ae:37:c3 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.76/24 brd 192.168.0.255 scope global dynamic enp0s3
valid_lft 258553sec preferred_lft 258553sec
inet6 fe80::3d29:1cc:c1ca:7437/64 scope link
valid_lft forever preferred_lft forever
3: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:61:b6:cf:1e:14 brd ff:ff:ff:ff:ff:ff
4: dummy1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 36:6f:b9:86:64:22 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 52:b8:91:9f:dc:91 brd ff:ff:ff:ff:ff:ff
inet 10.10.0.1/24 brd 10.10.0.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::50b8:91ff:fe9f:dc91/64 scope link
valid_lft forever preferred_lft forever
6: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 02:57:44:16:57:d2 brd ff:ff:ff:ff:ff:ff
inet 10.10.1.1/24 brd 10.10.1.255 scope global br1
valid_lft forever preferred_lft forever
inet6 fe80::57:44ff:fe16:57d2/64 scope link
valid_lft forever preferred_lft forever
7: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
8: tun0@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN qlen 1
link/ipip 192.168.0.76 peer 192.168.0.77
inet 192.168.254.1 peer 192.168.254.2/30 scope global tun0
valid_lft forever preferred_lft forever
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:8b:75:9a:4a brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
#ip r s
default via 192.168.0.1 dev enp0s3 proto static metric 100
10.10.0.0/24 dev br0 proto kernel scope link src 10.10.0.1
10.10.1.0/24 dev br1 proto kernel scope link src 10.10.1.1
10.20.0.0/24 dev tun0 scope link
10.20.1.0/24 dev tun0 scope link
169.254.0.0/16 dev br0 scope link metric 1005
169.254.0.0/16 dev br1 scope link metric 1006
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.0.0/24 dev enp0s3 proto kernel scope link src 192.168.0.76 metric 100
192.168.254.0/30 dev tun0 proto kernel scope link src 192.168.254.1
192.168.254.0/24 dev tun0 scope link
# ip ru
0: from all lookup local
32763: from 192.168.254.1 lookup tabletun0
32764: from all to 10.20.1.0/24 lookup tabletun0
32765: from all to 10.20.0.0/24 lookup tabletun0
32766: from all lookup main
32767: from all lookup default
# ip r s t tabletun0
default dev tun0 scope link
# iptables-save
# Generated by iptables-save v1.4.21 on Tue May 22 04:35:45 2018
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
COMMIT
# Completed on Tue May 22 04:35:45 2018
# Generated by iptables-save v1.4.21 on Tue May 22 04:35:45 2018
*filter
:INPUT ACCEPT [41:2780]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [21:2188]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
COMMIT
# Completed on Tue May 22 04:35:45 2018
#
# host2 config
#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ae:37:c3 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.77/24 brd 192.168.0.255 scope global enp0s3
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:feae:37c3/64 scope link
valid_lft forever preferred_lft forever
3: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN qlen 1000
link/ether da:ec:60:9c:fc:91 brd ff:ff:ff:ff:ff:ff
4: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 72:77:8b:b5:8b:1e brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether da:ec:60:9c:fc:91 brd ff:ff:ff:ff:ff:ff
inet 10.20.0.1/24 brd 10.20.0.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::d8ec:60ff:fe9c:fc91/64 scope link
valid_lft forever preferred_lft forever
6: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
7: tun0@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN qlen 1
link/ipip 192.168.0.77 peer 192.168.0.76
inet 192.168.254.2 peer 192.168.254.1/30 scope global tun0
valid_lft forever preferred_lft forever
8: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 8a:4e:6e:00:3e:24 brd ff:ff:ff:ff:ff:ff
inet 10.20.1.1/24 brd 10.20.1.255 scope global br1
valid_lft forever preferred_lft forever
inet6 fe80::884e:6eff:fe00:3e24/64 scope link
valid_lft forever preferred_lft forever
#ip r s
default via 192.168.0.1 dev enp0s3 proto static metric 100
10.10.0.0/24 dev tun0 scope link
10.10.1.0/24 dev tun0 scope link
10.20.0.0/24 dev br0 proto kernel scope link src 10.20.0.1
10.20.1.0/24 dev br1 proto kernel scope link src 10.20.1.1
169.254.0.0/16 dev br0 scope link metric 1005
169.254.0.0/16 dev br1 scope link metric 1008
192.168.0.0/24 dev enp0s3 proto kernel scope link src 192.168.0.77 metric 100
192.168.254.0/30 dev tun0 proto kernel scope link src 192.168.254.2
192.168.254.0/24 dev tun0 scope link
# ip ru
0: from all lookup local
32765: from 192.168.254.2 lookup tabletun0
32766: from all lookup main
32767: from all lookup default
# ip r s t tabletun0
default dev tun0 scope link
# iptables-save
# Generated by iptables-save v1.4.21 on Mon May 21 14:27:29 2018
*nat
:PREROUTING ACCEPT [85:7608]
:INPUT ACCEPT [16:2688]
:OUTPUT ACCEPT [2:152]
:POSTROUTING ACCEPT [2:152]
COMMIT
# Completed on Mon May 21 14:27:29 2018
# Generated by iptables-save v1.4.21 on Mon May 21 14:27:29 2018
*filter
:INPUT ACCEPT [20:1467]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [11:1104]
COMMIT
# Completed on Mon May 21 14:27:29 2018
#
# host1 debug
#
# ping 10.20.1.1
PING 10.20.1.1 (10.20.1.1) 56(84) bytes of data.
64 bytes from 10.20.1.1: icmp_seq=1 ttl=64 time=0.417 ms
64 bytes from 10.20.1.1: icmp_seq=2 ttl=64 time=1.28 ms
# tcpdump -nni any not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
04:41:36.016368 IP 192.168.0.63.137 > 192.168.0.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
04:41:36.159857 ARP, Request who-has 192.168.0.76 tell 192.168.0.77, length 46
04:41:36.159947 ARP, Reply 192.168.0.76 is-at 08:00:27:ae:37:c3, length 28
04:41:36.834556 IP 192.168.0.63.137 > 192.168.0.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
04:41:37.552177 IP 192.168.0.63.137 > 192.168.0.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
04:41:37.805327 IP 192.168.254.1 > 10.20.1.1: ICMP echo request, id 1743, seq 1, length 64
04:41:37.805345 IP 192.168.0.76 > 192.168.0.77: IP 192.168.254.1 > 10.20.1.1: ICMP echo request, id 1743, seq 1, length 64 (ipip-proto-4)
04:41:37.805663 IP 192.168.0.77 > 192.168.0.76: IP 10.20.1.1 > 192.168.254.1: ICMP echo reply, id 1743, seq 1, length 64 (ipip-proto-4)
04:41:37.805713 IP 10.20.1.1 > 192.168.254.1: ICMP echo reply, id 1743, seq 1, length 64
04:41:38.805772 IP 192.168.254.1 > 10.20.1.1: ICMP echo request, id 1743, seq 2, length 64
04:41:38.805815 IP 192.168.0.76 > 192.168.0.77: IP 192.168.254.1 > 10.20.1.1: ICMP echo request, id 1743, seq 2, length 64 (ipip-proto-4)
04:41:38.806769 IP 192.168.0.77 > 192.168.0.76: IP 10.20.1.1 > 192.168.254.1: ICMP echo reply, id 1743, seq 2, length 64 (ipip-proto-4)
04:41:38.806970 IP 10.20.1.1 > 192.168.254.1: ICMP echo reply, id 1743, seq 2, length 64
04:41:39.496721 ARP, Request who-has 192.168.0.113 tell 192.168.0.123, length 46
# ip r get 10.20.1.1
10.20.1.1 dev tun0 src 192.168.254.1
cache expires 396sec mtu 1480
# ping 10.20.1.1 -I br0
PING 10.20.1.1 (10.20.1.1) from 10.10.0.1 br0: 56(84) bytes of data.
^C
--- 10.20.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
# tcpdump -nni any not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
04:43:34.401275 ARP, Request who-has 10.20.1.1 tell 10.10.0.1, length 28
04:43:35.403213 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:35.403238 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:35.403254 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:35.403269 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:36.397348 ARP, Request who-has 10.20.1.1 tell 10.10.0.1, length 28
04:43:36.708938 ARP, Request who-has 192.168.0.76 (08:00:27:ae:37:c3) tell 192.168.0.83, length 46
04:43:36.708997 ARP, Reply 192.168.0.76 is-at 08:00:27:ae:37:c3, length 28
04:43:37.399550 ARP, Request who-has 10.20.1.1 tell 10.10.0.1, length 28
04:43:38.401714 ARP, Request who-has 10.20.1.1 tell 10.10.0.1, length 28
04:43:39.403103 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:39.403128 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:39.403144 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:39.403157 IP 10.10.0.1 > 10.10.0.1: ICMP host 10.20.1.1 unreachable, length 92
04:43:39.512526 ARP, Request who-has 192.168.0.113 tell 192.168.0.123, length 46