LINUX.ORG.RU
решено Форум — Admin  

Не работает DNAT (4 провайдера, маркировка пакетов, netfilter, iproute2,gentoo)

 , , ,


0

1

Доброго времени всем.

При настройке одной хитрой схемы циркуляции траффика столкнулся с проблемой: если интрефейс, на который приходить пакет для dnat-а не используется для связи со шлюзом по-умолчанию в таблице маршрутизации main, правило dnat жрет пакеты (т.е. пакеты не доходят даже до места назначения, не говоря уже о conntrack).

Ниже даны версии программ: 

gateway ~ # uname -a
Linux gateway 3.8.13-gentoo #5 SMP Wed Jun 5 12:50:19 YEKT 2013 x86_64 Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz GenuineIntel GNU/Linux
gateway ~ # iptables
iptables v1.4.16.3: no command specified
Try `iptables -h' or 'iptables --help' for more information.
gateway ~ # ip -V
ip utility, iproute2-ss130221
gateway ~ # cat /etc/gentoo-release
Gentoo Base System release 2.2

Ниже даны конфиги ip, iptables: 

gateway ~ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: enp10s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
4: enp3s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
5: enp3s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
6: enp3s0f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
7: enp3s0f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
9: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT qlen 100
    link/ether xx:xx:xx:xx:xx:xy brd ff:ff:ff:ff:ff:ff
10: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT qlen 500
    link/ether xx:xx:xx:xx:xx:yy brd ff:ff:ff:ff:ff:ff
11: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
12: vlan100@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
13: vlan101@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
14: vlan102@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
15: vlan103@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff


gateway ~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
3: enp10s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
4: enp3s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
5: enp3s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
6: enp3s0f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
7: enp3s0f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP
    link/ether d4:3d:7e:93:f6:99 brd ff:ff:ff:ff:ff:ff
9: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP qlen 100
    link/ether 00:a0:b0:c0:d0:f0 brd ff:ff:ff:ff:ff:ff
10: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP qlen 500
    link/ether 5e:5e:0f:0e:35:2e brd ff:ff:ff:ff:ff:ff
11: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:0a:0b:0c:0d:0f brd ff:ff:ff:ff:ff:ff
    inet 10.220.0.1/16 brd 10.220.255.255 scope global br0
12: vlan100@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:0a:0b:0c:0d:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.10/24 brd 192.168.100.255 scope global vlan100
13: vlan101@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:0a:0b:0c:0d:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.10/24 brd 192.168.101.255 scope global vlan101
14: vlan102@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:0a:0b:0c:0d:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.102.10/24 brd 192.168.102.255 scope global vlan102
15: vlan103@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:0a:0b:0c:0d:0f brd ff:ff:ff:ff:ff:ff
    inet xx.xx.xx.52/25 brd xx.xx.xx.127 scope global vlan103

gateway ~ # ip route show table all
default via 192.168.100.9 dev vlan100  table inet_isnet  metric 12
10.220.0.0/16 dev br0  table inet_isnet  proto kernel  scope link  src 10.220.0.1  metric 11
192.168.100.0/24 dev vlan100  table inet_isnet  scope link  metric 12
default via 192.168.101.9 dev vlan101  table inet_planeta  metric 13
10.220.0.0/16 dev br0  table inet_planeta  proto kernel  scope link  src 10.220.0.1  metric 11
192.168.101.0/24 dev vlan101  table inet_planeta  scope link  metric 13
default via 192.168.102.9 dev vlan102  table inet_nms  metric 14
10.220.0.0/16 dev br0  table inet_nms  proto kernel  scope link  src 10.220.0.1  metric 11
192.168.102.0/24 dev vlan102  table inet_nms  scope link  metric 14
default via xx.xx.xx.5 dev vlan103  table inet_nc  metric 15
10.220.0.0/16 dev br0  table inet_nc  proto kernel  scope link  src 10.220.0.1  metric 15
xx.xx.xx.0/25 dev vlan103  table inet_nc  scope link  metric 15
default via 192.168.102.9 dev vlan102  metric 10
default via xx.xx.xx.5 dev vlan103  metric 20
default via 192.168.101.9 dev vlan101  metric 30
default via 192.168.100.9 dev vlan100  metric 40
10.220.0.0/16 dev br0  proto kernel  scope link  src 10.220.0.1
xx.xx.xx.0/25 dev vlan103  proto kernel  scope link  src xx.xx.xx.52
127.0.0.0/8 via 127.0.0.1 dev lo
192.168.100.0/24 dev vlan100  proto kernel  scope link  src 192.168.100.10
192.168.101.0/24 dev vlan101  proto kernel  scope link  src 192.168.101.10
192.168.102.0/24 dev vlan102  proto kernel  scope link  src 192.168.102.10
broadcast 10.220.0.0 dev br0  table local  proto kernel  scope link  src 10.220.0.1
local 10.220.0.1 dev br0  table local  proto kernel  scope host  src 10.220.0.1
broadcast 10.220.255.255 dev br0  table local  proto kernel  scope link  src 10.220.0.1
broadcast xx.xx.xx.0 dev vlan103  table local  proto kernel  scope link  src xx.xx.xx.52
local xx.xx.xx.52 dev vlan103  table local  proto kernel  scope host  src xx.xx.xx.52
broadcast xx.xx.xx.127 dev vlan103  table local  proto kernel  scope link  src xx.xx.xx.52
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1
broadcast 192.168.100.0 dev vlan100  table local  proto kernel  scope link  src 192.168.100.10
local 192.168.100.10 dev vlan100  table local  proto kernel  scope host  src 192.168.100.10
broadcast 192.168.100.255 dev vlan100  table local  proto kernel  scope link  src 192.168.100.10
broadcast 192.168.101.0 dev vlan101  table local  proto kernel  scope link  src 192.168.101.10
local 192.168.101.10 dev vlan101  table local  proto kernel  scope host  src 192.168.101.10
broadcast 192.168.101.255 dev vlan101  table local  proto kernel  scope link  src 192.168.101.10
broadcast 192.168.102.0 dev vlan102  table local  proto kernel  scope link  src 192.168.102.10
local 192.168.102.10 dev vlan102  table local  proto kernel  scope host  src 192.168.102.10
broadcast 192.168.102.255 dev vlan102  table local  proto kernel  scope link  src 192.168.102.10

gateway ~ # cat /var/lib/iptables/rules-save
# Generated by iptables-save v1.4.16.3 on Wed Jul  3 20:07:49 2013
*raw
[0:0] -A PREROUTING -p tcp -m tcp --dport 9100 -j TRACE
[0:0] -A PREROUTING -s 91.207.40.45 -d 239.255.255.250 -j DROP
[0:0] -A PREROUTING -i vlan103 -d 255.255.255.255 -j DROP

COMMIT

*mangle
:PREROUTING ACCEPT [212339:99752202]
:INPUT ACCEPT [66456:44652965]
:FORWARD ACCEPT [142716:54058585]
:OUTPUT ACCEPT [75621:56964350]
:POSTROUTING ACCEPT [218337:111022935]
[108:7796] -A PREROUTING -s 10.220.1.8 -j MARK --set-mark 0x67
[108:7796] -A PREROUTING -i vlan100 -m conntrack --ctstate NEW -j MARK --set-mark 0x64
[109:7842] -A PREROUTING -i vlan101 -m conntrack --ctstate NEW -j MARK --set-mark 0x65
[107:38883] -A PREROUTING -i vlan102 -m conntrack --ctstate NEW -j MARK --set-mark 0x66
[11067:3566757] -A PREROUTING -i vlan103 -m conntrack --ctstate NEW -j MARK --set-mark 0x67
[398305:173708294] -A PREROUTING  -m connmark ! --mark 0x0 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
#[398305:173708294] -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED,SNAT,DNAT -m connmark ! --mark 0x0 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

[0:0] -A PREROUTING -m mark ! --mark 0x0 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff

[561:26928] -A PREROUTING -d 91.207.40.52/32 -p tcp -m tcp --dport 9100 -j LOG --log-prefix "mangle_prt_dport9100:"
[300:15212] -A FORWARD -p tcp -m tcp --dport 9100 -j LOG --log-prefix "mangle_fwd_dport9100:"
[0:0] -A POSTROUTING -p tcp -m tcp --dport 9100 -j LOG --log-prefix "mangle_pst_dport9100:"
COMMIT
# Completed on Wed Jul  3 20:07:49 2013
# Generated by iptables-save v1.4.16.3 on Wed Jul  3 20:07:49 2013
*filter
:INPUT DROP [7254:1549323]
:FORWARD ACCEPT [539203:178125572]
:OUTPUT ACCEPT [428565:293023088]
[288825315:386682075596] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[175471:34088366] -A INPUT -i br0 -j ACCEPT
[3100:183819] -A INPUT -i vlan103 -p tcp -m multiport --dports 80,443 -j ACCEPT
[236:13098] -A INPUT -s 188.226.62.64/32 -j ACCEPT
[0:0] -A INPUT -s 188.226.62.66/32 -p tcp -m tcp --dport 22 -j ACCEPT
[1346:58088] -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
[2:76] -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
[455537:27335100] -A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -i lo -j ACCEPT
[0:0] -A INPUT -s 195.64.208.186/32 -p tcp -m tcp --dport 22 -j ACCEPT
#[0:0] -A INPUT -d 91.207.40.52/32 -p tcp -m tcp --dport 2222 -j ACCEPT
#[0:0] -A INPUT -d 91.207.40.52/32 -p tcp -m tcp --dport 9006 -j ACCEPT
#[0:0] -A INPUT -d 91.207.40.52/32 -p tcp -m tcp --dport 9100 -j ACCEPT
[0:0] -A INPUT -p tcp -m tcp --dport 9100 --j LOG --log-prefix "filter_in_tcp9100:"
[0:0] -A FORWARD -p tcp -m tcp --dport 9100 --j LOG --log-prefix "filter_fwd_tcp9100:"
[0:0] -A OUTPUT -p tcp -m tcp --dport 9100 --j LOG --log-prefix "filter_out_tcp9100:"


COMMIT
# Completed on Wed Jul  3 20:07:49 2013
# Generated by iptables-save v1.4.16.3 on Wed Jul  3 20:07:49 2013
*nat
:PREROUTING ACCEPT [5215:1147937]
:INPUT ACCEPT [26:3548]
:OUTPUT ACCEPT [2226:159669]
:POSTROUTING ACCEPT [2712:193906]
#[5601:1216919] -A PREROUTING -m conntrack --ctstate NEW -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
[0:0] -A PREROUTING -p tcp -m tcp --dport 9100 --j LOG --log-prefix "nat_prt_tcp9100:"
[0:0] -A PREROUTING -d 91.207.40.52/32 -i vlan103 -p tcp -m tcp --dport 9100 -j DNAT --to-destination 10.220.10.8
[0:0] -A PREROUTING -d 91.207.40.52/32 -p tcp -m tcp --dport 9006 -j DNAT --to-destination 10.220.10.8:9006
[0:0] -A PREROUTING -d 91.207.40.52/32 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.220.10.8:2222
[0:0] -A POSTROUTING -m conntrack --ctstate DNAT -j LOG --log-prefix "nat_pst_ctsate_DNAT:"
[0:0] -A PREROUTING -m conntrack --ctstate NEW -p tcp -m tcp --dport 9100 --j LOG --log-prefix "[DNAT]nat_prt_ctstate_NEW_tcp9100:"
[0:0] -A PREROUTING -m mark ! --mark 0x0 --j LOG --log-prefix "nat_post_mark:"
[0:0] -A POSTROUTING -p tcp -m tcp --dport 9100 -j LOG --log-prefix "nat_pst_dport9100:"
[855827:65083086] -A POSTROUTING -s 10.220.0.0/16 ! -d 10.220.0.0/16 -j MASQUERADE
COMMIT
# Completed on Wed Jul  3 20:07:49 2013

Кусок логов: http://pastebin.com/DWF19PzN

P.S. xx.xx.xx.?? - ip и сеть провайдера №2

yy.yy.yy.64 - ip, с которого проводится проверка функционирования



Последнее исправление: cetjs2 (всего исправлений: 1)
← debian и suhosin
RHEV 3.2 →

Не особо разобрался в вашей ситуации, но что сразу бросилось в глаза:
1) по правилам таблицы NAT, должны DNAT-иться TCP-пакеты проходящие через этот роутер на 91.207.40.52:9100 попавшие с интерфейса vlan103 и проходящие на 91.207.40.52:9006 и 91.207.40.52:2222 попавшие с любого интерфейса
2) в логах этот ip-адрес не проскакивает нигде
3) в правилах счетчики по нулям
значит таких пакетов либо небыло, либо они не попали под эти правила.

DiMoN ★★★
()
Последнее исправление: DiMoN (всего исправлений: 5)
Ответ на: комментарий от DiMoN

91.207.40.52

Ooops, таки спалил ip. :)

А по теме, в логах этот адрес фигурирует, как xx.xx.xx.52

Статистика по правилам не нулевая (просто не делал iptables-save).

На текущий момент трейс выглядит так: 

[417497.737317] TRACE: raw:PREROUTING:policy:4 IN=vlan103 OUT= MAC=00:0a:0b:0c:0d:0f:00:80:48:5a:83:0b:08:00 SRC=188.226.62.64 DST=91.207.40.52 LEN=48 TOS=0x00 PREC=0x00 TTL=122 ID=19612 DF PROTO=TCP SPT=12283 DPT=9100 SEQ=198881654 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402)
[417497.737329] TRACE: mangle:PREROUTING:rule:5 IN=vlan103 OUT= MAC=00:0a:0b:0c:0d:0f:00:80:48:5a:83:0b:08:00 SRC=188.226.62.64 DST=91.207.40.52 LEN=48 TOS=0x00 PREC=0x00 TTL=122 ID=19612 DF PROTO=TCP SPT=12283 DPT=9100 SEQ=198881654 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402)
[417497.737334] TRACE: mangle:PREROUTING:rule:8 IN=vlan103 OUT= MAC=00:0a:0b:0c:0d:0f:00:80:48:5a:83:0b:08:00 SRC=188.226.62.64 DST=91.207.40.52 LEN=48 TOS=0x00 PREC=0x00 TTL=122 ID=19612 DF PROTO=TCP SPT=12283 DPT=9100 SEQ=198881654 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402) MARK=0x67
[417497.737339] TRACE: mangle:PREROUTING:policy:9 IN=vlan103 OUT= MAC=00:0a:0b:0c:0d:0f:00:80:48:5a:83:0b:08:00 SRC=188.226.62.64 DST=91.207.40.52 LEN=48 TOS=0x00 PREC=0x00 TTL=122 ID=19612 DF PROTO=TCP SPT=12283 DPT=9100 SEQ=198881654 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402) MARK=0x67
[417497.737344] TRACE: nat:PREROUTING:rule:1 IN=vlan103 OUT= MAC=00:0a:0b:0c:0d:0f:00:80:48:5a:83:0b:08:00 SRC=188.226.62.64 DST=91.207.40.52 LEN=48 TOS=0x00 PREC=0x00 TTL=122 ID=19612 DF PROTO=TCP SPT=12283 DPT=9100 SEQ=198881654 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B401010402) MARK=0x67

alxrt
() автор топика

РЕШЕНИЕ

sysctl net.ipv4.conf.vlan103.rp_filter=0
alxrt
() автор топика
Вы не можете добавлять комментарии в эту тему. Тема перемещена в архив.
← debian и suhosin
Admin
RHEV 3.2 →