利用raw表实现iptables调试

记录下调试iptables的过程。才知道iptables还有个raw表(￣.￣)+

背景

在配置folsom版openstack的quantum时出现vm无法ping通外网的问题，经过抓包分析确定问题是iptables中的snat规则不生效，需要调试iptables定位下有问题的iptables规则。 iptables有5个链: PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING，4个表: filter, nat, mangle, raw，其工作流在下图表示的很清楚。

从图中可见raw拥有最高的优先级，raw表使用PREROUTING和OUTPUT两个链，因此raw可以覆盖所有包。在raw表中支持一个特殊的目标:TRACE，使内核记录下每条匹配该包的对应iptables规则信息。使用raw表内的TRACE target即可实现对iptables规则的跟踪调试。

配置

假设需要对ipv4的ICMP包进行跟踪调试，抓取所有流经本机的ICMP包

# iptables -t raw -A OUTPUT -p icmp -j TRACE # iptables -t raw -A PREROUTING -p icmp -j TRACE

加载对应内核模组

modprobe ipt_LOG

调试信息记录在/var/log/kern.log文件。

调试

在vm内对外部作ping操作，vm的ip为10.0.0.4

[root@10-0-0-4 ~]# ping -c 1 192.168.0.19 PING 192.168.0.19 (192.168.0.19) 56(84) bytes of data.   --- 192.168.0.19 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms

在/var/log/kern.log中的对应调试信息如下

Apr 18 11:50:23 openstack-network kernel: [1038991.870882] TRACE: raw:PREROUTING:policy:2 IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870902] TRACE: nat:PREROUTING:rule:1 IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870909] TRACE: nat:quantum-l3-agent-PREROUTING:return:4 IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870915] TRACE: nat:PREROUTING:policy:2 IN=tap5c42978b-ac OUT= MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870938] TRACE: filter:FORWARD:rule:1 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870944] TRACE: filter:quantum-filter-top:rule:1 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870950] TRACE: filter:quantum-l3-agent-local:return:1 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870957] TRACE: filter:quantum-filter-top:return:2 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870962] TRACE: filter:FORWARD:rule:2 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870969] TRACE: filter:quantum-l3-agent-FORWARD:return:1 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870974] TRACE: filter:FORWARD:policy:3 IN=tap5c42978b-ac OUT=br-ex MAC=fa:16:3e:a7:0c:f3:fa:16:3e:a4:49:14:08:00 SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870979] TRACE: nat:POSTROUTING:rule:1 IN= OUT=br-ex SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1 Apr 18 11:50:23 openstack-network kernel: [1038991.870985] TRACE: nat:quantum-l3-agent-POSTROUTING:rule:1 IN= OUT=br-ex SRC=10.0.0.4 DST=192.168.0.19 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28976 SEQ=1

可见数据包流在nat表的quantum-l3-agent-POSTROUTING的第一条规则处被截断了，查看iptables中的nat表的规则如下

*nat :PREROUTING ACCEPT [99:21975] :INPUT ACCEPT [74:20608] :OUTPUT ACCEPT [181:30548] :POSTROUTING ACCEPT [26:13022] :quantum-l3-agent-OUTPUT - [0:0] :quantum-l3-agent-POSTROUTING - [0:0] :quantum-l3-agent-PREROUTING - [0:0] :quantum-l3-agent-float-snat - [0:0] :quantum-l3-agent-snat - [0:0] :quantum-postrouting-bottom - [0:0] -A PREROUTING -j quantum-l3-agent-PREROUTING -A OUTPUT -j quantum-l3-agent-OUTPUT -A POSTROUTING -j quantum-l3-agent-POSTROUTING -A POSTROUTING -j quantum-postrouting-bottom -A quantum-l3-agent-OUTPUT -d 192.168.0.16/32 -j DNAT --to-destination 10.0.0.4 -A quantum-l3-agent-OUTPUT -d 192.168.0.17/32 -j DNAT --to-destination 10.0.0.3 -A quantum-l3-agent-POSTROUTING ! -i qg-91757ded-c4 ! -o qg-91757ded-c4 -m conntrack ! --ctstate DNAT -j ACCEPT -A quantum-l3-agent-POSTROUTING -s 10.0.0.0/24 -d 192.168.1.1/32 -j ACCEPT -A quantum-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.1:8775 -A quantum-l3-agent-PREROUTING -d 192.168.0.16/32 -j DNAT --to-destination 10.0.0.4 -A quantum-l3-agent-PREROUTING -d 192.168.0.17/32 -j DNAT --to-destination 10.0.0.3 -A quantum-l3-agent-float-snat -s 10.0.0.4/32 -j SNAT --to-source 192.168.0.16 -A quantum-l3-agent-snat -j quantum-l3-agent-float-snat -A quantum-l3-agent-snat -s 10.0.0.0/24 -j SNAT --to-source 192.168.0.15 -A quantum-postrouting-bottom -j quantum-l3-agent-snat COMMIT

确定有问题的规则为

-A quantum-l3-agent-POSTROUTING ! -i qg-91757ded-c4 ! -o qg-91757ded-c4 -m conntrack ! --ctstate DNAT -j ACCEPT

把这条规则删掉后重启iptables，vm能顺利连接外网，问题解决。

参考文献

http:///2010/06/11/iptables-debugging/