OpenStack网络知识片断(持续更新) 来自:http://lynnkong./blog/1754524
LinuxBridge创建虚拟机
生成设备名:tap+portid 如果不是Xen,执行ip link show dev “tap+portid”看设备是否已存在,若存在返回;若不存在: ip tuntap add “tap+portid” mode tap ip link set “tap+portid” address <mac_address> ip link set “tap+portid” up
LinuxBridge agent
使用pydev库获取本机所有以tap开头的设备 对每一个tap设备(port)循环: 向Quantum获取port的详细信息 ip lingk show dev <dev> 获取设备所在的网桥(对于新增设备应该是空):在/sys/devices/virtual/net/目录下找到以brq开头的网桥名,对每一个网桥: 获取/sys/devices/virtual/net/<bridge>/brif/目录下所有设备 生成network对应的网桥(brq+networkid) 获取network对应phynet所对应的phyinterface(必须已存在) ip link add link <phyinterface> name <phyinterface.vlanid> type vlan id <vlanid> ip link set <phyinterface.vlanid> up brctl addbr <brq+networkid> brctl setfd <brq+networkid> brctl stp <brq+networkid> off ip link set <brq+networkid> up brctl addif <brq+networkid> <phyinterface.vlanid> brctl addif <brq+networkid> <dev> 循环结束
OVS run_instance(准备网络)
创建虚拟机,例如网卡portid:1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba ip link show dev qbr1e2b09d7-e9 #判断 brctl addbr qbr1e2b09d7-e9 #增加Linux网桥 ip link show dev qvo1e2b09d7-e9 #判断 ip link show dev qvb1e2b09d7-e9 #判断 ip link add qvb1e2b09d7-e9 type veth peer name qvo1e2b09d7-e9 #增加对等设备 ip link set qvb1e2b09d7-e9 up #激活设备 ip link set qvb1e2b09d7-e9 promisc on #混杂模式 ip link set qvo1e2b09d7-e9 up ip link set qvo1e2b09d7-e9 promisc on ip link set qbr1e2b09d7-e9 up #激活网桥 brctl addif qbr1e2b09d7-e9 qvb1e2b09d7-e9 #向Linux网桥添加设备 #下面的命令向OVS添加port ovs-vsctl
-- --may-exist add-port br-int qvo1e2b09d7-e9 -- set Interface
qvo1e2b09d7-e9
external-ids:iface-id=1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba
external-ids:iface-status=active
external-ids:attached-mac=fa:16:3e:ea:ad:8d
external-ids:vm-uuid=49b6d841-163f-4aab-b309-149727c227b4
OVS agent
初始化: ovs-vsctl -- --if-exists del-port br-int patch-tun ovs-ofctl del-flows br-int ovs-ofctl add-flow br-int hard_timeout=0,idle_timeout=0,priority=1,actions=normal 循环主体: 1) ovs-vsctl list-ports br-int 输出:qvo1e2b09d7-e9\nqvo2d58d5dc-db\nqvo2e505b97-bb\nqvo5739b2dc-78\nqvo69121bea-6a\nqvod58fde4e-5f\nqvoe0a0b269-53\n 2) 循环调用:ovs-vsctl get Interface qvo1e2b09d7-e9 external_ids,获取iface-id(portid) 输出: {attached-mac="fa:16:3e:ea:ad:8d",
iface-id="1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba", iface-status=active,
vm-uuid="49b6d841-163f-4aab-b309-149727c227b4"}\n 3) 根据portid循环:
a) 向Quantum查询信息,调用get_device_details接口 b) ovs-vsctl -- --columns=external_ids,name,ofport find Interface external_ids:iface-id="1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba",输出: external_ids :
{attached-mac="fa:16:3e:ea:ad:8d",
iface-id="1e2b09d7-e9d6-4aba-964e-79fdd8bcf3ba", iface-status=active,
vm-uuid="49b6d841-163f-4aab-b309-149727c227b4"} name : "qvo1e2b09d7-e9" ofport : 6 c) 给port所属的network自动分配(如果已记录过该net,直接跳到下一步)local vlan id(1-4094),且对于network对应的physical net,节点上要有一个OVS网桥与之对应,如果是vlan模式,需要做如下操作: # outbound,出口的vlan转换 br.add_flow(priority=4, in_port=self.phys_ofports[physical_network], #与br-int连接的port的标号 dl_vlan=lvid, #自动分配的local vlan,从1开始 actions="mod_vlan_vid:%s,normal" % segmentation_id) #segmentation_id是plugin分配的vlan号 # inbound,入口的vlan转换 self.int_br.add_flow(priority=3, in_port=self.int_ofports[physical_network], #与上面的br连接的port的标号 dl_vlan=segmentation_id, actions="mod_vlan_vid:%s,normal" % lvid) d) ovs-vsctl set Port qvo1e2b09d7-e9 tag=1 #这里的1是为network分配的local vlan id e) ovs-ofctl del-flows br-int in_port=6 #这里的6指port的标号,表示不允许数据流入
dhcp agent
dhcp agent需要为不同的plugin配置不同的interface_driver OVS:quantum.agent.linux.interface.OVSInterfaceDriver LinuxBridge:quantum.agent.linux.interface.BridgeInterfaceDriver
dhcp agent执行的命令及输出: #查看设备是否存在 ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip -o link show tap9739ea30-d6 '17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n' #查看设备IP ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip addr show tap9739ea30-d6 permanent scope global '17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \n link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n inet 10.10.11.2/24 brd 10.10.11.255 scope global tap9739ea30-d6\n' #查看DHCP进程信息,其中的进程号是从文件中获取/var/lib/quantum/dhcp/{netid}/pid cat /proc/13695/cmdline 'dnsmasq\x00--no-hosts\x00--no-resolv\x00--strict-order\x00--bind-interfaces\x00--interface=tap9739ea30-d6\x00--except-interface=lo\x00--domain=openstacklocal\x00--pid-file=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/pid\x00--dhcp-hostsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/host\x00--dhcp-optsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/opts\x00--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update\x00--leasefile-ro\x00--dhcp-range=set:tag0,10.10.11.0,static,120s\x00' #停止进程 ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 kill -9 13695 #启动进程 QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay
QUANTUM_NETWORK_ID=c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip netns exec
qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 dnsmasq --no-hosts
--no-resolv --strict-order --bind-interfaces --interface=tap9739ea30-d6
--except-interface=lo --domain=openstacklocal
--pid-file=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/pid
--dhcp-hostsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/host
--dhcp-optsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/opts
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update
--leasefile-ro --dhcp-range=set:tag0,10.10.11.0,static,120s
dhcp agent需要为不同的plugin配置不同的interface_driver
OVS:quantum.agent.linux.interface.OVSInterfaceDriver LinuxBridge:quantum.agent.linux.interface.BridgeInterfaceDriver
dhcp agent执行的命令及输出:
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip -o link show tap9739ea30-d6
'17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n'
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip addr show tap9739ea30-d6 permanent scope global
'17: tap9739ea30-d6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \n link/ether fa:16:3e:de:19:12 brd ff:ff:ff:ff:ff:ff\n inet 10.10.11.2/24 brd 10.10.11.255 scope global tap9739ea30-d6\n'
cat /proc/13695/cmdline
'dnsmasq\x00--no-hosts\x00--no-resolv\x00--strict-order\x00--bind-interfaces\x00--interface=tap9739ea30-d6\x00--except-interface=lo\x00--domain=openstacklocal\x00--pid-file=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/pid\x00--dhcp-hostsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/host\x00--dhcp-optsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/opts\x00--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update\x00--leasefile-ro\x00--dhcp-range=set:tag0,10.10.11.0,static,120s\x00'
ip netns exec qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 kill -9 13695
QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay
QUANTUM_NETWORK_ID=c6e38a5a-2adf-42a5-8c6f-5eab99208869 ip netns exec
qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 dnsmasq --no-hosts
--no-resolv --strict-order --bind-interfaces --interface=tap9739ea30-d6
--except-interface=lo --domain=openstacklocal
--pid-file=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/pid
--dhcp-hostsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/host
--dhcp-optsfile=/var/lib/quantum/dhcp/c6e38a5a-2adf-42a5-8c6f-5eab99208869/opts
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update
--leasefile-ro --dhcp-range=set:tag0,10.10.11.0,static,120s
l3 agent
l3 agent需要配置与plugin对应的interface_driver: OVS: quantum.agent.linux.interface.OVSInterfaceDriver,此时external_network_bridge为br-ex LinuxBridge: quantum.agent.linux.interface.BridgeInterfaceDriver
初始化:
1. 加载interface_driver 2. ip netns list #列出以'qrouter-'开头的namespace 输出: qdhcp-487f81ab-98d3-457a-b712-b29e71e89b52 qdhcp-084ae80a-b108-4f8a-90ca-f44aa1ca738a qdhcp-7c25296d-bc81-45f6-bcc0-37fa44588b83 qdhcp-c6e38a5a-2adf-42a5-8c6f-5eab99208869 qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc 循环: ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -o link list '9: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN \\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n25: qr-012c9d13-85: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:f2:a8:56 brd ff:ff:ff:ff:ff:ff\n26: qg-388798a1-55: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:f6:23:b9 brd ff:ff:ff:ff:ff:ff\n' 对命名空间内的设备名作循环(加粗字体): 如果是qr开头:ovs-vsctl --timeout=2 -- --if-exists del-port br-int qr-012c9d13-85 如果是qg开头:ovs-vsctl --timeout=2 -- --if-exists del-port br-ex qg-388798a1-55 循环结束 循环结束
工作任务:
1. 保证br-ex存在 2. 向Quantum获取router:external的网络(一个l3 agent只处理一个external网络,默认是br-ex,可以在配置文件中配置external_network_bridge) 3. 循环获取Quantum中的router对象(只处理连接外网的router): 增加qrouter-routerid命名空间 ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 sysctl -w net.ipv4.ip_forward=1 如果配置了metadata_ip,执行 rules.append(('INPUT',
'-s 0.0.0.0/0 -d %s -p tcp -m tcp --dport %s -j ACCEPT' %
(self.conf.metadata_ip, self.conf.metadata_port))) rules.append(('PREROUTING',
'-s 0.0.0.0/0 -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT
--to-destination %s:%s' % (self.conf.metadata_ip,
self.conf.metadata_port))) 应用iptable规则 ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-save -t filter ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-restore ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-save -t nat ip netns exec qrouter-0e38e30f-4fae-4f48-be4c-76d2fb803b23 /sbin/iptables-restore 对于router的新增内部port循环: ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -o link show qr-012c9d13-85,如果设备不存在: ovs-vsctl -- --may-exist add-port br-int qr-012c9d13-85 -- set Interface qr-012c9d13-85 type=internal -- set Interface qr-012c9d13-85 external-ids:iface-id=012c9d13-8554-4b39-96b8-e4bd2e787559 -- set Interface qr-012c9d13-85 external-ids:iface-status=active -- set Interface qr-012c9d13-85 external-ids:attached-mac=fa:16:3e:f2:a8:56 ip link set qr-012c9d13-85 address fa:16:3e:f2:a8:56 ip link set qr-012c9d13-85 netns qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip link set qr-012c9d13-85 up ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip addr show qr-012c9d13-85 permanent scope global ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -4 addr add 10.10.10.1/24 brd 10.10.10.255 scope global dev qr-012c9d13-85 如果router连接到外部网络,增加snat规则(将内部的IP转换为外网的IP),应用iptable规则 循环结束 对于router上删除的port循环: 在OVS上删除port 删除nat规则 循环结束
初始化gw_port:ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc ip -o link show qg-388798a1-55,如果设备不存在,同上(操作br-ex) ip netns exec qrouter-ccf5f323-2a41-41d1-8bb6-b772a8ae17fc route add default <gw_port的网关IP> 增加snat规则
处理router上的floatingIP,对br-ex上的port(qg-388798a1-55)配置外网地址,配置snat/dnat规则 循环结束
|