OpenStack networking juno l3 h-a, dvr
57 likes8,507 views
The document discusses Distributed Virtual Router (DVR) and L3 High Availability in OpenStack Networking (Juno). It describes DVR packet flow including SNAT on the network node, floating IP/DNAT on compute nodes, and East-West traffic flow between instances on different compute nodes using GRE tunnels. Compute nodes perform distributed routing functions using Open vSwitch and namespaces.
![[2018] 오픈스택 5년 운영의 경험](https://cdn.slidesharecdn.com/ss_thumbnails/cloudinfra05-190131073350-thumbnail.jpg?width=560&fit=bounds)
![[OpenInfra Days Korea 2018] Day 2 - CEPH 운영자를 위한 Object Storage Performance T...](https://cdn.slidesharecdn.com/ss_thumbnails/openinfradayobjectstorageperformancefinal2-180704062033-thumbnail.jpg?width=560&fit=bounds)
![[OpenStack Days Korea 2016] Track3 - 오픈스택 환경에서 공유 파일 시스템 구현하기: 마닐라(Manila) 프로젝트](https://cdn.slidesharecdn.com/ss_thumbnails/31netapp-160226172856-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to OpenStack networking juno l3 h-a, dvr (20)
![[오픈소스컨설팅] Linux Network Troubleshooting](https://cdn.slidesharecdn.com/ss_thumbnails/opensourceconsultingnetworktroubleshootingyjlee20210412-210413014206-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
OpenStack networking juno l3 h-a, dvr
- 1. OpenStack Networking - Juno - - DVR & L3 High Availability Paul Sim Technical Account Manager [email protected]
- 2. ● Distributed Virtual Router ○ Packet flow ○ Architecture ■ SNAT, ■ DNAT(Floating IP) ■ East<->West ● L3 High Availability Index
- 3. DVR (Distributed Virtual Router) - Installation Network node Neutron server eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3/dhcp- agent External network Compute node - 1 Nova compute eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3-agent Management Data Compute node - 2 Nova compute eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3-agent
- 4. DVR (Distributed Virtual Router) - Packet flow Compute node - 1 br-ex VM VM GRE Tunnel VM br-int Network node br-ex br-tun br-int br-tun Compute node - 2 VM VM VM br-int br-tun 1.SNAT External network 3. East-West traffic 2. Floating IP OVS bridge
- 5. DVR (Distributed Virtual Router) - SNAT : Network node Namespace OVS bridgeNetwork node qdhcp- br-tun eth0 br-int patch-tun patch-int gre~ snat- qrouter- tap taptap sg~ 50.50.6.2ns~ qr~ qg~ 192.168.10.109 SNAT br-ex tap
- 6. DVR (Distributed Virtual Router) - SNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge sg~on networknode packet flow
- 7. traffic flow DVR (Distributed Virtual Router) - SNAT : Compute node Namespace OVS bridge Linux bridge Compute node VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ sg~(50.50.6.2)on networknode ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter- 20838b7d-a7ac-4da9-92aa-adec797d600e ip rule ls 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 842139137: from 50.50.6.1/24 lookup 842139137 ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter- 20838b7d-a7ac-4da9-92aa-adec797d600e ip route show table 842139137 default via 50.50.6.2 dev qr-9722faba-b7
- 8. DVR (Distributed Virtual Router) - Floating IP/DNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow br-ex tap eth0 fip- fpr~ rfp~ fg~ RouteRoute NAT veth pair
- 9. DVR (Distributed Virtual Router) - Floating IP/DNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow br-ex tap eth0 fip- fpr~ rfp~ fg~ RouteRoute NAT veth pair ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9- 92aa-adec797d600e ip rule ls 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 32770: from 50.50.5.5 lookup 16 842138881: from 50.50.5.1/24 lookup 842138881 842138881: from 50.50.5.1/24 lookup 842138881 842139137: from 50.50.6.1/24 lookup 842139137 ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9- 92aa-adec797d600e ip route show table 16 default via 169.254.31.29 dev rfp-20838b7d-a
- 10. DVR (Distributed Virtual Router) - Floating IP/DNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow br-ex tap eth0 fip- fpr~ rfp~ fg~ RouteRoute NAT veth pair ubuntu@ubuntu-6:~$ sudo ip netns exec fip-02f9d340-2caa- 4c05-86fb-460c9580f9df ip route show default via 192.168.10.1 dev fg-f3887d61-2d 192.168.10.114 via 169.254.31.28 dev fpr-20838b7d-a
- 11. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request ICMP Reply i.e., ping 50.50.5.3 -> 50.50.6.3
- 12. DVR (Distributed Virtual Router) - East-West traffic flow : network topology
- 13. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ VM 50.50.6.3 ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac- 4da9-92aa-adec797d600e ip link 2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff 5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac- 4da9-92aa-adec797d600e ip link 2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff 5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff VM 50.50.5.3
- 14. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request 50.50.5.3 -> 50.50.6.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 SRC MAC : fa:16:3e:71:3d:5a SRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.6.3 SRC MAC : fa:16:3e:71:3d:5a SRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.6.3 SRC MAC : fa:16:3e:ce:8c:35 SRC IP : 50.50.5.3 DST MAC : fa:16:3e:15:1e:e0 DST IP : 50.50.6.3 GRE tunnel 0x3 SRC MAC : fa:16:3f:5e:a0:cf SRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.6.3
- 15. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Reply 50.50.6.3 -> 50.50.5.3 SRC MAC : fa:16:3e:15:1e:e0 SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.5.3 SRC MAC : fa:16:3e:15:1e:e0 SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35 DST IP : 50.50.5.3 SRC MAC : fa:16:3e:ff:85:9b SRC IP : 50.50.6.3 DST MAC : fa:16:3e:71:3d:5a DST IP : 50.50.5.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 GRE tunnel 0x1 SRC MAC : fa:16:3f:72:60:33 SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35 DST IP : 50.50.5.3
- 16. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request 50.50.5.3 -> 50.50.6.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 table=0, n_packets=9178, n_bytes=1009035, idle_age=17470, hard_age=65534, priority=1 actions=NORMAL table=0, n_packets=2066, n_bytes=214544, idle_age=5, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1) table=1, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=1,dl_vlan=2,dl_src=fa:16:3e:71:3d:5a actions=mod_dl_src:fa:16:3f:5e:a0:cf,resubmit(,2) table=2, n_packets=1849, n_bytes=183458, idle_age=5, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) table=20, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=2,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,set_tunnel:0x3,output:3
- 17. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request 50.50.5.3 -> 50.50.6.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 table=0, n_packets=1789, n_bytes=175146, idle_age=17, hard_age=65534, priority=2,in_port=3,dl_src=fa:16:3f:5e:a0:cf actions=resubmit(,1) table=1, n_packets=1765, n_bytes=172970, idle_age=17, hard_age=65534, priority=4,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,mod_dl_src:fa:16:3e:71:3d:5a,output:8 table=0, n_packets=1857, n_bytes=184993, idle_age=18, hard_age=65534, priority=1,in_port=2 actions=resubmit(,3) table=3, n_packets=1993, n_bytes=195880, idle_age=18, hard_age=65534, priority=1,tun_id=0x3 actions=mod_vlan_vid:2,resubmit(,9) table=9, n_packets=1789, n_bytes=175146, idle_age=18, hard_age=65534, priority=1,dl_src=fa:16:3f:5e:a0:cf actions=output:1
- 18. L3 High Availability - Installation Network node-1 Neutron server eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3/dhcp- agent External network Management Data KeepAlived Network node-2 Neutron server eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3/dhcp- agent KeepAlived Compute node - 1 Nova compute eth1 eth2 eth0 Neutron ML2 plugin Compute node - 2 eth1 eth2 eth0 Neutron ML2 plugin Nova compute
- 19. Network node-2 Compute node - 3 Compute node - 2 Network node-1 vRouter A - Master L3 High Availability Compute node - 1 Subnet 1 Subnet 3 Subnet 2 Subnet 5 vRouter B - Backup vRouter C - Backup vRouter D - Master vRouter C - Master vRouter D - Backup vRouter A - Backup Subnet 3 Subnet 4 vRouter B - Master Tenant X Tenant Y Tenant Z VRRP
- 20. L3 High Availability Namespace OVS bridge Network node-1 qdhcp- br-tun br-int qrouter- ha~ ns~ qr~qg~ br-ex Network node-2 qdhcp- br-tun br-int qrouter- qr~qg~ br-ex ns~ KeepAlived KeepAlived ha~ ubuntu@ubuntu-5:~$ sudo ip netns exec qrouter-d8625260-88a1-4312-b788-c04fc9094356 tcpdump -n -i ha-27fe59da-a8 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ha-27fe59da-a8, link-type EN10MB (Ethernet), capture size 65535 bytes 16:16:25.213440 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 16:16:27.214607 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 16:16:29.215796 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 16:16:31.216986 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20
- 21. L3 High Availability Network node-1 qdhcp- br-tun eth0 br-int patch-tun patch-int qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Network node-2 qdhcp- br-tun eth0 br-intpatch-tun patch-int gre~ qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Namespace OVS bridge gre~
- 22. L3 High Availability Network node-1 qdhcp- br-tun eth0 br-int patch-tun patch-int qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Network node-2 qdhcp- br-tun eth0 br-intpatch-tun patch-int gre~ qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Namespace OVS bridge gre~
- 23. L3 High Availability Namespace OVS bridge Network node-1 qdhcp- br-tun br-int qrouter- ha~ ns~ qr~qg~ br-ex KeepAlived ubuntu@ubuntu-5:~$ cat /var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/keepalived.conf vrrp_sync_group VG_1 { group { VR_1 } notify_master "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/notify_master.sh" notify_backup "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/notify_backup.sh" notify_fault "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/notify_fault.sh" } vrrp_instance VR_1 { state BACKUP interface ha-27fe59da-a8 virtual_router_id 1 priority 50 nopreempt advert_int 2 track_interface { ha-27fe59da-a8 } virtual_ipaddress { 192.168.10.118/24 dev qg-8fffbd7e-8a } virtual_ipaddress_excluded { 50.50.1.1/24 dev qr-dee474e1-1e } virtual_routes { 0.0.0.0/0 via 192.168.10.51 dev qg-8fffbd7e-8a } }
- 24. Network node Tenant A L3 High Availability Namespace OVS bridge br-tun br-int qrouter- ha~ br-ex KeepAlived qrouter- ha~ KeepAlived HA network : 169.254.192.0/18 Segmentation id : 0x6 Tenant B qrouter- ha~ KeepAlived qrouter- ha~ KeepAlived HA network : 169.254.192.0/18 Segmentation id : 0x7 ● One KeepAlived instance per vRouter ● One HA network per tenant ○ Each HA network has separate segmentation id ○ allow_overlapping_ips = True ● Maximum 255 HA routers per tenant.