Symantec SDN Deployment
Download as PPTX, PDF•1 like•1,364 views
Symantec deployed an SDN using OpenStack with the following key aspects: 1. They created different "Classes of Service" including a development environment and a production environment to onboard teams and manage workloads. 2. They provided self-service user onboarding through Horizon with automatic network creation to hide complexities. 3. They offered load balancing as a service using HA Proxy with various optimizations to achieve high performance. 4. They attached baremetal servers to the overlay network by launching them in network namespaces. 5. They aimed for over 99.95% control plane availability using a distributed controller and Cassandra setup with automation and monitoring.
Symantec SDN Deployment
- 1. Symantec SDN Deployment Jasmeet Sidhu, Rudrajit Tapadar Cloud Platform Engineering
- 2. Class of Service Copyright © 2015 Symantec Corporation 2
- 3. Class of Service • Dev – For developers to get familiar with OpenStack cloud – Each developer has a project • Production – For teams to onboard their members – Each team has a project – Manage user roles – Manage production workloads Copyright © 2015 Symantec Corporation 3
- 4. Self-Service User Onboarding Copyright © 2015 Symantec Corporation 4
- 5. Self-Service User Onboarding • Zero tickets for user onboarding – Provide sign up capabilities on Horizon • Provide easy networking on Dev CoS – Hide all complexities – Automatically create network – Allocate routable subnets by using Contrail VNC APIs – Create security group with proper rules – Create unique domain names for instances by using Designate for routable IPs Copyright © 2015 Symantec Corporation 5
- 6. Load Balancer as a Service Copyright © 2015 Symantec Corporation 6
- 7. Load Balancer as a Service •Out of the box – Icehouse, v1 APIs – Launch HA Proxy service instances on a single AZ – SSL Support: Wildcard cert • Symantec fixes –Multiple AZ, SSL Passthrough, Stats and Metrics • Performance: –~6.5 Gbps throughput with 10K parallel connections, VIP with 2 members –20K HTTPS requests/sec for 10K parallel connections with 1 million requests, 1K response size • Tuning - haproxy.cfg: maxconn 50K, nbproc 4, ulimit-n 200K, Cipher • Pain points –No control over ha proxy cfg –No control over resource allocations (cpu, etc) Copyright © 2015 Symantec Corporation 7
- 8. Baremetal on Overlay Copyright © 2015 Symantec Corporation 8
- 9. Baremetal on Overlay •Applications that run on baremetal but needs to be on the overlay – Example: swift proxy and data nodes – Launch them inside network namespaces – Plug them to the vRouter – East-West Traffic • Manual Setup via scripts – Nova is not aware but Contrail is. – Multiple nics sitting on multiple networks – Static IPs Copyright © 2015 Symantec Corporation 9
- 10. Availability::Control Plane Copyright © 2015 Symantec Corporation 10
- 11. Control Plane Availability • Goal - 99.95% Availability • 5 SDN controller VMs distributed over 3 racks • 5 Cassandra database baremetal nodes distributed over 3 racks – RF of 3 for analytics – RF of 5 for config – Compaction throughput 256 Mbps • Deployment Automation: Puppet • Issues seen: DB Timeouts, Version mismatch, admin token Copyright © 2015 Symantec Corporation 11
- 12. Failed Customer Interactions Copyright © 2015 Symantec Corporation 12
- 13. Failed Customer Interactions • Measure the control plane availability • Use Symantec’s Logging-Monitoring-Metering as a Service to parse Neutron logs • Compare response codes: 5XX counted as failures • Dashboards! Copyright © 2015 Symantec Corporation 13
- 14. Availability::Data Plane Copyright © 2015 Symantec Corporation 14
- 15. Data Plane Availability • Work in progress.. –FIP Availability –vDNS –Link Local –Private Network Copyright © 2015 Symantec Corporation 15
- 16. Seamless Upgrades Copyright © 2015 Symantec Corporation 16
- 17. Upgrade 1.20 to 2.0.1 • Goal - Zero Downtime • Controller upgrades – No in-place upgrades – Build a parallel control plane with new release – Add them to the VIP pool and gradually decommission old controllers • Database upgrades – Add new DB nodes one by one to the existing cluster – Repair the DB – Decommission old DB node one by one • Compute upgrades – Automate unloading and loading of kernel module in all computes Copyright © 2015 Symantec Corporation 17
- 18. Health Monitoring Copyright © 2015 Symantec Corporation 18
- 19. Health Monitoring • Volta –Logging •Logstash •Elasticsearch –Metrics •InfluxDB •Statsd •Collectd) –RESTful APIs make it easy: •Response Codes, Bytes Transfered, Time, Verb, etc. • OpsView / Zabbix Copyright © 2015 Symantec Corporation 19
- 20. Troubleshooting Copyright © 2015 Symantec Corporation 20
- 21. Troubleshooting • Most incidents are trivial – Known issues – Trivial fixes/workarounds • Some incidents are complex – RCA is very involved – Might have to wait for next code release for a fix – Quick and dirty solution – use auto healing scripts for workarounds •Periodically check system health (Synthetic Transactions) •Remediate known bugs •Fix problems as they are detected, Save pagers, run 24x7! (MX Encapsulation, Dead processes, etc.) Copyright © 2015 Symantec Corporation 21
- 22. Thank you! Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.
Editor's Notes
- #8: VIP Members - 16 CPU, 32GB RAM Stats - num connections, session active/total/drops, bytes in/out, response times
- #10: Two nics on two different networks (swift proxy network and replication network) Not production yet. Still exploring. Recent issues: Kernel panic on 3.16 kernel with network namespaces.