Tips Tricks and Tactics with Cells and Scaling OpenStack - May, 2015
2 likes945 views
The document discusses using cells in OpenStack to scale cloud infrastructure across multiple geographic locations. Key points include using cells to distribute OpenStack compute services around Australia, with over 6000 users, 700 hypervisors, and 30,000 cores spread across 8 sites and 14 cells. It also discusses strategies for operating, upgrading, and scheduling across multiple cells.
![• No formal way of disabling a cell
• Weighting helps – but is not absolute
– Weighting cell can still “win” scheduler calculation based on available RAM
• Solution: custom filter uses specific weight offset value to avoid scheduling (- 42)
32
Managing Cells – “Disable”
www.rackspace.com
class DisableCellFilter(filters.BaseCellFilter):
"""Disable cell filter. Drop cell if weight is -42.
"""
def filter_all(self, cells, filter_properties):
"""Override filter_all() which operates on the full list
of cells...
"""
output_cells = []
for cell in cells:
if cell.db_info.get('weight_offset', 0) == -42:
LOG.debug("cell disabled: %s" % cell)
else:
output_cells.append(cell)
return output_cells](https://image.slidesharecdn.com/tipstricksandtacticswithcellsandscalingopenstack-4-160303154118/85/Tips-Tricks-and-Tactics-with-Cells-and-Scaling-OpenStack-May-2015-32-320.jpg)
Tips Tricks and Tactics with Cells and Scaling OpenStack - May, 2015
- 1. Tips, Tricks and Tactics with Cells and Scaling OpenStack ● ● ●
- 2. OpenStack Summit - Paris 2014 Multi-Cell Openstack: How to Evolve your Cloud to Scale https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/multi-cell- openstack-how-to-evolve-your-cloud-to-scale
- 4. Australian Research Cloud ● Started in 2011 ● Funded by the Australian Government ● 8 institutions around the country ● Production early 2012 - Openstack Diablo ● Now running a mix of Juno and Icehouse ● Use Ubuntu 14.04 and KVM ● 100Gbps network connecting most sites (AARNET)
- 5. Reasons for using cells Single API endpoint, compute cells dispersed around australia. Simpler from users perspective. Single set of security groups, keypairs etc. Less openstack expertise needed as only one version of some core openstack services.
- 6. Size ● 8 sites ● 14 cells ● ~6000 users registered ● ~700 hypervisors ● 30,000+ cores People ● Core team of 3 devops ● 1-2 operators per site http://status.rc.nectar.org.au/growth/infrastructure/
- 7. Interaction with other services Each cell also has one or more: ● cinder-volume host, using Ceph, LVM and NetApp backends ● A globally replicated swift region per site ● Glance-api pointing to local swift proxy for images ● Ceilometer collectors at each cell push up to a central mongo ● Private L3 network spanning all cells - will be useful for neutron
- 8. Cells Infrastructure Each compute cell has: ● MariaDB Galera cluster ● RabbitMQ cluster ● nova scheduler/cells/vnc/compute/network/api-metadata ● glance-api ● swift region - proxy and storage nodes ● ceilometer-collectors to forward up to a global collector ● cinder-volume API cell: ● nova-api, nova-cells ● keystone ● glance-registry ● cinder-api, scheduler ● heat, designate, ceilometer-api
- 9. Scheduling ● Have some “private” cells only available to certain tenants. This is usually determined by funding source. ● Global flavours and cell local flavours ● Cell level aggregates for intra cell scheduling ● Some sites have GPUs, fast IO for their own use. ○ Introduced compute and ram optimised flavours ○ Not all cells support all flavours ● Each cell advertises 1 or more availability zones to use in scheduling. ○ Ties in with cinder availability zones
- 10. Bringing on new cells Want to test in production before opening to public Don’t want to flood brand new cells Scheduler filters ● Role based access to cell. Cells advertise what roles can schedule to them ● Direct only - Allow public to select that cell directly but global scheduler doesn’t count it
- 11. Operating cells Have a small openstack cluster to manage all global infrastructure Standard environment - use puppet Upgrade cells one at a time - live upgrades ● upgrade compute conductors ● upgrade API cell ● upgrade compute cells ● upgrade compute nodes Read access to compute cells RabbitMQs for troubleshooting and monitoring. Only real interface into each of the cells. Console-log is a good test of cells functionality - have one in each cell and monitor
- 12. Future plans Move to Neutron - in planning and testing stage ● Currently have a single public network per cell, want to provide tenant networks and higher level services Start off with a global neutron and simple shared flat provider networks per cell. All hypervisors talking to the same rabbit - scale issues? Also looking at other higher level openstack services (which there are many!)
- 14. CERN - Large Hadron Collider
- 15. CERN - Cloud Infrastructure ● ● ○ ● ● ○ ● ○ ○ ○ ○
- 16. CERN - Cloud Infrastructure ● ○ ○ ○ ○ ■ ● ○
- 17. CERN - Prune Nova DBs ● ● ● ● ○ ○ ○
- 18. CERN - Cells scheduling ● ○ ● ○ ■ ■ ■ ■
- 19. CERN - Flavors management ● ○ ● ○ ■ ● ○ ○
- 20. CERN - Testing Cells ● ○ ○ ● ○ ○ ○ …
- 21. CERN - Testing Cells
- 22. CERN - Testing Cells
- 23. Matt Van Winkle [email protected] @mvanwink www.rackspace.com Cells at Rackspace
- 24. • Managed Cloud company offering a suite of dedicated and cloud hosting products • Founded in 1998 in San Antonio, TX • Home of Fanatical Support • More than 200,000 customers in 120 countries Rackspace 24www.rackspace.com
- 25. • In production since August 2012 – Currently running: Nova; Glance; Neutron; Ironic; Swift; Cinder • Regular upgrades from trunk – Package built on trunk pull from mid March in testing now • Compute nodes are Debian based – Run as VMs on hypervisors and manage via XAPI • 6 Geographic regions around the globe – DFW; ORD; IAD; LON; SYD; HKG • Numbers – 10’s of 1000’s of hypervisors (Over 340,000 Cores, Just over 1.2 Petabytes of RAM) • All XenServer – Over 170,000 virtual machines – API per region with multiple Compute cells (3 – 35+) each Rackspace – Cloud Infrastructure 25www.rackspace.com
- 26. • Cells Infrastructure – Size between ~100 and ~600 hosts per cell – Different Flavor Types (General Purpose, HIgh I/O, Compute Optimized, etc) – Working on exposing maintenance zones or near/far scheduling (host, shared IP space, network aggregation) – Separate DB cluster for each cell • Run our Cells infrastructure in cells – Control Plane exists as instances in small OpenStack deployment – Multiple Hardware types – Separate tenants – Control plane instances from other internal users Rackspace – Cloud Infrastructure - Cells 26www.rackspace.com
- 27. • Multiple cells within each flavor class – Hardware Profile • Additionally, we group by vendor • Live migration needs matching CPUs – Range of flavor size within each cell (eg. General Purpose 1, 2, 4 and 8 Gig) • Tenant Scheduling – Custom filter schedules by Flavor class first • All General Purpose cells, for example – Scheduled by available RAM afterwards • Enhancements for spreading out tenant load and max IOPs per host – In some cases, filters can bind a cell to specific tenants (testing and internal use) • Work in Cells V2 to enhance scheduling – https://review.openstack.org/#/c/141486/ as one example 27 Cell Scheduling www.rackspace.com
- 28. • Common control plane nodes deployed by ansible play book – DB Pair – Cells service – Scheduler – Rabbit • Playbook Populates flavor info based on hardware type • Hypervisors bootstrapped once CP exists – Create Compute Node VM – Deploy Code and configure – Update routes, etc • Provision IP blocks • Test • Link via playbook 28 Deploying a Cell www.rackspace.com
- 29. • Larger region has run rate around 50,000 VMs • 1000’s of VMs created/deleted per hour in busiest regions • Downstream BI and Revenue assurance teams require deleted instance records be kept for 90 days • Current deleted instance counts range between 132,000 and 900,000 29 Rackspace – Purge Nova DBs www.rackspace.com
- 30. 30 Rackspace – Purge Nova DBs www.rackspace.com
- 31. • By Pass URL prior to linking a cell up – Test API endpoint: http://nova-admin-api01.memory1-0002.XXXX.XXXXXX.XXXX:8774/v2 • Full set of tests – Instance creates, deletes, resizes – Overlay network creation – Volume provisioning – Integration with other RS products • Trickier to test hosts being added to an existing cell – Hosts are either enabled or disabled – Targeting helps • --hint target_cell=’<cellname>’ • --hint 0z0ne_target_host=<host_name> 31 Testing Cells www.rackspace.com
- 32. • No formal way of disabling a cell • Weighting helps – but is not absolute – Weighting cell can still “win” scheduler calculation based on available RAM • Solution: custom filter uses specific weight offset value to avoid scheduling (- 42) 32 Managing Cells – “Disable” www.rackspace.com class DisableCellFilter(filters.BaseCellFilter): """Disable cell filter. Drop cell if weight is -42. """ def filter_all(self, cells, filter_properties): """Override filter_all() which operates on the full list of cells... """ output_cells = [] for cell in cells: if cell.db_info.get('weight_offset', 0) == -42: LOG.debug("cell disabled: %s" % cell) else: output_cells.append(cell) return output_cells
- 33. 33 Managing Cells – “Disable” www.rackspace.com
- 34. • Rackspace uses Quark Plugin – https://github.com/rackerlabs/quark • Borrowed old idea from Quantum/Melange days – Default tenant for each cell – Each cell is a segment – Provider subnets are scoped to a segment – Nova requests ports on provider network for the segment • Public • Private • MAC addresses too 34 Neutron and Cells www.rackspace.com
- 35. ? ● ● ●