Blue host using openstack in a traditional hosting environment

Using OpenStack
In A Traditional Hosting Environment

Jun Park (Ph.D.), Sr. Systems Architect
Mike Wilson, Sr. Systems Architect
EIG/Bluehost

OpenStack Summit 2013

• Scale @10x what we were in about a year’s time
• Needed a system to manage it all
• Provisioning these systems had to be automated
• System had to scale to multiple data centers
• Btw, we have 2 months to come up with this

EIG/Bluehost 2

High level requirements
• Centralized management, horizontal scalability.
• Abstractions for physical and logical deployments of
devices and resources.
• Open-source project with lots of support and momentum
• Typical Cloud features (migration, imaging, provisioning,
etc.)

Easy transition into future cloud offering with support for
industry standard APIs
=
OpenStack!

EIG/Bluehost 3

Bluehost Environment
• Scale
– More than 10,000 physical
servers
– Adding hundreds of nodes
per day
• Network
– Public network directly attached to VMs
– Plan on adding private network later
• OpenStack/Folsom Components
– Nova (compute, api, scheduler)
– Quantum (network)
– AMQP: Qpid (messaging)
– Mysql (db)

EIG/Bluehost 4

Outline
• Scalability/Stability
• Rethink Quantum Network
• Operational Issues

• Wrap-up/Conclusions

EIG/Bluehost 5

Why Difficult To Scale up?
• Components that don’t scale
– Messaging system Look at that line!

– Mysql server
– Heavy APIs
• Hard to Diagnose
– No simulator/emulator for high scalability testing
– No quantitative guide as to how to scale up
– No detailed error message or even not at all
– Requires detailed knowledge of codebase

EIG/Bluehost 6

Nova Enhancements
• Monitoring/troubleshooting
– Added service ping (similar to grizzly)
– Additional task_states
– Better errors to instance_faults
• Functionality
– Added LVM resize and injection
– Added stop_soft and stop_hard similar to reboot
• Stability/Scalability
– Some bug fixes to OVS VIF driver
– Added custom scheduler
EIG/Bluehost 7

MySQL/Innodb Concurrency
• Nova behavior
– Direct connection to DB (addressed in grizzly though)
– Too MANY queries; much more read than write
– Some queries often return huge number of results
– Periodicity of loads due to periodic tasks
• Mysql behavior
– innodb_thread_concurrency = num of threads that can
execute queries in the queue
– innodb_concurrency_tickets = number of rows you can
return before requeue (default 500!)
– With default settings your mysql server will fall over with a
few thousand compute nodes

EIG/Bluehost 8

So the answer is just increase concurrency and tickets right?

Sadly, no.

Tuning concurrency and tickets is a must! But we still requeue.
We don’t know why, but we have a workaround.

Our workaround:
Send most read queries to a cluster of mysql slaves.
I say most because many queries would be sensitive to
possible slave replication lag

EIG/Bluehost 9

Messaging System
• Qpid Chosen
– Speed and easy cluster ability
– Incredibly unstable at large scale (Even at small scale)
– Possibly poorly configured
– Older release in CentOS6
• Problem
– Broker bottleneck
– Unnecessarily complex
• Recommendation
– AVOID BROKER!
– ZeroMQ as replacement

EIG/Bluehost 10

BlueHost OpenStack

Group of Controllers mysql master
• Nova-api Read-only mysql slave
Read-only mysql slave
• Nova-scheduler Read-only mysql slaves
• Quantum-server
• Keystone

Load Balanced

Server Farm
> 10,000 physical servers
Host
BH-enhanced OVS
Qpid servers
Quantum Plugin
BH-enhanced Nova-
EIG/Bluehost compute

Rethink Quantum Network
• Problem
– Quantum API only for DB; no efficient API for Nova
– No API-around design for actual network objects
– Premature OpenvSwitch
(OVS) quantum plugin

• Our approach
– Adding intelligence to OVS plugin
– Optimizing API
• E.g., get_instance_nw_info() in nova; Python-level join operation
– Expand current implicit communication with nova-
compute (although not ideal)

EIG/Bluehost 12

OpenvSwitch (OVS)
• Support
– OpenFlow 1.0 (1.2, 1.3 in experiment)
– Various Hypervisors including KVM
– Merged into main stream since Linux 3.3

• Functionality
– Filtering rules and associated actions
• E.g., anti-IP spoofing, DMAC filtering
– Replacement or superset of ebtables/iptables/linuxbridge
– QoS or Linux traffic control (TC)
– Various third party controllers for full-fledged OF
functionality

EIG/Bluehost 13

Quantum With Nova-Compute
Implicit communication
with nova-compute
1. Nova boot API Why nova-compute
Controller still create tap?

Nova-api server Host
2. Call quantum-api to create port
Nova-compute
Quantum-server 4. Create OVS tap
6. Do the rest of steps
5. Set up external-ids on tap
for creating a new VM
3. Create port in DB
1. Monitor any change of external-
ids of the existing taps
Quantum 2. Call quantum-api to get port info Quantum-agent
DB
3. Modify external-ids on tap &
setup OVS flows
EIG/Bluehost 14

BH-Enhanced OVS Quantum Plugin
• Idiosyncratic Requirements
– Focus on a Zone; plan on addressing multiple-datacenters issue
– Direct public IP using flat shared provider network with No NAT
– Still required to be isolated while allowing intra-traffic among VMs
• Our Approach
– Developing “Distributed OpenFlow controller” using OVS plugin
– Either no-tag or 24bit ID (e.g., QinQ, VxLAN, GRE), but NO VLAN
– Caveats: Neither API-around approach nor Virtual Appliance
• New Features
– Anti-IP/ARP spoofing OF flows
– Multiple-IPs per public port
– Network bandwidth throttling (QoS)
– Optimal Intra-traffic flows
EIG/Bluehost 15

BH-Enhanced OVS Quantum Plugin
• DMAC matching for
internal traffic to VMs
• Deploy QoS for • DMAC matching for incoming
outgoing packets packets • Anti-IP
• TPA matching in ARP query spoofing: SRC
IP matching for
Tag=1 (MTU size bug) outgoing
VM1 pair of veth packets
tap1 br-int-eth0 phy-br-eth0
10 Mbps

br-int br-eth0 eth0
50 Mbps
Public Networks

VM2 tap2 Int-loopback phy-loopback
pair of veth
Tag=2
• DMAC matching for
EIG/Bluehost internal traffic to VMs 16

Operational Issues
• Reboot hosts
– Problem
• Circular dependencies between libvirtd and nova-compute
• OVS bug, allows to add non-existing tap interfaces
– Solution
• A simple workaround to restart services in rc.local
• Restart services
– Problem
• Quantum recreates taps, resulting in network hiccups
– Solution
• Check out existing taps, no touch when unnecessary
EIG/Bluehost 17

Operational Issues
• Monitor Health
– Problem
• Hard to track down
– Solution
• Adding health check APIs
• XML customization
– Problem
• No way to modify XML on the fly
• Not even allowed for static customization
– Solution
• Add more parameters
EIG/Bluehost 18

Wrap-up/Conclusions

EIG/Bluehost 19

Problem vs. Solution
Problem Solution
Monitoring/troubleshootin
Service ping, task_states, etc.
g
No LVM Add LVM driver
Nova
Overloaded scheduler Custom scheduler
OVS VIF driver issue Fix bugs
Overloaded Read-only Mysql slave server
Mysql
Innodb issue Optimized confs
Instability issue Clustered qpid
Qpid
Future plan No broker, ZeroMQ
Heavy API Optimized API
Quantum
Premature OVS plugin Add features (Anti-IP, No-tag flows, QoS)
Reboot, restart of services Simple workarounds
Operations
XML XML customization (cpu custom)

EIG/Bluehost 20

For True OpenStack Success
Scalability
 Scalable Messaging System & DB Infra

Networks
 Good Networks Abstraction

So … We don’t do live demo until …

EIG/Bluehost 21

We’re sorry, we haven’t had time to contribute a whole lot YET. But here’s
some code:

https://github.com/upoopoo for BH nova
https://github.com/JunPark for BH Quantum (as in live production)

Browse the branches, many have to do with features discussed today

EIG/Bluehost 22

Blue host using openstack in a traditional hosting environment

Recommended

More Related Content

What's hot (20)

Similar to Blue host using openstack in a traditional hosting environment (20)

More from OpenStack Foundation (20)

Recently uploaded (20)

Blue host using openstack in a traditional hosting environment

Editor's Notes