A closer look to locaweb IaaS
6 likes•1,321 views
This document provides an overview of Locaweb's Infrastructure as a Service (IaaS) engineering team and capabilities. It discusses the team size and responsibilities, defines IaaS, describes Locaweb's IaaS architecture including virtual/physical servers, open source software used, provisioning processes, and resource monitoring. It also lists several open source projects developed by Locaweb to support their IaaS operations.
![IaaS - Wikipedia
“In this most basic cloud service model, cloud providers offer computers, as
physical or more often as virtual machines, and other resources. The virtual
machines are run as guests by a hypervisor, such as Xen or KVM. Management
of pools of hypervisors by the cloud operational support system leads to the
ability to scale to support a large number of virtual machines. Other resources
in IaaS clouds include images in a virtual machine image library, raw (block) and
file-based storage, firewalls, load balancers, IP addresses, virtual local area
networks (VLANs), and software bundles.[46] IaaS cloud providers supply
these resources on demand from their large pools installed in data centers. For
wide area connectivity, the Internet can be used or—in carrier clouds --
dedicated virtual private networks can be configured.”
* http://en.wikipedia.org/wiki/Infrastructure_as_a_service#Service_models](https://image.slidesharecdn.com/acloserlooktolocawebiaas-120830152751-phpapp01/85/A-closer-look-to-locaweb-IaaS-14-320.jpg)
A closer look to locaweb IaaS
- 1. A closer look to Locaweb Iaas Gleicon Moraes Engineering Manager PaaS/IaaS @gleicon - http://blog.7co.cc
- 2. Agenda
- 3. Agenda • Engineering Team
- 4. Agenda • Engineering Team • IaaS
- 5. Agenda • Engineering Team • IaaS • Virtual/Physical servers
- 6. Agenda • Engineering Team • IaaS • Virtual/Physical servers • Architecture
- 7. Agenda • Engineering Team • IaaS • Virtual/Physical servers • Architecture • OSS
- 8. Agenda • Engineering Team • IaaS • Virtual/Physical servers • Architecture • OSS • Provisioning
- 9. Agenda • Engineering Team • IaaS • Virtual/Physical servers • Architecture • OSS • Provisioning • CMDB/Closed Loop
- 10. Agenda • Engineering Team • IaaS • Virtual/Physical servers • Architecture • OSS • Provisioning • CMDB/Closed Loop • Resource usage gathering
- 11. Agenda • Engineering Team • IaaS • Virtual/Physical servers • Architecture • OSS • Provisioning • CMDB/Closed Loop • Resource usage gathering • Software defined networks
- 12. Engineering Team • We aim to be efficient • DC and IaaS Automation • IaaS and PaaS products • Email and Domain Registration products • Coffee/psychological help/counseling • 40 people team (devs/architects/1 master devops)
- 13. IaaS - NIST definition “ The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls).” * http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
- 14. IaaS - Wikipedia “In this most basic cloud service model, cloud providers offer computers, as physical or more often as virtual machines, and other resources. The virtual machines are run as guests by a hypervisor, such as Xen or KVM. Management of pools of hypervisors by the cloud operational support system leads to the ability to scale to support a large number of virtual machines. Other resources in IaaS clouds include images in a virtual machine image library, raw (block) and file-based storage, firewalls, load balancers, IP addresses, virtual local area networks (VLANs), and software bundles.[46] IaaS cloud providers supply these resources on demand from their large pools installed in data centers. For wide area connectivity, the Internet can be used or—in carrier clouds -- dedicated virtual private networks can be configured.” * http://en.wikipedia.org/wiki/Infrastructure_as_a_service#Service_models
- 15. IaaS - tl;dr “Automate infrastructure such as the customer will not know the underlying details, will not manage them and can provision services automagically.”
- 16. IaaS - building blocks • Servers: virtual and physical • Storage area • Network devices: firewall, switches, load balancer
- 17. IaaS - High Level • Automation • Resource Management • Install, Uninstall, Migrate • High Availability, Scalability, Capacity Planning
- 18. IaaS at Locaweb • 3 DCs, 6k Servers (physical), 1k storages 6PB area, 12K network equipments/ports, > 100 Km of cables • 10k VMs, 3.2M email accounts, 250k hosting customers, ~500k sites, ~600k DB • 130 people at day to day 24/7 Operations team (from DC basics to managing apps and platforms), < 40 sysadmins • Currently ~ 18 people from Engineering team taking care of IaaS
- 19. Virtual and Physical • Single tenant per Physical Server • Single tenant per VM • Multiple tenants per VM • Multiple tenants per Physical Server • Multiple VMs per Physical Server
- 20. Virtual and Physical • Single tenant per Physical Server • Single tenant per VM • Multiple tenants per VM • Multiple tenants per Physical Server • Multiple VMs per Physical Server = Cloud
- 21. Cloud • Check back NIST definition • Hypervisor + set of servers + set of storages + network = time sharing • Distinct capacity planning than physical servers • Flexible configuration options • Vertical Scaling • Horizontal scaling
- 22. Architecture - Cloud Internet Main Network Network Physical Servers Firewall hypervisor ovs Gear Simplestack SimpleNet/Quantum
- 23. Architecture - Physical Internet Main Network Why not ? Network Physical Servers Firewall Gear Simplestack SimpleNet/Quantum
- 24. OSS
- 25. OSS • Ruby, Rails, Python, CFEngine, PostgreSQL, MySQL, Cassandra, Redis, XEN, KVM, Haskell, Cyclone.io, bottle.py, Quantum, R, EjabberD, Resque, lots of gem, eggs
- 26. OSS • Ruby, Rails, Python, CFEngine, PostgreSQL, MySQL, Cassandra, Redis, XEN, KVM, Haskell, Cyclone.io, bottle.py, Quantum, R, EjabberD, Resque, lots of gem, eggs • Up-to-date technology
- 27. OSS • Ruby, Rails, Python, CFEngine, PostgreSQL, MySQL, Cassandra, Redis, XEN, KVM, Haskell, Cyclone.io, bottle.py, Quantum, R, EjabberD, Resque, lots of gem, eggs • Up-to-date technology • No lock-ins
- 28. OSS • Ruby, Rails, Python, CFEngine, PostgreSQL, MySQL, Cassandra, Redis, XEN, KVM, Haskell, Cyclone.io, bottle.py, Quantum, R, EjabberD, Resque, lots of gem, eggs • Up-to-date technology • No lock-ins • Vendor neutral
- 29. OSS • Ruby, Rails, Python, CFEngine, PostgreSQL, MySQL, Cassandra, Redis, XEN, KVM, Haskell, Cyclone.io, bottle.py, Quantum, R, EjabberD, Resque, lots of gem, eggs • Up-to-date technology • No lock-ins • Vendor neutral • We contribute back
- 31. Our projects
- 32. Our projects • Leela - Data collection monster
- 33. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy
- 34. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy • SimpleNet - OVS and FW controller
- 35. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy • SimpleNet - OVS and FW controller • NET/L2 - Controller/Inventory for network equipment
- 36. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy • SimpleNet - OVS and FW controller • NET/L2 - Controller/Inventory for network equipment • BrickLayer - packaging for normal people
- 37. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy • SimpleNet - OVS and FW controller • NET/L2 - Controller/Inventory for network equipment • BrickLayer - packaging for normal people • Logix - Graylog2 message bus for log streams
- 38. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy • SimpleNet - OVS and FW controller • NET/L2 - Controller/Inventory for network equipment • BrickLayer - packaging for normal people • Logix - Graylog2 message bus for log streams • xenapi-ruby - XEN API bindings for Ruby
- 39. Our projects • Leela - Data collection monster • SimpleStack - Provisioning made easy • SimpleNet - OVS and FW controller • NET/L2 - Controller/Inventory for network equipment • BrickLayer - packaging for normal people • Logix - Graylog2 message bus for log streams • xenapi-ruby - XEN API bindings for Ruby • otto, debundler, bpmachine and more each week
- 41. Our Contributions • Contributed to Quantum, from Openstack
- 42. Our Contributions • Contributed to Quantum, from Openstack • Snorby/snort contributions
- 43. Our Contributions • Contributed to Quantum, from Openstack • Snorby/snort contributions • Mod_security for Nginx and helping on IIS
- 44. Our Contributions • Contributed to Quantum, from Openstack • Snorby/snort contributions • Mod_security for Nginx and helping on IIS • hired consulting from grsecurity and dovecot teams - we support OSS companies
- 45. Bricklayer
- 46. Bricklayer • First opensource project from Locaweb
- 47. Bricklayer • First opensource project from Locaweb • Package builder (deb + rpm) straight from git
- 48. Bricklayer • First opensource project from Locaweb • Package builder (deb + rpm) straight from git • 150+ projects, 500+ builds/day
- 49. Bricklayer • First opensource project from Locaweb • Package builder (deb + rpm) straight from git • 150+ projects, 500+ builds/day • tag your project, get the packages done and on repositories
- 50. Logix
- 51. Logix • We have lots of logs. Everything broke.
- 52. Logix • We have lots of logs. Everything broke. • 26.753.205.474 lines of log/day
- 53. Logix • We have lots of logs. Everything broke. • 26.753.205.474 lines of log/day • Highly distributed: local syslog daemon to RabbitMQ
- 54. Logix • We have lots of logs. Everything broke. • 26.753.205.474 lines of log/day • Highly distributed: local syslog daemon to RabbitMQ • Elastic search + graylog2 to store, search and filter
- 55. Provisioning • Ruby: Panel, Control panel, Scheduler • Python: Provisioning, Server management, Metric collection • REST APIs to Hypervisor, Network, Firewall, XMPP
- 56. Provisioning - Cloud Internet Cloud Control Panel API Sales Main Network Network Physical Servers Firewall hypervisor ovs Gear Provisioner Simplestack SimpleNet/Quantum
- 57. Provisioning - Servers Dedicated Servers Control Panel API Closed Loop Racked Servers Sales Internet Main Network Network Physical Servers Firewall hypervisor ovs Gear Simplestack SimpleNet/Quantum
- 58. Provisioning - Managed Servers Managed Servers Control Panel Sales Dedicated Servers Control Panel PaaS API Closed Loop Provisioner Racked Servers Sales Internet Cloud Control Panel API Sales Main Network Network Physical Servers Firewall hypervisor ovs Gear Provisioner Simplestack SimpleNet/Quantum
- 59. Cloud provisioner Jobs DHCP CMDB FW API quantum/ Control Panel core Resque simplenet Sales simplestack Notifications Leela console
- 60. Closed loop The closed loop process CMDB API Conductor Network Futurama Cobbler Hardware
- 61. Closed loop
- 62. Closed loop • All servers get racked, wired, tested and configured
- 63. Closed loop • All servers get racked, wired, tested and configured • Power management discovery
- 64. Closed loop • All servers get racked, wired, tested and configured • Power management discovery • Network configuration
- 65. Closed loop • All servers get racked, wired, tested and configured • Power management discovery • Network configuration • OS install: Windows, Linux and OpenSolaris aware
- 66. Closed loop • All servers get racked, wired, tested and configured • Power management discovery • Network configuration • OS install: Windows, Linux and OpenSolaris aware • Server life cycle: once deactivated it gets back to the pool to be used again
- 67. CMDB Futurama Power audit Ops NET/L2 Frontend Controllers Database API Product Resque provisioners IP provisioning Server provisioning IT chg SAP management
- 68. Futurama Management CFEngine Server side Planet Express Leela- CF-Agent Server Conductor- CMDB Leela-agent bkp-agent audit Cegonha Asdrubal CFTools
- 69. Resource Metering and Monitoring - Leela Cassandra Leela- Cassandra Cassandra Lasergun Leela-agent Leela- Reader Cassandra Cassandra Cassandra API
- 70. Resource Metering and Monitoring - Leela
- 71. Resource Metering and Monitoring - Leela • 18k writes/sec
- 72. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster
- 73. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics
- 74. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics • ~600GB/mo
- 75. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics • ~600GB/mo • 1M keys (~5k servers)
- 76. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics • ~600GB/mo • 1M keys (~5k servers) • Write latency: 15 us
- 77. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics • ~600GB/mo • 1M keys (~5k servers) • Write latency: 15 us • Read latency: 1s to read 1mo worth of data
- 78. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics • ~600GB/mo • 1M keys (~5k servers) • Write latency: 15 us • Read latency: 1s to read 1mo worth of data • Down to minute resolution
- 79. Resource Metering and Monitoring - Leela • 18k writes/sec • 6 TB total per cluster • 13 baseline metrics + 68 distinct metrics • ~600GB/mo • 1M keys (~5k servers) • Write latency: 15 us • Read latency: 1s to read 1mo worth of data • Down to minute resolution • http://leela.readthedocs.org/en/latest/intro/archnut.html
- 80. Resource Metering and Monitoring - Leela
- 81. Resource Metering and Monitoring - Leela • Map/Reduce with SQL like interface:
- 82. Resource Metering and Monitoring - Leela • Map/Reduce with SQL like interface: -SELECT mov_avg_samples = 7 (function)
- 83. Resource Metering and Monitoring - Leela • Map/Reduce with SQL like interface: -SELECT mov_avg_samples = 7 (function) -FROM cpro9559.cpu.cpu8.idle (metric)
- 84. Resource Metering and Monitoring - Leela • Map/Reduce with SQL like interface: -SELECT mov_avg_samples = 7 (function) -FROM cpro9559.cpu.cpu8.idle (metric) -WHERE timestamp >= 1346279003 (timeframe)
- 85. Resource Metering and Monitoring - Leela
- 86. Resource Metering and Monitoring - Leela • Create charts
- 87. Resource Metering and Monitoring - Leela • Create charts - var widget = LEELA.widget(jQuery.(“#target”));
- 88. Resource Metering and Monitoring - Leela • Create charts - var widget = LEELA.widget(jQuery.(“#target”)); - jQuery.ajax(“/v1/pastweek/ cpro9559.cpu.cpu8.idle”, {dataType: “jsonp”, success: widget.render});
- 89. Software defined network
- 90. Software defined network • Traditional equipment: local config and controller
- 91. Software defined network • Traditional equipment: local config and controller • SDN: flows (commands), openflow 1.0, central controller, distributed data plane
- 92. Software defined network • Traditional equipment: local config and controller • SDN: flows (commands), openflow 1.0, central controller, distributed data plane • Abstraction over VLANs with ACLs, Tunnels or even VLAN QoQ
- 93. Software defined network Switch Vendor A Switch Vendor B Data path (hardware) Data path (hardware) Control path Openflow Control path Openflow API Controller OpenVSwitch
- 94. Software defined network Cisco Force 10 HP OpenVSwitch Firewalls API Net/L2 Quantum CMDB
- 95. Ruby @ Locaweb Not only for front-end
- 96. ?
- 97. Thanks !