SkyBase - a Devops Platform for Hybrid Cloud

Editor's Notes

• #4: Decisions about: Which CM: Chef, Puppet, Salt, Ansible, ...? Hybrid Cloud Management System -Rightscale, Scalr, ..., or create our own How Continuous Integration and Delivery pipeline should look like? After trying different tools and approaches we discovered that with all those tools out there we were still missing something. Here I’d like to tell the story of our journey and present our rationale for the choices we made and describe the solution, which we are currently developing I hope that this talk will be helpful for those of you who are in the same position as we are and who feel the same pain as we are. .
• #5: all at the same time is a very tricky situation
• #6: To give an idea what we are dealing with here. 1. Multiple Apps (2 major and some other standalone services) 2. Each app is customized and deployed for multiple customers 3. Different versions of App are potentially running for different customers 4. Different services deployed separately but depend on and interact with each other 5. Multiple Teams develop and deploy their services into production It’s not just a simple move from Dev-QA-Prod for a single app.
• #7: The role of Ops team is to provide common tools for the other DevOps teams to deploy and manage their services.
• #8: This is our pipeline. It’s more or less standard pipeline, where during each phase different groups of servers are used (orange blocks). These groups are usually called environments: Dev, QA, Stage, Prod. Later we’ll see that word environment might be confusing in situations with multiple locations and clouds. Usually pipelines are promoting new release automatically up to QA environment. After that people are usually using the approval gates to UA and Prod.
• #9: We were focusing in several areas. We came up with two main concepts: Arti-Ball and Planet. What are these?
• #10: Decoupling of CI and CD is achieved by creating an ArtiBall , so CI creates AB, then CD uses AB.
• #11: From our experience, when discussing CD pipeline and even simply describing location of the server or service term “EVIRONMENT” is often ambiguous and confusing. And even in the same conversation different people mean different things. Is it DEV/QA/PROD or it’s a colocation cage, or is it Chef Server Environment and so on. Let’s consider this matrix and define planet as Universe-Provider-Region. We can think of planet as a VPC. These 3 parameters is enough, because there is no need to have 2 VPCs in the same region.
• #12: 1. “Planet” is based on a set of subnets grouped by location and purpose. ( Dev-AWS-us-west-1, Prod-AWS-eu-west-2) 2, “Planet” contains supporting infrastructure to provision, install and maintain Our Services. Usually One Our Service runs in one planet. One Deployment is always putting one ArtiBall into one Planet Multi-Region deployments will spend across two regions.
• #13: After we introduced Planets the implementation of deployment pipeline simplified. Any Pipeline can be configured as sequence of single planet deployments
• #14: Any deployment system has to include these steps.
• #15: Now let’s talk about implementation of concepts described earlier. But first this...
• #16: Let’s talk about our approach but first an old joke: Once two men came to Hodja Nasreddin and asked him to resolve their dispute. Hodja agrreed to do this. First man says: “I think that this thing has to be done from right to left” – Hodja says: “You are absolutely right!” Second man says: “I thing that same thing has to be done from left to right: - Hodja says: “And you are absolutely right!” Both men left with joy. A bystander, who saw this, says to Hodja: “Hodja, it is impossible that both men can be both right at the same time!” So Deployment Pipeline is a such thing. “And you are absolutely right!” – says Hodja Nasreddin.
• #18: CI process dumps AB to an S3 bucket Process AB Process target planet data Update Support services Update Chef server Create and apply CF templates for the stack Stack is created and Chef client pointed to Chef server Chef Server completes the install
• #19: Client WebApp with REST API Worker (executes a request) SkyBase Clases perform actions on AB and Planet)
• #21: Decoupling of the CI and CD allows to develop these processes independently at different pace – This is accomplished by using ArtiBalls ( Release Bundles ) Defining a target for deployment as a planet allows greatly simplify management of stack deployment templates ( e.g. Cloud Formation or Heat templates) thus accelerating transition of developers into DevOps. There several fundamental steps in the deployment, and with correct architecture it possible to change the mechanisms of each step without affecting other – which allows to create a functioning system while some parts of the design are still in discussion or if the new system has to coexist with legacy mechanisms Skybase by definition is a central location, which has access to all elements of infrastructure and can easily be extended with new functionality, and as such it presents a convenient DevOps platform for all
• #22: External services like monitoring, cost analytics, ... Openstack “Mistrel”