Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER
1 like1,019 views
This document discusses continuous integration (CI) for open source software on OpenPOWER systems. It provides background on CI, OpenPOWER systems, and the Cloud Foundry platform. It then describes using the Concourse CI tool to continuously build a Concourse project from a GitHub repository. Key steps involve deploying OpenStack, setting up a Docker registry, installing BOSH and Concourse, defining a Concourse pipeline, and updating the pipeline to demonstrate the CI process in action. The document emphasizes the importance of CI for open source projects and how it benefits development on OpenPOWER systems.
![#ibmedge
OpenPOWER: Open Hardware for High Performance
9
Systems designed for
big data analytics
and superior cloud economics
Upto:
12 cores per cpu
96 hardware threads per cpu
1 TB RAM
7.6Tb/s combined I/O Bandwidth
GPUs and FPGAs coming…
OpenPOWER
Traditional
Intel x86
http://www.softlayer.com/bare-metal-search?processorModel[]=9](https://image.slidesharecdn.com/iop1603continuousintegrationwithcloudfoundryconcourseanddockeronopenpowerv11-160927165636/85/Continuous-Integration-with-Cloud-Foundry-Concourse-and-Docker-on-OpenPOWER-10-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (15)
Ad
Similar to Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER (20)
Ad
More from Indrajit Poddar (10)
Recently uploaded (20)
Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER
- 1. #ibmedge© 2016 IBM Corporation Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER Yulia Gaponenko [email protected] Indrajit Poddar [email protected] IBM Systems Sep 22, 2016
- 2. #ibmedge About Indrajit (a.k.a. I.P) Expertise: • Accelerated Cloud Data Services, Machine Learning and Deep Learning • Apache Spark, TensorFlow… with GPUs • Distributed Computing (scale out and up) • Cloud Foundry, Spectrum Conductor, Mesos, Kubernetes, Docker, OpenStack, WebSphere • Cloud computing on High Performance Systems • OpenPOWER, IBM POWER 1 Indrajit Poddar Senior Technical Staff Member, Master Inventor, IBM Systems [email protected] Twitter: @ipoddar
- 3. #ibmedge About Yulia 2 Yulia Gaponenko Software Engineer, IBM Systems [email protected] github: barthy1 Expertise: • Cloud Foundry committer (BOSH project) • Distributed Computing (scale out and up) • Cloud Foundry, Docker, OpenStack, WebSphere • Open Source • Linux, Eclipse • Cloud computing on High Performance Systems • OpenPOWER, IBM POWER
- 4. #ibmedge Please Note: • IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice and at IBM’s sole discretion. • Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. • The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. • The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. • Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. 3
- 5. #ibmedge Agenda • Continuous Integration (CI) : What is it? • OpenPOWER Systems: What are these? • CI for Open Source Software on OpenPOWER: Why is it important? • Cloud Foundry: What is it? • Concourse: What is it? • Scenario: Continuously build a Concourse project from GitHub 4
- 6. #ibmedge Change is the only constant… 5 Collaborative Open Source Development has to embrace change e.g. Linux has 2,000 new developers and gets 10,000 patches for each version http://arstechnica.com/information-technology/2015/02/linux-has-2000-new- developers-and-gets-10000-patches-for-each-version/
- 7. #ibmedge What is the elephant like? 6 Ubuntu CentOS Fedora Intel - X86_64 Debian POWER – ppc64le ARM – arm64 https://en.wikipedia.org/wiki/Blind_men_and_an_elephant Open Source Development has to embrace variations Microprocessor Architectures Linux OS Distros
- 8. #ibmedge Continuous Integration for Open Source Development • Merge developer copies into shared mainline several times a day https://en.wikipedia.org/wiki/Continuous_integration • Best Practices • Continuous Build • Continuous Test • Continuous Deployment • Tools: • Jenkins • Travis • Concourse • Many more… 7
- 9. #ibmedge IBM POWER Systems Cloud Performance on Open Source • Runs popular Linux Distros • Runs Docker containers at a high density per system https://g01acxwass069.ahe.pok.ibm.com/cms/developerworks/linux/perfcol/cloud.html 8
- 10. #ibmedge OpenPOWER: Open Hardware for High Performance 9 Systems designed for big data analytics and superior cloud economics Upto: 12 cores per cpu 96 hardware threads per cpu 1 TB RAM 7.6Tb/s combined I/O Bandwidth GPUs and FPGAs coming… OpenPOWER Traditional Intel x86 http://www.softlayer.com/bare-metal-search?processorModel[]=9
- 11. #ibmedge OpenPOWER Worldwide Development Community 10 continents 20+ technologies revealed innovations under way100+ members200+ 24 6countries OpenPOWER is an open development community, using the POWER Architecture
- 12. #ibmedge IBM OpenPOWER Systems with Open Source Software 11 Open Source Software works great on POWER GoLang And many more …
- 13. #ibmedge Continuous Integration for Platform-as-a-Service (PaaS) • Develop, run, manage applications without worrying about the complexity of the infrastructure • Often runs on top of a Infrastructure-as-a- Service e.g. OpenStack • Examples: • Apache Mesos, Kubernetes with Docker e.g. docker run –t ppc64le/ubuntu • Cloud Foundry e.g. cf push helloworld.js • CI and PaaS naturally go hand in hand 12
- 14. #ibmedge Continuous Integration for OpenPOWER How to ensure that latest versions of open source software packages always run on OpenPOWER? • Source code changes from multiple developers do not cause breakages • Particularly when development is done on x86 • Changes are automatically tested on POWER architecture • Binary packages for POWER are always available at the most recent level • in repository as binary blob • as Docker image • Automatic versioning 13 Docker image Source code Github Building next version Testing on Power Repository
- 15. #ibmedge Concourse: Continuous Integration for Cloud Foundry 14 • Pipeline-based CI system written in Go. • Website: concourse.ci
- 16. #ibmedge 15 Cloud Foundry. What is it? • Open source cloud computing Platform-as-a-Service (PaaS) • Faster and easier to build, test, deploy, and scale applications • Choice of clouds, developer frameworks, and application services. Pivotal Cloud Foundry, HP Helion, Predix GoLang Commercial PaaS products based on open source Cloud Foundry:
- 17. #ibmedge Scenario: Continuously Build a Concourse project from GitHub • https://github.com/barthy1/git-resource • tracks commits in a branch of a Git repository • Get: • Github project (git-resource) • Docker image with git (git-image) • Put: • New Docker image with git-resource to Docker registry (git-resource-image) • Action: • Build and test (git-resource) 16
- 18. #ibmedge Scenario: CI Tools • Docker Registry: to distribute container images • Open Stack: to create connected virtual machines on multiple servers • BOSH: to deploy Cloud Foundry releases on Open Stack • Github: to collaboratively develop Open Source Software • Concourse: to continuously integrate open source development and test – Concourse cli called fly • Concourse CI descriptor, called a pipeline: to design and execute a workflow for continuous integration 17
- 19. #ibmedge Scenario: CI Tools Interaction 18 Bare metals VM VM Concourse Server Docker Containers VM VM BOSH Image Registry Object Store Concourse fly client or browser client DJ1. Sets up CI infrastructure DJ2. Adds CI pipeline DJ3. Commit code CS1. Monitor CS2. Build CS3. Test 4. Output Developer Jane
- 20. #ibmedge 1. Deploy your own or use a managed OpenStack • Oregon State University Open Source Lab OpenStack on OpenPOWER systems • SoftLayer bare metals with Ubuntu OpenStack on OpenPOWER 19
- 21. #ibmedge 2. Setup a Docker Private Registry or use DockerHub Docker Registry: Docker Image storage DockerHub: Public Docker registry • Docker private registry • Install Docker • Clone https://github.com/barthy1/distribution • Build Docker image • Configure TLS and basic auth • Run Docker private registry 20
- 22. #ibmedge 3. Install BOSH BOSH: Tool to simplify deployment of other projects (Concourse, CF…) • Deploy project using release packages • Update deployment with new versions of the release packages • Monitor state of the deployment • Fix problems with VMs or connectivity • Scale out with additional VMs • Bosh installation: • Binaries available from: ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/ – bosh-init - BOSH installer – BOSH release – bosh cli – interact with BOSH server • Sample deployment manifest: https://www.cloudfoundry.org/bosh-on-openpower-systems/ – update only OpenStack connection details and links to binaries 21
- 23. #ibmedge 4. Install Concourse Concourse: Project to continuously run tests and build binaries • Concourse contains 2 releases: • Concourse • Garden-runc • Concourse installation: • Binaries available from: ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/ • Sample deployment manifest • Installation steps: https://www.cloudfoundry.org/concourse-release-for-openpower – upload all releases – upload deployment manifest – start deployment 22
- 24. #ibmedge 5. Concourse pipelines Pipeline: Set of steps to run the tests and build the result • Various input and output data types – Git - pull and push to git repositories – Time - start on schedule or timestamp output – Docker-image - fetch, build and push Docker images – Semver – set or bump version numbers – Amazon S3 - fetch from and upload to s3 buckets – …. – Custom resources • Color of the box shows type and state – Black – resources – Green – successfully finished most recent build – Red – test/build failed • All the tests and the logs are available • Group by tests for every commit • History of the Github project updates 23
- 25. #ibmedge 5. Examine an existing pipeline • Pipeline sample: • Input resources: – Github project – Docker image • Test/build stage – Clone/update Github project – Pull/refresh Docker image – Run the test – Build new Docker image – Push Docker image to Docker registry • Output resources: – New Docker image • Logs are available for public/private access • New test can be started via Web • Access to the container/s with tests • Information about current commit 24
- 27. #ibmedge 6. Update the existing pipeline • Existing pipeline to show live how it works • Update the pipeline descriptor with new parameters • Upload new pipeline to the server with Concourse cli • Run new pipeline • Show execution steps and logs • Login to the running container with tests • Run latest Docker image for POWER 26
- 28. #ibmedge Summary • Continuous Integration is essential for open source projects • CI benefits OpenPower • Concourse tool is great for CI! 27
- 29. #ibmedge Resources • Pointers to blogs on how to deploy BOSH, Cloud Foundry, Concourse on OpenPOWER http://blog.altoros.com/cloud-foundry-on-ibm-power.html https://www.cloudfoundry.org/bosh-on-openpower-systems/ https://www.cloudfoundry.org/cf-release-for-openpower/ https://www.cloudfoundry.org/concourse-release-for-openpower • How to setup Docker private registry: https://gist.github.com/barthy1/246cd6892d5db2cec9b6f866de70c089 • Latest Docker binaries for POWER: https://master.dockerproject.org/ • Latest Cloud Foundry binaries for POWER: ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/ • Oregon State University Open Source Lab with OpenStack: http://osuosl.org/services/powerdev 28
- 30. #ibmedge Notices and Disclaimers 29 Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.” Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
- 31. #ibmedge Notices and Disclaimers Con’t. 30 Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
- 32. © 2016 IBM Corporation #ibmedge Thank You