Олександр Щедров — Build your application in seconds and optimize workflow as much as you can using DevOps
0 likes213 views
The document discusses optimizing application builds and workflows through DevOps, highlighting the build process, challenges with large projects, and the importance of clean code and fast delivery. It details tools and practices for automating builds, using Docker for virtualization, and integrating with GitHub for testing and status reporting. The presentation emphasizes efficiency improvements in build timelines and database optimization strategies.
![GitHub API for lazy builders
sudo jo state=pending target_url=http://URL/job/PR_BUILDER_TESTS/$
{BUILD_NUMBER}/console description="Tests run has been started."
context="CIBox tests" > pending.json
curl -u [bot_name]:[token] https://api.github.com/repos/[org]/
[repo]/statuses/${PR_SHA1} --request POST --data @pending.json
Pending status](https://image.slidesharecdn.com/alexanderschedrov-160907073416/85/Build-your-application-in-seconds-and-optimize-workflow-as-much-as-you-can-using-DevOps-34-320.jpg)
![GitHub API for lazy builders
# Run tests.
cd /var/www/build${PARENT_BUILD_NUMBER}
ansible-playbook tests.yml -i 'localhost,' --connection=local
# Set status and post comment to GitHub.
cd ${WORKSPACE}
sudo jo body="Behat test results file URL: http://URL/build$
{PARENT_BUILD_NUMBER}/build_reports/behat_report.html" >
comment.json
curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/
[repo]/issues/${PR_ID}/comments --request POST --data @comment.json
sudo jo state=success target_url=http://URL/job/PR_BUILDER_TESTS/$
{BUILD_NUMBER}/console description="Tests run has been finished."
context="CIBox tests" > success.json
curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/
[repo]/statuses/${PR_SHA1} --request POST --data @success.json
Tests + success status](https://image.slidesharecdn.com/alexanderschedrov-160907073416/85/Build-your-application-in-seconds-and-optimize-workflow-as-much-as-you-can-using-DevOps-36-320.jpg)
![GitHub API for lazy builders
cd ${WORKSPACE}
sudo jo state=failure target_url=http://URL/job/PR_BUILDER_TESTS/$
{BUILD_NUMBER}/console description="Tests run has been failed."
context="CIBox tests" > failed.json
curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/
[repo]/statuses/${PR_SHA1} --request POST --data @failed.json
Failed status](https://image.slidesharecdn.com/alexanderschedrov-160907073416/85/Build-your-application-in-seconds-and-optimize-workflow-as-much-as-you-can-using-DevOps-38-320.jpg)
![Remote access API
http://[JENKINS_URL]/job/[JOB_NAME]/api/json?tree=allBuilds[*]](https://image.slidesharecdn.com/alexanderschedrov-160907073416/85/Build-your-application-in-seconds-and-optimize-workflow-as-much-as-you-can-using-DevOps-60-320.jpg)
Олександр Щедров — Build your application in seconds and optimize workflow as much as you can using DevOps
- 1. Alexander Schedrov aka sanchiz DrupalCamp Lviv, Sep 2016 Build your application in seconds and optimize workflow as much as you can using DevOps
- 2. What is build? Build is the process of integrating, building and compiling the software that is produced.
- 3. – Кузьма Иванович, водовоз “Без воды - и не туды, и не сюды”
- 4. – Кузьма Иванович, водовоз “Без воды - и не туды, и не сюды”
- 5. – Щедров Александр, билдовоз :) “Без билда - и не туда, и не сюда”
- 9. What is a big project?
- 10. Big data?
- 11. Big codebase?
- 12. Big team?
- 13. Big client?
- 14. Big money!
- 15. Big money!
- 16. What is the most valuable thing for client in project?
- 18. Custom modules?
- 19. Tests coverage?
- 21. – Andrii Podanenko, FFW “Success depends on how fast you can show changes to client.”
- 22. Our project
- 23. In terms of build • $$$ :) • Huge database • ~300mb *.sql.gz • ~ 4gb *.sql • D8 multisite - 3 sites • Automated tests • Code sniffers
- 24. In terms of workflow • Builds queue • ~50 per day • Server space • 80GB droplet • ~10 active builds + 5 feature dev sites
- 28. + 2 subsites :(
- 29. Let The Machines Do The Work
- 30. Lazy builders
- 31. Primary build and secondary builds 1. Primary build 2. Other builds in parallel • Slave sites • Sniffers • Tests
- 34. GitHub API for lazy builders sudo jo state=pending target_url=http://URL/job/PR_BUILDER_TESTS/$ {BUILD_NUMBER}/console description="Tests run has been started." context="CIBox tests" > pending.json curl -u [bot_name]:[token] https://api.github.com/repos/[org]/ [repo]/statuses/${PR_SHA1} --request POST --data @pending.json Pending status
- 36. GitHub API for lazy builders # Run tests. cd /var/www/build${PARENT_BUILD_NUMBER} ansible-playbook tests.yml -i 'localhost,' --connection=local # Set status and post comment to GitHub. cd ${WORKSPACE} sudo jo body="Behat test results file URL: http://URL/build$ {PARENT_BUILD_NUMBER}/build_reports/behat_report.html" > comment.json curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/ [repo]/issues/${PR_ID}/comments --request POST --data @comment.json sudo jo state=success target_url=http://URL/job/PR_BUILDER_TESTS/$ {BUILD_NUMBER}/console description="Tests run has been finished." context="CIBox tests" > success.json curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/ [repo]/statuses/${PR_SHA1} --request POST --data @success.json Tests + success status
- 38. GitHub API for lazy builders cd ${WORKSPACE} sudo jo state=failure target_url=http://URL/job/PR_BUILDER_TESTS/$ {BUILD_NUMBER}/console description="Tests run has been failed." context="CIBox tests" > failed.json curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/ [repo]/statuses/${PR_SHA1} --request POST --data @failed.json Failed status
- 41. Build hierarchy Primary Build Slave 1 Slave 2 Sniffers Tests Comment + status on GitHub Comment + status on GitHub Comment + status on GitHub Comment + status on GitHub Comment + status on GitHub
- 44. Optimize SQL import SQL import is the most intensive operation • Warm up databases for build? • Bandwidth speed for downloading dumps locally? • Maintenance?
- 46. Create Docker Image 1. Run local Docker repository(per project) 2. Pull base image cibox/mysql 3. Import database from production dump 4. Create/Update image after import 5. Push to local Docker repository(per project)
- 47. Build pipeline 1.Pull changes from Docker Hub + Custom Repository 2.Run Docker container 3.Specify in settings container’s IP 4.Container will be killed via cleaner
- 48. Benefits 1. Seconds to spin up container 2. Non-blocking operations for concurrent builds 3. Pull only changes(commits) to local environment 4. Update Docker Image daily
- 51. New Primary build timeline
- 53. Why it faster now? • ~2 seconds to spin up container • ~2 seconds to clear Drupal 8 cache • Non-Blocking operation in filesystem • Isolated MySQL • Run fast tests on build • Run full tests on demo/dev site
- 55. Numbers • Primary build • ~15 mins -> ~3 mins • Overall time • ~3 mins before testing of main product • ~6-8 mins before testing of secondary products • ~6-8 mins before sniffers & test results
- 56. Metrics - measure your success
- 58. Time in comments
- 59. https://wiki.jenkins-ci.org/display/JENKINS/Global+Build +Stats+Plugin
- 61. DevOps life
- 62. – Chang Xiao, FFW “I swear to god our vagrant is like the nerdiest thing ever.” “It should be like making a ANSCII sandwich, brewing some tea, waiting for your local environment to be fully baked.”
- 64. Drupal.org: https://www.drupal.org/u/sanchiz GitHub: https://github.com/Sanchiz Blog: http://sanchiz.net Email: [email protected] Twitter: @alexschedrov Thank you!