Optimizing Your CI Pipelines
0 likes81 views
Take your CI to the next level! Learn how to optimize your pipelines for faster and more efficient builds through parallelization, caching, failing early, and more.
![With DAG build_3.8:
stage: build
script:
- docker build -f Dockerfile_3.8
- docker push $CI_REGISTRY_IMAGE:3.8
test_3.8:
stage: test
needs: ["build_3.8"]
script:
- docker pull $CI_REGISTRY_IMAGE:3.8
- docker run $CI_REGISTRY_IMAGE:3.8 pytest
release_3.8:
stage: release
needs: ["test_3.8"]
script: echo "Release script"](https://image.slidesharecdn.com/optimizingyourcipipelines-230525144844-1d320116/85/Optimizing-Your-CI-Pipelines-31-320.jpg)
![Downstream (child) pipelines
frontend:
trigger:
include: frontend/.gitlab-ci.yml
strategy: depend
rules:
- changes: [frontend/*]
backend:
trigger:
include: backend/.gitlab-ci.yml
strategy: depend
rules:
- changes: [backend/*]](https://image.slidesharecdn.com/optimizingyourcipipelines-230525144844-1d320116/85/Optimizing-Your-CI-Pipelines-34-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Optimizing Your CI Pipelines (20)
Ad
More from Sebastian Witowski (6)
Ad
Recently uploaded (20)
Optimizing Your CI Pipelines
- 1. PyCon Italia 2023 Sebastian Witowski Optimizing Your CI Pipelines
- 2. Simple pipelines are relatively easy
- 4. Roadmap Better Docker setup Make things run faster Run less often, stop fast Tips & tricks
- 5. Choosing a CI system
- 6. Choosing a CI system
- 10. Example project
- 11. Example project
- 12. Example project ... build: stage: build script: - docker compose build test: stage: test script: - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest deploy: stage: deploy script: echo "Here goes deployment script" environment: production
- 13. Certificate of ABSOLUTELY NOT PRODUCTION GRADE!
- 14. Docker
- 15. Optimize your Docker configs! Layers caching, tags, etc.
- 16. Improve your Docker config FROM python:3.10-slim-buster FROM python:3.10-alpine
- 17. Improve your Docker config FROM python:3.10-slim-buster Bigger image Shorter build time FROM python:3.10-alpine Smaller image Longer build time
- 18. Push (pull) image to (from) registry ... build: stage: build script: - docker compose build test: stage: test script: - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest
- 19. Push (pull) image to (from) registry ... before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - docker compose build - docker push registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev test: stage: test script: - docker pull registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest
- 20. Push (pull) image to (from) registry ... before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - docker compose build - docker push registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev test: stage: test script: - docker pull registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest
- 23. Make things run faster (in parallel)
- 24. Make things run faster (in parallel)
- 25. Make things run faster (in parallel)
- 26. Make things run faster (in parallel) There is an open issue about this from 2018.
- 27. Directed Acyclic Graph One job starts after another finishes, regardless of what stage they are in.
- 28. Without DAG
- 29. Without DAG
- 30. With DAG
- 31. With DAG build_3.8: stage: build script: - docker build -f Dockerfile_3.8 - docker push $CI_REGISTRY_IMAGE:3.8 test_3.8: stage: test needs: ["build_3.8"] script: - docker pull $CI_REGISTRY_IMAGE:3.8 - docker run $CI_REGISTRY_IMAGE:3.8 pytest release_3.8: stage: release needs: ["test_3.8"] script: echo "Release script"
- 32. Downstream (child) pipeline Separate mini-pipelines that can be triggered from your main pipeline.
- 34. Downstream (child) pipelines frontend: trigger: include: frontend/.gitlab-ci.yml strategy: depend rules: - changes: [frontend/*] backend: trigger: include: backend/.gitlab-ci.yml strategy: depend rules: - changes: [backend/*]
- 36. Run tests in parallel pytest-xdist Run tests across multiple CPUs $ pip install pytest-xdist $ pytest -n auto
- 37. Run tests in parallel pytest-xdist Run tests across multiple CPUs pytest-test-groups Run tests across multiple runners
- 38. Run tests in parallel pytest-xdist Run tests across multiple CPUs pytest-test-groups Run tests across multiple runners # requirements.in pytest-test-groups # .gitlab-ci.yml test: stage: test parallel: 5 script: - pytest --test-group-count $CI_NODE_TOTAL --test-group=$CI_NODE_INDEX
- 39. Run tests in parallel pytest-xdist Run tests across multiple CPUs pytest-test-groups Run tests across multiple runners
- 40. Run less and stop fast • Interruptible jobs
- 41. Run less and stop fast • Interruptible jobs build: stage: build interruptible: true script: - docker compose build
- 42. Run less and stop fast • Interruptible jobs • Stop fast (pytest -x) build: stage: build interruptible: true script: - docker compose build
- 43. Tip 5: Not running things in the CI
- 44. Not every check is mandatory in the CI
- 45. Not every check is mandatory in the CI Some can run only on the main branches. Others can be triggered manually.
- 47. Caching and cache policies You can use caching to, well, cache stuff between jobs. But you can also specify if you want to push or pull stuff to cache using policy key. default: cache: &global_cache key: $CI_COMMIT_REF_SLUG paths: - .cache/pip - some/other/path/ policy: pull-push job: cache: # inherit all global cache settings <<: *global_cache # override the policy policy: pull
- 48. Fast zip For caching/artifacts, you can choose different level of compression (low level of compression runs faster, but results in a larger zip file). variables: FF_USE_FASTZIP: "true" # Available options are: # fastest, fast, default, slow, # or slowest ARTIFACT_COMPRESSION_LEVEL: "fastest" CACHE_COMPRESSION_LEVEL: "fastest"
- 50. Use your own runners
- 51. Use your own runners
- 52. Takeaways
- 53. Takeaways • Learn concepts, not tools
- 54. Takeaways • Learn concepts, not tools • There are no silver bullets python-alpine or python-debian? Pull an image or build it?
- 55. Takeaways • Learn concepts, not tools • There are no silver bullets python-alpine or python-debian? Pull an image or build it? • Not every check has to run in every pipeline make MR pipelines fast and main branch pipelines thorough
- 56. Takeaways • Learn concepts, not tools • There are no silver bullets python-alpine or python-debian? Pull an image or build it? • Not every check has to run in every pipeline make MR pipelines fast and main branch pipelines thorough • Outdated CI setup is also a technical debt
- 58. • Slide 1: https:/ /www.midjourney.com/app/search/?jobId=6f707c18-bf19-4fac-b87f-622ff3e1561b • Roadmap - containers: https:/ /www.midjourney.com/app/search/?jobId=28e92637-bfd6-4d07-ae36-a279dd35c2d4 • Roadmap - rocket: https:/ /www.midjourney.com/app/search/?jobId=eaeeb022-17af-4b71-8438-a8eb05afc298 • Roadmap - traffic lights: https:/ /www.midjourney.com/app/search/?jobId=89159672-debf-4845-a822-be159983f172 • Roadmap - chest: https:/ /www.midjourney.com/app/search/?jobId=1bb66ddc-84a4-4ac1-a36a-7c7f8dbd1c04 • Gitlab logo: https:/ /www.midjourney.com/app/search/?jobId=a5491c2d-199e-43dd-b65f-9ebefc031f65 • Ribbon badge: https:/ /www.clipartmax.com/download/m2i8H7d3Z5i8G6K9_certificate-ribbons-ribbon-badge-vector-png/ • Containers: https:/ /www.midjourney.com/app/search/?jobId=f14ee5f1-b3a7-4bca-9907-e67607beff90 • DAG image: https:/ /www.midjourney.com/app/search/?jobId=4609a9aa-7154-49f2-9b10-38d600aa04f2 • Child pipelines: https:/ /www.midjourney.com/app/search/?jobId=39e19a04-1196-4c2e-99e9-f242464b5d7c • Rube Goldberg machine: https:/ /www.midjourney.com/app/search/?jobId=60ab5692-a07a-4992-8641-13dfbe6193f2 Attributions Most images comes from midjourney.com Drawings were done with excalidraw.com