Continuos Integration @Knetminer
Download as PPTX, PDF•0 likes•450 views
The document presents a detailed overview of continuous integration (CI) practices, outlining typical CI build workflows and various tools used over the years, including Jenkins and GitHub Actions. It emphasizes the importance of customizing CI processes with script hooks to improve maintainability and portability. Additionally, it provides examples and resources for further implementation and customization of CI jobs.
![Typical CI build workflow
1. Is build needed (no changes in a scheduled job or "[ci skip]")? => STOP
2. Get and set variables (eg, github and Nexus credentials)
3. Are we releasing? => change version in Maven POMs (eg, 2.0, parameter sent by the user)
4. "mvn deploy", or "mvn install"
1. depending on conditions (eg, if we're in the master branch and not in a pull request)
5. Are we releasing?
1. Tag github with new version and git-commit
2. Change POMs with next snapshot version (eg, 2.0.1-SNAPSHOT, parameter sent by the user)
6. Local git commits (eg, because of POM's new versions and tagging)? => git push](https://image.slidesharecdn.com/ci-at-knetminer-210331143826/85/Continuos-Integration-Knetminer-3-320.jpg)
![Customisation with script hooks
1. Is build needed (no changes in a scheduled job or "[ci skip]")? => STOP
2. Get and set variables (eg, github and Nexus credentials)
build-begin.sh
3. Are we releasing? => change version in Maven POMs (eg, 2.0, parameter sent by the user)
4. build-before.sh
( "mvn deploy", or "mvn install" ) OR build-body.sh
build-after.sh
5. Are we releasing?
1. Tag github with new version and git-commit
2. Change POMs with next snapshot version (eg, 2.0.1-SNAPSHOT, parameter sent by the user)
6. Local git commits (eg, because of POM's new versions and tagging)? => git push
build-end.sh](https://image.slidesharecdn.com/ci-at-knetminer-210331143826/85/Continuos-Integration-Knetminer-10-320.jpg)
Continuos Integration @Knetminer
- 1. Marco Brandizi <[email protected]> 30/4/2021 Continuos Integration @Knetminer Find this presentation on SlidesShare Background source: https://tinyurl.com/ydq2e6xn
- 2. What's Continuous Integration, by the way? Image source: https://pt.slideshare.net/thyagk/continuous-integration-system/3
- 3. Typical CI build workflow 1. Is build needed (no changes in a scheduled job or "[ci skip]")? => STOP 2. Get and set variables (eg, github and Nexus credentials) 3. Are we releasing? => change version in Maven POMs (eg, 2.0, parameter sent by the user) 4. "mvn deploy", or "mvn install" 1. depending on conditions (eg, if we're in the master branch and not in a pull request) 5. Are we releasing? 1. Tag github with new version and git-commit 2. Change POMs with next snapshot version (eg, 2.0.1-SNAPSHOT, parameter sent by the user) 6. Local git commits (eg, because of POM's new versions and tagging)? => git push
- 4. Our (and not only) Issues 1. Pre-check if build is needed 2. Settings 3. Pre-build Release ops 4. Maven (includes deployment) 5. Post-build release ops 6. Git Push • Pretty much the same for all Java/Maven projects • Copy-pasting is hard to maintain • Can't we factorise?
- 5. Our (and not only) Issues • CIs in my past five years: • Until 2016: Bamboo • 2017-2021: Jenkins • 2018-2020: Travis • 2020-2021: GitHub Actions • 2022 Yet another one?! • Every time: • New system to learn • New syntax to describe CI jobs to learn • Migrating previous jobs and test • Migrating settings (eg, credentials) • Can't we make it portable? • (At least a little more?)
- 6. My Approach
- 7. Just plug-in in your CI 1. Pre-check if build is needed 2. Settings 3. Pre-build Release ops 4. Maven (includes deployment) 5. Post-build release ops 6. Git Push Your CI job (see archetype-resources/.github/workflows/build.yml) mkdir -p ci-build <Download install.sh> ./ci-build/install.sh # was downloaded together with others (eg, maven-settings.xml) ./ci-build/build.sh
- 9. Customisation 1. Pre-check if build is needed 2. Settings 3. Pre-build Release ops 4. Maven (includes deployment) 5. Post-build release ops 6. Git Push • Uhm... aren't you throwing your CI workflow at me? • I've had the problem myself • Solution: script hooks
- 10. Customisation with script hooks 1. Is build needed (no changes in a scheduled job or "[ci skip]")? => STOP 2. Get and set variables (eg, github and Nexus credentials) build-begin.sh 3. Are we releasing? => change version in Maven POMs (eg, 2.0, parameter sent by the user) 4. build-before.sh ( "mvn deploy", or "mvn install" ) OR build-body.sh build-after.sh 5. Are we releasing? 1. Tag github with new version and git-commit 2. Change POMs with next snapshot version (eg, 2.0.1-SNAPSHOT, parameter sent by the user) 6. Local git commits (eg, because of POM's new versions and tagging)? => git push build-end.sh
- 11. Customisation with script hooks Your CI job 1. Download 2. Run 1. Pre-check if build is needed 2. Settings build-begin.sh 3. Pre-build Release ops 4. build-before.sh Maven (includes deployment) OR build-body.sh build-after.sh 5. Post-build release ops 6. Git Push build-end.sh Just add your hooks as needed 1. to <your-repo>/ci-build/ 2. Only the ones you need 3. They see build variables (eg, IS_RELEASE, NEW_RELEASE_VER) 4. They can expose their own variables (eg, for the next hook) 1. Or change existing ones (be careful)
- 12. Need an example? https://github.com/Rothamsted/knetminer/tree/master/ci-build • You'll see customisations for: • Settings custom variables (docker_tag) • Build on top of our own Docker-based environment, instead of GH Actions environment • Much more portable • post-build actions to push new Docker images (including release images) • Invocation of Jenkins API to trigger internal deployment jobs (which redeploy from the new Docker images)
- 13. Final notes • build.sh isn't the bible, we can change/improve as we need, eg • more hooks • See if it can be adapted to other languages (eg, Python) or a new one is needed • DO NOT copy it locally just because it doesn't fit your needs, try to reuse stuff! • Please let me know about proposed/made changes • Are there alternatives? => https://github.com/rosineygp/mkdkr • + Entirely based on Docker + Make, more portable • - Not sure using Docker even for the simplest builds is worth • - I know Make very little, not sure I want to depend on it • ~ Requires some CI configuration anyway (eg, to set up credentials and alike, should be solved with a credential manager) • - Presumably, would require volume mapping to manage the cache (and cache configuration at CI level)