Continuous Delivery NYC: From GitOps to an adaptable CI/CD Pattern for Kubernetes
1 like346 views
The document discusses transitioning from GitOps to an adaptable CI/CD pipeline for Kubernetes, emphasizing the need to modernize sub-optimal software delivery processes. It outlines current challenges in adopting Kubernetes-native best practices and presents a proposed approach that integrates existing methodologies with Kubernetes capabilities. Additionally, the document suggests a flexible mental model for managing application updates and environment configurations, alongside best practices for implementing deployment pipelines.
Continuous Delivery NYC: From GitOps to an adaptable CI/CD Pattern for Kubernetes
- 1. From GitOps to an adaptable CI/CD pattern for Kubernetes Andrew Phillips Continuous Delivery : NYC; Nov 1, 2018
- 2. From GitOps to an adaptable CI/CD pattern...for everything? Andrew Phillips Continuous Delivery : NYC; Nov 1, 2018
- 3. The bio slide ● Been on most sides of this space: developer, infra builder, product owner, evangelist and more ● Long-standing open-source contributor ● Author and regular conference and meetup presenter ● Co-organizer of ContainerDays Boston & NYC
- 4. Agenda 1. What’s the challenge? 2. CI/CD practices & patterns today 3. What does Kubernetes add into the mix? 4. A CI/CD pipeline for Kubernetes 5. Next steps for your scenario 6. Q&A
- 5. 1. The context ● Lots of organizations looking at Kubernetes right now ● Trying to use this also as an opportunity to “clean up” sub-optimal software delivery pipelines ● How to do this right?
- 6. 1. The challenge ● Lots of new runtime-specific concepts to deal with ● “Kubernetes-native” best practices still very much in flux ● Wrapping your head around “new” practices (e.g. GitOps) is hard ● Extrapolating from new practices in theory to actual, working implementations is even harder ● Figuring out how all this new stuff relates to accepted industry practices and tools is harder still
- 7. Failure mode: CCAOA Cargo Cult All Over Again
- 8. 1. The approach ● Recall good practices that we’d like to retain ● Incorporate refinements related to Kubernetes to create a straw man setup ● Develop a mental model to understand advantages and shortfalls of the setup ● Refine the setup based on tradeoffs to be made related to each user’s particular situation ● Implement using appropriate tools
- 9. 2. Existing good practice ● Reproducible builds ● Store source and derived artifacts appropriately ● Minimize duplication, especially around environment config ● Keep the business process flexible and the env automation robust ● Support 4 related processes: ○ Application update ○ Environment (config) update ○ Environment spin-up/restore ○ Environment drift detection and remediation ● Specifically, support env-specific (e.g. log settings change) and cross-env (e.g. new app version rollout) processes concisely
- 10. Deployment execution: business process vs. technical process Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 11. 2. The four related processes ● Application update ○ “I want to validate a new release candidate and promote it through envs to prod” ● Environment (config) update ○ “I want to change the attributes of a particular env only” ● Environment spin-up/restore ○ “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point in time” ● Environment diff, drift detection and remediation ○ “I want to understand how the actual config of an env relates to the intended config, also potentially across different environments”
- 12. 3. What’s new with Kubernetes ● “as-code” description of what an environment should look like (“environments-as-code”) ● Actuation based on reconciliation engine built into runtime with continuous enforcement ○ Interest in “pull-based”, async invocation via repo-watching ● Out-of-the-box support for some types of rollout via Deployment object, as well as CRDs to define your own ○ But can also manipulate underlying objects directly ● GitOps ~ environments-as-code + async invocation + repo workflow for business process
- 13. Deployment execution: adding in Kubernetes Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment “environments-as-code” repositories Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 14. TL;DR Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment “environments-as-code” repositories Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace Imperative pipeline across environments Declarative spec for each environment (with support for some imperative “cheating” where necessary)
- 15. 3. The straw man
- 17. App1 update
- 18. team1-repo env-repo applied-staging-branch applied-prod-branch master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }}
- 19. team1-repo env-repo applied-staging-branch app1.yaml applied-prod-branch master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }} Step 1 apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app: app1 env: staging --- apiVersion: apps/v1beta2 kind: Deployment metadata: name: app1 namespace: staging ... --- apiVersion: v1 kind: ConfigMap ...
- 20. team1-repo env-repo applied-staging-branch app1.yaml applied-prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }} Step 2 apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app: app1 env: production --- apiVersion: apps/v1beta2 kind: Deployment metadata: name: app1 namespace: production ... --- apiVersion: v1 kind: ConfigMap ...
- 21. App2 update
- 22. team1-repo env-repo applied-staging-branch app1.yaml applied-prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config app.version: 0.0.34 port: 8090 image: gcr.io/image@sha256:{{ container.image.digest }}
- 23. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 1app.version: 0.0.34 port: 8090 image: gcr.io/image@sha256:{{ container.image.digest }} apiVersion: apps/v1beta2 kind: Deployment metadata: name: app2 namespace: staging ... spec: replicas: 1 ... image: gcr.io/image@sha256:abcdef ports: - name: http containerPort: 90 --- ... --- apiVersion: v1 kind: ConfigMap ... data: config.yaml: | 'app.version': 0.0.4
- 24. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 2 app.version: 0.0.34 port: 8090 image: gcr.io/image@sha256:{{ container.image.digest }} apiVersion: apps/v1beta2 kind: Deployment metadata: name: app2 namespace: production ... spec: replicas: 1 ... image: gcr.io/image@sha256:abcdef ports: - name: http containerPort: 90 --- ... --- apiVersion: v1 kind: ConfigMap ... data: config.yaml: | 'app.version': 0.0.4
- 25. App2 prod config change
- 26. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config apiVersion: v1 kind: ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': INFOTRACE
- 27. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 1 apiVersion: v1 kind: ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': INFOTRACE apiVersion: v1 kind: ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': TRACE
- 28. 3. The straw man ● Source config in app repo, “compiled” (a.k.a. hydrated) config in env repo(s) ○ Cf. source code in app repo, compiled code in artifact repository ● One app repo per app/team, shared env repos for environments ● Use branches where possible to represent different environments ○ If greater separation via access control is needed, use different repos or consider other storage ● Keep config in app repo unless it needs to be independent of the app lifecycle or you really, really need to be able to change it for one env ○ Allows you to use templating/overriding to cut down on duplication ○ The fewer places to keep track of config, the better ● Source config doesn’t have to be “raw” YAML, can be more suitable abstraction! ○ Cf. higher-level language source code vs. low-level assembly code
- 29. 3. The straw man ● Prefer explicit invocation over “repo-watching” ○ Avoids “root-level” process running inside cluster, and provides more flexibility for multi-step rollouts ○ “Repo-watching” makes visualizing current status harder, and requires some sort of feedback mechanism to distinguish successful from failed deployments in the repo ○ Hard to support phased/multi-step application ○ Easier to reproduce/simulate
- 30. 3. The straw man ● Prefer explicit invocation over “repo-watching” ○ Avoids “root-level” process running inside cluster, and provides more flexibility for multi-step rollouts ○ “Repo-watching” makes visualizing current status harder, and requires some sort of feedback mechanism to distinguish successful from failed deployments in the repo ○ Hard to support phased/multi-step application ○ Easier to reproduce/simulate ● Commit after successful application, not before ○ Avoids having to distinguish attempted from successful deployments in the repo ○ Allows for richer pre-application validation than code diff in a PR (e.g. three-way diff against actual environment) ○ Avoids commit rights to repo being equivalent to deploy rights to env (and your automation will need commit rights to make pull requests, unless you used forked repos) ○ Harder if pull requests are used for the business process (as in vanilla GitOps) - requires multiple branches or “on approve” deployment
- 31. 4. The mental model ● Understand how your app is updated across two dimensions: ● code change promoted through to prod ● common externalized config setting, e.g. localized title dependent on app version independent of environment
- 32. 4. The mental model ● Understand how your app is updated across two dimensions: ● code change promoted through to prod ● common externalized config setting, e.g. localized title ● adding debug logging to staging ● configuring a scaling policy in prod based on a new metric dependent on app version independent of environment dependent on environment
- 33. 4. The mental model ● Understand how your app is updated across two dimensions: ● code change promoted through to prod ● common externalized config setting, e.g. localized title ● rotating database credentials for prod ● updating discovery service endpoint ● adding debug logging to staging ● configuring a scaling policy in prod based on a new metric dependent on app version independent of environment dependent on environment independent of app version
- 34. 4. The mental model ● Understand how your app is updated across two dimensions: Application update Static environment config update (App-linked) environment config update dependent on app version independent of environment dependent on environment independent of app version
- 35. 4. The mental model ● Understand how your app is updated across two dimensions: Application update Static environment config update (App-linked) environment config update dev responsibility ~ app repo platform responsibility ~ env repo
- 36. 4. The mental model 1. Application update a. “I want to validate a new release candidate and promote it through envs to prod” 2. Environment (config) update a. “I want to change the attributes of a particular env only” 3. Environment spin-up/restore a. “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point in time” 4. Environment diff, drift detection and remediation a. “I want to understand how the actual config of an env relates to the intended config, also potentially across different environments”
- 37. ● App repo is driving deployment, env repo is snapshotting cluster state ○ Think source code in github is to docker image in registry as template in app repo is to manifest in env repo ● Env repo is just a checkpoint in time, cluster can evolve ○ Kubernetes applies changes to manifests ○ Strategies like exponential rollouts or traffic shifting apply changes over time ● Env repo is not a guaranteed healthy state ○ We can defer snapshotting until some health/success metric… but rollback has no silver bullet ● Deletion from either repo ≠ deletion from cluster ○ Challenge is: not trivial to know if cluster depends, or will depend on manifest not submitted to repo ○ kubectl apply --prune attempts to solve this with a lot of (scary) caveats 4. The mental model
- 38. 5. Tuning for your scenario ● Appropriate level of abstraction? How much “raw” Kubernetes YAML should our developers have access to? ● Where should the abstraction live? In templates? In CRDs? In the automation tool? ● What to use for templating? Token replacement or overrides? ● When do you snapshot/publish to the env repo? On every change to the cluster? Or when a desired end-state is reached? (think multi-step rollout) ● Access control: how many repos do you need? Are code repositories right for your use case, or e.g. a blobstore better? ● Am I distributing or deploying? Helm is much better suited for distribution than deployment ○ Its templating capability is often used as part of deployment flows, though
- 39. 5. Implement ● Choose storage implementations and partitioning strategies for your environments-as-code ● Define the appropriate level of abstraction for your developers and choose tools to support it ● Choose a flexible automation tool for your deployment business process ● Define an appropriate definition of deployment health/success to determine when a deployment is “good” ● Decide which of the four processes - app update, env config update, env restore and env drift detection - you want to support ● Build pipelines ● Done!