Observability and Orchestration of your GitOps Deployments with Keptn
Download as pptx, pdf0 likes34 views
GitOps has become the default way to manage configuration in cloud native environments with tools like Argo or Flux keeping Git and K8s in sync. But GitOps lacks end-2-end traceability when GitOps operators make changes on the target environments. And as k8s lacks application awareness its hard to enforce pre- and post-deployment orchestration task such as sending notifications upon successful app delivery or validating all SLOs are healthy for a new version. The CNCF project Keptn is addressing those challenges by automatically providing End-2-End Observability through OpenTelemetry as well as introducing an application deployment lifecycle events enabling pre- and post-deployment checks natively on k8s. Keptn therefore extends your GitOps approach with the missing observability and orchestration needed for successful cloud native development.
More Related Content
Similar to Observability and Orchestration of your GitOps Deployments with Keptn (20)
More from Andreas Grabner (20)
Recently uploaded (20)
Observability and Orchestration of your GitOps Deployments with Keptn
- 1. Andreas Grabner DevOps Activist @ Dynatrace DevRel & Maintainer @ Keptn @grabnerandi, https://linkedin.at/grabnerandi @keptnProject, https://www.keptn.sh Observability and Orchestration of your Deployments Taming the operational complexity of GitOps with Keptn
- 2. KeptnV1: General Purpose Orchestration for Cloud and Non-Cloud-Native Tooling on K8s Keptn can connect to any of your tools through an open event standard to automate delivery and remediation sequences Started in 2019 and since 2022 we are proud to be a
- 3. But in the recent past, Cloud Native GitOps has shifted complexity from Dev to Ops Complex Dev & Continuous Integration (CI): Composition, Validation, Security Feature A Feature B Backend Svc Authentication Service Frontend Components Service A (v1) Service B (v2, v3) Service C (v1) Service D (v3,v4) Infra & Cloud (vY) Svc A (v1) Svc B (v2) Svc C (v1) Svc D (v3) Istio/Prom/ xxx X Monolithic World Cloud Native World Simpler Ops: Deploy, observe and operate a well-defined app Simpler Dev: Service-based Continuous Delivery (CD) Complex Ops: Composition, Validation, Security, Observability, Orchestration Svc A (v1) Svc B (v3) Svc C (v1) Svc D (v3) Istio/Prom/ xxx Y Well-defined Infra, e.g: VMWare, EC2 … with diff flavors (EKS, AKS, OpenShift …) Test Security Test, security, validation not part of GitOps Validate
- 4. This shift also brought us a lot of feedback for KeptnV1 Single Pod-Ready != Application Ready That’s why 90% Keptn Users adopted our SLO-based Validation Zero Integration It’s hard integrating new tools into EVERY pipeline. Especially in organizations with > 800 Global Dev Teams. Need a zero-integration approach Configuration as CRDs Many teams are adopting ArgoCD, Flux … non CRDs in external repository doesn’t work in a “GitOps world” OpenObservability The CNCF community has agreed on OpenTelemetry and Prometheus for observability data. Why store it in MongoDB? Application Awareness While we deploy individual workloads (=microservices) we need visibility into applications that are made up of one or many services
- 5. Solution: A K8s Operator* to Observe and Orchestrate App-aware Workload Lifecycle My-Application:2.0 ** Frontend-Svc:2.0 Backend-Svc:1.5 Storage-Svc:1.0 Post Pre Post Pre Post Pre Timespan & Result for each single deployment Pre-App-Deployment Post-App-Deployment Timespan Time & Result for whole app deployment Tasks: Dependency, Env Health, Certificates, Approval, ... Evaluations: SLOs, Error Budgets ... Timespan & result of each task / evaluation Tasks: Tests, Security Scans, Cleanups, Promote ... Evaluations: SLOs, User Experience, Adoption ... Timespan & result of each task / evaluation * K8s Operators can leverage K8s webhooks and extend K8s scheduler for pre- and post-deployment hooks ** K8s doesn’t yet have a standard application concept but Delivery SIG is working on it Observe: Metrics (DORA) & Traces Orchestrate: Pre-Deploy Orchestrate: Post-Deploy
- 6. Introducing Keptn Lifecycle Toolkit: App-aware Workload Lifecycle Observability and Orchestration kind: Deployment name: simplenode ... template: metadata: annotations: keptn.sh/workload: simplenode keptn.sh/version: 3.0.1 keptn.sh/pre-tasks: notify keptn.sh/pre-eval: check-error-budget keptn.sh/post-tasks: api-tests, notify keptn.sh/post-eval: evaluate-slo ... kind: KeptnApp name: simplenode-app spec: workloads: - name: simplenode-frontend-svc version: 3.0.1 - name: simplenode-backend-svc version: 2.0.1 pre-task: check-approval pre-eval: check-error-budget post-tasks: functests, notify post-eval: evaluate-slo Example from upcoming Live Demo: Single deployment with Keptn annotations, Keptn Tasks and optional Keptn App Step 1: Annotate your Deployment & StatefulSets! Step 2: Define Keptn Tasks and Evaluations! kind: KeptnTaskDefinition name: notify ... spec: function: | let text = Deno.env.get("SECURE_DATA"); let context = Deno.env.get("CONTEXT"); let resp = await fetch("https://hooks.slack.com/xxxx"); console.log("Sending slack message") Step 3: (optional) KeptnApp for app-awareness ! kind: KeptnEvaluation name: evaluate-slo Spec: source: prometheus objectives: - name: cpu capacity query: "sum(kube_node_status_capacity{resource='cpu’} " evaluationTarget: ">4"
- 7. Keptn Lifecycle Toolkit: App-aware Workload Lifecycle OpenTelemetry Trace Pre- App Post- App Application Post- WL 1 Pre- WL 1 Pre-App Pre-WL 1 Deploy 1 Post-WL 1 Post-App Pre-WL X Deploy X Post-WL X $ kubectl apply *.yaml Workload
- 8. Keptn Lifecycle Toolkit: App-aware Workload Lifecycle Prometheus Metrics (DORA Metrics)
- 9. Keptn Lifecycle Toolkit: Keptn CRDs visualized in ArgoCD
- 10. Live Demo
- 11. Keptn Lifecycle Toolkit: Observing and Orchestrating Lifecycle of K8s Deployments Your GitOps Your Deployment: image: mysvc:1.0 Your Observability tool Dev Staging Prod Visibility across all stages using observability standards promote promote deployment-traces deployment-traces notify notify notify on-pre: dependency Keptn-Task: (tooling for cluster) name: dependency function: exec.checkDBAvail() --- name: notify function: slack.send() --- name: validate_slo promql: errors<5% && rt<100ms Keptn-App: (app-scope checks) name: mybusinessapp workloads: mysvc,yoursvc,xsvc on-post: validate_slo,notify on-post-success: promote Extend your GitOps Day2Ops on-pre-deploy dependency on-pre-deploy dependency on-pre-deploy dependency approval sync on-post-deploy api_tests validate_slo on-post-deploy perf_tests security validate_slo on-post-deploy func_tests security validate_slo cleanup $ $ git add “keptn CRD files” $ git commit $ kubectl apply –f install_keptn.yaml Keptn Lifecycle Toolkit installed!
- 12. Keptn Lifecycle Toolkit tames the complexity that GitOps shifted towards Ops Cloud Native World Simpler Dev: Service-based Continuous Delivery (CD) Simpler Ops: Deploy, observe and operate a well-defined app Service A (v1) Service B (v2, v3) Service C (v1) Service D (v3,v4) Infra & Cloud (vY) Svc A (v1) Svc B (v2) Svc C (v1) Svc D (v3) Istio/Prom/ xxx X Svc A (v1) Svc B (v3) Svc C (v1) Svc D (v3) Istio/Prom/ xxx Y -native on any flavor and tooling Extends GitOps with Test, security, validation Declarative in Git
- 13. Andreas Grabner DevOps Activist @ Dynatrace DevRel & Maintainer @ Keptn @grabnerandi, https://linkedin.at/grabnerandi @keptnProject, https://www.keptn.sh Observability and Orchestration of your Deployments Taming the operational complexity of GitOps with Keptn
Editor's Notes
- #2: Find more: https://www.keptn.sh/ https://lifecycle.keptn.sh/ https://github.com/keptn-sandbox/klt-on-k3s-with-argocd/ https://twitter.com/keptnProject https://slack.keptn.sh
- #15: Find more: https://www.keptn.sh/ https://lifecycle.keptn.sh/ https://github.com/keptn-sandbox/klt-on-k3s-with-argocd/ https://twitter.com/keptnProject https://slack.keptn.sh