David Kirk_Local Microservice Development in EKS.pdf
0 likes•24 views
This document describes a hybrid approach to local microservice development against an EKS cluster. The key points are: - Developing locally but against an EKS cluster is challenging due to differences in environments. - The proposed solution is to run all services in the EKS cluster except the one being developed locally, replacing it with a proxy that redirects traffic to the local machine. - This allows using local development tools while still interacting with the full production environment in the cluster. It can be done through exclusion, scaling the local service to 0, or interception using an nginx proxy to redirect traffic. - A case study is provided of how this approach is implemented at NTWRK using Tail
![Case Study: Using this at NTWRK
- We have “personal environments”
- Single centralized helm repo defines everything that’s deployed
- `make build-dev` builds all containers
- `make deploy-dev` takes helm and deploys it into personal namespace
- Intercept & Exclude at deploy time
- `make deploy-dev-tailscale INTERCEPT=[chart_name]`
- `make deploy-dev-tailscale EXCLUDE=[chart_name]`
- Can intercept & exclude in one deployment](https://image.slidesharecdn.com/davidkirklocalmicroservicedevelopmentineks-230614011628-bc0c7658/85/David-Kirk_Local-Microservice-Development-in-EKS-pdf-15-320.jpg)
David Kirk_Local Microservice Development in EKS.pdf
- 1. Local Microservice Development in EKS David Kirk
- 2. About Me - 10 years as a Dev, 8 of which in DevOps - Staff Engineer at NTWRK - AWS & Kubernetes have been main focuses for 6 years - [email protected] or [email protected]
- 3. What’re we talkin’ about? - Describe the problem - how do we develop against EKS? - Attempted & Potential Solutions - Hybrid Environments - the solution we’re using at NTWRK - Case-study of how it’s implemented at NTWRK - Suggestions of quick wins w/ this technique
- 4. I’ve done a talk like this before - I’ve learned lessons since then about how to do this better - That talk proposed something that took 6+ months to fully build - The goal here is to describe the solution and give practical wins you can bring home
- 5. The Problem - Developing against EKS - Most development happens locally - Tools are designed with this in mind - Debuggers, profilers, build tools, etc - Developers are used to this mental framework - EKS is not running locally - But it’s where production is running - How do we bridge this gap? - Ideal solution is one that allows us to use EKS and local tools
- 6. Why is this worth solving? - Reduce cognitive overhead - There’s a barrier to entry here - Worth it in the long run - fewer variations to consider - Minimize internal tooling & configuration complexity - The fewer environmental variations, the less work to align them - Devs get Ops experience - Hey look, it’s DevOps! - If devs gain k8s experience while dev-ing, they know better how to work with k8s in prod
- 7. Summary of Goals - Easy to use & implement - Useful - Devs can tie their local machine into EKS - Ideally vice-versa as well
- 8. High Level Solution - Redirect Traffic to Local - This is the basis for all proposed solutions - Run the thing you’re developing locally - Run everything else somewhere else - Send traffic that would go to the thing you’re developing to localhost Everything Else Runs Here Your Laptop Service A Service B Dummy Service C Actual Service C - Dummy service replaces yours in the cluster - Receives traffic like normal - Routes all traffic to your localhost Network Traffic
- 9. Potential Solution: docker-compose - Pro: Simple to configure - It’s entry-level deployment yaml, IMO - Very readable, and in one file - Con: It’s not kubernetes - Duplicating configuration - effort to maintain - Devs aren’t exposed to kubernetes - Run everything not being actively developed in docker-compose - Substitute the thing you’re developing w/ an nginx container - Forward all traffic it gets to `host.docker.internal` - Run the thing you’re developing locally
- 10. Potential Solution: Minikube - Pro: It’s kubernetes! - You get some kubernetes experience - But also you don’t get multi-node kubernetes - Con: Configuration complexity - All of your charts/definitions need to support both minikube & EKS - That’s gonna be some complex charts - Run everything not being actively developed in Minikube - Substitute the thing you’re developing w/ an nginx container - Redirect its traffic to `host.minikube.internal` - Run the thing you’re developing locally
- 11. Running Everything Locally Has Limits - Things will get slow - You’re better at creating stuff for your CPU to do than Intel & Apple are at making CPUs that can do stuff fast - Even beefy machines will slow down - This slows down everything else around development - You can deploy subsets of services - Again introducing configuration complexity - Hard to maintain, prone to issues
- 12. Proposed Solution - Hybrid EKS Development - Run everything in EKS except what you’re developing - Substitute the thing you’re developing w/ an nginx container - Forward all traffic it gets to your IP address - Run the thing you’re developing locally - Use a VPN to access the cluster - Reach into cluster w/ it - use k8s DNS for your cluster’s CIDR - Reach out of the cluster w/ it - use VPN IP addresses to talk to dev machines
- 13. Exclusion & Interception - Two modes of network traffic interaction - These cover all of the use cases we’ve seen internally - Exclusion - Scale the deployment you’re developing to 0 - Run it locally, and use k8s DNS through the VPN to interact w/ cluster - Used if it doesn’t receive requests - e.g. Kafka Consumers & Cronjobs - Interception - Replace the pod you’re developing w/ nginx - Redirect that traffic to your laptop’s VPN IP address - Run it locally, again use k8s DNS through the VPN
- 14. Exclusion & Interception Examples EKS Cluster Your Laptop Kafka Broker Service A Service B nginx Service C Actual Service C Network Traffic over VPN Consumer D - Exclude D - It doesn’t receive requests - it consumes from Kafka - Intercept C - Capture requests & send to local machine
- 15. Case Study: Using this at NTWRK - We have “personal environments” - Single centralized helm repo defines everything that’s deployed - `make build-dev` builds all containers - `make deploy-dev` takes helm and deploys it into personal namespace - Intercept & Exclude at deploy time - `make deploy-dev-tailscale INTERCEPT=[chart_name]` - `make deploy-dev-tailscale EXCLUDE=[chart_name]` - Can intercept & exclude in one deployment
- 16. Interception Details - How do we route that traffic to the VPN IP address? - We use Tailscale as our VPN - All examples will reference those specifics, but concepts can be tweaked based upon your needs - Replace containers in intercepted pod - nginx routes traffic to VPN IP address - set at deploy time - Tailscale side car gives nginx access to VPN over pod’s localhost
- 17. Interception Details: Actual Code Edition Use this template if intercepting Default deployment configuration
- 18. Interception Details: Actual Code Edition pt 2 Internal Traffic External Traffic Send it to local machine’s VPN IP
- 19. Interception Details: Actual Code Edition pt 3 nginx container nginx configuration Tailscale sidecar grants VPN access
- 20. Potential Quick Wins - You don’t need personal environments - Run command locally to intercept static environment - Could be staging or UAT - Be careful about collisions - Team-specific long-lived environments - Less concern about collisions