Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, the Ugly
3 likes2,386 views
CloudNativeCon North America 2017, Austin (Texas, USA): Talk by Josef Adersberger (@adersberger, CTO at QAware) Abstract: Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like: - What technical constraints of Kubernetes can be obstacles for applications and how to tackle these? - How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security? - How to industrialize and govern the migration process? - How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
Ad
More Related Content
What's hot (20)
Similar to Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, the Ugly (20)
![[Capitole du Libre] #serverless - mettez-le en oeuvre dans votre entreprise...](https://cdn.slidesharecdn.com/ss_thumbnails/capitoledulibre2018serverless-181120135943-thumbnail.jpg?width=560&fit=bounds)
Ad
More from QAware GmbH (20)
Ad
Recently uploaded (20)
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, the Ugly
- 1. Migrating Hundreds of Legacy Applications to Josef Adersberger, CTO, QAware @adersberger }THE GOOD, THE BAD, THE UGLY proud CNCF member
- 3. CIOLet’s bring all our web applications into the cloud! I’ve already canceled our current data center contract.
- 4. Chief Architect Great goal! But we’ll need our time.
- 5. Excellent! You’ve got one year.
- 6. CIOMy priorities: (1) Security level (2) Time (3) OpEx savings (4) Migration costs
- 9. AND WE HAD … IMPEDIMENTS
- 10. THE GOOD
- 11. 11 Visibility
- 13. PACKAGED AND DISTRIBUTED AS CONTAINERS BUILD AND COMPOSED AS MICROSERVICES DYNAMICALLY EXECUTED IN THE CLOUD CLOUD NATIVE APPLICATIONS 3KEYPRINCIPLES
- 14. OMG - no greenfield approach!? Hundreds of legacy systems!?
- 15. Questionnaire: Typical Questions And Their Motivation 1. Technology stack (e.g. OS, appserver, jvm) 2. Required resources (memory, CPU cores) 3. Writes to storage (local/remote storage, write mode, data volume) 4. Special requirements (native libs, special hardware) 5. Inbound and outbound protocols (protocol stack, TLS, multicast, dynamic ports) 6. Ability to execute (regression/load tests, business owner, dev knowhow, release cycle, end of life) 7. Client authentication (e.g. SSO, login, certificates) What images to provide? How many applications will be hard/ inefficient to schedule (>3 GB RAM, > 2 cores)? What storage solutions to provide? What applications will be hard to impossible to be containerized? Are there any non cloud-friendly protocols? How risky is the migration? Is the migration maybe not needed? What IAM and security mechanisms have to be ported to the cloud?
- 16. 16 QAVALIDATOR SONARQUBE EAM TOOL QUESTIONNNAIRES JIRA XLS STATIC ANALYSIS IBM MIGRATION TOOL … MIGRATION TASKS BASIC TOUR-DE-MIGRATION SYSTEM PROPERTIES dashboard for information radiator freestyle analysis with Tableau MIGRATION DATABASE CLOUDALYZER
- 17. 17 Emergent design of software landscapes
- 18. OLD APPLICATIONS (~200) MORE MODERN APPLICATIONS (~200) ALL WEB APPLICATIONS (~400) Re-architect to run on k8s on AWS lift & shift VMs to AWS EC2
- 19. APPLICATION HTTPD WEB LAYER ALMIGHTY LEGACY FRAMEWORK J2EE 1.4 APPSERVER JVM 1.6 DB MQ HOST BATCH FS CLIENTS TLS 1.0+ mostly non-TLS; TCP-Binary, WS, REST, C:D, LDAP Corba, SMTP, FTP, NAS, … RACF ESB ONPREM DATA CENTER ONPREM DATA CENTER DB MQ HOST BATCH FS RACF ESB KUBERNETES / OPENSHIFT (CAAS) DOCKER JVM 8 ALMIGHTY FRAMEWORK NG INNER APPLICATIONS AWS WEB LAYER AWS CLIENTS TLS 1.2 all TLS 1.2 JEE 7 APPSERVER API GATEWAY OUTER APPLICATIONS all 2-way TLS 1.2 & OIDC identity token
- 20. APPLICATION HTTPD WEB LAYER ALMIGHTY LEGACY FRAMEWORK J2EE 1.4 APPSERVER JVM 1.6 DB MQ HOST BATCH FS CLIENTS TLS 1.0+ mostly non-TLS; TCP-Binary, WS, REST, C:D, LDAP Corba, SMTP, FTP, NAS, … RACF ESB ONPREM DATA CENTER ONPREM DATA CENTER DB MQ HOST BATCH FS RACF ESB KUBERNETES / OPENSHIFT (CAAS) DOCKER JVM 8 ALMIGHTY FRAMEWORK NG INNER APPLICATIONS AWS WEB LAYER AWS CLIENTS TLS 1.2 all TLS 1.2 JEE 7 APPSERVER API GATEWAY OUTER APPLICATIONS all 2-way TLS 1.2 & OIDC identity token
- 21. 2 Can we evolve existing enterprise applications into the cloud with reasonable effort? Containerization 12-Factor App Principles Microservices Cloud-native Apps Monolithic Deployment Traditional Infrastructure CLOUD FRIENDLY CLOUD NATIVECLOUD ALIEN
- 22. 2 Can we evolve existing enterprise applications into the cloud with reasonable effort? Containerization 12-Factor App Principles Microservices Cloud-native Apps Monolithic Deployment Traditional Infrastructure Sweetspot time and value (security, opex) Put the monolith into a container … and enhance the application according the 12 factors CLOUD FRIENDLY
- 23. Edge Service to the Rescue
- 25. MONOLITH EDGE SERVICE CLIENTS TOKEN PROVIDER TLS 1.2 + diverse user contexts (SAML, Cookie, Header, Certificate, User/Pwd, …) Change user context to OIDC identity token IAM SYSTEMS API GATEWAY TLS 1.2 + identity token TLS 1.2 + identity token Redirect to (S)SO if user context not set / valid but required user context > 10
- 26. Edge Service <<pod>> Edge Service <<container>> Edge Service Chassis <<spring boot app>> Spring Framework Spring Boot Netflix Zuul / Spring Cloud Zuul Edge Service Filters Redis Edge Library Token Service Token Cache SPI Token SPIToken APIToken Handling Filter <<pre>> Static Content Filter <<routing>> Security Filter <<pre>> Auth Plugins Token Provider Web Layer API Gateway
- 27. Sidecars to the Rescue
- 28. Container Patterns Applied • Log extraction / re-formatting (fluentd) • Scheduling (Quartz) Sidecar: Enhance container behaviour Ambassador: Proxy communication Adapter: Provide standardized interface • Configuration (ConfigMaps & Secrets to files) • TLS tunnel (Stunnel, ghosttunnel) • Circuit Breaking (linkerd) • Request monitoring (linkerd) Pod Application Container Pattern Container Other Container “Design patterns for container-based distributed systems”. Brendan Burns, David Oppenheimer. 2016
- 29. Kubernets Constraints Initially we thought we’ll run into k8s restrictions on our infrastructure like: ‣ No support for multicast ‣ No RWX PVC available We did. But cutting these application requirements lead to a better architecture in each and every case.
- 30. THE BAD
- 31. STATE
- 32. Databases ONPREM DATA CENTER AWS DATABASES MONOLITH Databases stay in onprem data center • Advantages: • onprem/cloud version of the application can run in parallel • privacy • Disadvantages: • latency (no real problem) Activate TLS for all database connections (low effort on application-side but separate project on the database side)
- 33. Files File persistence is very restricted to keep persistent state completely out of AWS (privacy). ‣ No file writes allowed into container ‣ No RWX PVC available ‣ Files with application data written to PVC must be deleted after 15mins ‣ No NAS mounts from onprem data centers into containers allowed Migration tasks for affected applications ‣ Store files as BLOB in database or use FTPS
- 34. Session State 90% OF THE APPLICATIONS HAVE SESSION STATE 100% OF THE APPLICATIONS HAVE MORE THAN 1 INSTANCE Session Stickyness: not within the cloud! Session Persistence: ‣ Within existing application database: performance impact too high for most ‣ Redis: no transport encryption out-of-the-box and separate infrastructure required Session Synchronization: ‣ Application server: nope, no dynamic peer lookup within k8s ‣ In-memory data grid: Hazelcast. Nope. $$$ for TLS. ‣ In-memory data grid: Apache Ignite. Done. ‣ Embedded within application or standalone in a separate container. ‣ Little bit cumbersome but working k8s peer lookup ‣ Runs into JIT bug on IBM JVM (some methods have to be excluded from JIT)
- 38. Metrics Events / LogsTraces • High effort to instrument for valuable insights •Scalability unclear for hundreds of applications •Applications have no time to run their own Prometheus instance •Scalability unclear for hundreds of applications (Jaeger & ZipKin) •Applications have no time to run their own instance •Scalability unclear (a lot of events lost) •Applications have no time to run their own EFK instance •Non-standardized log format requires custom log rewrite adapter but no fluentd DaemonSet Diagnosability
- 41. We Came Far BACKEND MONOLITH EDGE SERVICE TLS 1.2 + user context API GATEWAY Two-way TLS 1.2 + identity token Two-way TLS 1.2 + identity token TOKEN CHECK TLS 1.2 + user context secrets injected at container startup security filters included CLIENTS CLIENTCERT ACL EGRESS RULES
- 42. The Bad: Certificate Management VAR 1: CLOUD NATIVE STYLE CERTIFICATE MANAGEMENT (SPIFFE-BASED, AT SERVICE MESH OR APPLICATION LEVEL) VAR 2: REPLACE BY POLICIES (AT NETWORKING LEVEL) e.g. e.g. SPIRE
- 43. THE UGLY String hostRequest = new HostRequest().hostusPokus(message);
- 44. 44 CLOUD ENABLING CLOUD ALIENS
- 45. TOXIC TECH
- 46. The Almighty Legacy Framework • “worry-free package framework” from the early 2000s with about 500kLOC and 0% test coverage • Migration tasks • from J2EE 1.4 to JEE 7 and Java 6 to 8 • add identity token check and token relay • modify session handling (synchronization) • modify logging (to STDOUT) • modify configuration (overwrite from ConfigMap) • enforce TLS 1.2 • place circuit breakers • predefined liveness and readiness probes • Strategies: • the hard way: migrate manually and increase coverage • decorate with ambassadors, sidekicks and adapters • do not migrate parts and replace that API within the applications APPLICATION ALMIGHTY LEGACY FRAMEWORK J2EE 1.4 APPSERVER JVM 1.6
- 47. 47 ~ 100 systems live on target by end of this year (after 8mo) ~ 200 systems live on target by end of first quarter 2018 other ~200 systems migrated by end of first quarter 2018 via virtual lift & shift. They will be migrated onto Kubernetes afterwards
- 48. That’s what we’ve learned from migrating hundreds of J2EE legacy apps onto Kubernetes?
- 49. No stupid “as-is into containers” approach!
- 50. Getting as close as possible to cloud friendly application principles. Increasing the security level by an order of magnitude!
- 51. Q -> A proud member of the CNCF
- 52. TWITTER.COM/QAWARE - SLIDESHARE.NET/QAWARE Thank you! [email protected] @adersberger
- 53. BONUS SLIDES
- 55. ARCHITECTURE TEAM (~2 FTE) SUPPORT TEAM (~10 FTE) DOZENS OF MIGRATION PROJECTS RUNNING IN PARALLEL (UP TO ~80) ‣ Training sessions ‣ Support sessions ‣ Guidance (cookbook, sample application) ‣ Development environment (SEU-as-Code) ‣ Standard images ‣ Automated Refactorings (RefactoBot) ‣ Pre-migrated frameworks ‣ Solutions: Edge service, ambassadors INDUSTRIALIZATION TEAM (~6 FTE) ‣ Application blueprint (target architecture, migration rules) ‣ Migration database
- 56. NO SPEAK CLOUD
- 57. DMS System APPLICATION DMS Proprietary (binary) TCP-level on fixed ports Non-encrypted Multiple target servers DMS ADAPTER Hard-coded DNS names as reference to other servers in response APPLICATION DMS ADAPTER STUNNEL STUNNEL DMS ONPREM KUBERNETES ON AWS POD STUNNEL STUNNEL PODS SVC.HOST1 SVC.HOST2 SVC.HOST3 lbl:host1 lbl:host2 lbl:host3 ONPREM dnshostname1.com.myapp.svc.cluster.local dnshostname1.com
- 59. IDE (ECLIPSE, INTELLIJ) ARTIFACTORY Kubernetes Cluster RUNNING TESTED APPLICATIONS INFORMATION RADIATOR GITHUB ENTERPRISE ACCOUNTABILITY PLUGIN JENKINS JARs Docker image Docker image deploy Trigger Code Code Status