Microservices Delivery Platform. Tips & Tricks
- 1. Microservices Delivery Platform. Tips and Tricks
- 2. Solution Architect @ EPAM Andriy Trubitsyn 17 years of software development; Solution architect @ microservice accelerator in EPAM Speaker @ XP Days Ukraine Speaker @ meetups and events
- 3. Microservices delivery platform = Microservices platform + CI/CD +
- 4. What Microservices Platform is ? https://dzone.com/articles/deploying-microservices-spring-cloud-vs-kubernetes
- 5. One Microservices Platform 7 ECS Cluster ECS Cluster Amazon API Gateway Amazon RDS Amazon DynamoDB Amazon ElastiCache Amazon ES Amazon S3 AWS CloudFormation Amazon CloudWatch Legacy DB Amazon SQS Amazon SNS AWS Lambda for AAA Sumologic OpsGenie Parameter Store ๏ง Mixed AWS Cloud & OnPrem ๏ง Platforms relies on Netflix OSS ๏ง Dozens ECS clusters ๏ง Hundreds of microservices ๏ง Heterogenous ecosystem: Java, .Net, NodeJS ๏ง Synchronous and asynchronous microservices communication
- 6. 8
- 8. CI/CD Pipelines Code Build DevQA INT LIVE ECR registry AWS CloudFormation Image Build CI/CDCI OWASP ZAP Code Build
- 9. CI/CD Pipelines shared library & templates CI/CD pipeline templateShared libraries โข Pipeline Building blocks โข API for tools โข Agents injection โข Default CI/CD pipelines โข Sample book
- 10. Lessons Learnt 1 C I / C D P I P E L I N E S A R E V E RY S I M I L A R 5 2 3 4 M A K E S S E N S E TO D I F F C I A N D C D P I P E L I N E S T E M P L AT E P I P E L I N E I S H E L P F U L T E M P L AT E P I P E L I N E I S TO B E C U S TO M I Z A B L E T E M P L AT E U P DAT E S D I S T R I B U T I O N I S A D I S A S T E R
- 12. Microservices generator โข Integrates with Service Discovery Eureka โข Sends tracing meta information to a tracing tool Zipkin. โข Sends metrics to a monitoring tool DataDog โข Sends logs to SumoLogic โข Cloudformation script includes ECS Service and TaskDefinition configuration AWS CloudFormation A buildable java project with CI/CD pipeline which
- 13. Microservices generator โข Default scaling strategy by CPU โข Can be build and deployed to all environments DEV, INT, LIVE โข Configuration for Cost Optimization function โข Imported CI/CD pipeline in Jenkins with configured hooks and integrated with SonarQube and Jfrog artifactory AWS CloudFormation A buildable java project with CI/CD pipeline which
- 14. Lessons Learnt 1 G E N E R ATO R S S AV E I N C R E D I B L E A M O U N T O F E F F O R T S 2 3 4 O N E G E N E R ATO R N E E D S P E R E A C H L A N G UA G E G E N E R ATO R S B R I N G V E R S I O N I M P L I C I T C O N F L I C T S V I R T UA L T E A M TO O W N G E N E R ATO R S
- 15. Dedicated repo for DTO objects and Contracts DTO
- 16. Dedicated repo for DTO objects
- 17. Dedicated Artifactory Section for Contracts Catalog microservice Order microservice Search microservice
- 18. Lessons Learnt 1 S I N G L E S O U R C E O F T R U T H F O R A L L M I C R O S E R V I C E S 2 3 4 A R C H I T E C T S G O V E R N A C T I V I T Y V I A N F R S A R C H I T E C T S O W N S R E P O S N E E D S A C H A N N E L O F S C H E M A U P DAT E N O T I F I C AT I O N S
- 20. Naming Strategy my.little.pony Entire System Domain Sub Domain Microservice Microservice Microservice
- 21. Versioning Strategy Microservice Order Endpoint Microservice OrderExtendedSchema Non Backward Compatible Endpoint
- 22. Lessons Learnt 1 S I M P L E B U T I M P O R TA N T O T H E R W I S E TO TA L M E S S 5 2 3 4 A R C H I T E C T S T R A C K A L L N E W R E P O & M I C R O S E R V I C E S A LWAY S L AT E S T I S S I M P L E B U T M U C H D U P L I C AT I O N S T I L L U S E V E R S I O N S F O R S H A R E D T E M P L AT E S / L I B S H A R D TO I N V E N T N E W N A M E S
- 23. 24/7 support
- 25. OpsGenie 27
- 26. Lessons Learnt 1 H A R D TO E S TA B L I S H T H E P R O C E S S 2 3 C O D E Q UA L I T Y B E C O M E S M U C H B E T T E R T E A M V E LO C I T Y S O M E W H AT D E C R E A S E D 4 S O M E P EO P L E R E F U S E W O R K S H I F T S
- 28. Infrastructure Cost Function microservice Amazon S3 Amazon RDS Amazon ElastiCache Tagging in CF: โข Domain โข SubDomain โข Team
- 29. Lessons Learnt 1 TA G YO U R R E S O U R C E S 2 3 T R A C K I N F R A C O S T O F A N Y C H A N G E S C L E A N R E S O U R C E S A U TO M AT I C A L LY 4 D E V E LO P E R S D O E S N O T U S E R E S O U R C E S E F F I C I E N T LY
- 31. Dedicated Platform Team https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/ Team Team Team โฆ Domain/ART Platform Team SA SM PO On Call Support
- 32. Lessons Learnt 1 T H E R E I S W O R K N O T F O R C R O S S F U N C T I O N A L T E A M S 2 3 D I R E C T I N VO LV E M E N T W O R K S C R O S S F U N C T I O N A L T E A M S A R E N O T F U L LY I N D E P E N D E N T
- 34. Architecture Documentation โข Entire Domain C&C diagram โข Microservice C&C diagrams โข Microservice interfaces & batch jobs sequence diagrams
- 35. Architecture Documentation โข Microservice cost estimations โข Dedicated project with Swagger documentation โข NFRs check lists
- 36. Lessons Learnt 1 2 3 D O C U M E N TAT I O N T E M P L AT E S M A N DATO RY T E X T I S N O T U S E F U L , D I A G R A M S I S M O R E H E L P F U L D I F F E R E N T D O C U M E N TAT I O N F O R D I F F E R E N T P U R P O S E S 4 N F R S C H E C K L I S T A R E R EQ U I R E D
- 38. Dedicated Test Projects โข Dedicated e2e tests ( nightly run ) - Positive & negative scenarios - Cloud & OnPrem โข Dedicated Performance tests ( by demand ) - Scaling - Resource usage - Performance Microservices In Cloud On Prem Apps On Prem DBs together
- 39. Lessons Learnt 1 2 3 4 P E R F O R M A N C E T E S T S A R E Q U I T E E X PA N S I V E D E D I C AT E D A Q A T E A M M AY B E A N A N S W E R A L L K I N D S O F T E S T S D O N โ T F I T O N E M I C R O S E R V I C E P E R F O R M A N C E T E S T S A R E Q U I T E T I M E C O N S U M I N G
- 40. Conclusion
Editor's Notes
- #7: Briefly discuss about all microservices concerns shown on the slide Tell about attempts to build a microservices reference architecture and platforms then
- #8: 1. Describe this case study microservices platform
- #9: Letโs start talking about practices. What kind of practices I am talking about some things which can not be found in internet
- #10: Many microservices with similar pipelines PoC showed we need a templates
- #11: Jenkins, groovy Describe steps Continuous delivery Three AWS environments Each commit to master goes to LIVE in ~25 minutes Dozens cross-functional teams
- #12: Library and utils to build pipelines Agents to collect logs and metrics Stop deployment if this is a branch
- #19: JS consumes JSON JSON๏ JAVA POJO ๏ JFROG โ consumed by JAVA
- #22: Hard to find a miroservice at GitHub โOrder managementโ is taken
- #23: 1. Domain driven design
- #24: Can add changes backward compatible Hard to name Always last version Simple to support changes Code duplication
- #26: AWS do 24/7
- #27: Talk to teams Schedule shifts ( with reward ) Create escalation rules
- #31: Why we need to track costs (experiments, tests, inefficient resource usage, ELBs) Why we need to tag resources Untagged to be deleted in 5 minutes AWS gives infrastructure grouping/tag grouping. Because organizational structure does not reflect AWS resources. Some teams use AWS resources in efficient way ( too much cpu, memory, disk. The same for all three envs โ serverless RDS, no HA in Dev, etc)
- #33: Even with cross-functional teams we have the classic dev & ops problem.
- #34: SAFe, Spotify, Cross-functional teams โ BE+AQA, FE+BE+AQA, (+DBA) , (+DevOps), BIs Need of a platform team for user and roles creation, platform deployment, CI/CD and tools support Problems with the dedicated platform team OnCall support from the platform team
- #36: We have MRA, PoCs.
- #37: Entire domain C&C diagram shows connections between microservices, gives a high level vision (no details) C&C diagrams to understand how many new microservices we need, what microservices to change, integration points, work spread Microservice interface description Sequence diagrams for developers it means they understand steps to implement business logic, sync & async intercommunication
- #38: Cost is to permanently grow. We need to estimate cost before microservice release to plan our infrastructure spending's NFRs check list to ensure a microservice is built according to microservice reference architecture It is useful to have a dedicated project with all swagger documents ( refer to sandox) for external B2B integration
- #40: Tell about different type of tests and why we need them Describe envs to involve, when each of them is to start E2E involve Cloud & Prem TDD & BDD Developers write tests Each commit takes 20 minutes to release Priority queues.
- #41: Problems: 1. Time consuming vs 20 minutes release cycle what to do?
- #43: Microservices delivery platform is not a simple thing, I will have to build it as your scale growths. But now you know my mistakes and can your life a little easier But definitely during this year you and I will have to overcome many challenges at our way So see you next time and we will share our finding and become wiser.