The road to continuous deployment (DomCode September 2016)
- 1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL ROOK
- 2. ▸ Java, PHP & Scala developer ▸ Consultant, trainer, speaker ▸ make.io ▸ Dutch Web Alliance ▸ Maintainer of Phing ▸ @michieltcs
- 3. THIS TALK ▸ Background ▸ The approach ▸ Process / standards ▸ Build pipelines ▸ Results & lessons learned
- 5. THE SYSTEM - SAN DIEGO ▸ ... or the Big Ball Of Mud ▸ Large legacy monolith ▸ Generates significant income ▸ Slow & complex ▸ Technical debt
- 6. SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR
- 7. THE SYSTEM - SAN DIEGO ▸ Infrequent, manual releases ▸ Fragile tests ▸ Low velocity ▸ Frequent outages / bugs / issues ▸ Frustrated team ▸ Low confidence modifying existing code
- 8. GOALS ▸ Reduce issues ▸ Reduce cycle time ▸ Increase productivity ▸ Increase motivation
- 10. APPROACH ▸ Services per domain object (job, jobseeker, ...) ▸ Strangler pattern ▸ Proxy to switch between old/new ▸ Migrate individual pages
- 12. APPROACH ▸ Services behind load balancers ▸ Access legacy db’s ▸ Continuous deployment ▸ Docker containers ▸ Frontends are services
- 13. SAN DIEGO ELASTIC SEARCHLEGACY DB JOB SERVICE RMQ ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVBINTERMEDIAI
- 14. PROCESS
- 15. STARTING OFF ▸ Scrum, 1 week sprints ▸ TDD / BDD ▸ Definition of Done ▸ Team mindset / experience ▸ Focus on value ▸ Replace old features with new (legacy becomes obsolete)
- 17. DEV BUILD / TEST CONTINUOUS INTEGRATION
- 18. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY
- 19. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DEPLOYMENT
- 20. WHY CONTINUOUS DEPLOYMENT ▸ Small steps ▸ Early feedback ▸ Less overhead ▸ Reduce cycle time ▸ Reduce risk
- 21. ONLY COMMIT TO MASTER
- 22. NO BRANCHES
- 24. PAIR PROGRAMMING
- 25. BOY SCOUT RULE
- 26. QUALITY GATES
- 28. FEATURE TOGGLES, A/B TESTS
- 29. DASHBOARDS
- 30. BUILD PIPELINE
- 32. EVERY COMMIT GOES TO PRODUCTION
- 33. DEFENSE IN DEPTH UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS
- 34. DEFENSE IN DEPTH UNIT TESTS INTEGRATION TESTS ACCEPTANCE UI TESTS public function testJobCannotBeFound() { $jobRepository = $this->prophesize(JobRepository::class); $jobRepository->getById(EXPECTED_JOB_ID) ->shouldBeCalled() ->willReturn(false); $jobService = new JobService($jobRepository->reveal()); $this->assertFalse($jobService->getById(EXPECTED_JOB_ID)); }
- 35. DEFENSE IN DEPTH UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS public function testFindJob() { $expectedJob = $this->loadFixture('active_job.yml'); $actualJob = $this->repository->getById($expectedJob->getId()); self::assertInstanceOf(Job::class, $actualJob); self::assertEquals($expectedJob->getId(), $actualJob->getId()); }
- 36. DEFENSE IN DEPTH UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTSScenario: Link to related job Given a job exists And there are related jobs available When that job is viewed Then a list of related jobs is shown And each related job links to the detail page of the related job
- 37. DEFENSE IN DEPTH UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS
- 38. DEFENSE IN DEPTH MANUAL TESTING? UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI
- 39. CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS SMOKE TESTS Cost Speed Exploratory testing Monitoring
- 40. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER
- 41. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull
- 42. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run
- 43. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15
- 44. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri: url: http://localhost:8080/_health status_code: 200 timeout: 30
- 45. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded
- 46. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded
- 47. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm
- 48. PIPELINE AS CODE node { stage 'Run tests' sh "phpunit" sh "behat" stage 'Build docker image' sh "phing build" sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" stage 'Deploy acceptance' sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i acc deploy.yml" stage 'Deploy production' sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i prod deploy.yml" }
- 49. BUILD PIPELINE
- 50. FEEDBACK!
- 51. RESULTS
- 52. RESULTS ▸ Total build time per service < 10 minutes ▸ Significantly improved page load times ▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc) ▸ Increased confidence and velocity ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ More fun
- 53. LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸ Overhead of weekly sprint; requires discipline ▸ Docker orchestration ▸ Issues with traffic between Amazon <-> on-premise datacenter ▸ Javascript testing
- 54. LESSONS LEARNED ▸ Experience with new tech ▸ Stability of build pipelines ▸ Management/leadership buy-in ▸ Business alignment ▸ Not enough focus on replacing legacy application