Perfomance tuning on Go 2.0
Download as PPTX, PDF•0 likes•2,635 views
This document discusses performance tuning of a continuous integration/delivery pipeline system built using Go and related tools. It describes problems the system was having with slow page loads, long job queues, rescheduling jobs, and timeouts. Profiling revealed thread blocking and database connection issues as culprits. Solutions introduced included caching, query tuning, locking optimizations, and JRuby/Rails configuration tweaks to reduce memory usage and lock contention. The results were a system with fast response times and high concurrency.
![“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,building]”Pipeline-nameStage detailStage detailKey-identifier](https://image.slidesharecdn.com/perfomancetuningongo2-0-xconfpune-101019043428-phpapp01/85/Perfomance-tuning-on-Go-2-0-62-320.jpg)
![“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,passed]”Pipeline-nameStage detailStage detailKey-identifier“pipelineDashboardFragment|dev|1[compile,1,passed][test,1,building]”](https://image.slidesharecdn.com/perfomancetuningongo2-0-xconfpune-101019043428-phpapp01/85/Perfomance-tuning-on-Go-2-0-63-320.jpg)
Perfomance tuning on Go 2.0
- 1. Performance Tuning on Go 2.0http://www.thoughtworks-studios.com/go/Yogi Kulkarni
- 2. GoContinuous integration Release management
- 4. PipelineStage-2Unit TestStage-3Acceptance TestStage-1CompileRun Unit Test Suite - 1Accetance Test Suite - 1CompileJobRun Unit Test Suite - 2Acceptance Test Suite – 2Run Unit Test Suite - 3
- 8. The Problem
- 10. Jobs taking long to get assigned to agents
- 12. Artifact uploads timing out
- 14. Lots of concurrent activity
- 15. What was happening under the hood?Profiler revealed the culprits
- 16. Thread blocks
- 17. Threads blocked on the database connection pool
- 18. So we introduced asynchronicity
- 19. Big mistake!
- 20. Find the root cause
- 24. ObjectiveNo request should take > 100 msMost should be under 50 ms
- 25. Setup & Tools
- 27. “Performance Server”Dell 620 laptop2 Cores, 2GB RAM
- 28. Large Dataset5GB merged databaseLarge config file with 100 pipelinesReal-world data
- 29. Extreme load70 agentsEach Job generating lots of console outputEach Job uploading 10 MB artifactsApache Bench / Httperf
- 30. Automation for fast feedbackCapistrano scriptFeedback cycle down from hours to 10 min
- 31. Caching
- 32. Single application-wide cacheDomain objectsDTOsRails view fragments
- 35. Transaction rollbacks == Incorrect cached dataIgnore cache puts in transactionsInvalidate caches on transaction commit only
- 36. Spring to the rescue
- 37. Fast deep-cloning of cached objectsJava Deep Cloning Library: http://robust-it.co.uk/clone/index.php
- 38. Locks
- 39. Be suspicious of synchronized methods
- 40. Lock on interned stringsFine granularityProfiler supportAlso used as cache keys
- 42. Use ReadWriteLocks for highly contended code
- 43. Use “volatile” fields for lockless reads
- 44. Slow
- 45. Not synchronized
- 46. Acquire locks before starting transactionsTo enforce aggregate invariants
- 47. Thread 1Stage(Building)Job – 1(Completed)Stage(Building)Stage(Building)Job – 2(Building)Job – 1(Completed)Job – 1(Building)Thread 2Job – 2(Cmpleted)Job – 2(Building)Stage(Building)Job – 1(Building)Job – 2(Completed)
- 49. Query Tuning
- 50. Use a large databaseUse the query analyzer to find missing indexes
- 51. Views are great… sometimes10x-20x faster for “TOP n” queriesDepends on database implementation
- 52. Query selectivity is important200 msPipelines(20,000 rows)Stages(70,000 rows)BuildStateTransitions(4,000,000 rows)
- 53. Introduce an n+1 query2 ms
- 54. JRuby on Rails Hacks
- 55. Single JRuby runtime + Rails multi-threaded mode Much lower memory footprintBut completely uncharted territory
- 56. Severe thread blocks in JRubyReported to JRuby communityFixed immediately in 1.5.0 & 1.5.1
- 57. Rails nested partials Mysterious lock contention10 concurrent requests take 90 secs to completeStill a mystery!
- 58. Desperate situations, desperate measuresStatically inline partials at build time!
- 59. Routes subsystem is slowCache url_for
- 60. LoggingReplace Rails BufferedLogger with Java Log4jLoggerTurn off Rails logging in production
- 61. Fragment cachingGenerate a cache-key that incorprates all fields that are needed to uniquely identify that object’s current stateLRU cache will evict it on state change
- 64. Lessons LearnedFind the root causeUse a profilerMeasure before and after each fix
- 65. How we got from that to thisINSERT IMAGEThreads after performance tuning