Performance Tuning 101
5 likes985 views
The slide deck used for performance tuning basics for an app in managed environment. Uses ThoughtWorks Studios CI/CD server named Go as an example application. Goes into finding performance problems, pitfalls, anti-patterns, problem patterns for typical performance issues and war-stories extracted from Go performance tuning work.
Ad
More Related Content
Viewers also liked (18)
Similar to Performance Tuning 101 (6)
Ad
Recently uploaded (20)
Ad
Performance Tuning 101
- 1. Problem Approach Nature of fixes War Stories Performance Tuning 101 An introduction to performance tuning an application in managed environment Pavan KS1 Janmejay Singh2 1 mail: [email protected] blog: http://itspanzi.blogspot.com 2 mail: [email protected] blog: http://codehunk.wordpress.com Rootconf Bangalore, May 2012
- 2. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems.
- 3. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal.
- 4. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel.
- 5. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure.
- 6. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools.
- 7. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact.
- 8. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first.
- 9. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance.
- 10. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app).
- 11. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted.
- 12. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted. Don’t jump in the lake without knowing whats waiting inside.
- 13. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite)
- 14. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp.
- 15. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment.
- 16. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment. GNU/Linux
- 17. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment. GNU/Linux Approach and learning is independent of tech-stack.
- 18. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 19. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 20. Problem Approach Nature of fixes War Stories Don’t go looking for performance issues Doing it proactively is a wild goose chase! Varying parameters and running into a real problem is improbable, if not impossible.
- 21. Problem Approach Nature of fixes War Stories Remember... Don’t be out there with a gun looking for problems. Its easier to reproduce and fix a performance issue when you have seen it happen in a real setup. That way, you also know for sure, it is real.
- 22. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow
- 23. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow
- 24. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule
- 25. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late
- 26. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late SCM changes reflect after a long time
- 27. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late SCM changes reflect after a long time Process is hogging resources(requires 16 cores, 8 gig ram for a basic setup)
- 28. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 29. Problem Approach Nature of fixes War Stories Benchmark How slow is slow?
- 30. Problem Approach Nature of fixes War Stories Benchmark How slow is slow? How fast is fast?
- 31. Problem Approach Nature of fixes War Stories Benchmark How slow is slow? How fast is fast? Benchmark using realistic setup.
- 32. Problem Approach Nature of fixes War Stories Benchmark How slow is slow? How fast is fast? Benchmark using realistic setup. Use a repeatable and automated setup.
- 33. Problem Approach Nature of fixes War Stories Remember... Do not go overboard. Know your end goal.
- 34. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 35. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix?
- 36. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix? What if I stop at fix?
- 37. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix? What if I stop at fix? Why do I need to iterate?
- 38. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix? What if I stop at fix? Why do I need to iterate? How long should I iterate?
- 39. Problem Approach Nature of fixes War Stories Remember... Intution tends to work very poorly. Belive proof, not gut feel.
- 40. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 41. Problem Approach Nature of fixes War Stories What to measure Easy and obvious: Page response time
- 42. Problem Approach Nature of fixes War Stories What to measure Easy and obvious: Page response time Not so easy: App throughput Enqueue/Dequeue time for message queues CPU/IO churn Database performance Lock contention Managed environment specific parameters(Memory usage, GC churn etc). Environmental issues(too many processes, too low memory, very slow IO).
- 43. Problem Approach Nature of fixes War Stories Remember... Understand your environment. Know what to measure.
- 44. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf
- 45. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts
- 46. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics.
- 47. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify.
- 48. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler.
- 49. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler.
- 50. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler. Managed env specific params: GC logs, GC profiling.
- 51. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler. Managed env specific params: GC logs, GC profiling. Environmental issues: ps, dmesg, /proc/<pid>/status, /proc/net/dev, syslog, messages.
- 52. Problem Approach Nature of fixes War Stories Remember... Round pegs for round holes. Use appropriate tools. Configure them correctly with least possible overhead.
- 53. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 54. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code.
- 55. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical.
- 56. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem.
- 57. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix.
- 58. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous.
- 59. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads.
- 60. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads. Attack one and only one problem.
- 61. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads. Attack one and only one problem. Having said all that, correctness always comes first.
- 62. Problem Approach Nature of fixes War Stories Remember... Requires mechanical approach Attack worst problem first Reason every fix Reasoning is of paramount importance in multithreaded app Do one fix at a time Allows easy reasoning and verification, can be reverted Prefer correctness over performance
- 63. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 64. Problem Approach Nature of fixes War Stories Lock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse.
- 65. Problem Approach Nature of fixes War Stories Lock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse. More than one shared variable mandates synchronization usually, watch out!
- 66. Problem Approach Nature of fixes War Stories Lock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse. More than one shared variable mandates synchronization usually, watch out! Atomic operations and globaly unique objects(like interned symbols) are your friends.
- 67. Problem Approach Nature of fixes War Stories Remember... Jumps places, unlike functional bugs Look beyond local impact of a fix
- 68. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 69. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library
- 70. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs
- 71. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs Profile to see if majority of allocations are from just a handful of calls If its several calls with each contributing very little to the pie, don’t even try fixing it.
- 72. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs Profile to see if majority of allocations are from just a handful of calls If its several calls with each contributing very little to the pie, don’t even try fixing it. Understand GC ergonomics and each algorithm(if you have alternatives) in detail. Tweek your max/min permitted allocation judiciously, malloc/free are expensive system calls.
- 73. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 74. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human).
- 75. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball.
- 76. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball. Understand your DB settings, MVCC, table-locking scenarios, row-locking scenarios, transaction isolation tradeoffs.
- 77. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball. Understand your DB settings, MVCC, table-locking scenarios, row-locking scenarios, transaction isolation tradeoffs. Contd...
- 78. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends.
- 79. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity.
- 80. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity. Every DB may be subtly different(joins can be more expensive than materialzed views(if you have em))
- 81. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity. Every DB may be subtly different(joins can be more expensive than materialzed views(if you have em)) Denormalization can help crack tough nuts.
- 82. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 83. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt.
- 84. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching.
- 85. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold.
- 86. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold. Too frequent IO will cause io-wait, hence switch. Use of swap can completely kill performance. Swap is for emergency, not for regular usage. Over logging can hurt, be careful! Concept of buffered IO exists for a reason, use it! Use memory mapped IO if it makes sense.
- 87. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold. Too frequent IO will cause io-wait, hence switch. Use of swap can completely kill performance. Swap is for emergency, not for regular usage. Over logging can hurt, be careful! Concept of buffered IO exists for a reason, use it! Use memory mapped IO if it makes sense. Assigning very low max memory limit can cause frequent GCs, allocate enough memory so its your functional thread that works and not GC threads.
- 88. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 89. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause
- 90. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit.
- 91. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad.
- 92. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad. Atomic operations and globaly unique objects(like interned symbols) are your friends.
- 93. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad. Atomic operations and globaly unique objects(like interned symbols) are your friends. Unchartred waters ERB garbage churn Strange JVM allocations Single threaded database JRuby class hirarchy modification(runtime level) locks.
- 94. Problem Approach Nature of fixes War Stories Know what you are getting into... Know what you are getting into...
- 95. Problem Approach Nature of fixes War Stories Gist, to reiterate... Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted. Don’t jump in the lake without knowing whats waiting inside.
- 96. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 97. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements
- 98. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering
- 99. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader
- 100. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader Stream being dereferenced into a string
- 101. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader Stream being dereferenced into a string Thread locals to the rescue(used for caching big reusable objects)
- 102. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 103. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking
- 104. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods
- 105. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock)
- 106. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate
- 107. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency
- 108. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety
- 109. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety Take difficult design tradeoffs to keep code requiring a class of lock in one place
- 110. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety Take difficult design tradeoffs to keep code requiring a class of lock in one place Double check locking can be very helpful in frequently called code
- 111. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 112. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice)
- 113. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used
- 114. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes
- 115. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes Double check cache invalidation flows, its hard, sometimes bleeds across classes.
- 116. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes Double check cache invalidation flows, its hard, sometimes bleeds across classes. Be vigilant when using hirarchical caches(no associative-arrays or maps should be cached), its easy to run memory out with this.
- 117. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this).
- 118. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation.
- 119. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation. Understand your caching library, too low overflow limit may cause IO issues.
- 120. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation. Understand your caching library, too low overflow limit may cause IO issues. Use one and only one cache instance(its easy to end up in a memory runout with this). We had view fragments, urls, domain objects and view models cached in the same instance.
- 121. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
- 122. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt
- 123. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops.
- 124. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences.
- 125. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible.
- 126. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code.
- 127. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn.
- 128. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn. Use tools like Chromium dev toolset to nail leaks and profile methods.
- 129. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn. Use tools like Chromium dev toolset to nail leaks and profile methods. Auto-refreshing dashboards should refresh only visible and changed area.
- 130. Problem Approach Nature of fixes War Stories Questions? Questions?