Blackboard DevCon 2011 - Developing B2 for Performance and Scalability
1 like527 views
The document discusses performance optimization and anti-patterns when developing applications. It provides examples of common anti-patterns like inefficient filtering, unnecessary processing, and applications that do not scale well over time. Specific low-level anti-patterns are also described for databases, user interfaces, and application code. Solutions and best practices are suggested to avoid these anti-patterns and improve performance.
![Top 10 violated low-level performance
anti-patterns fixed in Blackboard Learn™
Missing or non-optimal index [db]
Inefficient or erroneous criteria [db]
Unnecessary data retrieval [app}
Too many objects in memory [app}
Presentation of unlimited results [ui]
Frequency (%)
Inefficient access path [ui]
No data caching [app}
Not releasing resource (immediately) after use [app}
Incompatiblity with HA deployment [app}
Excessive DB round trips [app}
0 5 10 15 20 25](https://image.slidesharecdn.com/7112011titian22031300tatsuminoriakiweisserkevin-120721190755-phpapp02/85/Blackboard-DevCon-2011-Developing-B2-for-Performance-and-Scalability-9-320.jpg)
Blackboard DevCon 2011 - Developing B2 for Performance and Scalability
- 1. Developing B2 for Performance and Scalability Noriaki Tatsumi Kevin Weisser
- 2. Why Performance Matters? • User experience – Your users don’t like slow and unresponsive apps – Slow applications lose users in frustrations – Users tend to wander off beyond captive response time of 7 – 10 seconds • User adoption – Google saw 25% decline in searches within 6 weeks after introducing 0.5s delay (~$2.5B loss) – Wikia observed up to 20% user loss as the page load time reached 5 seconds
- 3. Wikia – Exit rate (abandon) http://assets.en.oreilly.com/1/event/27/Varnish%20-%20A%20State%20of%20the%20Art%20High-Performance%20Reverse%20Proxy%20Presentation.pdf
- 4. “With great power comes great responsibility” – Uncle Ben, Spiderman • Reputations at stake • System resources are shared and limited • B2s with scalability problem impacts system performance – System-wide performance degradation – Server downtime
- 5. What is an anti-pattern? • Some repeated pattern of action, process or structure that initially appears to be beneficial, but ultimately produces more bad consequences than beneficial results • A refactored solution exists that is clearly documented, proven in actual practice and repeatable. - Definition from Wikipedia
- 6. How can anti-patterns help? • Prevents performance as an afterthought • Helps identify performance issues that typically don’t arise until deployed in production • Prevents pre-mature optimizations
- 7. How do I use anti-patterns? • Identify and correct the occurrences of anti- patterns during: – Design review – Implementation/code review – Execution of experiments and profiling – Static code analysis
- 8. Top 10 violated high-level performance anti-patterns fixed in Blackboard Learn™ Inefficient Filtering Unnecessary Processing The Ramp Data Overreach Frequency (%) Unlimited Resource Usage Functional Bug Excessive Granularity More is Less Usage of Outdated Tools and APIs Mishandled Concurrency 0 5 10 15 20 25 30
- 9. Top 10 violated low-level performance anti-patterns fixed in Blackboard Learn™ Missing or non-optimal index [db] Inefficient or erroneous criteria [db] Unnecessary data retrieval [app} Too many objects in memory [app} Presentation of unlimited results [ui] Frequency (%) Inefficient access path [ui] No data caching [app} Not releasing resource (immediately) after use [app} Incompatiblity with HA deployment [app} Excessive DB round trips [app} 0 5 10 15 20 25
- 10. Most violated high-level anti-pattern #1: Inefficient Filtering • Description – Querying data using inefficient logic, algorithm, or technique • Problem – Consumes resources that can be used by other processes – Increases execution/response time • Solution – Store data in both centralized and organized manner – Search against smaller data set first, fail fast, and don’t look at unnecessary data – Leverage indexing and other efficient techniques/algorithms
- 11. Most violated high-level anti-pattern #1: Inefficient Filtering Missing or non-optimal index (DB) • Most violated low-level anti-pattern • Problem – Missing appropriate index on search columns leads to inefficient data retrieval operation by SQL query optimizer – Having index can inversely add unnecessary overhead if the column is never used for searched criteria • Solution – Analyze SQL execution plan to verify usage of proper index – Create index on foreign keys – Create unique index on unique columns – Use an index to eliminate unnecessary full-table or index scans to access small percentage of data on a large table – Use covering index for key lookup
- 12. Creating Database Objects with B2 • Release 9.1 SP1 and above – B2s authored by independent developers can now define and install schema elements • Not enabled by default – Administrators must enable this functionality from the B2 management section of the admin panel • Documentation – http://www.edugarage.com/display/BBDN/Creating+ Database+Objects+with+Building+Blocks
- 13. Most violated high-level anti-pattern #1: Inefficient Filtering Inefficient or erroneous criteria (DB) • The second most violated low-level anti-pattern • Problem – Duplicate, unnecessary, incorrect, or missing predicate checks and joins exist in SQL – Increases response time, system resource consumption, and locking • Solution – Be specific on what data you need – Keep it simple and avoid unnecessary checks – Use JOIN criteria that aligns with index – Analyze execution plan to derive efficient and balanced query
- 14. Most violated high-level anti-pattern #1: Inefficient Filtering Wrapping filtering column in a function (DB) • Problem – The query optimizer does not see the column – Available index likely does not get used • Solution – Rewrite query to avoid using function if possible – Format bind variables in application tier
- 15. Example of bad T-SQL function usage - Without -
- 16. Example of bad T-SQL function usage - With LOWER-
- 17. Example of bad T-SQL function usage - Comparison - Without T-SQL function With T-SQL function Logical I/O reads 3 3238 Estimated CPU cost 0.396307 15.454153
- 18. Most violated high-level anti-pattern #1: Inefficient Filtering Other low-level anti-pattern examples • Filtering larger query first • Implicit data type conversion • Inefficient CSS selectors • Inefficient DOM element lookup • Inefficient regex • Unnecessary query on VIEW instead of table • Usage of "NOT IN" comparison rather than JOIN • Usage of wrong index • Using UNION when returned records always unique
- 19. Most violated high-level anti-pattern #2: Unnecessary Processing • Description – Processing when not needed or not needed at the time – Excessive or duplicate executions • Problem – Consumes resources that can be used by important work – Increases execution/response time • Solution – Remove, postpone (asynchronous), prioritize, or reorder the processing steps – Leverage caching to reuse loaded or calculated data
- 20. Most violated high-level anti-pattern #2: Unnecessary Processing Presentation of Unlimited Results (UI Design) • 5th most violated low-level anti-pattern • Problem – Users typically do not consume all data when presented with large amount so the used resources are wasted – Severe system performance issue can arise depending on the volume of prepared results • Solution – Paginate results – Guide users to narrow down search criteria – Limit amount of data that can be returned
- 21. Example usage of pagination and limiting the number of returned results
- 22. Most violated high-level anti-pattern #2: Unnecessary Processing Inefficient access path (UI Design) • 6th most violated low-level anti-pattern • Problem – Room for optimization exists in actions/clicks users must take to complete his/her task – Results to creating additional work for the system (also impacts user’s time to task) • Solution – Refactor UI or click path to let users accomplish their tasks more efficiently
- 23. Example of enhancing access path
- 24. Most violated high-level anti-pattern #2: Unnecessary Processing Other low-level anti-pattern examples • Excessive executions of maintenance task • HTTP 404 error on cacheable static content • Missing short-circuit logic • No data caching • No http caching • Unnecessary cache reload • Unnecessary conditional check • Unnecessary object creation • Updating unchanged data
- 25. Most violated high-level anti-pattern #3: The Ramp • Description – Processing time increases or the system crashes as the application gets used over a period of time • Problem – The application cannot handle the increased volume of data/traffic – Resource consumption increases or exhausts • Solution – Select algorithms or data structures based on maximum size expected – Use algorithms/designs that adapt to the increasing size or doesn’t get effected by it (ex. flyweight pattern, streaming technique)
- 26. Most violated high-level anti-pattern #3: The Ramp Too many objects in memory • #4 most violated low-level anti-pattern • Problem – Loading large amount of data or objects at once for processing can result to out-of-memory error and other system-level performance impact – Examples include unmarshalling entire DB resultset in heap and generation of large XML documents using DOM object • Solution – Flyweight and lazy loading patterns – Streaming and chunking techniques
- 27. Example of streaming database result set - Overview - • DataList/DataListIterator – Interface for a list/iterator data structure materialized from a data source, such as a database or a file – Extends the List/Iterator interface to leverage polymorphism • ResultSetDataList – Implementation of DataList that’s materialized from a SQL resultset – Unmarshalls the resultset rows one by one as the getter method is called • ResultSetDataListIterator – Iterator that’s materialized from ResultSetDataList
- 28. Example of streaming database result set - Usage considerations - Avoid out-of-memory and SQL timeout • Use ResultSet.TYPE_FORWARD_ONLY • Set Statement fetch size – Default for Oracle JDBC = 10 – Default for i-net MERLIA = 0 • ‘0’ uses client cursor that reads all rows into memory • Specify fetch size to use cursor on server side • Use flyweight pattern in the unmarshaller if possible
- 29. Example of streaming database result set - Code sample - • See Appendix A
- 30. Most violated high-level anti-pattern #3: The Ramp Not releasing resource (immediately) after use • #8 most violated low-level anti-pattern • Problem – Cause of connection leaks, deadlocks, performance degradation and other unexpected behavior – Threads, sockets, database connection, file handler, and other resources can be a victim • Solution – Use finally block to ensure release – Release resource as quickly as possible
- 31. Example of resource management - CSContext - CSContext csCtxt=null; try { csCtxt = CSContext.getContext(); // your code } catch (Exception e) { csCtxt.rollback(); csCtxt=null; } finally { if (csCtxt!=null) { csCtxt.commit(); }
- 32. Most violated high-level anti-pattern #3: The Ramp Other low-level anti-pattern examples • Exceeding max number of expressions in a list • No data caching • No priority logic • Not releasing resource (immediately) after use • Presentation of unlimited results • Transaction isolation unnecessarily wide • Unlimited cache storage
- 33. Low-level performance anti-patterns fixed in Blackboard Learn™ by tier Application Database Frequency (%) UI Design Browser Network 0 10 20 30 40 50 60
- 34. Tools to identify anti-patterns: First line of defense • Peer code review – Atlassian Crucible – Review Board • Static code analyzer – FindBugs – PMD – Sonar – Custom rules • Performance unit testing – JUnitPerf
- 35. Tools to identify anti-patterns: Deeper analysis • Java – VisualVM – Memory Analyzer (MAT) – YourKit • Browser – Firebug – Google Page Speed – dynaTrace AJAX Edition – Fiddler – WebPagetest
- 36. Tools to identify anti-patterns: Deeper analysis • Database – MS SQL Server • SQL Server Management Studio • SQL Server Profiler • Performance Dashboard – Oracle • SQL*Plus • Automatic Workload Repository (AWR) • Active Session History (ASH)
- 37. Takeaways • The importance of performance and scalability for your B2 • Top violated performance anti-patterns in Blackboard Learn™ • Forensics Tools
- 38. Please provide feedback for this session by emailing [email protected]. The title of this session is: Developing B2 for Performance and Scalability
- 39. Appendix A: Example of streaming database result set
- 40. Example of streaming database result set - DataList.java - public interface DataList<E> extends List<E> { /** Closes the list and releases all of the resources that back the list */ void close(); }
- 41. Example of streaming database result set - ResultSetDataList.java - public <X> ResultSetDataList( DataListSelectQuery<X> query ) { // make sure the query object and resultset object in the query object are not null … try { if ( query.getRst().isClosed() ) { throw new IllegalArgumentException( "Result set must not be closed" ); } // Use this information to restrict the usage of methods that require scrolling _resultSetType = query.getRst().getType(); } catch ( SQLException e ) { throw new IllegalArgumentException( "Unable to validate input result set", e ); } _query = query; _rst = query.getRst(); _unmarshaller = query.getUnmarshaller(); …
- 42. Example of streaming database result set - ResultSetDataList.java - public E get( int index ) { if ( _resultSetType == ResultSet.TYPE_FORWARD_ONLY ) { String msg = "You must use the Iterator because your ResultSet is set to TYPE_FORWARD_ONLY."; throw new IllegalArgumentException( msg ); } if ( index < 0 ) { throw new IndexOutOfBoundsException(); } try { if ( _rst.absolute( index + 1 ) ) { if ( _rst.getRow() != index + 1 ) { throw new IndexOutOfBoundsException(); } return (E) _unmarshaller.unmarshall(); } else { throw new IndexOutOfBoundsException(); }…
- 43. Example of streaming database result set - ResultSetDataList.java - /** The close() method closes the prepared statement and result set wrapped in this object. This is called by the finalize method that garbage collector executes when appropriate. However, it should still be called as soon as this object is no longer necessary by the programmer. Please do not rely on the garbage collector. */ @Override protected void finalize() throws Throwable { try { close(); } finally { super.finalize(); } }
- 44. Example of streaming database result set - ResultSetDataListIterator.java - @Override public E next() { if ( !hasNext() ) { throw new NoSuchElementException(); } try { E element = (E) _list._unmarshaller.unmarshall(); _elementRetrieved = true; return element; } //handle exceptions …
Editor's Notes
- #3: Q: Do you like waiting for applications?Q: Have you guys seen the study done by Google?
- #4: Illustrates the direct correlation of site exit rate to increase in page load time = this is why you should care about performance and scalability of your B2s
- #9: Anti-patterns produced from root-cause analysis of performance bugs that were fixed since 8/2009 (~200 bugs).
- #10: More concrete examples of anti-pattern violations.
- #13: Q: Why should I care about table indexes? A: Because you can create database objects in 9.1 SP1 and above!
- #20: Solution:Postpone -> asynchronous
- #29: ResultSet.TYPE_FORWARD_ONLY:Cursor may move forward only
- #32: - The top level state manager for a series of Blackboard Content System calls- For every set of actions grouped together into one transaction, a new context is created and used- A Context instance contains transactional state for that request (i.e. commit, rollback)
- #34: Segway to tools
- #35: Eclipse integration available for:FindBugsPMDSonarJUnitPerf
- #37: SQL Server Management Studio:- Use to get execution plans and run DMV queriesSQLDiag:http://msdn.microsoft.com/en-us/library/ms162833.aspxSimplifies diagnostic information gathering for SQL ServerPAL:http://pal.codeplex.com/Reads in a performance monitor counter log and analyzes it using known thresholdsAWR:http://www.oracle-base.com/articles/10g/AutomaticWorkloadRepository10g.phpUsed to collect performance statistics (ex. Wait events, resource intensive SQL statements, system and session statistics) ASHhttp://www.orafaq.com/wiki/ASHTracks session activity and simplify performance tuning
- #42: DataListSelectQuery extends