Wait! What’s going on inside my database? (PASS 2023 Update)
0 likes60 views
PostgreSQL 9.6 introduced wait events and PostgreSQL 10 progressed them, but what are they? What do they mean? How do I find them and how do I make them go away? Wait events are one of the most significant advancements in observability for PostgreSQL databases; their usefulness is unparalleled in troubleshooting performance. This talk will go into all that and more as we explore the world of PostgreSQL wait events.
![Mariinsky Theatre, St. Petersburg
by Sandra Cohen-Rose and Colin Rose (Montreal, Canada)
CC BY-SA
#PASSDataCommunitySummit
• Version 9.6
• Aa65de0 – 11 Sep 2015 – Autogenerate lwlocknames.[c|h]
• 53be0b1 – 10 Mar 2016 – Heavy/Lightweight Locks, Buffer Pins
Wait Events](https://image.slidesharecdn.com/pg-wait-eventspass23upload-231121002235-6af2d77f/85/Wait-What-s-going-on-inside-my-database-PASS-2023-Update-28-320.jpg)
![Mariinsky Theatre, St. Petersburg
by Sandra Cohen-Rose and Colin Rose (Montreal, Canada)
CC BY-SA
#PASSDataCommunitySummit
• Version 9.6
• Aa65de0 – 11 Sep 2015 – Autogenerate lwlocknames.[c|h]
• 53be0b1 – 10 Mar 2016 – Heavy/Lightweight Locks, Buffer Pins
• Version 10
• 6f3bd98 – 4 Oct 2016 – Latches & Sockets, Clients, Main Loops
• 249cf07 – 18 Mar 2017 – I/O
• Fc70a4b – 26 Mar 2017 – Background and Auxiliary Processes
• Version 11
• 1804284 – 20 Dec 2017 – Parallel-Aware Hash Joins
Wait Events](https://image.slidesharecdn.com/pg-wait-eventspass23upload-231121002235-6af2d77f/85/Wait-What-s-going-on-inside-my-database-PASS-2023-Update-29-320.jpg)
Ad
More Related Content
Similar to Wait! What’s going on inside my database? (PASS 2023 Update) (20)
Recently uploaded (20)
Ad
Wait! What’s going on inside my database? (PASS 2023 Update)
- 1. Wait! What’s going on inside my database? PostgreSQL and the Science of Database Performance Jeremy Schneider
- 2. Jeremy DB & Perf Engineer / AWS Slack: pgtreats.info/slack-invite Blog: ardentperf.com Twitter/X: @jer_s Organizer / Seattle PG User Group • Engineering for Aurora and RDS Open Source: Aurora PostgreSQL first GA release (2017), ICU & collation, data durability & corruption protection, performance, fleet data collection and analysis, early logical replication work, new major versions, addressing complex operational issues, recruiting & training • Launched an internal Amazon-wide dedicated PostgreSQL chat channel in 2018 which now has thousands of members and numerous technical discussions daily • Founder of "RAC Attack" Community Driven Oracle Cluster Database Workshop – almost 40 events across 15 countries between 2011 and 2016 • Participant, speaker and/or organizer at Linux and Oracle user groups & conferences since the early 2000’s. PostgreSQL user groups & conferences since 2017. • Database blog since 2007, Oracle ACE Alumni, Oak Table Schneider
- 3. #PASSDataCommunitySummit About PostgreSQL 1970: Mathematician Edgar F. Codd, working as researcher for IBM, publishes “A Relational Model of Data for Large Shared Data Banks” 1973: Michael Stonebraker and Eugene Wong at University of California Berkeley seek funding and begin development of a relational database called INGRES 1986: Michael Stonebraker and Lawrence A. Rowe at University of California Berkeley publish “The Design of POSTGRES” – a new database that is the successor to INGRES 1994: Andrew Yu and Jolly Chen at University of California Berkeley add support for the SQL language 1996: Transition to non-university core team of volunteers, official release under new name POSTGRESQL 1985
- 8. #PASSDataCommunitySummit Delfador Chibi by Peileppe CC0 https://openclipart.org/detail/19811/delfador-chibi About Database Performance 1990’s Manager: “Dear DBA: Expert consultants have taught us that if the Buffer Cache Hit Ratio (BCHR) is below 90% then the system immediately needs an expensive tuning engagement. Please report any databases that have BCHR < 90%.”
- 9. #PASSDataCommunitySummit Delfador Chibi by Peileppe CC0 https://openclipart.org/detail/19811/delfador-chibi About Database Performance 1990’s Manager: “Dear DBA: Expert consultants have taught us that if the Buffer Cache Hit Ratio (BCHR) is below 90% then the system immediately needs an expensive tuning engagement. Please report any databases that have BCHR < 90%.”
- 10. #PASSDataCommunitySummit Nørgaard, Mogens et al. Oracle Insights: Tales of the Oak Table. Berkeley, CA: Apress/OakTable Press, 2004. p76-77. About Database Performance
- 11. #PASSDataCommunitySummit About Database Performance Millsap, Cary V. Optimizing Oracle Performance. Sebastopol, CA: OReilly, 2003. p225, 240, 258-259 R = S + W “How long the SQL takes to run” See also: • Shallahamer, Craig. Forecasting Oracle Performance. Berkeley, CA: Apress, 2007.
- 12. #PASSDataCommunitySummit About Database Performance Active Session Sampling (JB’s notebook, 2004) Images & Quotes Used With Permission
- 13. #PASSDataCommunitySummit About Database Performance January 2022 March 2023
- 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA
- 16. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA Wait Events
- 17. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit • 1990s: Database kernel instrumentation: • Counters and tools to snapshot/compare them • Events (log a message under certain circumstances) • 1992: Unable to solve a performance problem, engineers added event code in version 7.0.12 capable of emitting log messages when the database waited for something • First exposed in V$SESSION_WAIT and later in V$SESSION (equivalent of pg_stat_activity) • PostgreSQL recognized and built on concepts that had become widespread across the industry Wait Events
- 18. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events
- 19. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events Millsap, Cary V. Optimizing Oracle Performance. Sebastopol, CA: OReilly, 2003. p225, 240, 258-259 R = S + W “How long the SQL takes to run” See also: • Shallahamer, Craig. Forecasting Oracle Performance. Berkeley, CA: Apress, 2007.
- 20. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events Active Session Sampling (JB’s notebook, 2004) Images & Quotes Used With Permission
- 21. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit “But why are these events called wait events? … In short, when a session is not using the CPU, it may be waiting for a resource, an action to complete, or simply more work. Hence, events that associated with all such waits are known as wait events.” Shee, Richmond, Kirtikumar Deshpande, and K. Gopalakrishnan. Oracle Wait Interface a Practical Guide to Performance Diagnostics & Tuning. New York: London, 2004. p16 Wait Events
- 22. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit High-Level Idea: Caveats: • OS scheduling/runqueue • Measurement overhead • OS kernel or underlying library CPU time (e.g. I/O) Wait Events The database is WAITING any time when it’s not running on the CPU
- 23. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Idle Waiting Cpu Wait Events psql> SELECT…
- 24. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events
- 25. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events
- 26. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events https://youtu.be/RpoC3UcG5YA
- 27. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events
- 28. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit • Version 9.6 • Aa65de0 – 11 Sep 2015 – Autogenerate lwlocknames.[c|h] • 53be0b1 – 10 Mar 2016 – Heavy/Lightweight Locks, Buffer Pins Wait Events
- 29. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit • Version 9.6 • Aa65de0 – 11 Sep 2015 – Autogenerate lwlocknames.[c|h] • 53be0b1 – 10 Mar 2016 – Heavy/Lightweight Locks, Buffer Pins • Version 10 • 6f3bd98 – 4 Oct 2016 – Latches & Sockets, Clients, Main Loops • 249cf07 – 18 Mar 2017 – I/O • Fc70a4b – 26 Mar 2017 – Background and Auxiliary Processes • Version 11 • 1804284 – 20 Dec 2017 – Parallel-Aware Hash Joins Wait Events
- 30. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit • Version 12 • Add a wait event for fsync of WAL segments (Konstantin Knizhnik) • Ensure that TimelineHistoryRead and TimelineHistoryWrite wait states are reported in all code paths that read or write timeline history files (Masahiro Ikeda) • Version 13 • Rename various wait events to improve consistency (Fujii Masao, Tom Lane) • Report a wait event while creating a DSM segment with posix_fallocate() (Thomas Munro) • Add wait event VacuumDelay to report on cost-based vacuum delay (Justin Pryzby) • Add wait events for WAL archive and recovery pause (Fujii Masao) • The new events are BackupWaitWalArchive and RecoveryPause. • Add wait events RecoveryConflictSnapshot and RecoveryConflictTablespace to monitor recovery conflicts (Masahiko Sawada) • Improve performance of wait events on BSD-based systems (Thomas Munro) • Version 14 • Add wait event WalReceiverExit to report WAL receiver exit wait time (Fujii Masao) • Wake up for latch events when the checkpointer is waiting between writes. This improves responsiveness to backends sending sync requests. The change also creates a proper wait event class for these waits. (Thomas Munro) • Version 15 • Add wait events for local shell commands. The new wait events are used when calling archive_command, archive_cleanup_command, restore_command and recovery_end_command. (Fujii Masao) • Correct the name of the wait event for SLRU buffer I/O for commit timestamps. This wait event is named CommitTsBuffer according to the documentation, but the code had it as CommitTSBuffer. Change the code to match the documentation, as that way is more consistent with the naming of related wait events. (Alexander Lakhin) • Re-activate reporting of wait event SLRUFlushSync. Reporting of this type of wait was accidentally removed in code refactoring. (Thomas Munro) • Version 16 • Add wait event SpinDelay to report spinlock sleep delays (Andres Freund) • Create new wait event DSMAllocate to indicate waiting for dynamic shared memory allocation. Previously this type of wait was reported as DSMFillZeroWrite, which was also used by mmap() allocations. (Thomas Munro) • Allow parallel application of logical replication. Wait events LogicalParallelApplyMain, LogicalParallelApplyStateChange, and LogicalApplySendData were also added. Column leader_pid was added to system view pg_stat_subscription to track parallel activity. (Hou Zhijie, Wang Wei, Amit Kapila) • Have wal_retrieve_retry_interval operate on a per-subscription basis. Previously the retry time was applied globally. This also adds wait events >LogicalRepLauncherDSA and LogicalRepLauncherHash. (Nathan Bossart) • Version 17 • Support custom wait events for wait event type "Extension” (Masahiro Ikeda) Wait Events
- 31. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events src/include/pgstat.h
- 32. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events doxygen.postgresql.org
- 33. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Wait Events
- 34. Mariinsky Theatre, St. Petersburg by Sandra Cohen-Rose and Colin Rose (Montreal, Canada) CC BY-SA #PASSDataCommunitySummit Gaps after migrating to Open Source/Community PostgreSQL • Wait Event Counters and Cumulative Times • Wait Event Arguments (object, block, etc) • Comprehensive tracking of CPU time (POSIX rusage) • Ability to find previous SQL for COMMIT/ROLLBACK • Needed to identify which transaction is committing. (SQL comments on commit?) • On-CPU State • SQL Execution Stage (parse/plan/execute/fetch) • SQL Execution Plan Identifier in pg_stat_statements • Current plan node • Progress on long operations (e.g. large seqscan) • Better runtime visibility into PLs Wait Events
- 35. I can haz Wait Events? Solving Problems with Wait Events in PostgreSQL By Antony Griffiths (Flickr), CC BY
- 37. #PASSDataCommunitySummit Solving Problems With Wait Events
- 38. #PASSDataCommunitySummit Solving Problems With Wait Events pid | state | wait_event_type | wait_event | xact_runtime | query_short --------+-------------+-----------------+---------------------+-------------------------+---------------------------------------------------- 8135 | active | | | -00:00:00.000941 | autovacuum: VACUUM pghist.pg_stat_statements_20190 8168 | active | | | 00:00:00 | SELECT col1, col2, | | | | | 108975 | | Activity | WalWriterMain | | 108976 | | Activity | AutoVacuumMain | | 108973 | | Activity | CheckpointerMain | | 108974 | | Activity | BgWriterMain | | 108979 | | Activity | LogicalLauncherMain | | 8185 | active | | | 00:00:00.07941 | autovacuum: VACUUM pghist.pg_stat_sys_indexes_2019 8212 | active | | | 00:00:00.349238 | autovacuum: VACUUM pghist.pg_stat_statements_20190 115699 | active | Lock | relation | 00:30:01.170404 | SELECT proc('param1') 103268 | active | IO | DataFileRead | 00:46:46.277548 | select count(*) from some_ones_table a , (select c 95936 | active | LWLock | buffer_io | 00:56:57.327904 | SELECT col1 FROM some_ones_table a, (SELECT col1 a 95935 | active | IO | DataFileRead | 00:56:57.328169 | SELECT col1 FROM some_ones_table a, (SELECT col1 a 95921 | active | LWLock | buffer_io | 00:56:57.393765 | SELECT col1 FROM some_ones_table a, (SELECT col1 a 56628 | active | IO | DataFileRead | 01:47:55.333596 | select col1 from some_ones_table WHERE err_id in ( 53981 | active | IO | BufFileRead | 01:51:40.986659 | SELECT col1 FROM some_ones_table a, (SELECT asin a 49386 | active | LWLock | buffer_io | 01:58:13.166389 | SELECT count(*) FROM some_ones_table a, (SELECT co 29172 | active | IO | BufFileRead | 02:04:09.108342 | SELECT count(*) FROM some_ones_table a, (SELECT co 43208 | active | LWLock | buffer_io | 02:06:39.296499 | SELECT count(*) FROM some_ones_table a, (SELECT co 43207 | active | IO | DataFileRead | 02:06:39.29666 | SELECT count(*) FROM some_ones_table a, (SELECT co 31401 | active | IPC | MessageQueueReceive | 02:06:39.370239 | SELECT count(*) FROM some_ones_table a, (SELECT co 12387 | active | IO | DataFileRead | 02:46:50.262871 | select count(*) from some_ones_table a , (select c 12386 | active | IO | DataFileRead | 02:46:50.263142 | select count(*) from some_ones_table a , (select c 12385 | active | IO | DataFileRead | 02:46:50.266696 | select count(*) from some_ones_table a , (select c 83681 | active | BufferPin | BufferPin | 15:24:45.260184 | autovacuum: VACUUM schema1.some_ones_table (to prev 23340 | active | LWLock | buffer_io | 1 day 16:39:18.732685 | select column_001,column2,column3,column000004, 24074 | active | LWLock | buffer_io | 1 day 16:41:55.91496 | WITH this_subquery_01 as (select column_001,PIPELI 8110 | active | LWLock | buffer_io | 1 day 17:03:52.767838 | WITH this_subquery_01 as (select column_001,PIPEL 51767 | active | LWLock | buffer_io | 1 day 19:03:47.006302 | WITH this_subquery_01 as (select column_001,PIPEL 9217 | active | LWLock | buffer_io | 1 day 20:01:58.572314 | WITH this_subquery_01 as (select column_001,PIPEL 6086 | active | IO | DataFileRead | 1 day 20:06:08.584313 | WITH this_subquery_01 as (select column_001,PIPEL 115385 | active | LWLock | buffer_io | 1 day 20:35:27.617606 | WITH this_subquery_01 as (select column_001,PIPEL 94256 | idle in trx | Client | ClientRead | 27 days 02:33:48.940102 | select subquery00_.column_001 as COLUMN01_2_0_, a (33 rows)
- 39. #PASSDataCommunitySummit Solving Problems With Wait Events
- 40. #PASSDataCommunitySummit Solving Problems With Wait Events
- 41. 401 What happened? (It’s not happening now.)
- 42. #PASSDataCommunitySummit Repository of Historical Performance Data (pg_stat_activity) Top SQL Top Wait Events EXPLAIN ANALYZE with Buffers, IO timing, etc Investigate WAIT EVENT & STEP Taking The Most TIME Solving Problems With Wait Events
- 43. #PASSDataCommunitySummit Repository of Historical Performance Data (pg_stat_activity) Top SQL Top Wait Events EXPLAIN ANALYZE with Buffers, IO timing, etc Investigate WAIT EVENT & STEP Taking The Most TIME Solving Problems With Wait Events
- 44. #PASSDataCommunitySummit Repositories of Historical Performance Data (Active Session Sampling of Wait Events) • https://wiki.postgresql.org/wiki/Monitoring • Amazon RDS Performance Insights • RDS for PostgreSQL 10+ • Aurora PostgreSQL-Compatible Edition 9.6+ (v10 Wait Events were backported) • Rolling 7 days of history is free. Up to 2 years on paid tier. Solving Problems With Wait Events
- 45. #PASSDataCommunitySummit Repository of Historical Performance Data (pg_stat_activity) Top SQL Top Wait Events EXPLAIN ANALYZE with Buffers, IO timing, etc Investigate WAIT EVENT & STEP Taking The Most TIME Solving Problems With Wait Events
- 46. #PASSDataCommunitySummit Solving Problems With Wait Events
- 47. #PASSDataCommunitySummit Solving Problems With Wait Events (one of many options)
- 48. #PASSDataCommunitySummit Solving Problems With Wait Events
- 49. #PASSDataCommunitySummit Solving Problems With Wait Events
- 50. #PASSDataCommunitySummit Solving Problems With Wait Events
- 51. #PASSDataCommunitySummit Repository of Historical Performance Data (pg_stat_activity) Top SQL Top Wait Events EXPLAIN ANALYZE with Buffers, IO timing, etc Investigate WAIT EVENT & STEP Taking The Most TIME Solving Problems With Wait Events
- 52. #PASSDataCommunitySummit Solving Problems With Wait Events
- 53. #PASSDataCommunitySummit Solving Problems With Wait Events
- 54. #PASSDataCommunitySummit Solving Problems With Wait Events
- 55. #PASSDataCommunitySummit Solving Problems With Wait Events
- 56. #PASSDataCommunitySummit Solving Problems With Wait Events Aurora PostgreSQL-Compatible: • AWS Documentation covers Aurora-Specific Wait Events • Shares Code With Community PostgreSQL (and merges regularly)
- 57. Thank you aws.amazon.com/rds/postgresql Jeremy Schneider Slack: pgtreats.info/slack-invite Blog: ardentperf.com Twitter/X: @jer_s
- 59. Stuffed Elephant Store • A unique service that produces on- demand stuffed elephants • Multiple sizes with long or short fur • Any color the customer wants, as long as its blue Breibeest (Flickr), CC BY
- 61. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- 62. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- 63. #PASSDataCommunitySummit See the query text and the wait events by query Look back 7 days or as much as 2 years to find activity Amazon RDS Performance Insights
- 64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. explain.depesz.com
- 65. #PASSDataCommunitySummit Explains how Postgres plans to execute a query Shows the type of operation, the estimated cost, and the estimated number of rows Execution Plans
- 66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- 67. #PASSDataCommunitySummit Contains the structure of all objects in the database Statistics views shows usage of the objects System Catalogs
- 68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- 69. #PASSDataCommunitySummit PostgreSQL has a rich set of index types Base functionality can be enhanced by specialized extensions Indexes
- 70. It’s so simple! Solving Problems With Wait Events
- 72. Session evaluation Your feedback is important to us Evaluate this session at: www.PASSDataCommunitySummit.com/evaluation
- 73. Thank you aws.amazon.com/rds Jeremy Schneider Slack: pgtreats.info/slack-invite Blog: ardentperf.com Twitter/X: @jer_s