Understanding PostgreSQL LW Locks
12 likes13,083 views
Lightweight locks (LWLocks) in PostgreSQL provide mutually exclusive access to shared memory structures. They support both shared and exclusive locking modes. The LWLocks framework uses wait queues, semaphores, and spinlocks to efficiently manage acquiring and releasing locks. Dynamic monitoring of LWLock events is possible through special builds that incorporate statistics collection.
Understanding PostgreSQL LW Locks
- 1. Understanding PostgreSQL LWLocks Jignesh Shah Staff Engineer, VMware Inc PgCon 2011 - Ottawa
- 2. About MySelf § Joined VMware in 2010 q PostgreSQL performance on vSphere § Previously at Sun Microsystems from 2000-‐2010 q Database Performance on Solaris/Sun Systems § Work with PostgreSQL Performance Community q Scaling, BoIlenecks using various workloads § My Blog: hIp://jkshah.blogspot.com © 2011 VMware Inc 2
- 3. Content v What are PostgreSQL LWLocks ? v Understanding modes of LWLock v LWLocks API v Architecture of the LWLocks Framework v Internals of LWLock Wait List v Ways to monitors the events related to LWLocks v Top LWLocks. © 2011 VMware Inc 3
- 4. What are PostgreSQL LWLocks? § LWLocks – Light Weight Locks of PostgreSQL § Primary Intent q Mutually Exclusive access to shared memory structures § Offers Shared and Exclusive mode § Not to be confused with the PostgreSQL Locks § PostgreSQL Locks depends on LWLocks to protect its shared state § Uses SpinLockAcquire/SpinLockRelease and PGSemaphoreLock/PGSemaphoreUnlock © 2011 VMware Inc 4
- 5. Understanding Modes of LWLocks ExclusiveLock § For a parcular lockid there can be only 1 exclusivelock held (no shared locks and no other exclusive lock for the lockid) SharedLock • For a parcular lockid there can be 1 or more SharedLock held (no exclusive lock for the lockid) © 2011 VMware Inc 5
- 6. LWLocks API § LWLockAssign(void) § Get a dynamic LWLock number (mostly at module load) § LWLockAcquire(LWLockId lockid, LWLockMode mode) § Waits ll Lock is acquired in EXCLUSIVE or SHARED mode § LWLockCondionalAcquire(LWLockId lockid, LWLockMode mode) § Returns False if it not available (Non-‐blocking) § LWLockRelease(LWLockId lockid) § Releases the lock § LWLockReleaseAll (void) § Used aaer an ERROR to cleanup § LWLockHeldByMe (LWLockId lockid) § Debug support test © 2011 VMware Inc 6
- 7. Architecture Flow of LWLocks LWLockCondionalAcquire • Easy lock it if available else return false (Non-‐blocking) LWLockAcquire • Lock if available or else put me in wait queue (FIFO) and got to sleep on a semaphore (and loop) • Only return when lock is available (Blocking) LWLockRelease • Release the Lock and if there are no more shared locks remaining then wake up the next process waing in the wait queue © 2011 VMware Inc 7
- 8. Wait Queue List • Wait Queue is protected by a mutex which is accessed using SpinLocks • When the lock is released and there are no more shared locks pending then the process releasing the lock will wake the next waiter as follows • If the next process in wait queue is waing for an EXCLUSIVE mode , only that process is removed from the queue and wakes them up • Or since the next process is waing for SHARED mode, it will try to remove as many consecuve SHARED mode processes and wake them up © 2011 VMware Inc 8
- 9. Lock Acquisition Internals Spins on ‘lock’ mutex Sleep on its process wait semaphore Lock • If Not Add to Wait Queue List Available • Release ‘lock’ mutex Release ‘lock’ mutex © 2011 VMware Inc 9
- 10. Lock Release Internals Spins on ‘lock’ mutex Release Lock • Wake Up process next in FIFO Shared • If it is Shared wake up all sequenal shared waiters Locks ? Release ‘lock’ mutex © 2011 VMware Inc 10
- 11. LWLocks Wait Queue List FIFO Example • Consider something 80-‐20 Shared /Exclusive rao S S S S E S E S S S S It will wake up first 5 shared locks then wait ll all of them releases the lock before waking up the process asking for Exclusive It will do only exclusive one then and one it releases the lock, wakes up the next 1 shared one © 2011 VMware Inc 11
- 12. Observations about LWLocks Wait Queue List • All Operaons on Wait Queue List are serialized • Not scalable on SMP architecture • Currently only FIFO supported • No restricon on shared wakeups © 2011 VMware Inc 12
- 13. LWLocks – Defined in lwlock.h typedef enum LWLockId { BufFreelistLock, ShmemIndexLock, OidGenLock, XidGenLock, SInvalWriteLock, ProcArrayLock, WALInsertLock, SInvalReadLock, WALWriteLock, ControlFileLock, CheckpointLock, CLogControlLock, SubtransControlLock, MultiXactGenLock, MultiXactOffsetControlLock, MultiXactMemberControlLock, RelCacheInitLock, BgWriterCommLock, TwoPhaseStateLock, TablespaceCreateLock, BtreeVacuumLock, AddinShmemInitLock, AutovacuumLock, AutovacuumScheduleLock, SyncScanLock, RelationMappingLock, AsyncCtlLock, AsyncQueueLock, SerializableXactHashLock, SerializableFinishedListLock, SerializablePredicateLockListLock, OldSerXidLock, SyncRepLock, /* Individual lock IDs end here */ FirstBufMappingLock, FirstLockMgrLock = FirstBufMappingLock + NUM_BUFFER_PARTITIONS, /*16 */ FirstPredicateLockMgrLock = FirstLockMgrLock + NUM_LOCK_PARTITIONS, /* must be last except for MaxDynamicLWLock: */ NumFixedLWLocks = FirstPredicateLockMgrLock + NUM_PREDICATELOCK_PARTITIONS, MaxDynamicLWLock = 1000000000 } LWLockId; © 2011 VMware Inc 13
- 14. Monitoring LWLOCK_STAT • Special build with LWLOCK_STAT defined LOCK_DEBUG • Puts debug messages in system alert log DYNAMIC TRACING (DTrace, SystemTap) • Recommended • Useful to find hot locks • Not for producon use, because it can make the server unstable © 2011 VMware Inc 14
- 15. Dynamic Tracing for LWLocks • postgresql-‐lwlock-‐wait-‐start(lockid,mode) • Postgresql-‐lwlock-‐wait-‐done(lockid,mode) • Postgresql-‐lwlock-‐acquire(lockid,mode) • Postgresql-‐lwlock-‐condacquire(lockid,mode) • Postgresql-‐lwlock-‐condacquire-‐fail(lockid,mode) • Postgresql-‐lwlock-‐release(lockid) © 2011 VMware Inc 15
- 16. Example Monitoring using Systemtap (DBT2) LOCKNAME LWID M W/A COUNT SUM-TIME(us) MAX-TIME(us) AVG-TIME(us) WALInsertLock 7 Ex W 14013 2746505 3955 195 WALWriteLock 8 Ex W 10006 25508653 286749 2549 LockMgrLock 55 Ex W 2035 203429 3323 99 LockMgrLock 45 Ex W 932 54297 2860 58 LockMgrLock 54 Ex W 673 24362 1062 36 ProcArrayLock 4 Ex W 515 15907 666 30 ProcArrayLock 4 Sh W 176 6064 97 34 LockMgrLock 56 Ex W 171 5826 376 34 CLogControlLock 11 Sh W 111 22490 6127 202 LockMgrLock 57 Ex W 101 5524 1326 54 LockMgrLock 59 Ex W 79 2883 347 36 CLogControlLock 11 Ex W 58 8543 4439 147 LockMgrLock 49 Ex W 57 1848 76 32 LockMgrLock 47 Ex W 57 3166 1468 55 © 2011 VMware Inc 16
- 17. Top Locks - WALWriteLock • Protects writes on WAL • Acquired when WAL records flushed to disk • Acquired when WAL Log switch occurs • Improve the underlying storage of pg_xlog • Synchronous_commit=off helps indirectly ( Do not wait for flush to the disk) • Full_page_writes=off also helps reduce the stress (but not recommended since resiliency goes down) © 2011 VMware Inc 17
- 18. Top Locks - WALInsertLock • Protects WAL Buffers • Increasing wal buffers may help though to only certain extent • Snychnrous_commit off will lead to increased pressure on this lock ( Not a bad thing ) • However eventually not much can be done once it gets to a big problem without new commits • Full_page_writes=off certainly helps (again not recommended since it reduces the resiliency of the database from write errors) © 2011 VMware Inc 18
- 19. Top Locks - ProcArrayLock • Protects ProcArray structure • It used to be that every transacon actually acquired this lock in exclusive mode before commit causing it to be a top lock • Fixed in 9.0 © 2011 VMware Inc 19
- 20. Top Locks - SInvalidReadLock • Protects sinval array • Readers take “Shared” SInvalReadLock • SICleanupQueue and other array-‐wide updates take “Excluslive” SInvalReadLock to lock out all readers • Long wait mes to acquire SInvalidReadLock generally results when the Shared Buffer pool is being stressed • Increase shared_buffers in postgresql.conf corresponding to acve data size © 2011 VMware Inc 20
- 21. Top Locks - CLogControlLock • Protects CLogControl structure • Generally not a problem • If it shows on the top lists check • $PGDATA/pg_clog should be on buffered file system © 2011 VMware Inc 21
- 22. Example Monitoring using Systemtap (Sysbench simple read) LOCKNAME LWID M W/A COUNT SUM-TIME(us) MAX-TIME(us) AVG-TIME(us) LockMgrLock 45 Ex W 85343 469682510 13152 5503 LockMgrLock 57 Ex W 57547 30903727 8313 537 LockMgrLock 44 Ex W 390 34061 1670 87 LockMgrLock 59 Ex W 375 41570 2032 110 LockMgrLock 56 Ex W 361 39685 1889 109 LockMgrLock 47 Ex W 344 24548 1564 71 LockMgrLock 54 Ex W 335 67770 2319 202 LockMgrLock 50 Ex W 325 44213 1690 136 LockMgrLock 49 Ex W 325 39280 1475 120 LockMgrLock 55 Ex W 323 39448 1584 122 LockMgrLock 48 Ex W 323 26982 1669 83 © 2011 VMware Inc 22
- 23. Top Locks - LockMgrLocks • Protects relaons • Sets of about 16 Lock Parons by default to handle all relaons • Each relaon is part of only one paron (irrespecve of size) © 2011 VMware Inc 23
- 24. Other Locks - BufMappingLocks • Protects regions of Buffers • Sets of about 16 Regions of Buffers by default to handle the whole Bufferpool • Only taken shared access © 2011 VMware Inc 24
- 25. Problems with LWLock • Overall system gravitates to certain top locks • Single mutex lock protects adding and depleng the wait queue • Not SMP Scalable • Chance for opmizaon out there • Performance limited to serialized rate the locks are processed © 2011 VMware Inc 25
- 26. Questions / More Information v Email: [email protected] v Learn more about PostgreSQL q hIp://www.postgresql.org v Blog: hIp://jkshah.blogspot.com © 2011 VMware Inc 26