PostgreSQL: meet your queue
7 likes5,887 views
This document summarizes a presentation about using PostgreSQL with message queues. It discusses how PostgreSQL can integrate with other systems by using the pg_amqp extension to publish messages to an AMQP broker on transaction commit. This allows actions like cache purges or search index updates to happen automatically when related database records change. While the current implementation uses AMQP 0.8, future support for AMQP 1.0 would add two-phase commit to make the process safer.
![[@NaukriEngineering] Messaging Queues](https://cdn.slidesharecdn.com/ss_thumbnails/queueprocessing-161111064335-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (17)
Ad
Similar to PostgreSQL: meet your queue (20)
Ad
More from Theo Schlossnagle (20)
Recently uploaded (20)
PostgreSQL: meet your queue
- 1. PostgreSQL: meet your Queue / Presentation / Theo Schlossnagle 1
- 2. PostgreSQL is Awesome • Fast. • Extensible. • Tablespaces. • Robust data types. • Partitioning (albeit fake). • Partial and functional indexes. • Extremely supportive community. • Extremely compliant with database standards. 2
- 3. PostgreSQL is not the “world” • Inevitably, we must interact with the rest of the world. • “non-SQL” components: • nosql systems • caching systems • search systems (solr/lucene) • management processes 3
- 4. Appropiare Typicalis • Enforce in the application: • the application code that updates the price or description of a product in the products table; • the application submits the updates to the search index system. • The flaw: • psql# UPDATE products SET description = REPLACE(description, ‘behaviour’, ‘behavior’); • Administrative fixes like that require out-of-band dependency handling. 4
- 5. A Solution • Ideally, the database would notify all of these systems. • The most common case I see: memcached. • app: pull from memcached user::[email protected] if not found: select * from users where email=‘[email protected]‘ put row in memcached at user::[email protected] • app: update users set mood=‘happy‘ where email=‘[email protected]‘ (a) purge memcached record (b) get full row and replace in memcached • hence: pgmemcache • Problem: • need a Postgres module for each remote component 5
- 6. Enter Queueing • Queueing? • A generic message bus that allows PostgreSQL to communicate with other components in the architecture. • Enter AMQP: “Advanced Message Queueing Protocol” • Why not STOMP? • Why not Starling? • AMQP has been around the block, and the specification is quite complete. • Almost every “real” message broker implementation supports AMQP 6
- 7. Setups: Installing • svn export https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp • cd pg_amqp • make USE_PGXS=1 • make install • add to postgresql.conf: shared_preload_libraries = 'pg_amqp.so' • (re)start postgres • load the pg_amqp.sql file into your database. 7
- 8. Setup: configuring your broker INSERT INTO amqp.broker (host,port,vhost,username,password) VALUES (‘localhost’,5672,NULL,‘guest’,‘guest’) RETURNING broker_id 8
- 9. Setup: declaring an exchange • This can often be done outside of the AMQP client • using an AMQP management process (that is just, in fact, an AMQP client) • Often, another component has already created the exchange. • If you really need to do it within PostgreSQL: SELECT amqp.declare_exchange(broker_id, exchange_name, exchange_type, passive, durable, auto_delete) 9
- 10. Usage: sending messages • How do I connect? • It’s implicit... you don’t need to. • How do I disconnect? • Broker connections are cached and live across transactions. • If you want to explicitly disconnect: SELECT amqp.disconnect(broker_id); 10
- 11. Usage: sending messages for real • Sending messages as a part of my transaction: SELECT amqp.publish(broker_id, exchange, routing_key, message); • This will publish the “message” over the specified “exchange” using the specified “routing_key,” but only on transaction commit within Postgres. • Sending messages NOW: SELECT amqp.autonomous_publish(broker_id, exchange, routing_key, message); • Publish the same message, but do it immediately. 11
- 12. A dark side: unsafe? WTF? • Currently, pg_amqp uses the AMQP 0.8 specification. • The AMQP 1.0 specification introduces formal 2PC. • It is possible in the current implementation to have AMQP transaction fail when we commit it (in postgres’s commit txn hook) in a fashion that we cannot act on. Ouch. • This only happens when the AMQP broker crashes between the last in-transaction publish call and the COMMIT on the database side. • Once RabbitMQ supports AMQP 1.0, I’ll update the driver to use 2PC. 12
- 13. Thank you for listening. https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp/ / Presentation 13