PostgreSQL: meet your queue
7 likes5,892 views
This document summarizes a presentation about using PostgreSQL with message queues. It discusses how PostgreSQL can integrate with other systems by using the pg_amqp extension to publish messages to an AMQP broker on transaction commit. This allows actions like cache purges or search index updates to happen automatically when related database records change. While the current implementation uses AMQP 0.8, future support for AMQP 1.0 would add two-phase commit to make the process safer.
PostgreSQL: meet your queue
- 1. PostgreSQL: meet your Queue / Presentation / Theo Schlossnagle 1
- 2. PostgreSQL is Awesome ā¢ Fast. ā¢ Extensible. ā¢ Tablespaces. ā¢ Robust data types. ā¢ Partitioning (albeit fake). ā¢ Partial and functional indexes. ā¢ Extremely supportive community. ā¢ Extremely compliant with database standards. 2
- 3. PostgreSQL is not the āworldā ā¢ Inevitably, we must interact with the rest of the world. ā¢ ānon-SQLā components: ā¢ nosql systems ā¢ caching systems ā¢ search systems (solr/lucene) ā¢ management processes 3
- 4. Appropiare Typicalis ā¢ Enforce in the application: ā¢ the application code that updates the price or description of a product in the products table; ā¢ the application submits the updates to the search index system. ā¢ The flaw: ā¢ psql# UPDATE products SET description = REPLACE(description, ābehaviourā, ābehaviorā); ā¢ Administrative fixes like that require out-of-band dependency handling. 4
- 5. A Solution ā¢ Ideally, the database would notify all of these systems. ā¢ The most common case I see: memcached. ā¢ app: pull from memcached user::[email protected] if not found: select * from users where email=ā[email protected]ā put row in memcached at user::[email protected] ā¢ app: update users set mood=āhappyā where email=ā[email protected]ā (a) purge memcached record (b) get full row and replace in memcached ā¢ hence: pgmemcache ā¢ Problem: ā¢ need a Postgres module for each remote component 5
- 6. Enter Queueing ā¢ Queueing? ā¢ A generic message bus that allows PostgreSQL to communicate with other components in the architecture. ā¢ Enter AMQP: āAdvanced Message Queueing Protocolā ā¢ Why not STOMP? ā¢ Why not Starling? ā¢ AMQP has been around the block, and the specification is quite complete. ā¢ Almost every ārealā message broker implementation supports AMQP 6
- 7. Setups: Installing ā¢ svn export https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp ā¢ cd pg_amqp ā¢ make USE_PGXS=1 ā¢ make install ā¢ add to postgresql.conf: shared_preload_libraries = 'pg_amqp.so' ā¢ (re)start postgres ā¢ load the pg_amqp.sql file into your database. 7
- 8. Setup: configuring your broker INSERT INTO amqp.broker (host,port,vhost,username,password) VALUES (ālocalhostā,5672,NULL,āguestā,āguestā) RETURNING broker_id 8
- 9. Setup: declaring an exchange ā¢ This can often be done outside of the AMQP client ā¢ using an AMQP management process (that is just, in fact, an AMQP client) ā¢ Often, another component has already created the exchange. ā¢ If you really need to do it within PostgreSQL: SELECT amqp.declare_exchange(broker_id, exchange_name, exchange_type, passive, durable, auto_delete) 9
- 10. Usage: sending messages ā¢ How do I connect? ā¢ Itās implicit... you donāt need to. ā¢ How do I disconnect? ā¢ Broker connections are cached and live across transactions. ā¢ If you want to explicitly disconnect: SELECT amqp.disconnect(broker_id); 10
- 11. Usage: sending messages for real ā¢ Sending messages as a part of my transaction: SELECT amqp.publish(broker_id, exchange, routing_key, message); ā¢ This will publish the āmessageā over the specified āexchangeā using the specified ārouting_key,ā but only on transaction commit within Postgres. ā¢ Sending messages NOW: SELECT amqp.autonomous_publish(broker_id, exchange, routing_key, message); ā¢ Publish the same message, but do it immediately. 11
- 12. A dark side: unsafe? WTF? ā¢ Currently, pg_amqp uses the AMQP 0.8 specification. ā¢ The AMQP 1.0 specification introduces formal 2PC. ā¢ It is possible in the current implementation to have AMQP transaction fail when we commit it (in postgresās commit txn hook) in a fashion that we cannot act on. Ouch. ā¢ This only happens when the AMQP broker crashes between the last in-transaction publish call and the COMMIT on the database side. ā¢ Once RabbitMQ supports AMQP 1.0, Iāll update the driver to use 2PC. 12
- 13. Thank you for listening. https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp/ / Presentation 13