PostgreSQL 9.4 JSON Types and Operators

Editor's Notes

• #3: I am a Systems Integration Engineer. To me that means that I take different systems written in different ways with different technology, connect these systems to produce new value with the correlated data or through the interaction of those previously separate systems. Doing this means knowing what technology you already have or you may be planning to deploy can do for you. I am also an advocate for several open source projects and platforms.
• #4: One big aspect that a lot of engineers face is how to talk SQL, JSON, XML, et al to one data system of record efficiently with minimized implementation and maintenance overhead. A great platform to accomplish this is with PostgreSQL. You can transform, select specific sections of data, and use JSON data in expressions in database queries without any round trips to your application implementation layer. REVIEW SLIDE The goal of this particular talk is to inform you of the features PostgreSQL 9.4 JSON types and operators. These features are an efficient and convenient way to store, query, and update JSON data structures that your applications need to use.
• #5: As of PostgreSQL 9.4, there are two JSON types to be aware of that may apply to your implementation needs: JSON and JSONB
• #6: REVIEW SLIDE GIN stands for Generalized Inverted Index. GIN is designed for handling cases where the items to be indexed are composite values, and the queries to be handled by the index need to search for element values that appear within the composite items. For example, the items could be documents, and the queries could be searches for documents containing specific words.
• #7: Some of you might be saying, hey bruh, what about HSTORE ? REVIEW SLIDE
• #8: If you are going to use JSON types, then you should know what the JSON operators are and how you can use them in value expressions and WHERE clauses.
• #9: When an object is returned, it is either JSON or JSONB - the same as the left hand operand. Hash arrow #> operators take array-of-string arguments. Notice the object value at array-of-string path A COMMA TWO is 3 and not 2, as object elements are zero indexed.
• #10: JSONB types have additional operators for comparing their fields and sets of those fields.
• #12: We see that the doc column Storage Strategy is extended meaning it allows for both compression and out-of-line storage if the row is still too big to fit on an 8k page. This is explained in more detail in the PostgreSQL documentation under TOAST - an acronym for The Oversized-Attribute Storage Technique.
• #14: An exact copy of the JSON inserted has been stored. REVIEW SLIDE We can still extract elements of the data with JSON operators such as ->> dash arrow arrow.
• #15: The dash arrow arrow operator says, within the doc, look up the field and return it as text. REVIEW SLIDE You typically do not want to do this unless you know you do not plan to do anymore processing or selection of the JSON data such as to pipe the JSON data it to the user’s browser for use in a widget.
• #16: There is also dash arrow which doesn’t convert the field to text and leaves it as a JSON element. This can be used to navigate JSON objects and subsections such as the address element of a greater JSON document containing a person’s details. REVIEW SLIDE
• #18: Now let’s create a table with a column of type JSONB
• #19: Let’s insert the same JSON document into the JSONB table.
• #20: The data is returned with normalized formatting, white space is gone. REVIEW SLIDE We can still extract elements of the data with JSON operators such as dash arrow arrow ->>.
• #21: REVIEW SLIDE The JSON object select is returned with normalized formatting, no white space of the address element is returned.
• #22: We can use question mark operator ? does field key exist to select rows that have a specific element key. I know that my sample data only has line2 for address about half the time. So how do I get rows that only have line2 specified ? REVIEW SLIDE
• #23: What if we have data as a JSON array and we need to use it as a row result set? Consider this example where we have shopping lists as JSONB documents and the items to purchase are stored as a JSON array.
• #24: We can expand JSON arrays as rows with jsonb_array_elements_text() for display in a checklist or other item iterator collection.
• #25: What if we have another row and query the table resulting in more than one row matching? jsonb_array_elements_text() has collected all rows’ item arrays and turned them into a result set of 6 rows. There are two cheese curls value rows as both shopping lists contained that value in the items array. What if I limit 1 SELECT distinct on ( item ) shopping_list_id, jsonb_array_elements_text(shopping_list_doc->'items') AS item, shopping_list_doc->'listName' as name FROM shopping_lists ;
• #27: For the somejson table with 100101 rows, the cost is 0 startup cost, and 4671.77 to retrieve every matching row, with an execution time of 366.303 ms
• #28: Using a JSON text expression index for zip code, the cost is now 12.18 startup and 1317.64 cost to retrieve all rows instead of 4171.32 units, with execution time down to 5.214 ms from 366.303 ms - a big execution time savings.
• #29: On a the JSONB version of the address document table with 10104 rows, the cost is 0 startup cost, and 4171.32 to retrieve every matching row, with an execution time of 64.235 ms JSONB query plans in this example are already measuring faster without an explicit zipcode expression index.
• #30: What if we add the JSON selection value expression index, the same as we use in the where clause, Cost has changed from 0 startup cost, 4171.32 to retrieve all matching rows with 64.235 ms execution time down to 12.22 startup cost, all row retrieval cost at 1253.10 units which is over a one thirds savings at 1265.32 total cost, running 10 times faster at 6.161 ms execution time Less planner cost than JSON column expression index at 1329.82 units for the somejson table, but slower than 5.214 ms
• #31: What about if we need contains expression efficiency? Generalized Inverted Indexes and the JSON Contains Operators are often faster especially in situations with complex JSON documents and many rows. With the AT ARROW @> left contains right operator, we already see the cost to find the same rows with zipcode 11100 has gone from 4171.32 to 3665.80 with no startup cost, 64.843 ms execution time You might be asking, can we make that base line more efficient with an index?
• #32: The answer is - ussuuuuaaaally! We can add an index to support our where clause that is using a @> left contains right operator on the JSON document for selection. Let’s say we know that our application is usually doing WHERE clauses with the left contains right operator for zipcode match clauses, We specify a GIN index with the jsonb_path_ops operator class to limit the index to the @> left contains right operator and its inverse. This makes it a more size efficient index compared to creating a GIN index without the jsonb_path_ops operator class specifier. This takes our cost from 0 startup and 3665.80 for full retrieval down to 12.78 startup with only 349.75 cost to retrieve all rows. That’s a total cost of only 362.53 to get all matching rows for the 11100 test zipcode in 6.800 ms. That’s 10 fold query cost and time savings - the best overall as measured in our example index variants.
• #33: And now are you asking yourself - in this particular case did we just prove that a string element selection expression is faster than a Generalized Inverted Index ? Yes we did - but at what cost! AT WHAT COST! Remember, the somejsonb table has 101104 rows in it, taking up 19 MB of space. The _zipcode index is 2232 kB in size and only indexes the value of the zipcode selection expression. The _gin index is 1680 kB and indexes all json paths to their values for use by the contains operators. So the GIN method is half a millisecond slower but takes up only 75% of the space, and can be used by contains operators for any path value expression comparisons for the whole JSON document. Optimization through indexing all depends what your application needs to query efficiently.
• #34: The examples so far have been contrived to give you a sense of how JSON data types and operators work. If your task is to intake JSON documents and process them, then those examples certainly apply. However, I also want to make sure I review something important for your implementations: And that is that I do NOT recommend JSON for primary storage of application data. For system of record storage, I advocate to use normalized relational design whenever possible. That’s not to say that PostgreSQL should not be the means by which you transform your data to JSON, so let’s go through an example of how you might do that.
• #35: This application database has users, and users post blog entries, and interested users come read them, and we keep track of that. This is the basic users table with user id, name, rank, display name, and profile. And here ... ADVANCE SLIDE
• #36: … is the post data model and how we keep track of how has read what posts. For the sake of this example, user_id’s are emails and post_id’s are slugs so we can readily digest the data without a reference sheet. REVIEW SLIDE Let’s go see how we can create JSON views of this data for use by the blog application presentation layer.
• #37: This is what a view might look like to select user details as a JSON document.
• #38: This is what a view might look like to select post details as a JSON document.
• #39: And a select from that post JSON view.
• #40: Those are basic views. How about a view of posts that a specific user has not read yet? REVIEW SLIDE Let’s try this view.
• #41: There are two posts in the system, one by Beth and one by Rick.
• #42: Here are the posts that Morty has not read yet. He has not read any of the two posts in the system.
• #43: Here are the posts that Rick has read. He has read all two posts in the system.
• #44: Here are the posts that Beth has not read. She has read her own post. Rick’s new space van post is still unread to Beth.
• #45: What if you need to optimize or determine which view is faster? Consider this alternate view definition that uses INNER JOIN ON NOT IN instead of CROSS JOIN WHERE NOT EXISTS EXPLAIN ANALYZE IN TERMINAL CREATE VIEW unread_posts_v2_json AS -- users that have not read posts -- alternate version SELECT ur.*, row_to_json(ur, TRUE) AS json FROM ( SELECT posts.post_id, posts.post_date, posts.post_summary, users.user_id -- take the cross product of users and posts FROM users INNER JOIN posts ON ( -- filter out the ones that exist in posts_read (users.user_id, posts.post_id) NOT IN (SELECT user_id, post_id FROM posts_read) ) ) AS ur; SELECT COUNT(*) FROM posts; SELECT COUNT(*) FROM posts_read; SELECT COUNT(*) FROM unread_posts_json WHERE user_id = '[email protected]'; EXPLAIN ANALYZE SELECT json FROM unread_posts_json WHERE user_id = '[email protected]'; EXPLAIN ANALYZE SELECT json FROM unread_posts_v2_json WHERE user_id = '[email protected]'; The query planner costs are similar for the inner loops, but lower for the hash anti join versus the loop, this is because some rows can be dropped during first pass filtering. This query profiles better, and may be the better choice to go live with. Need to do more testing to be sure.
• #46: There is one limitation I know some of you may be aware of that I would like to address. And that is that There is no in-place editing of JSON elements in 9.4 In PostgreSQL 9.5, the jsonb_set() function allows you to replace sections of a JSON document in place. Look into jsonb_set() usage for more details on this. In 9.4, one approach to do in-place updating is to append child elements with updated values and allow the storage behavior of JSONB last-element-wins behavior to update your element values. There is also always the rebuild the JSON document approach where you read and modify JSON document and overwrite the previous version entirely.
• #47: Can you foreign key JSON documents to another table? (check this) You can for anything that supports a GIN index, so you can with JSONB