Dev Jumpstart: Schema Design Best Practices

Editor's Notes

• #2: HELLO! This is Buzz Moschetti at MongoDB, and welcome to today’s webinar entitled “Schema Design” If your travel plans today do not include exploring schema design with MongoDB then please exit the aircraft immediately and see an agent at the gate Otherwise – WELCOME ABOARD for about the next hour.
• #3: There are three themes that are important to grasp when thinking about schema design in MongoDB and these will be reinforced throughout the presentation. First, great schema design…. I’ll repeat it: great… This is not new or something revolutionary to MongoDB. It is something we have been doing all along in the construction of solutions. Sometimes well, sometimes not. It’s just that the data structures and APIs used by MongoDB make it much easier to satisfy the first two bullet points. Particularly for an up-stack software engineer kind of person like myself, the ease and power of well-harmonizing your persistence with your code – java, javascript, perl, python – is a vital part of ensuring your overall information architecture is robust. And WRT legacy issues, we need to confess to ourselves that even after 40 years, RDBMS design, software integration, and day 2 change still have many hurdles like field overloading and flattening Boils down to success = “schema” + code
• #4: Potentially very provocative. In the old days, we had long requirements cycles and one or both of the following things happened: Requirements changed midstream, invalidating a lot of schema design already performed especially for certain 1:1 entity relationships High quality problem: The solution was asked to scale well beyond the original goals Increasingly, it is no longer acceptable to have a 6 or 12 or 18 month requirements phase to build a system that lasts 10 years or more. You want to get the initial information architecture done in a couple of weeks, expand upon it incrementally, and build a system that doesn’t have to live 10 years to “justify its development time.”
• #5: Because MongoDB is impedance matched to the software around it, you have a great deal of design flexibility and even innovation in the way you construct the overall solution. Certainly, there are best practices to guide you to a particular design choice, but in most cases, you still have the ability to change course without a complete rethinking of the solution.
• #6: Let’s talk about some the terms. JOINS: RDBMS uses Join to stich together fundamentally simple things into larger, more complex data. MongoDB uses embedding of data within data and linking to produce the same result
• #7: A document is not a PDF or a MS word artifact. A document a term for a rich shape. Structures of structures of lists of structures that ultimately at the leaves have familiar scalars like int, double, datetimes, and string. In this example we see also that we’re carrying a thumbnail photo in a binary byte array type; that’s natively supported as well. This is very different than the traditional row-column approach used in RDBMS. Another important difference is that In MongoDB, the data is the schema, and it is expected that the code layer above MongoDB is responsible for all validation of content, types, and shape. Truth is in most non-trivial systems, even with RDBMS and stored procs, etc. plenty of that validation and logic is being handled outside the database; the MongoDB paradigm is to put it all there. Now here is something very important: For the purposes of the webinar, we will be seeing this “to-string” representation of a document as it is emitted from the MongoDB CLI. This is easy to read and gets the structural design points across nicely. But make no mistake: you want most of your actual software interaction with MongoDB (and frankly any DB) to be via high fidelity types, not a big string with whitespaces and CR and quotes and whatnot.
• #8: Drivers in each language represent documents in a language-native form most appropriate for that language. Java has maps, python has dictionaries. You deal with actual objects like Dates, not strings that must be constructed or parsed. Another important note: We’ll be using query functionality to kill 2 birds with one stone: To show the shape of the document To show just a bit of the MongoDB query language itself. Note that in MongoDB, documents go into collections in the same shape they come out so we won’t focus on insert. This is a very different design paradigm from RDBMS, where, for example, the read-side of an operation implemented as an 8 way join is very different than the set of insert statements (some of them in a loop) required for the write side.
• #9: Let’s get back to schemas. … This is largely because the design goals and constraints of legacy RDBMS engines heavily influence the data being put into them. Overall, the platform is focused on the physical representation. For example, although most have been conflated, the legacy types of char, varchar, text, CLOB etc. to represent a string suggest a strong coupling to byte-wise storage concerns.
• #10: Documents, on the other hand, are more about business entities. You’ll want to think of your data moving in and out as objects, even though the analogy is not – and should not – be 100% true because we’re only dealing with data here, not in memory state and certainly not methods and behaviors. No compile time binding either!
• #11: So what better way to show this than diving right into a problem?
• #12: Today we’ll explore data structures and schema for a library management application Good example because in general most of you have some familiarity with the entities involved and we can explore some 1:1, 1:n and other design options
• #13: To make things a little more interesting, we’ll capture our emerging requirements in the form of questions the application needs to ask the database. This is useful because, again, it helps us keep our eye on the software layer above the database Imagine a patron walks up to the counter and presents his/her library card to check out some books. The first thing a librarian might want to do is confirm the patron’s address so as to have a place to send the library police when the book isn’t returned in a timely manner.
• #14: This might be the very first cut at such a thing, with 2 collections, patrons and addresses. Both share a common key space on the _id unique ID field. To get name and address will require 2 queries. Very important: You notice here that a list of favorite genres has been set up. It is a LIST of strings. Not a semicolon or comma delimited string. Each element of that list is completely addressable, queryable, and updatable via dotpath notation. As is the entire list. And should we require it, it is indexable. But let’s not get too far ahead. Can we do better?
• #15: I think we can: embed the address substructure into the patron document. Only 1 query is necessary. You notice here that we treated address like an Object. We did not “faltten” everything out into addr_street, addr_city, addr_state, etc. We’ll come back to that again in just a few slides and see the power it provides.
• #16: A common concern that pops up at this point is that every query will act like “SELECT *” and drag back potentially a LOT of data. You can control that with projection as we see here. This operation is of course processed server side and is efficient. Note that we can both pick up an entire substructure as a unit OR JUST specific subfields using “dot notation”. Very powerful.
• #17: Let’s summarize one-to-one relationships. Read performance is optimized because we only need a single query and a single disk/memory hit.. MongoDB is especially useful in dealing with 1:1 belongs-to relationships Not every non-scalar datum is going to force you to create a new table, a foreign key, and key management.
• #18: Business Requirements Change! People may have more than one address. And: we acknowledge that the legacy practice of assuming some small number of instances and flattening out addresses into address1 and address2 is bound to bump up against a situation where someone has 3 addresses. We’ve seen where that leads traditional schema design. We don’t want to repeat it.
• #19: Now, just model addresses as an array. That was easy. No need to create a new table, set up foreign keys, do key management, etc. We combine the power of arrays with substructures. At times it takes longer to describe the change than it does to implement it and have it available as part of the information architecture including querying and indexing.
• #20: No suppose this didn’t happen on day 2 but instead after the beta of the system was released. What about the old documents that had address, not addresses? There are 2 aspects to this issue. The MongoDB paradigm is that the code layer just above the DB is responsible for managing these types of things. The same code that is responsible for the higher-level business validation and verification is also responsible for selectively using (both on a read and write basis) the address or addresses field. There are several migration strategies that one could employ in conjunction with functionality at the code layer.
• #21: Migrate on release: If address exists, write into new array of size 1 of new fields addresses and remove address field.
• #22: Before I mentioned we’d see more about the power of substructure. Substructure is the concept that allows your schema to well-harmonize with software. By being able to pull out the entire address structure – either as a single field or part of the larger array of addresses, you can pass it to a function that does something interesting with no matter what fields are or are not set. And what’s really great is that these substructures are generic Maps or dictionaries – NO COMPILE TIME dependencies!
• #23: Moving on, let’s look at another relationship in our emerging information architecture: publisher and books.
• #24: Shameless plug! Note that a book may have multiple authors….
• #25: Here’s a first cut at the design. Here, the 1:! Belongs-to relationship between book and publisher is not nearly as strong as patron to address But for the moment, let’s choose to duplicate publisher in every book that the publisher has published. Data duplication is OK because the publisher is largely immutable.
• #26: Here’s an even better one: making the author name more granular by making it a substructure with separate first and last name elements. In a slightly different problem space, like real estate, imagine if you have lots of names: buyername, sellername, lawyername, executorname, guarantorname.
• #27: This design will permit very high speed display of relevant book information. And we can still quickly look up all books for a given publisher.
• #28: Suppose publisher info is not so static. Maybe we want to take a more traditional route and separate publisher information out into it’s own collection. We need to assign _ids and we’ll use simplified examples here. Note that in MongoDB each document needs a unique _id but it’s not important for today’s talk to get into the advanced topic of _id construction If you don’t supply one, one will be created automatically. And here’s a teaser: _id does not have to be a string or number or scalar: it can be a multi-field substructure! Note that the belongs-to 1:1 relationship of locations carries over nicely without fuss.
• #29: It now takes 2 queries to pull the information. Now this isn’t the end of the world. It’s effectively no more code work than a JOIN. Instead of decomposing a resultset or passing the result into an ORM, your code layer deals directly with the database.
• #30: If we want to get more than one book, no problem. Get the books and as you’re processing the book data, capture the publisher ID. We store it in a map to remove duplicate IDs Then we turn around and do an efficient lookup by primary key into the publishers collection. This works well regardless of whether the list is 1 long or 10,000. This is effectively a JOIN. An interesting thing here is the network hit. For relatively low numbers of publishers WRT to books, coupled with a good amount of publisher information, the network hit will be LOWER with this logic rather than a traditional JOIN! This is because the JOINed publisher data is NOT repeated over and over again in the result set.
• #31: We have already seen a way to do this : by performing a find() on the Books collection with a particular publisher name OR publisher ID. But is there another way?
• #32: As much as arrays are very useful in MongoDB, beware of the temptation to treat them as “rows.” Arrays should be “slow-growing” and not without a reasonable bound. For example, you wouldn’t carry system of record customer transactions as a array inside the customer. You MIGHT, however, for the purposes of speed at scale, duplicate the last 3 transactions as an array in the customer record – but that array is not growing without bound. From an underlying technology basis, this is less of an issue now with the WiredTiger storage engine available in 3.0 but you should still be mindful of it.
• #33: As our schema is evolving, we’re beginning to see that an increasing number of questions can be efficiently answered by existing set of entities and relationships. This holds for authors; recall we set up the substructure of first and last names. But suppose we want more detail on an author?
• #34: Let’s go ahead and create the Authors collection and link to it with a Author ID. Yes, we are using some fancier code here to extract the author IDs but in the end, it is still only 2 queries. The authors documents contain all sorts of information about each author. We have left the first and last names in the books document for the purposes of speed But we could get rid of them if we wanted….
• #35: With this next question, we’re starting to get into the world of tradeoffs for performance and speed.
• #36: Here’s one way to do it: Acknowledging that most authors are not steven king or danielle steele and have something of upper bound on the books they’ve written, we can doubly link the books and authors collection. In general the data is static so referential integrity is not as much as an issue as performance. Again, we’ve kept title in the stucture within the books field for performance purposes (to keep it to a single query) when looking up basic info about an author. But if so desired, we could just have books be an array of book IDs and use a 2-query approach as we’ve seen in prior slides.
• #37: Remember: all fields including those inside structures inside arrays are indexable.
• #39: Suppose the library wants to add value by having book clubs enhance the authors information with bespoke information. There are at least 3 challenges here: 1. the book club is operating semi-independently from the main system designers Clearly, we don’t know early on what the book club is going to add Each author may have a different set of information, create by a different member of the book club.
• #40: This is a very, very exciting part of MongoDB. No need to come up with userDefined column 1, column 2, etc. We see here that Kristina and Mike have very different substructures inside the personalData field. We call this polymorphism: the variation of shape from document-to-document within the same collection. The library application logic only is looking for a field called “personalData”; actions will be taken dynamically based on the shape and types in the substructure! For example, It is a very straightforward exercise to recursively “walk” the structure and construct a panel in a GUI – especially if you are using AngularJS and the MEAN stack (MongoDB / Express / Angular / Node.js ) No need to use XML or blobs or serialized objects. It’s all native MongoDB -- and documents are represented in the form most easily natively manipulated by the language Every field is queryable and if so desired, indexable!
• #41: Polymorphism is such an important topic it deserves another slide, even if it is independent of the library model. Here’s an events collection. We’ve taken out _id for some extra space. Type and ts and data are the well-known fields, and there is probably an index on type and ts. But the data field contents are specialized for each type. No need for complex tables or blobs or XML. And very easily extensible to new types we may need to add on day 2. This model extends to products, transactions, actors in a system. If you think about your data in terms of entities, you’ll discover that you probably only need a handful. That plus polymorphism within the entity gets you a long way. Because of this design approach, rarely will a MongoDB database have 100s of collections
• #42: This is an example of faceted search or tagging in general.
• #43: We can begin simply by implementing Categories as an array We probably will only have a few and they don’t change very often and have a natural upward bound. An index on categories will allow rapid retrieval. Again, a very appropriate use of belongs-to 1:1
• #44: If we want a little more hierarchy, we can use regex operations. The category field is indexed as before, but now as a string Uses an index because of the anchored regular expression. There is a downside: when a category hierarchy gets changed, all documents will need to be re-categorized.
• #46: On behalf of all of us at MongoDB , thank you for attending this webinar! I hope what you saw and heard today gave you some insight and clues into what you might face in your own schema design efforts. Remember you can always reach out to us at MongoDB for guidance. With that, code well and be well.