MongoDB.local Dallas 2019: Pissing Off IT and Delivery: A Tale of 2 ODS's

Editor's Notes

• #3: Hello everyone My name is Scott Jones and this is E d Bozovsky. We are with Acxiom Corporation. We are here to talk about our journey of modernizing our Operational Data Store, ODS, using MongoDB technologies while at the same time pissing off our IT and Delivery, a little :) I will talk about our journey from the old way to the modern way. Ed will talk about specifics around how we utilize Stitch in our new ODS.
• #4: A little back ground on Acxiom LLC Acxiom specializes in making sense out of all our client’s data and enabling that data for use in multiple solutions. Acxiom has been a leader in identity and ethical data use for more than 48 years. Acxiom provides the data foundation for the world’s best marketers. Acxiom makes marketing relevant
• #5: We will walk you through our journey from using RDBMs to using document data store for our Solution <click> We will review how we did it before <click> To making the decision to take the green pill and change our ways <click> To some of our pain points and the separation anxiety that IT and Delivery had as we progressed <click> To full enlightenment.Speed, flexibility without the brittleness
• #6: Let’s go over our specific business problem we are trying to solve. <click> We manage marketing system for our clients. We start with ingesting data from their various systems. <click> We map those data files into standard models / interfaces, by industry, and load the data into a warehouse <click> From the warehouse we create marts for specific needs. Each mart is a set of tables that are built for a predetermined need <click> The most common marts enable third party campaign tools and third party reporting tools <click> The other common area is to make a subsection of the client's warehouse and/or marts available for real-time access.This is what our team works on. <click> The major issue with doing this is how we can have Common Set of code to support multiple clients Allow customization of what data is exposed by client And How to create a standard API around this use case
• #7: So I am going to go into how we did this in the past and how MongoDB’s technology has allow us to modernize our system
• #8: <click> The traditional approach was to load the needed data into its own database instance.So we have an example here of exposing loyalty data <click> You will notice that we have designed our database schema to have extension tables. This allows each Acxiom client that implements this solutions to have extra attributes on our standard data model. Here we have 3 extra attributes for our first client <click> We put an API layer to access the data. This layer needs to translate the database schema into JSON schema. This layer also has to be able to handle the extension attributes in a general way. <click> We then have to open up firewall to allow the client apps to hit our API <click> To get all of this accomplish we engage with our IT and start opening tickets Deploy DB <click> Deploy Code <click> Deploy Firewall <click> Then we get our second Acxiom Client that want our solution. We have to design their extension table for the extra attributes they need. Then we open up another set of IT ticket to get items deployed
• #9: How can this scale? How can we make money if every install is taking so many people to implement How can we support this if every install is just a little bit different
• #10: Let’s start talking about how we transitioned to the new technology The first step was to look at the access patterns in all of our current solutions. <click> You will recall that we take in files from multiple systems and standardize them into our warehouse. <click> For example we could take files from systems for Loyalty, offers, accounts and purchase But what happens if our client has multiple systems that have conflicting keys <click> Here is an example of two loyalty systems that use sequence numbers as their keys. If we we try to load both into our warehouse we would have a conflict So how did we fix that in our systems? <click> We add the source system as part of the key. We call the key that comes from the client’s system the “Natural Key” Why do I talk about all of this? It is because one of the main access patterns to our system is for the client to call us with a key from their system and ask us to return related data. This is what we call a HardKey lookup. It is just reading data by a primary key that is already known.
• #11: The other main access pattern is to use PII to retrieve data. (Name, Address, Email and/or Phone combinations) <click> Again we take in files from all of our customer systems. We need to be able to determine the same individual across their files. <click> So are these two records the same individual or different individuals <click> Acxiom has a unique identity graph that will convert PII data into a canonical ID / Key. This identity graph will only handout a canonical ID. So if I give it my wife’s maiden name at our address and my wife’s married name at our address - it will return the same canonical ID / Key. <click><click> We utilize this identity graph in most of our systems to lookup data. It is a much simpler pattern because we have centralize the system for converting PII to Canonical ID. So most other systems just have to store the canonical ID in their data store. We add this canonical id to records in our warehouse. Acxiom calls the id a consumer link.
• #12: So let’s give real world examples of these to pattern <click> One very common approach is looking data up by a key that was handed out. Our clients will send out offers via email <click> When the consumers click on the link. <click> The link contains a key that is used by the client system to lookup the specific data So this is what we call a hard key lookup. Lots of our current clients utilize this method to pull up offer or account information Basically they have the key we have the detail data
• #13: <click> Our clients have systems that either collect PII information or already has the PII information. <click> The client system will utilize this information and <click> send it to our system to lookup other data. <click> Our API sends that information to our identity graph system and gets an canonical id back. That id is used to lookup data in the ODS 
• #14: So we have two usage patterns Lookup by hardkey Lookup by canoncial id / consumerLink <click> Our main collection / document can look something like this. <click>hardkey as the PK You will recall our hardkey/primary key is call natural key and is a compound key <click> canonical id as a foreign key Our canonical id is called consumerLink and we allow multiple of them per record. This allows us to associate n number of individual to one source record.For instance a loyalty record can have both my wife and I on it <click> Looking at this document / collection, do I really care about anything else? Not really. The plus about documents stores is that each document in a collection can be different. <click> So here is a structure that stores loyalty information.It has all the key information in it, plus a bunch of other stuff.Our APIs don’t care about the other stuff, it can just return it. This is the great thing about having a JSON Document store. It is the same structure as what our REST services returns. <click> Here is the structure of contact histories.It has the same keys structure but addition attributes for offers
• #15: So with our identification of the usage patterns We created two main APIs to fulfil the patterns <click> A hardkey lookup that allows the end user to pass in a key to retrieve data. This use the natural key as the lookup <click> A PII Lookup that hits our identity graph, converts the pii to our canonical id / consumer link and looks up documents by that id <click> Because a collection can have different structures per document, we added a type attribute to indicate the type of document. The client tells us the list of document types they want and we filter the search by that. We just return the documents that meet the criteria passed in . The API does not care what is in the document, as long as the keys structure is the same Each client can define loyalty or offers the way they want, no changes to the plumbing
• #16: So how did we pick Mongo <click> We started three years ago using MongoDB as our metadata store for our overall marketing system. We had great luck is using a document store for administrative data. We spent much, much less time on the API to maintain our systems <click> A group of us came to Mongo World Last year <click> We went to sessions on Atlas. We had not considered using MongoDB’s cloud offering up to that point <click> Then we went to a session and key note about Stitch <click> Well we came home and said let's try that. I had already done a white paper on how we might build a new ODS system using a document store. But I had not thought about using Atlas cloud deployment and I knew nothing about Stitch. So in one week we build a full working POC I did the work in Atlas, getting sample data Karen did the work in Stitch, getting our APIs create She built a hello world She added a call to Acxiom’s Identity Graph system She was even able to incorporate our security system At the end of the week the system work. A few weeks later we had a new customer that request an ODS along with their larger marketing environment. So we said why not try it. It just worked
• #17: Now we are going to transition to the details around using stitch for our project. Karen Wilson, took on the POC from day one. She had been our hero when comes to all the stitch coding <click> Some days it has been a bit of a beast <click> But most day she has been riding the wave and getting stuff DONE!!!
• #18: So now I am going to switch over to the details around using MongoDB Stitch for our solution Stitch is Mongo’s backend as a service offering
• #19: So what have we done normally to start development Normally – Open Ticket to set up an api server Open Ticket to set up a database Open Ticket to poke holes in the firewall so the machines can talk to each other Open Ticket install libraries Open ticket to have DB installed
• #20: This process is then repeated for each environment we have to set up Development Test integration production
• #21: But what if we didn’t do what we’ve always done? IT is busy – and if we can reduce or even eliminate our reliance on them for a lot of these tasks, they can focus on their primary responsibilities and we can focus on just writing the code, instead of having to manage a bunch of tickets/requests
• #22: So now I am going to switch over to the details around using MongoDB Stitch for our solution Stitch is Mongo’s backend as a service offering
• #23: So let’s try something different – what would that look like? With Atlas and stitch stitch We expose our APIs in Stitch via the cloud The APIs call Acxiom for services when needed APIs call into Atlas – <C> but with the tight integration between Stitch and Atlas, it’s kind of like them being in the same area
• #24: What does this mean for us? It means we don’t have to spend our time trying to develop strategies for things like <C>Server maintenance The systems are cloud based and managed by Mongo<C> <C>Firewall requests They’re already in the same place, they automatically know how to talk to each other<C> <C>Build & deployment infrastructure Because the systems are cloud based, this is all handled<C> <C>Disaster/recovery planning No need to set up redundant systems. With Atlas automatic backups are enabled. And Stitch’s export mechanism makes it easy to ensure you always have the latest saved off somewhere (bitbucket, etc)<C>
• #25: So what is stitch? It is a backend as a service that integrates with mongo’s DBaaS Atlas. Allows a tighter, cleaner integration with mongodb while elimination a lot of the overhead with developing separate applications. Several components <C>Values Allow you set up client specific configuration items or things that may change from client to client <C>Access rules Allows you to create functions or specific queries that determine whether users can view specific collections, documents, or even fields within a document <C>Functions Javascript-based capability to interact with mongo, services, values <C>Webhooks Allows users to get into the system from outside/use for an API
• #26: One of the things we use values for is mapping acxiom user profiles to stitch users and then specifying what those users have access too. So we have something that inspects the token the user logs in with and gives us their acxiom user name (acxiom_user)<C> We can then look and see the associated stitch user id <C> and then we can also set a pretty label for their name<C> Allowed_sources <C> takes the stitch user id – and lists the document types <C> that user should have access to, we can list the types the user has access individually or use key word of all or none.
• #27: Access rules can do all sorts of things Ours uses a function We give it the documents type And the user It returns yes/no – should that user see that document type – based on the setting discussed in values
• #28: Here is one of our functions <C>It gets the ‘authentication’ field from values <C>Looks for the special cases of ‘all’ or ‘none’ <C>Looks for this specific document type
• #29: 3 webhooks /sources – this is a basic document lookup – GET /sources-by-pii – this makes an API, then does a document lookup – POST /feedback – this is a way for users to add data to the document store – POST All of our webhooks response with result, you could opt not to if yours were just to trigger something like a counter increase We use run webhook as -> script to have the logs show which user executed the call (this uses the user mapping in values)
• #30: With MongoDB Stitch you are provided with endpoints for each service, but they are long and complex So we created a proxy that would take a simple URL and rebuild the complex version of Stitch’s URL API Gateway to forward ods.myacxiom.com traffic Lambda to create Javascript that would rebuild URL S3 to store swagger JSON contracts At the same time we are able to validate the API JSON (Query parameters, header rows, and the body)
• #32: canonical ID = abilitec ID
• #34: As we told you earlier we want to have a single set of code for all of our clients. We have common exit code at the start of each business function to allow extra validation – Ensure company code is included in body We have common exit code at the end of each business function to allow extra customization – Filter certain affiliate codes even though they are stored in MongoDB
• #36: One developer person working on the API was Karen Wilson One Data scientist creating and loading the data was Scott In one week we had a basic POC Hello API call Document store lookup Called out to an external API Document update
• #37: We had Drastic performance improvements <C><C> Latency was reduced by 1/3 26 million documents loaded We set off 35 simultaneous request threads And a data load <C>Data Load testing showed we were able to load 17,000,000 documents in just over 13 mins – a rate of abt 21.5k documents per second – and this has even been improved upon
• #38: We started the POC in July and by December we had our our fully fledged system up and our first production client
• #39: 5 clients running on one code base
• #40: 5 clients running on one code base
• #41: Custom authentication not available for webhooks Debugging can be a challenge Stitch is still young Big changes by them == Big changes needed by us Not a lot of documentation Stitch’s implementation of MongoDB commands are limited compare to standard MongoDB User Management can be tricky MongoDB user Atlas User Stitch User Acxiom User
• #42: IT gets to do other things besides managing our tickets We are adding new features … Asynchronous API calls streaming to/from Acxiom’s UDL Over the last year we have needed only 3 developers working on this system. Recently we have added 2 additional developers and 1 tester
• #45: Database Triggers Authentication Triggers
• #46: So what about all the infrastructure <C>Disaster Recovery <C> Complete <C>Backups <C> Complete Upgrades <C>
• #47: Maintenance and upgrades require minimal downtime Migrations from 1 node to another are dependent on the size of the MongoDB Atlas size but generally its minutes Upgrades are EASY Stitch Code typically takes 1 to 2 minutes and require 0 downtime from customer The system allows to set alerts for thresholds so you know when it’s time to make upgrades There are several installed on startup You can also setup custom alerts
• #48: Less Reliance on IT No server setup required No firewalls to be configured A new client can be configured and loaded in about 20 minutes
• #49: Increased profitability Because less people are needed, overhead costs are lower, and profit margins are greater
• #50: Born on June 2017
• #51: Multi-environment code management dev, test, cert, client environments. You need to plan on how you are going to manage this User Management can be tricky MongoDB user Atlas User Stitch User Acxiom User Multiple developers working can be tricky -- We now create an environment for each developer on our team