Bulletproof Backend Architecture: Building Adaptive Services with Self-Describing Messages - Sean Travis Taylor, Redeem

Editor's Notes

• #1: Hello, good afternoon and welcome. My name is Sean Taylor. If you’re here for a talk titled Bulletproof Backends you’re in the right place. If you are not here for a talk titled Bulletproof Backends you’re in the right place. You cannot leave. This talk is about something I call adaptive services. What are adaptive services? Adaptive services are services that can absorb change. How many of you or your engineering teams have grumbled at new or emerging business requirements that require change? Then you are part of the problem. But my name is Sean and I’m here to help. First: who am I? When I’m not cross-hatching the globe giving talks. I’m Head of Platform at a small startup called Redeem. Redeem is a personal fintech platform; our lead product is a Chrome Extension that automatically recommends the best credit card to use online to maximize your benefits, reward and offers. But I’m not here to talk about that. Let’s get on with adaptive services. (Next)
• #2: Now for what it is. I’m going to make a modest proposal for changing the way you think about how web services should communicate with each other using something called Self-Describing Messages. (Next)
• #3: I’m going to demo two services communicating with those self-describing messages with a real world business case. The business case is going to show how a service can radically change the contract in real time without affecting any consuming services. I’m going to show how building services in this way can reduce developer toil and kill meetings…which is the real point of this talk. (Next)
• #4: Raise your hand if you like meetings. (Next)
• #5: First let’s map the current terrain. Starting right at the top. So, why do web services break? Internal Errors: (also known as bugs) Network Errors: …or conditions of all kinds that may beyond our control Authentication/Authorization Errors: maybe a requestor doesn’t have the right permissions or credentials to access a resource) Breaking changes: something somewhere has changed and something else broke in response. For this talk we’re focused on those breaking changes. So let’s drill down a bit further. (Next)
• #6: New Required Field: A field that was previously optional is now required. Maybe we’re requiring first and last name of the customer where before we only needed the first name. Changed Message Structure: Changed field names or locations. Maybe the customer first name is now nested under a customer object. Maybe firstName becomes givenName and lastName becomes surname. Changed Meaning: Changed field types. Maybe the shopping cart is now an array of objects containing information about each item instead of a set of key/value pairs where the key is the item id and the value is an object with product information inside. All of these are changes that would break our service clients. (Next)
• #7: Let’s talk about message structure and its importance. I contend the structure of messages is of less importance than the meaning of messages. The structure of the messages is an implementation detail–an important implementation detail but an implementation detail that we are not the best at addressing in current web service architecture. In fact our typical approach to handling this detail is binding the object models of clients and servers, which guarantees system fragility – a change to one requires a change to the other. How many of you are integrating multiple third party APIs into your services? How many of you are creating mappers or mapping layers to reconcile different external data formats? Once you bind directly to an external data format, you’ve given up the game. Your calendar is full of meetings.
• #8: Here are some examples of messages that share the same meaning but have a different structure. Sales receipt from a hotel restaurant Sales receipt from a convenience store Email receipt from a online purchase Despite their structural differences all these messages mean the same thing–they are proof of a transaction for the purchaser’s records.
• #9: Humans are good at filling in the semantic gaps between messages that mean the same thing but are structured differently. Machines less so. Systems today cannot do this an any meaningful way. This makes them brittle. Services designed to address message semantics are more flexible than services designed without these considerations. (Next)
• #10: What is a self-describing message? It is a message that contains all the data needed to process the message within the message itself. These self-describing messages are key to system adaptability. More on these later. This image of a package offers a great example. All kinds of information: where the package is going and where it has been there may be hints about what’s inside (e.g. fragile or perishable goods) or how it should be processed (e.g. urgent) which guide us in processing the message there is tracking information that allows us to get more data about the package from the courier’s system We glean all this without knowing what’s inside the package or understanding how to process the package ahead of time.
• #11: Now for a small example and a demo. I call this a Tale of Two Companies. We’ll explore a pretty common business problem and we’ll see how self-describing messages can provide a solution.
• #16: This is actually a very typical problem: an unstable API a communication problem with a partner failure to build consensus on implementation approach
• #18: JSON Patch is a formal specification for describing changes to a JSON document. We start on the left with a candidate object. Our JSON Patch-aware web service uses the patch instructions in the middle to produce the desired object shape on the right. What’s great about JSON Patch is that it’s just JSON. This means it can be serialized and sent over wire. This also means we can provide a link to this JSON Patch documents in our messages. Can you see where I’m going with this?
• #19: In the demo we’re going to see the Muzak backend contacting the Metamusic API. We will see all (3) of the API formats returning from the Metamusic API in real-time at random with no logic that inspects the shape of the response payload. The only help we’ll get from the Metamusic API is a link to the JSON Patch spec for the incoming response payload. Muzak will fetch this patch document and apply the patch to transform the response payload–transforming a volatile system to a stable one.
• #20: So we’ve discussed what Self-Describing messages are and how they contribute to more robust service design. We’ve seen JSON-Patch documents in action and watched as Self-Describing Messages keep our services online even in the face of change. I hope you’ll consider this design in your next application and that it will save your dev team some toil and maybe even get them out of a few meetings!
• #23: This is not a hypermedia talk. Though hypermedia is in the title, this is actually the second-to-last time I’ll mention this concept. I find it fascinating but I think because of a lot talks that discuss this concept can get abstract, they end up boring people…kinda like Functional Programming talks. (Next)
• #24: Communication is in the meaning of the message. Diverse structures of messages with similar meanings are around us all the time.
• #25: On the far left, the candidate is obsessed with the beauty and majesty of the camel and believes the first, best and only way to implement an API response is in camel case. In the middle, the candidate feels an API should be implemented by way of vibes; there isn’t a whole lot of rhyme or reason to any of the field names. On the far right, the candidate is obsessed with the philosophy of Blaise Pascal and believes the first, best and only way to implement an API response is in Pascal case.
• #26: Here is an example of the reverse. These two objects share same structure but a slightly different meaning owing to the type change of the cart field.