Modern Databases for Modern Application Architectures: The Next Wave of Design, Development and Delivery

Editor's Notes

• #2: Thank you for joining today. This should be interesting, since it is not the typical webinar... We are going to review the five trends that precipitated the change in application design, the four major outcomes, the need for an alterative approach to data and the four gaps these changes have created in people, process and technology. The next wave of design, development and delivery is upon us and yet most are ill equipped. Worse yet, they do not know it. In order to chart a course forward, it helps to know how we got here. Once you understand the makeup of the people, processes and technologies at play, acting decisively is easy.
• #3: Now, I am making some potentially provocative statements and so, I’ll spend just a few minutes introducing myself. My career started at IBM where I spent over a decade leading teams as a developer and architect; Building all sorts of compelling innovation that changed the face of the company; I left IBM as a Distinguished Engineer having most recently influencing the cloud strategy. JPMorgan was another interesting adventure, where I built an organization around advanced and predictive analytics. Let your imagination run wild and that’s about what we did. That brings us to MongoDB, where I am an enterprise architect, which means I am also a customer advocate, but better yet, a partner in understanding the technologies of the time and where MongoDB fits.
• #4: Most of the businesses doing most of the world’s business are trying to sort out the trend from the transform – there is a fundamental shift and we are going to explore this confluence and intersection.
• #5: Lets take a look at these four tends…
• #6: Open Source If you listened to the zealots a decade ago you would believe that there is no reason to ever pay for software, that it should be free. Thankfully for most software businesses, open source or not, there viability in software as a business. The impact of Open Source on the world was disruptive… Disruptive to the institution and Drives viable alternatives in the face of commercial conflict. http://commons.wikimedia.org/wiki/File:Free_Software_and_Open_Source_Software_Composite_Logo.svg
• #7: SOA Another massive trend … the minute there were web services, there was a need for some to package it all up into an architecture that helped position products. SOA is good. All of SOA is probably less so. It is very likely that some of you have been working on getting service oriented for a few years with little movement. It is hard to justify expense to a CIO when there is no apparent change in the IT. It takes some vision to believe the expenditure is worth it. That said, we are at a point where all modern applications demonstrate this level of solution design. http://blog.cobia.net/cobiacomm/2014/01/27/defining-a-service-oriented-architecture-soa-mindset-big-soa-or-small-soa/
• #8: App Stores The rise of the app stores where all the plumbing to write software to deliver, sell and manage gets rolled out creating an explosion of software producers – not engineers, just people building things. http://commons.wikimedia.org/wiki/File:Free_Software_and_Open_Source_Software_Composite_Logo.svg
• #9: Everyone is a builder. The maker movement is a huge deal … its relatively more recent, but underscores the fact that people are way more able than before and the technology and tools around us make it even easier to DIY.
• #10: Everyone may not be a software engineer, but there are many more programmers and technologies that enable people to build things that they use to pay someone else to do.
• #11: Lets take a look at this set of trends …
• #12: Virtualization Virtualization existing in mainframes and as notch software until … boom … everyone had that ability to carve up IT resources without having to managing the physical elements all of the time.
• #13: Automation In came automation … frameworks, workflow etc. … make things consistent, repeatable and less error prone http://www.marketingwhatsnew.com/2012/01/who-needs-marketing-automation/
• #14: Orchestration Add to the mix orchestration … the management of workflows and you get … https://store.sap.com/sap/cpa/doc/?uuid=00163E04FB111ED485C5E5FB3C13E4CB&revision=20120613&modified=20140728081429.8887260
• #15: Cloud computing introduces self-service hosting and more importantly a richer view of a platform.
• #16: And … technology dissemination is spreading faster than ever. This chart was created by Nicholas Felton of the New York Times – you probably have seen it before. It took decades for the telephone to reach 50% of households, but 5 years fro cellphones to accomplish the same penetrate in 1990. http://blogs.hbr.org/2013/11/the-pace-of-technology-adoption-is-speeding-up/
• #17: Things are speeding up, with fewer barriers to entry… Most everyone has a mobile device and increasingly one that can do most of what laptops do. http://blogs.hbr.org/2013/11/the-pace-of-technology-adoption-is-speeding-up/
• #18: Social software and social media rocked the way people we using the Internet. We went from a consumption model to a production consumption model. To the point where its hard to know if the hype is real or if it’s a side effect. One thing is for sure, the combination of Mobile and Social created an explosion of data (digital finger prints to photos to video to blog posts etc.) From forums to creating content to making a living out of being networked.
• #19: So, Mobile and Social are another set of collaborative trends contributing to the current landscape.
• #20: Lets do a quick review of the following trends – different space.
• #21: Parallel computing … The most exemplar expression is in super computers, where a massive amount of design thinking goes into the most powerful arrangement of hardware technology. Of course its hard to program in these environments … in fact it wasn’t until the last several years that you even had modern day applications leveraging multiple cores, let alone the Sequoia’s 20 petaflops / second capability.
• #22: Grid computing … moving data to compute Popularized by the search for extraterrestrial intelligence or computing for the greater good was something grids were useful for … however …
• #23: People started to realize this was only true if the work units were small enough and if you actually didn’t need access to do the computation. In walks Hadoop … moving the compute to data That at nifty programming approach (MapReduce) and a new generation of parallel computing is born.
• #24: Edward Tufte (Tuff tee) grew in popularity over the last decade and the design consciousness of data was born. There were certainly others, but Tufte’s influence is now seen everywhere.
• #25: Mainstream data visualization in the Atlantic magazine… http://cdn.theatlantic.com/static/coma/images/issues/201101/numbers.jpg
• #26: Infographics as shown in this example from EMC
• #27: Analytics is the final trend ….
• #28: These are the five clusters of confluence.
• #29: So, we started out with this statement … there has been a fundamental shift in application design … Someone might say, yeah but those trends cover the last decade … I agree … Yet, we are using for the most part antiquated approaches to modern applications. Things have changed but most of our architecture have remained the same. Modern day systems aspire for more and the amount of goopy code written to accommodate it getting unmanageable.
• #30: Web, mobile and wearables are the primary platform… yet most companies still maintain a large rich client footprint. Companies are even leading with mobile first strategies as a way to reinvent their user experiences without having to deal with their legacy rich client and web deployments.
• #31: Breaking an application into components is where you decide … am I making a baby step or creating a platform. When you consider cloud computing, it can either be about outsourcing your hosting – which is actually kind of boring and expensive OR It can be about building out a platform so that, instead of solutions being a set of monolithic offerings, they are weaved together with a set of components, each of which is independently scalable, performant, resilient etc.
• #32: The maker culture and supporting technology nurtures the rise of innovators and early adopters, driving shorter cycles from experimentation to value. Companies are hungry for the possibility of being agile and iterative and less exposed.
• #33: Analytics moves beyond the most basic of Business Intelligence… the technology and brains are emerging for a new order of data science that is no longer for the select few… in a decade the data scientist will be as common as software engineers.
• #34: Consider the following in support of these statements …
• #35: Web 2.0 represented not just a technology shift but a conceptual shift … components, aggregation, collaboration, sharing Almost a revolt against the machine, where crowd sourcing is actually seemingly better than all the number crunching available.
• #36: Loosely coupled, well defined extension Consider Eclipse (Open Source Software for rich client platforms) IDEs etc. Wordpress one of the most popular content management systems for creating websites and blogs Both are well-defined engines that intentionally expose extension points to that the community is freed up to build loosely coupled but closely integrated offerings. This approach nurtures the programming democracy and DIY culture shift with plugins that do everything abound.
• #37: Quickly lets look back at the Web 2.0 image … The bottom half is really about Engagement – creating content and leaving fingerprints Content sharing, tagging, rating, recommending etc. Socially connected … not just me and the content, but me and everyone I know.
• #38: This is a creative depiction from Ogilvy captured about a talk on recommendation engines at South by Southwest. All this data … fingerprints and all … drives the possibilities of getting the right content to the right person at the right time. http://ogilvynotes.com/49786/438586/sxsw-2011/recommendation-engines
• #39: Consider the ultimate in personalization – Prototype – a game trailer that embeds your Facebook content to create the ultimate in personalization. Its not like Facebook did anything special or Prototype did any heaving crunching … this is about understanding who you are and showing you elements that shift the perception from generic to tailor made.
• #40: The evolution of technology, Consumer driven experiences Cultural shifts Give rise to NoSQL
• #41: The change is substantial. It use to be a single request as a socket open call with a response on the same socket. Not the first socket is actually a list of all the other calls to make. One call is now 10, 30 or 50 calls to render a much greater application. To the point where we have optimizers from Yahoo and others that count these requests and offer strategies to optimize. Things are getting smaller … simpler application layers like Node.js
• #42: Everyone wants to shift to an agile process to reduce overall risk while releasing early and often, capturing the eye-balls and wallets of their target users.
• #43: We have content proliferation and rich personalization – together they create fodder for social, crowd sourced information and…
• #44: …more advanced and “relevant” analytics.
• #45: There is a progression that people seem to find themselves in moving from a RDBMS to more alternative and distributed systems … Both to find the right fit And To achieve different operating characteristics
• #46: If there is a commercial in this entire presentation its this slide. I am bullish on MongoDB, and have been since I was at IBM, because it addresses all the things I am trying to do with my applications and architecture … It is not a niche technology So there are faster cars that drive in a straight line. Most of use are experienced with problems that are more dynamic than that… in which case a more balanced sports car that is looking to win an endurance race is the answer. And it delivers on the performance, scale flexibility, consistency, resiliency and automation everyone is trying to achieve. http://www.cartype.com/pages/5891/automotive_blueprints
• #47: We currently run legacy applications and yet today’s application designs are diverging in aspiration from that base. Technology we are running may be what we are skilled up for, but it is less than adequate too build the future.
• #48: Consider this is what we are saying are characteristics of new application design. There is a considerable shift from what we are asking our systems to do … To how we want it to run … To how we want to interact with it…
• #49: Don’t blink … modern machines are distributed systems and this seems to be taboo, since as far as I can tell this is a rarer skill to find. Most of what we have build in the past is not really distributed and if you were to argue it is, it is only in the most simple form.
• #50: What we are talking about is one system with many layers Development to platform to application … these are the focus areas today.
• #51: There are gaps and this is where your architectural brain should be churning. http://en.wikipedia.org/wiki/Mind_the_gap#mediaviewer/File:Mind_the_gap_2.JPG
• #52: Data scientists are the next kings and queens of the land … most people think you can take a software engineer and make them learn data, but these guys are an entirely different breed. Hire one data scientist from MIT and you will face the challenge of consuming the output.
• #53: Everyone thinks they have this one down, but here is what I see… Companies are doing waterfall in smaller units. Companies are changing the rules of scrum and losing the social dynamics that make it work Companies are running agile, but allowing technology decisions, standards etc. to convert them back to waterfall during certain stages.
• #54: You need to know what to use when … If a distributed hashtable is the same thing to you as a document store then we have a problem. This is the gap we are talking about. Knowing when to use what and why. MongoDB makes for an easy answer because it is the only database alternative that actually provides all the capabilities you expect from how most people use RDBMs, which means it works for everything you do.
• #55: Distributed system are different. Distributed systems are different. My guess is you know a handful that actually understand what this means … have built these kinds of systems and understand the operating characteristics of them. For example, consider what it takes to be able to take a point in time backup of a distributed system? I bet you could design a solution on the whiteboard. That’s the kind of thinking we have done at MongoDB in the MMS offerings you can use to backup and restore MongoDB.
• #56: So in summary …
• #57: Consider the layers … We have the user We have the app We have the platform We have been creating pressure at the data layer and it only makes sense to meet the increasing personalization, experiences and expectations The single fat pipe to your single database is no longer viable. (either through cost or technologically)
• #58: A cluster of confluences drove the need to an alternative approach do data management
• #59: What we are running is no longer an obvious fit for our current and future work
• #60: And you already know the answer. So, its not that you wont use other technologies, but there is a very good chance that you will be using MongoDB. It does what we all need it to do, in a way that lets us act and manage data in ways we have aspired to.