Big Data, Hadoop, Hortonworks and Microsoft HDInsight

Editor's Notes

• #6: For the visual thinkers out there, let’s expand our mathematical model to show some concrete examples.ERP, SCM, CRM, and transactional Web applications are classic examples of systems processing Transactions. Highly structured data in these systems is typically stored in SQL databases.Interactions are about how people and things interact with each other or with your business. Web Logs, User Click Streams, Social Interactions & Feeds, and User-Generated Content are classic places to find Interaction data.Observational data tends to come from the “Internet of Things”. Sensors for heat, motion, pressure and RFID and GPS chips within such things as mobile devices, ATM machines, and even aircraft engines provide just some examples of “things” that output Observation data.Most folks would agree that video is “big” data. The analysis of what’s happening in that video (ie. What you, me, and others are doing in the video) may not be “big” but it is valuable and it does fit under our umbrella.Moreover, business data feeds and publicly available data sets are also “big data”.So we should not minimize our thinking to just data that flows through an organization.Ex. The mortgage-related data you may have COULD benefit from being blended with external data found in Zillow, for example.The government, for example, has the Open Data Initiative. Which means that more and more data is being made publicly available.One of the use cases I find interesting is the Predictive Policing use case where state/local law enforcement is using analytics applied to crime databases and other publicly available data to help predict where and when pockets of crime might be springing up. These proactive analytics efforts have yielded real reductions in crime!Anyhow, this is what Big Data means to me…hopefully it makes sense to you. It is important to note that we think of big data beyond the traditional concepts of volume, velocity and variety into transactions, interactions and observations. In reality, this IS the big data our customers are dealing with.
• #25: Gray Systems Lab, Dr. David DeWittFuture of query processingOne interface to query relational & Hadoop dataQuery data without moving itExpanding to other data sources in the futureSeamless integration with unstructured data & hadoopBreakthrough technologyGrey systems lab - DeWitt It’s going to dramatically simplify how users query relational and Hadoop dataFuture of query processingPioneered in the Jim Gray Systems Labs by David DeWitt, PolyBase is a federated query processor in SQL Server 2012 Parallel Data Warehouse which represents a breakthrough innovation from traditional query processing to join structured and unstructured data from Hadoop together. Without manual intervention, PolyBase Query Processor can accept a standard SQL query and combine tables from a relational source with tables from a Hadoop source directly through external tables.  As well, PolyBase Query Processor parallelizes the ability to import/export data to and from Hadoop giving PDW speed, simplicity, and responsiveness in addressing these new types of queries.Ability to issue standard T-SQL that joins relational data with unstructured data in Hadoop PolyBase rapidly imports/exports data between Hadoop and PDW in parallel3) PolyBase can query data in Hadoop directly without movement (with external tables)4) Created in “Gray Systems Labs” by David DeWitt
• #27: And that's the second thing I wanted to share with you this afternoon
• #29: We believe that Hadoop can be in a position to process more than half the world’s data. I’ve talked to a variety of industry analysts, and there’s not a big argument over Hadoop’s opportunity to achieve this. Some would argue it should be 2016 or 2017, rather than 2015. But we believe aggressive goals help focus people on the right things, so let’s keep it 2015 for now, and let’s see how close we can get. The point here is that this statement can act as our “north star” and help guide our way as we focus on our list of 5 items we can be doing:Be diligent stewards of the open source coreBe tireless innovators beyond the coreProvide robust data platform services & open APIsEnable ecosystem at each layer of the stackMake platform enterprise-ready & easy to use