BIG DATA ANALYTICS MEANS “IN-DATABASE” ANALYTICS
2 likes•2,217 views
The document discusses big data analytics focusing on 'in-database' analytics and the challenges associated with managing extreme amounts of data. It outlines the need for robust analytic architectures, including storage, processing capabilities, and tools for data analysis, visualization, and model deployment. The summary concludes that effective analytics on big data necessitates high-performance, parallel systems and an interactive interface for data discovery and management.
BIG DATA ANALYTICS MEANS “IN-DATABASE” ANALYTICS
- 1. Teradata Proprietary and Confidential BIG DATA ANALYTICS MEANS “IN-DATABASE” ANALYTICS Dr. Bruce Aldridge Sr. Industry Consultant Hi-Tech Manufacturing Teradata 760.458.1376 [email protected]
- 2. 2 7/30/2013 Teradata Confidential Overview of Topics • “Big Data” Analytics > The problems of extreme data > Key principles for analytic engines • Analytic Technologies > Changing from sequential to parallel > Design for analytics • Operationalizing Analytics > Analytic life cycle management > Visualization / interacting
- 3. 3 7/30/2013 Teradata Confidential What is “Big Data”? Big Data: any information that’s too fast, too large or doesn’t fit what you are using Data Explosion > Automation of equipment and business processes > Sensor integration > Communication (networks / web) > Compliance
- 4. 4 7/30/2013 Teradata Confidential Using Big Data • Collecting data and using data are different things > Data Lakes serve as high volume low cost repositories for collection > Data may be semi-structured or structured - frequently the conversion happening within the repository > Large amounts of data may be stored for reporting, compliance or investigations • Unusual or new events provide learning (Most “big data” will not provide new information or knowledge)
- 5. 5 7/30/2013 Teradata Confidential Guidelines for Big Data • Collecting ≠ learning ≠ Using data > Data stored on appropriate system for use > Data mining and statistic tools for learning > Model publication (PMML) & monitor for deployment > Visualization tools critical for all
- 6. 6 7/30/2013 Teradata Confidential6 > 7/30/2013 Extreme data brings new challenges • New techniques to limit variables for analysis / modeling • Emergence of columnar analytics • Wealth of data results in more variables than responses 𝑦 𝑚 = 𝑓 𝑥1, 𝑥2, 𝑥3, 𝑥4, … , 𝑥 𝑛 where n>m • Data organization struggles with wide data (>100,000 columns) Id V1 V2 V3 V4 V5 V6 V7 V8 v9 AA 1.2 3.1 41 56 ‘a’ 9 0.2 ? ? AB 0.9 2.7 41 62 ‘a’ 8 0.2 1.1 7 BA 1.0 2.9 42 57 ‘b’ 9 0.1 1.1 ? Id Col ID Val AA V1 1.2 AA V3 41 AB V1 0.9 AB V8 1.1 AB V2 2.7 “pivot” Id V1 V2 V3 V4 V5 V6 V7 V8 v9 CA 1.2 3.1 41 56 ‘a’ 9 0.2 ? ? CB 0.9 2.7 41 62 ‘a’ 8 0.2 1.1 7 BB 1.0 2.9 42 57 ‘b’ 9 0.1 1.1 ? Id Col ID Val AA V1 1.2 AA V3 41 AB V1 0.9 AB V8 1.1 AB V2 2.7 CA V4 56 CB V3 41 BB V1 1.0 Multiple tables add more rows
- 7. 7 7/30/2013 Teradata Confidential Technology Requirements for “Big Data” Analytics • Need for large amounts of data storage • Ability to get at the data (SQL) • Availability of tools for > Visualization > Characterizing, organizing and cleaning data > Summarizing (descriptive statistics) > Analyzing (predictive models, data discovery) > Monitoring & reporting • Analytic Fault Tolerance (massive systems imply more failures) • Dynamic growth – ability to add more capability without “starting over” – mixing technologies • ROI llll llll
- 8. 8 7/30/2013 Teradata Confidential Analytic Tools • Faster analytics require a different approach – Parallel > Sequential processing will be limited > Parallel analytics distributes calculations across multiple nodes with each node having the data necessary > Management of calculation (distribution) and collection • Because data is generally stored on multiple nodes, so….. No choice but to bring the analytics to the data. Data Analytic Modeling Tools Business Results Local Data repository Parallel Analytic Procedures Simple reporting / management tools Data
- 9. 9 7/30/2013 Teradata Confidential Putting it all together: Analytic Architecture LANGUAGES MATH & STATS DATA MINING DISCOVERY PLATFORM LOW COST – HIGH CAPACITY PARALLEL DATA LAKE CAPTURE | STORE | REFINE LANGUAGES MATH & STATS DATA MINING BUSINESS INTELLIGENCE APPLICATIONS FLEXIBLE ANALYTIC / DISCOVERY PLATFORM REPORTING / MONITOR SYSTEM OF RECORD - DATA WAREHOUSE AUDIO & VIDEO IMAGE S TEXT WEB & SOCIAL MACHINE LOGS CR M SCM ER P Environment for: • Low Cost high capacity storage • High power analytics • Fault tolerant high performance reporting • Exploration / visualization across all areas Visualization exploration
- 10. 10 7/30/2013 Teradata Confidential Data Preparation Transform, clean and aggregate data to form data set suitable for analysis Monitor / Model Deployment Deploy statistical model to run iroutinely - automatically monitoring for control Data Exploration Explore all data with statistical profiling and visualization Understand / Model the data Apply mathematical / relational models to test hypotheses about the data Modeling ADS Sample Data Build ADS Production ADS Automated process Analytics Process SQL In-dbs Function PMML or UDF Models
- 11. 11 7/30/2013 Teradata Confidential • Business / Data understanding > Defining objects and requirements of the business > Data collection and data profiling / characterization • Data preparation – joins between tables, attribute selection, cleaning, building new values • Modeling: Analytic algorithms applied and parameters adjusted • Evaluation: results scored according to objectives and requirements • Deployment: Models and parameters put into on demand or automatic execution on new data Analytic discovery process CRISP – Cross Industry Standard Process – data mining
- 12. 12 7/30/2013 Teradata Confidential Analysis – The generation of knowledge Generation of knowledge is iterative and interactive • An idea related to a problem or observation is formulated • Data is collected to support or refute the idea (deduction – what kind of data is necessary?) • Analysis is made on the data to validate or refute (induction) • Results either support / reject idea or suggest modifications Monitoring • Known analytic models used for prediction / verification • Adjust / control based on prediction vs. observation • Business scoring used to prioritize Data (facts, phenomena) Idea (model, hypothesis, theory, conjecture) Monitor / controlValidate Revise
- 13. 13 7/30/2013 Teradata Confidential Establishing a Robust Environment Quality Information Master Data Management Data Profiling / visualization Logical/Physical Model Data “correction” Data Steward/Cleansing processes Discovery Statistical / Data Mining Tools Secure access Robust Analysis capabilities Visualization / understanding Clear and significant results Flexibility in data and models Automation and Alerts Simple publication of discovery knowledge Automated pattern/anomaly detection Business scoring for notification and escalation Clear communication of results Visualization and Reports Choice of the tools to match needs (e.g. Dashboard vs. Engineering views) Timing and need for data refresh Reporting on Core or staging Consistent use of metrics/results (e.g. analytics in database vs. at the reporting layer)
- 14. 14 7/30/2013 Teradata Confidential Analytics – Key Requirements • Performance: > Parallel processing - true shared nothing architecture > Data structure influences analytics (order of magnitude) > Management of analytics and data critical • Fault tolerance > More nodes WILL result in more failures > Analytic Fault Tolerance is more than database fault tolerance – the ability to avoid restarting the analytics • Different node performance > Execution in parallel will never be identical – adjust for node differences > System expansions must be compatible • Flexible analytics > Big data analytics combine queries with analytic functions > Analytic languages not parallel (in general) – need ability to add / customize new functions
- 15. 15 7/30/2013 Teradata Confidential Analytic Applications • Existing parallel analytics > In-database proprietary > In-database addons (Fuzzy Logix, SAS, Partial R) > Hybrid (Aster) – Database architecture supporting MAP-Reduce functions • Many existing applications moving parallel > SAS: Partnered with Teradata for seamless in- database execution of more analytics > R: Partnered with Revolution R for rapid data extraction and execution of some analytics in- database AND in parallel > Spotfire: Execution of aggregation analytics and ability to define in-database analytic functions. Embedded TERR (Tibco Enterprise Runtime R) • Write your own > Map reduce framework > User defined functions
- 16. 16 7/30/2013 Teradata Confidential 16 > 7/30/2013 Analytic Libraries and Enhancements Database built in: • Descriptive Statistics • Basic data mining models (regression, cluster, trees, PCA) • User defined functions Partners • Revolution R, SAS, Fuzzy Logix, Spotfire, … Enhancements • High Speed connections • “Native” data storage
- 17. 17 7/30/2013 Teradata Confidential Device Lot Raw Data Wafer Dashboard as an Analytic Tool • The Dashboard becomes a 2-way interface • User interaction parameterizes and launches new analytics
- 18. 18 7/30/2013 Teradata Confidential Integration of “Dashboard” • Reporting / visualization tool with ability to execute custom functions in-database > Empower all users - ability to publish in-database analytics to users
- 19. 19 7/30/2013 Teradata Confidential Monitor Analytics • Analytic models generally are published into SQL compatible queries • Applying models to data involves: > Gather and format data for analytic > Group data into consistent sets > Screen data > Apply algorithms > Evaluate results • Complete Sequence applied to > Massive amounts of analyses > Repetitive / automated analyses > Scoring / Triage to identify most significant results
- 20. 20 7/30/2013 Teradata Confidential An Analytic Monitor approach User direct edit of Group Description Table (very infrequent) View for Instances Stage Data Group Instance Table Group Description Table Core EDW Data Model Reporting, BI & Alert management tools Alert settings Core EDW Data Model Core EDW Data Model Core EDW Data Model Standard ETL Views for Data (Group Instance) Creation of Views to evaluate load data (installation / dba level user) Analytic procedures 1) Update Group instance Table with new / changed data 2) Identify new & core data for required calculations 3) Perform calculations with standard or custom libraries 4) Compare results to business rules or statistical tests 5) Update alert and report flags Result Table Work Tables (optional) Workflow Status / Control Table ETL starts Analytic Stored Procedure And verifies completion
- 21. 21 7/30/2013 Teradata Confidential Statistical summaries vs…… Powerful in-DB Analysis enables the use of Simple BI Tools
- 22. 22 7/30/2013 Teradata Confidential Data Graphs Powerful in-DB Analysis enables the use of Simple BI Tools Analysis of Big Data: 76M rows of Telemetry (16 of 159 plots/units shown ) graphed and stored in- database for evaluation. Engine hrs. vs. date
- 23. 23 7/30/2013 Teradata Confidential Summary • Analytics on “Big Data” will require one or more high performance (parallel) systems connected to an interactive interface • Support combinations of high volumes (data lakes), high performance and flexible advanced analytics • Tools for understanding, cleansing, discovery AND monitoring necessary • Interactive Visualization support across all systems • Management of analytics and fault control