InfoSphere BigInsights - Analytics power for Hadoop - field experience
1 like•2,631 views
This document provides an overview and summary of InfoSphere BigInsights, an analytics platform for Hadoop. It discusses key features such as real-time analytics, storage integration, search, data exploration, predictive modeling, and application tooling. Case studies are presented on analyzing binary data and developing applications for transformation and analysis. Partnerships and certifications with other vendors are also mentioned. The document aims to demonstrate how BigInsights brings enterprise-grade features to Apache Hadoop and provides analytics capabilities for business users.
InfoSphere BigInsights - Analytics power for Hadoop - field experience
- 1. InfoSphere BigInsights Analytics power for Hadoop – field experience Wilfried Hoge IT Architect Big Data @wilfriedhoge Stephan Reimann IT Specialist Big Data @stereimann
- 2. © 2015 International Business Machines Corporation 2 IBM BigInsights – Open Source and IBM Value Adds Real-time Analytics InfoSphere Streams Enterprise Performance Adaptive Map Reduce & Big SQL Storage Integration GPFS POSIX Distributed Filesystem Data Governance and Security Data Click, LDAP and Secured Cluster Search BigIndex and Data Explorer Data Exploration BigSheets “schema-on-read” tooling MapReduceHDFS HBase Flume Pig Lucene Jaql ZooKeeperOozie Hive Sqoop HCatalog 100% based on Apache Open Source Hadoop Components Predictive Modeling BigR scalable data mining” on R Text Analytics Text processing with AQL Application Tooling Toolkits and accelerators ANSI SQL BigSQL Optimized SQL support
- 3. © 2015 International Business Machines Corporation 3 Key Differentiators for BigInsights Enterprise Performance & Integration Analytics Usability & Productivity • Workload / performance optimization • GPFS • Security • Key integrations & Connectors with Enterprise Ecosystem • Text analytics • Social Data Analytics Accelerators • Machine Data Analytics Accelerators • Execute R in an integrated application • Big SQL • BigSheets • Development Tools • Web Console
- 4. © 2015 International Business Machines Corporation 4 Field experience – analyzing binary data The challenge • Use case – Enable users to analyze data that is provided in binary format without the need to run scripts • Challenges – Binary to csv transformation – Access csv data on HDFS to directly analyze content – Access csv data from BI tools through SQL – Possibility to analyze the data for technical business users – Flexible automation capabilities (scheduling)
- 5. © 2015 International Business Machines Corporation 5 Field experience – analyzing binary data The binary file – direct analysis not possible
- 6. © 2015 International Business Machines Corporation 6 Running Applications on Big Data • Browse available applications • Deploy published applications (administrators only) • Launch (or schedule for launch) a deployed application • Monitor job (application) execution status • Predefined applications • Import & Export Data • Database & Files • Web and Social • Analyze and Query • Predictive Analytics • Text Analytics • SQL/Hive, Jaql, Pig, Hbase • Accelerators
- 7. © 2015 International Business Machines Corporation 7 7 Editors • A workflow editor that greatly simplifies the creation of complex Oozie workflows with a consumable interface • A Pig/Jaql Editor with content assist and syntax highlighting that enables users to create and execute new applications using Pig or Jaql in local or cluster mode from the Eclipse IDE Application development & deployment • Enablement of BigSheets macro and BigSheets reader development • Text Analytics development, including support for modular rule sets • Publish new application: BigSheets Macro, BigSheets Reader, AQL module, Jaql module Tools for Developers 1. Sample your Data 2. Develop your application using BigInsights tools 3. Test your application 4. Package and publish your application 5. Deploy your application on the cluster
- 8. © 2015 International Business Machines Corporation 8 Field experience – analyzing binary data Developing and publishing a transformation application
- 9. © 2015 International Business Machines Corporation 9 Field experience – analyzing binary data The transformation application – user can convert binary data to csv
- 10. © 2015 International Business Machines Corporation 10 BigSheets to analyze and visualize • Model “big data” collected from various sources in spreadsheet-like structures • Filter and enrich content with built-in functions • Combine data in different workbooks • Visualize results through spreadsheets, charts • Export data into common formats (if desired) No programming knowledge needed!
- 11. © 2015 International Business Machines Corporation 11 Field experience – analyzing binary data The csv file – BigSheets offers easy analysis
- 12. © 2015 International Business Machines Corporation 12 Field experience – analyzing binary data An analytical result with BigSheets
- 13. © 2015 International Business Machines Corporation 13 Field experience – analyzing binary data The loader application – create tables for analysis
- 14. © 2015 International Business Machines Corporation 14 Big SQL 3.0 – Architected for Performance • Leverage IBM's rich SQL heritage, expertise, and technology – Modern SQL:2011 capabilities – DB2 compatible SQL PL support • SQL bodied functions and stored procedures • Application logic/security encapsulation • Architected from the ground up for performance – low latency and high throughput • MapReduce replaced with a modern MPP architecture – Compiler and runtime are native code (not java) – Big SQL worker daemons live directly on cluster – Continuously running (no startup latency) – Processing happens locally at the data • Operations occur in memory with the ability to spill to disk – Supports aggregations and sorts larger than available RAM • Integration with BigSheets (source & target) InfoSphere BigInsights Big SQL SQL MPP Runtime Data Sources Parquet CSV Seq RC Avro ORC JSON Custom SQL-based Application IBM Data Server Client
- 15. © 2015 International Business Machines Corporation 15 Big SQL 3.0 – Architecture cont. • Head (coordinator / management) node – Listens to the JDBC/ODBC connections and compiles / optimizes the query – Coordinates the execution of the query – Optionally store user data in traditional RDBMS table (single node only) • Big SQL worker processes reside on compute nodes (some or all) • Worker nodes stream data between each other as needed • Workers can spill large data sets to local disk if needed – Allows Big SQL to work with data sets larger than available memory Mgmt Node Big SQL Mgmt Node Hive Metastore Mgmt Node Name Node Mgmt Node Job Tracker ••• Compute Node Task Tracker Data Node Compute Node Task Tracker Data Node Compute Node Task Tracker Data Node Compute Node Task Tracker Data Node••• Big SQL Big SQL Big SQL Big SQL GPFS/HDFS
- 16. © 2015 International Business Machines Corporation 16 Big SQL 3.0 – Features Data shared with Hadoop ecosystem Comprehensive file format support Superior enablement of IBM software Enhanced by Third Party software Modern MPP runtime Powerful SQL query rewriter Cost based optimizer Optimized for concurrent user throughput Results not constrained by memory Distributed requests to multiple data sources within a single SQL statement Main data sources supported: DB2 LUW, DB2/z, Teradata, Oracle, Netezza Advanced security/auditing Resource and workload management Self tuning memory management Comprehensive monitoring Comprehensive SQL Support IBM SQL PL compatibility Application Portability & Integration Federation Performance Enterprise Features Rich SQL
- 17. © 2015 International Business Machines Corporation 17 Field experience – analyzing binary data Run complex SQL on generated tables INSERT INTO Sites (Counter,Tested,Site1,Site_num1,Number_of_xxxx_tested, XA1,Percentage_of_xxxx_per_yyyy, Counter_plus_one,Pass,Site2,Site_num2,Number_of_pass_xxxx, ZB2,xxxxx_of_site_num,xxxx_file_name) SELECT 12000 + ROW_NUMBER() OVER () * 10,'Tested','Site’,tab1.Site_num, (SELECT sum(tab2.piece_count) FROM tab2 WHERE tab2.site_num=tab1.site_num) as num_xxxx_tested, 'PA',(SELECT sum(tab2.piece_count) FROM tab2 WHERE tab2.site_num=tab1.site_num and tab2.head_num=255), 34000 + ROW_NUMBER() OVER () * 10 + 1,'Pass','Site',tab1.site_num, (SELECT COUNT(*) FROM tab1 as tab12 WHERE tab1.site_num=tab12.site_num and tab1.piece_Flg=0) as num_xxxx_passed, 'PA',((SELECT sum(tab2.piece_count) FROM tab2 WHERE tab2.site_num=tab1.site_num) / NULLIF(0.001,(SELECT COUNT(*) FROM tab1 as tab12 WHERE tab1.site_num=tab12.site_num and tab1.piece_Flg=0))), tab1.xxxx_file_name FROM tab1 as tab1, tab2 as tab2 GROUP BY tab1.site_num, tab1.piece_Flg, tab1.xxxx_file_name; rank function subselects
- 18. © 2015 International Business Machines Corporation 18 Application linking and interfaces to build new apps • Compose new applications from existing applications and BigSheets • Invoke analytics applications from the web console, including integration within BigSheets • REST data source App that enables users to load data from any data source supporting REST APIs into BigInsights, including popular social media services • Sampling App that enables users to sample data for analysis • Subsetting App that enables users to subset data for data analysis 18
- 19. © 2015 International Business Machines Corporation 19 Field experience – analyzing binary data User builds his/her own application flow
- 20. © 2015 International Business Machines Corporation 20 Field experience – analyzing binary data What was achieved 1/2 – Conversion from binary to csv (Transformation App) • Customer provided Java classes that read binary file and produced csv output • Developer embedded java code in an BigInsights application • User can provide source and target path • User can provide filters if not the whole data set should be extracted • User can schedule the application (with parameters) • Application automatically has a REST interface for external scheduling • Application uses map/reduce for scaling if larger number of files have to be transformed – User can analyze the csv files with BigSheets
- 21. © 2015 International Business Machines Corporation 21 Field experience – analyzing binary data What was achieved 2/2 – Create SQL tables from csv (Loader App) • Developer embedded necessary SQL in App • User can create tables from csv files – User can run complex SQL on tables with preferred Front-End tool – User can combine Apps and create his/her own flow
- 22. © 2015 International Business Machines Corporation 22 IBM BigInsights brings efficient integration of R with Big R • R as a big data query language – Outside-in execution • R as a statistical language for deep computing – Inside-out execution – Partitioning of large data (“divide”) – Parallel cluster execution of pushed down R code (“conquer”) – Almost any R package can run in this environment • R as the gateway to scalable machine learning – A scalable ML engine that provides canned algorithms, and an ability to author new ones, all via R R Clients Scalable ML Engine Data Sources Embedded R Execution R Packages R Packages Pull data (summaries) to R client Or, push R functions right on the data
- 23. © 2015 International Business Machines Corporation 23 SystemML – Declarative high-level language SystemML tokens documents 1 1 0.10 1 2 0.30 1 3 0.22 1 4 1.24 : : : : : : W H Ktopics tokensK topics documents 1 1 0.10 1 2 0.30 : : : Topic Detection in Social Media § Modeled after R syntax and semantics § Expressivity – Express a wide class of algorithms: Descriptive statistics, linear & logistic regression, decision trees, SVM, MCMC simulation, etc. § Productivity – Enable programmer productivity: algorithm developer does not have to worry about scalability, numeric stability and optimizations § Performance and Scalability – Optimizer to generate low-level executions plans • Cost-based operator selection based on − data characteristics (dimensions, sparsity) − cluster characteristics (memory, parallelism) • Generation of runtime execution plan § Big Data – Sparsity-driven data representation and operator implementations for data sets with Billions of non-zero values
- 24. © 2015 International Business Machines Corporation 24* Requires Service Engagement ISV Partner Solution Type BigInsight Version Certified ISV Partner Solution Type BigInsight Version Certified Data Integration 2.1 (3.0 in process 4Q) Reporting 2.1 & 3.0 Data Security 2.1.2 Customer Analytics 2.1.2 Cluster Mgt 3.0 Analytics 2.1.2 (3.0 in process) Data Vis 2.1 (3.0 in process) Visual Reporting 2.1 & 3.0 Data Virtual- ization 2.1.2 & 3.0 TDHC 3.0 Analytics 2.1.2&3.0 Aster 3.0 * Data Integration 2.1 (3.0 in process 3Q) Backup & Recovery 2.1.2 IBM Product Solution Type BigInsight Version Certified IBM Product Solution Type BigInsight Version Certified Business Intelligence 2.1.2 (3.0 end of Nov’14) Predictive Analytics 2.1.2 (3.0 mid4Q) InfoSphere Information Server v11.3 Data Integration 3.0 SPSSv10.2.1 AS v1.0.1 BigInsights Certifications
- 25. © 2015 International Business Machines Corporation 25 lHelium SW BigInsights ISV Partner Ecosystem
- 26. © 2015 International Business Machines Corporation 26 Get started with BigInsights • Hadoop Dev: links to videos, white papers, lab, . . . . http://developer.ibm.com/hadoop/ • BigInsights Trials http://ibm.com/software/data/infosphere/hadoop/trials.html
- 27. IBM big data • IBM big data • IBM big data IBM big data • IBM big data • IBM big data IBMbigdata•IBMbigdata IBMbigdata•IBMbigdata THINK
- 28. © 2015 International Business Machines Corporation 28 BigInsights has a simple but effective security system based on a gateway to Hadoop • All Hadoop servers are connected over a private network • Unrestricted communication between cluster servers on the private network • BigInsights Web Console acts as a gateway into the cluster • Authentication through PAM, LDAP Kerberos • Role based authorization • Authorization will be enforced at 3 levels: – UI level – Data level – Map-Reduce level • Authorization also respected by services (e.g. SQL) Authentication Authority Gateway / Web Console External Sources Users Services Data Nodes Infrastr. Nodes Distributed Filesystem
- 29. © 2015 International Business Machines Corporation 29 Manage your HDFS Files • Navigate the distributed file system to see what’s stored • Create/remove/rename directories • Modify permissions • Upload / download files, remove/rename files, Edit files • Execute Hadoop file system shell commands
- 30. © 2015 International Business Machines Corporation 30 About the Hadoop-DS Benchmark § Created by IBM § The Big Data Decision Support Benchmark (Hadoop-DS) is inspired by, and is highly compliant with TPC-DS - Fully complies with the TPC-DS schema requirement - Uses all 99 queries - Meets the multi-user requirement - Has been audited by a TPC-DS auditor but as a non-TPC benchmark § Select deviations from TPC-DS due to Hadoop limitations: - No data maintenance operations, referential integrity enforcement, or ACID property validation as these are not feasible with HDFS - Additional statistics used - Metric adjustments - No price/performance measures included - Not an official TPC benchmark result
- 31. © 2015 International Business Machines Corporation 31 IBM Big SQL – Runs 100% of the TPC-DS queries Key points § With competing solutions, many queries needed to be re-written, some significantly § Owing to various restrictions, some queries could not be re- written or failed at run-time § Re-writing queries in a benchmark scenario where results are known is one thing – doing this against real databases in production is another Competitive environments require significant effort
- 32. © 2015 International Business Machines Corporation 32 IBM Big SQL – Leading performance 0 2.000 4.000 6.000 8.000 10.000 12.000 14.000 16.000 18.000 Big SQL Impala Hive Power run (single-stream) - seconds As measured across the subset of queries that Impala and Hive can both run 3.6x FASTER!! 48:29 2:55:35 4:30:35 3.6x faster than Impala, 5.6x faster than Hive * Subject to findings of TPC auditor – full disclosure report expected late October 2014
- 33. © 2015 International Business Machines Corporation 33 IBM Big SQL – Leading performance 0 10.000 20.000 30.000 40.000 50.000 60.000 70.000 Big SQL Impala Hive Throughput run – (four streams) - seconds As measured across the subset of queries that Impala and Hive can both run 1:54:02 4:08:39 16:32:12 2.2x FASTER!! 2.2x faster than Impala, 4x faster than Hive * Subject to findings of TPC auditor – full disclosure report expected late October 2014