Smart Enterprise Big Data Bus for the Modern Responsive Enterprise
4 likes1,291 views
The document discusses the need for a "Smart Enterprise Big Data Bus" to efficiently synchronize data in real-time between various enterprise applications and Hadoop, as existing integration approaches are unable to meet the demands of real-time analytics. It proposes using a streaming analytics platform called StreamAnalytix to create a high-speed data bus that can ingest, transform, analyze, and synchronize data across the enterprise. Examples of implementing such a solution for a healthcare company are also provided.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
![Inspiring Travel at Airbnb [WIP]](https://cdn.slidesharecdn.com/ss_thumbnails/june91205pmairbnbqiancheng-150616222059-lva1-app6891-thumbnail.jpg?width=560&fit=bounds)
Ad
Similar to Smart Enterprise Big Data Bus for the Modern Responsive Enterprise (20)
Ad
More from DataWorks Summit (20)
Recently uploaded (20)
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise
- 1. Smart Enterprise Big Data Bus ForThe Modern Responsive Enterprise AnandVenugopal June 10, 2015
- 2. THE JOURNEY SO FAR… • ImpetusTechnologies – Leading Big Data Solutions Provider • StreamAnalytix – Enterprise Class Streaming Analytics platform – Early Access Program: Aug 2014 – Pilot Projects: October 2014 – February 2015 – GA Launch: February 2015 – Today – Sharing one of the key insights and use-case patterns from pilot
- 3. SMART ENTERPRISE BIG DATA BUS - AGENDA Why ? • Case Study • Emerging Big Data Landscape and its Challenges What ? • Solution Characteristics • Components Required How ? • Implementation Using StreamAnalytix • Technology Stack Q&A • Summary • Follow up
- 4. CASE STUDY - HOW IT BEGAN Healthcare giant is speeding up critical business processes by using a streaming analytics platform for real-time data synchronization between their Hadoop platform and their enterprise NoSQL database App-1 App-2 App-3 Enterprise NoSQL Enterprise Data Hub/Lake Hadoop
- 5. EMERGING ENTERPRISE BIG DATA LANDSCAPE Hadoop Stack – Becoming the Center of the Enterprise Data Universe Business Critical Applications Are Still ''Silos'' Real-time Streaming Analytics Adoption Rapidly Increasing
- 6. HADOOP – IT MAKES COFFEETOO! Active Archive ETL Single Source of Truth BI + Analytics Hadoop Streams/ Transactions
- 10. WHAT’S MISSING FROMTHESE PICTURES?
- 11. HOW HADOOP CONNECTSTOTHE REST OF IT ALL ! Active Archive ETL Single Source of Truth BI + Analytics Hadoo p Streams/ Transactions Enterprise Data Hub/Lake Hadoop ERP CRM Current EDW/ ADW App Server 1 App Server 2 Hub and Spoke Architecture The IntegrationChallenge – Developing, Maintaining, Managing the Hub-spoke System
- 12. EMERGING ENTERPRISE BIG DATA LANDSCAPE Hadoop Stack – Becoming the Center of the Enterprise Data Universe Business Critical Applications Are Still ''Silos'' Real-time Streaming Analytics Adoption Rapidly Increasing
- 13. SILOED ENTERPRISE APPLICATIONS MAYTAKE MANYYEARS TO INTEGRATE WITH HADOOP Billing Customer Service Transaction Processing Provisioning
- 14. THE DATA INTEGRATION CHALLENGE IS HERETO STAY Enterprise Data Hub/Lake Hadoop ERP CRM Current EDW/ADW App Server 1 App Server 2 Hub and Spoke Architecture The Integration Challenge – Developing, Maintaining, Managing the Hub-spoke System
- 15. EMERGING ENTERPRISE BIG DATA LANDSCAPE Hadoop Stack – Becoming the Center of the Enterprise Data Universe Real-time Streaming Analytics Adoption Rapidly Increasing Business Critical Applications Are Still ''Silos''
- 16. The Modern Enterprise Expected To Be ''Real-time'' Or ''Near Real-time'' Sales Web Site Billing Customer Service Growth of Internet ofThings (IoT) and Sensor/ Machine Data Sources Context-sensitive Customer Service
- 17. The Modern Enterprise Expected To Be ''Real-time'' Or ''Near Real-time'' Sales Web Site Billing Customer Service Mobile Location Based Offers Internet Advertisements Call-center Agent Interactions
- 18. CONTEXT AWARE POSITIVE CUSTOMER EXPERIENCE Multi-channel engagement in real-time Context Sensitive service Happy customers, Loyalty, Revenue, Profits, Growth
- 19. Real-time Responses Need Hi-speed Any-to-Any Data Synchronization & Analytics Over Streaming Data Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop Developing and managing all the peer-to-peer data interfaces and the hub-spoke model together would be very hard
- 20. EMERGING ENTERPRISE BIG DATA LANDSCAPE Hadoop Stack – Becoming the Center of the Enterprise Data Universe Business Critical Applications Are Still ''Silos'' Real-time Streaming Analytics Adoption Rapidly Increasing
- 21. SMART ENTERPRISE BIG DATA BUS - AGENDA Why ? • Case Study • Emerging Big Data Landscape and its Challenges What ? • Solution Characteristics • Components Required How ? • Implementation Using StreamAnalytix • Technology Stack Q&A • Summary • Follow up
- 22. A HI-SPEED BIG DATA BUS ARCHITECTURE WOULD BE AN EFFICIENT ANY-TO-ANY DATA SYNCHRONIZATION MECHANISM Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop
- 23. A HI-SPEED BIG DATA BUS ARCHITECTURE WOULD BE AN EFFICIENT ANY-TO-ANY DATA SYNCHRONIZATION MECHANISM Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop
- 24. A HI-SPEED BIG DATA BUS ARCHITECTURE WOULD BE AN EFFICIENT ANY-TO-ANY DATA SYNCHRONIZATION MECHANISM Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop
- 25. THE HI-SPEED BIG DATA BUSWOULDALSO NEEDTO SUPPORT ''ONTHEWIRE'' COMPUTATION FOR DATA ANALYTICS,TRANSFORMS, CEP Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop Transformation Analytics,Alerting
- 26. CAPABILITY LIST FORTHE ''SMART ENTERPRISE BIG DATA BUS'' • Ingest • Parse • Filter • Transform • Move • Store AT SCALE, AND FAST ! • Read • Synchronize • Analyse • Predict • Alert • Visualise Provisioning Machine Data Processing Billing Enterprise Data Hub/Lake Hadoop Transformation Analytics, AlertingCustomer Service
- 27. THE ''SMART ENTERPRISE BIG DATA BUS''
- 28. ''Stations'' in the Data Transit System Read-Write Adapters Reliable, Fault-tolerant, Elastic Scalable Distributed Stream Processing and Transport Fabric Stream Processing Services provided by the ''Smart Enterprise Big Data Bus'' include UI for Work-flow Orchestration, Management and Monitoring THE SMART ENTERPRISE BIG DATA BUS
- 29. ESB (VS. SMART ENTERPRISE BIG DATA BUS) • Were architected for a different workload in a different era • Designed for light weight remote service invocations – not as a heavy throughput full peer-to- peer data transfer mechanism • No compute / analytics capability on the wire • Expensive vertical scaling vs. distributed elastic scale-out with commodity hardware • Monolithic workflows vs. independent control and elastic scalability of each stage in a workflow based on compute needs
- 30. SMART ENTERPRISE BIG DATA BUS - AGENDA Why ? • Case Study • Emerging Big Data Landscape and its Challenges What ? • Solution Characteristics • Components Required How ? • Implementation Using StreamAnalytix • Technology Stack Q&A • Summary • Follow up
- 31. SMART ENTERPRISE BIG DATA BUS IMPLEMENTATION • Kafka • Rabbit MQ • Apache Storm • Operators • RT Dashoards • Websockets • CEP • Filter • Indexer • NoSQL • HDFS, Hbase • PMML AT SCALE, FAST, EASY !
- 32. STREAMANALYTIX ARCHITECTURE BLOCK DIAGRAM
- 33. STREAMANALYTIX – ENTERPRISE BIG DATA BUS – CREATION
- 34. VISUAL CONFIGURATION – NO CODE – NO COMPLEXITY
- 37. SAMPLE DEPLOYMENT/HARDWARE SPECIFICATION External systems Data Store Integration layer Processing andAnalytics layer Storm RHEL 4 cores, 16GB RAM vm06dev222 x 1 ElasticSearch RHEL 8 cores, 32GB RAM vm06dev218 x 1 PostgreSQL RHEL 4 cores, 8GB RAM vm06dev222 x 1 Meta Information Store Zookeeper x 1 Kafka+ZK RHEL 4 cores, and 8GB RAM vm06dev220 x 1 Free Switch x 2 Graphite x 1 Web UI Layer RHEL 4 cores, and 2GB RAM vm07eng29, vm07eng31 IR(Platform) x 1 RHEL 4 cores, and 2GB RAM vm07dev97 RHEL 4 cores, 16GB RAM vm06dev222 RHEL 4 cores, 4GB RAM vm06dev224 Tomcat(Query Server) RHEL 4 cores, 8GB RAM vm06dev218 x 1 Tomcat(Admin+Log UI) RHEL 4 cores, 16GB RAM vm06dev222 x 1 Couchbase RHEL 8 cores, 32GB RAM vm06dev224 x 1 RabbitMQ RHEL 4 cores, and 4GB RAM vm06dev224 x 1
- 38. 39 Any real time source Filter Transfor m Analyze Transport Store Pull PushWeb Front End WWW App Server 1 App Server 2 P.O.S. Business Transaction AD W Analyst Machine Learning Pattern Recognition Enterprise Data Hub/Lake Hadoop Real-time Dashboards CEOs Office ERP ERP Current EDW Rules Alerts Downstream Apps The Smart Enterprise Big Data Bus - Big Data River ?
- 39. www.streamanalytix.com Email: [email protected] Thank you. Questions?? We are Hiring!! Free Storm Book Demo Booth S9
Editor's Notes
- #2: One intro slide of DSP 10 mins for DSP neustar use case (focus on algorithms) Not techy but more business 10 mins for other customer complaints use cases and demo 30 Mins for Impetus Corp 15 mins for BDP
- #19: Data source – listener for Active MQ Secure data streaming from remote servers Alert for call drop events in the main pipeline