Data Integration and Advanced Analytics for MongoDB: Blend, Enrich and Analyze Disparate Data in a Single MongoDB View
Download as PPTX, PDF11 likes3,183 views
The document discusses blending disparate data sources like stock quotes, news, and Twitter sentiment data into a single MongoDB view for analytics using Pentaho tools. It provides an example of blending intraday Tesla stock quote data from a web service with real-time Twitter data from the Twitter API about Tesla to inform investment decisions. Pentaho data integration is used to extract, transform, and load the data into MongoDB, and Pentaho analytics tools like the new Analyzer for MongoDB allow visualizing and analyzing the blended data.
![Competitive edgewithmongod bandpentaho_2014sep_v3[1]](https://cdn.slidesharecdn.com/ss_thumbnails/competitiveedgewithmongodbandpentaho2014sepv31-141022155232-conversion-gate02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (14)
Ad
Similar to Data Integration and Advanced Analytics for MongoDB: Blend, Enrich and Analyze Disparate Data in a Single MongoDB View (20)
Ad
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
Data Integration and Advanced Analytics for MongoDB: Blend, Enrich and Analyze Disparate Data in a Single MongoDB View
- 1. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75551 Bo Borland VP, Field Technical Sales Blending disparate data into a single MongoDB view for analytics Pentaho Analytics for MongoDB
- 2. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75552
- 3. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75553 Tesla Motors Inc $34/per share $229/per share January 2013 thru June 20, 2014
- 4. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75554 Tesla Motors Inc Current month stock performance Tesla announces it will share all patents in spirt of open source movement Source: http://www.teslamotors.com/blog/all-our-patent-are-belong-you June 12th - Open Source Announcement 1. Attract and motivate the world’s most talented engineers 2. Collaborate with auto-makers on a common, rapidly evolving technology platform (the charging station network) 3. Accelerate the creation and advancement of electronic vehicles (sustainable transport) to address carbon crises 10% increase
- 5. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75555 Tesla Motors Inc Mixed reaction to open source news June 12th - Open Source Announcement June 12, 2014 - Tesla Motors Inc. to Give Away Patents -- Good or Bad for Investors? June 15, 2014 - Electric car groups eye collaboration over charging technology
- 6. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75556 Supercharger Charging Stations Open Now & By 2015 June 16, 2014 - Tesla Motors, Inc. Stock Jumps on News of Possible Charging Collaboration “Tesla stock is up about 3% this morning (at the time of this writing) after Financial Times author Henry Foy reported that Tesla, Nissan, and BMW, the world's three largest manufacturers of electric vehicles, are in "keen talks" regarding a new level of collaboration on charging networks, according to sources at all three companies.”
- 7. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75557 Twitter Data “When important news is shared on Twitter, traders and investors need to be able to access it, and validate its importance in order to incorporate that information into their decision making process,” said Jean-Paul Zammitt, head of sales and product development for the Bloomberg Professional service. Source: http://www.bloomberg.com/now/press-releases/bloomberg- integrates-live-twitter-feeds-with-financial-platform/
- 8. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75558 Market-Moving Insights Blending sources that can influence stocks and financial markets Twitter Sentiment issued by corporations, executives, government officials, economist, commentators, media outlets. High-quality news from various news organizations and industry bloggers Intra-day buy/sell stock quote data
- 9. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-75559 • Web service api call for live intraday equity quote data • Level 1, tick-by-tick quote data for given stock and date range • Enrich the data and load to MongoDB • Relevant fields: • Date • Time • Bid Price • Offer Price • Bid Qty • Offer Qty Intraday Equity Quotes
- 10. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755510 • Live Twitter API search query for Telsa Motor stock • Unstructured search result is parsed • Enriched with Pentaho predictive analytics model to derive sentiment score • Enriched dataset loaded to MongoDB • Relevant fields: • User • Tweet text • Retweet count • Followers • Search term • Sentiment index Twitter Sentiment Data +1 (Positive) 0 (Neutral) -1 (Negative) Advanced Analytics Data Science Pack: Pentaho 5.1 capability with a text classification predictive model to score each tweet and load the tweets and scores into MongoDB.
- 11. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755511 Market-Moving Insights • Intra-day buy/sell stock quote data provided by a web service • Real time Twitter data provided by Twitter API • Pentaho Data Integration to blend disparate data sources • Pentaho Predictive Analytics to enrich data with twitter sentiment score Single MongoDB view with Pentaho Analysis to make stock investment decisions
- 12. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755512 Analyzer for MongoDB Released Today as part of Pentaho 5.1 Release Analyze and visualize MongoDB data thru a drag-n-drop interface. • Native OLAP analysis on real-time data in MongoDB • Easy to use, award-winning OLAP interface for building visualizations that can be added to dashboards • Native integration with MongoDB leveraging the power of the MongoDB query language and aggregation framework. Analyze blended June data to make an investment decision on Tesla Motors?
- 13. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755513 Analyzer Demonstration
- 14. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755514 Market-Moving Insights Blending sources that can influence stocks and financial markets Twitter Sentiment issued by corporations, executives, government officials, economist, commentators, media outlets. High-quality news from various news organizations and industry bloggers Intra-day buy/sell stock quote data Blend Enrich Predict Load Query Aggregate Notify Inform
- 15. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755515 mongoDB clusterPDI ETL Analytics Broad Connectivity Broad connectivity combined with powerful data integration
- 16. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755516 Concept – Data Transformations INPUT(S) – PROCESS(ES) – OUTPUT(S)
- 17. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755517 Data Integration Demonstration
- 18. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755518 Pentaho Visual Development Eliminates the Need for Complex Coding Would you rather do this? Scheduling Modeling Ingestion / Manipulation / Integration … or this?
- 19. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755519 Equity Data Ingestion INPUT(S) – PROCESS(ES) – OUTPUT(S)
- 20. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755520 Twitter API Data Ingestion INPUT(S) – PROCESS(ES) – OUTPUT(S)
- 21. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755521 Pentaho Job Automates Data Load Sequence START – CHECK – WATCH – EXECUTE – NOTIFY - FINISH
- 22. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755522 Equity Quotes Schema Eliminates the need for table joins RDBMS Star Schema (6-9 tables) Mongo Hierarchy Schema (1 collection) quotes tweets company dimension date dimension time dimension tweet dimension record type dmension
- 23. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755523 Entity 360 – NoSQL Architecture A Blended View to Drive Revenue Growth and Service Improvements MongoDB CRM System Claims Admin. Info PDI Online Transactions Documents & Images PDI Advanced Analytics Business Analysis Analyzer for MongoDB Data Science Pack Aggregation Framework Data Science Pack
- 24. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755524 Pentaho Analytics for MongoDB
- 25. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755525 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755525 OPERATIONAL DATA BIG DATA DATA STREAMPUBLIC/PRIVATE CLOUDS DBA/ETL/BI DEVELOPER BUSINESS USER BUSINESS/DATAANALYST Data Integration + Analytics + Predictive Extensive community of developers Pluggable, extensible, java architecture Single, Open Platform for Analytics
- 26. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755526 Powerful Technologies Online / Offline Data For creating a single view in MongoDB
- 27. © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-755527 Bo Borland VP, Field Technical Sales [email protected] Thank You
Editor's Notes
- #8: The 4 days between the press release and the the big price jump is a time where Twitter Sentiment would be extremely valuable in determining expert and public opinion on the Tesla’s decision to open-source it’s patents. Tweets issued by corporations, executives, government officials, economist, commentators, media outlets are influential voices that can influence markets. Bloomberg views Twitter data as an important part of their platform and critical for investment decision-making.
- #9: What I want to do in today presentation is use Pentaho to blend data from June and analyze data a few days prior to the price increase. We will see how Pentaho Analytics can blend Intra-day stock quote data, news feeds, and Twitter sentiment data into a single MongoDB view. And then use Pentaho to analyze that data.
- #14: Demo of building a transformation to load CSV equity quote data to MongoDB
- #18: Demo of building a transformation to load CSV equity quote data to MongoDB