Big Data at CME Group: Challenges and Opportunities
2 likes2,033 views
Presentation given on September 18, 2012 at the 'Hadoop in Finance Day' conference held in Chicago and organized by Fountainhead Lab at Microsoft's offices.
Ad
More Related Content
Viewers also liked (16)
Similar to Big Data at CME Group: Challenges and Opportunities (20)
Ad
More from Slim Baltagi (6)
Ad
Recently uploaded (20)
Big Data at CME Group: Challenges and Opportunities
- 1. Big Data at CME Group: Challenges and Opportunities Rick Fath & Slim Baltagi 9/18/2012
- 2. © 2012 CME Group. All rights reserved Agenda 1. CME Group Overview 2. Big Data at CME Group 3. Big Data Use Cases at CME Group 4. Big Data Challenges and Opportunities at CME Group 5. Big Data Key Learning’s at CME Group 6. Questions
- 3. 1. CME Group Overview 3
- 4. © 2012 CME Group. All rights reserved 4 CME Group Overview • #1 futures exchange in the U.S. and globally by 2011 volume • 2011 Revenue of $3.3bn • 11.8 million contracts per day • Strong record of growth, both organically and through acquisitions – Dow Jones Indexes – S&P Indexes – BM&FBOVESPA – CME Clearing Europe – CME Europe Ltd Combination is greater than the sum of its parts
- 5. © 2012 CME Group. All rights reserved 5 Forging Partnerships to Expand Distribution, Build 24-Hour Liquidity, and Add New Customers CBOT Black Sea Wheat Partnerships include: • Equity investments • Trade matching services • Joint product development • Order routing linkages • Product licensing • Joint marketing • European clearing services • Developing capabilities globally • Expanding upon global benchmark products • Positioned well within key strategic closed markets •Recently announced application to FSA for CME Europe Ltd. – expected launch mid-2013
- 6. Big Data: Transactions + Interactions + Observations 2. Big Data at CME Group
- 7. © 2012 CME Group. All rights reserved When you Data gets too Big!! 7
- 8. © 2012 CME Group. All rights reserved Big Data and CME Group: a natural fit Definition (Strategy) of Big Data at CME Group • CME Group is a Big Data factory! 11 million contracts a day… • Data is not merely a byproduct, but an essential deliverable • Daily market data generated by the exchange and also historical data • Growing partnerships around the world • Meeting new regulatory requirements (CFTC, SEC) • Trading shift from floor to electronic • Algorithmic trading improvements-high volumes, low latency, and historical trends
- 9. © 2012 CME Group. All rights reserved Solving Big Data at CME Group • Leverage legacy tools vs. adoption of new Big Data technologies • Transactional data (trades, market data, order data, etc.) • Missed opportunities with existing Legacy Big Data applications: Not all data is being stored and quality is lacking. Learn from this lesson • Raw data being sold without analytics • Risk and Maturity of new Big Data Technologies • Quality of Historical data: different protocol, format.... 9
- 10. 3. Big Data Use Cases at CME Group 10
- 11. © 2012 CME Group. All rights reserved Use Case 1: CME Group's BI solution Description: Provide complex reporting and data analysis for internal Business teams. Challenges: • Performance Needed: Reports and Analysis are time sensitive. (Existing queries taking 18+ hours) • Established tools and partnerships (Informatica, Business Objects, and established business users) • Mission critical queries, timely analytics, complex aggregation required • Risk adverse business customers 11
- 12. © 2012 CME Group. All rights reserved Use Case 1: CME Group's BI solution Opportunities: • Structured RDBMS • Minimize integration impact • Enhance customer analytics • Deliver faster time to market of new Market Regulation requirements • Improve and increase fact based decision making • Reduce batch processes execution duration Solution: 12
- 13. © 2012 CME Group. All rights reserved Why we chose a Data Warehouse Appliance? • No need to re-engineer the existing Oracle Database applications • Based on CME evaluation of Data Warehouse Appliances, Exadata is best suited to CME environment • DWA performs consistently better than our DB production environment. • Product maturity • Faster queries compared to Oracle DB 13
- 14. © 2012 CME Group. All rights reserved Description: Reduce reliance on SAN while increasing query performance. Challenges: • Expensive storage cost. • Oracle queries cannot keep up with inserts and do not meet SLAs Opportunities: • Reduce storage cost with non specialized hardware. “not commodity” • Fast Parallel queries Solution: 14 Use Case 2: CME Group’s Rapid Data Platform
- 15. © 2012 CME Group. All rights reserved Description: Provide historical Market Data to exchange customers who want to understand market trends, conduct market analysis, and buildtest trading algorithms. Challenges: • Expensive storage cost. • Legacy downstream application with limited support • Historical data from acquisitions and mergers…”Don’t look for problems because you will find them” • 100TB data and growing…. • Data Redundancy and Quality (limited awareness) • Business users dependent on Technology staff to use the data. 15 Use Case 3: CME Group’s Historical Data Platform
- 16. © 2012 CME Group. All rights reserved Opportunities: • Reduce storage cost with non specialized hardware. “not commodity” • Reduce duplication of data (derive data on demand fast) • Improve Data Quality with better interrogation • Grow the business (new datasets, mash-ups, analytics) • Separate Delivery from Storage: Store everything, deliver what you need. • Enable Business users (Ad Hoc queries, define new datasets) • Reduce TCO (support, improve reliability) Solution: 16 Use Case 3: CME Group’s Historical Data Platform
- 17. © 2012 CME Group. All rights reserved Why we chose Hadoop? • Solution built for scale and growth • Structure of data doesn’t matter. Data is the Data. • Hadoop as ETL solution • Performance POC with Oracle – Beat Oracle queries and removes duplication • Decouple Storage and Delivery. Store raw data, deliver enhanced data. • Ecosystem reduces development time with proven solutions to common problems • Structure is fluid, perfect for historical data 17
- 18. 4. Big Data Challenges and Opportunities at CME Group 18
- 19. © 2012 CME Group. All rights reserved Enterprise Challenges • Open Source Paradigm • Enterprise Maturity- built for failure • Operations Readiness • Framework can mask problems • Educate the enterprise about Hadoop • Evolution of Hadoop 19
- 20. © 2012 CME Group. All rights reserved Deployment Challenges • Selection of Hadoop distribution • Growing cluster equals growing support fees. How to lower maintenance fees • Future Project candidates (super cluster vs. individual clusters) • Distributions support and alignment • Maximize investment in IT assets (Informatica, BusinessObjects, ExaData) • DR and Backup Solutions 20
- 21. © 2012 CME Group. All rights reserved Potential Hadoop Use Cases for Future 21 • Compliance and regulatory reporting • Trade surveillance • Abnormal trading pattern analysis • Integrate disparate datasets and unlock the value at the intersection of data. • Identify new Big Data projects as a new source of Revenue • Reduce enterprise data redundancy (ETL, storage, analytics), • Use as alternative for costly CEP solutions where applicable
- 22. 5. Big Data Key Learning’s at CME Group 22
- 23. © 2012 CME Group. All rights reserved What we Learned • Embrace open source- “We are not the first ones to solve this problem!!” • Enterprise readiness: DR, Backup, Security, HA • Solving operational maturity (server to admin ratio) • Leverage existing IT investments • Adapt standards to make cluster more supportable • Tackling the learning curve: keep close to community • Leverage the Hadoop Ecosystem… • Capture everything (More capture, support for structured and semi- structured • Capture data first and then figure out new opportunities (schema- less) • Hadoop can be complimentary to existing IT assets. 23
- 24. 6. Questions 24
- 25. © 2012 CME Group. All rights reserved 25 Futures trading is not suitable for all investors, and involves the risk of loss. Futures are a leveraged investment, and because only a percentage of a contract’s value is required to trade, it is possible to lose more than the amount of money deposited for a futures position. Therefore, traders should only use funds that they can afford to lose without affecting their lifestyles. And only a portion of those funds should be devoted to any one trade because they cannot expect to profit on every trade. The Globe Logo, CME®, Chicago Mercantile Exchange®, and Globex® are trademarks of Chicago Mercantile Exchange Inc. CBOT® and the Chicago Board of Trade® are trademarks of the Board of Trade of the City of Chicago. NYMEX, New York Mercantile Exchange, and ClearPort are trademarks of New York Mercantile Exchange, Inc. COMEX is a trademark of Commodity Exchange, Inc. CME Group is a trademark of CME Group Inc. All other trademarks are the property of their respective owners. The information within this presentation has been compiled by CME Group for general purposes only. CME Group assumes no responsibility for any errors or omissions. Although every attempt has been made to ensure the accuracy of the information within this presentation, CME Group assumes no responsibility for any errors or omissions. Additionally, all examples in this presentation are hypothetical situations, used for explanation purposes only, and should not be considered investment advice or the results of actual market experience. All matters pertaining to rules and specifications herein are made subject to and are superseded by official CME, CBOT, NYMEX and CME Group rules. Current rules should be consulted in all cases concerning contract specifications.