H2o storm
- 1. Real-Time Predictions H2O // Storm H2 O.ai Spencer Aiello [email protected] Jan 15, 2015
- 2. H2 O.aiOverview: ● Introductions ● Real Time Analytics ● The Speed of Information ● The Analytics Workflow ● H2O // Storm ● Demo
- 3. H2 O.ai Real Time Analytics: Then & Now 1930 - 1940s Kerrison Predictor ENIAC - Weather Modeling (pseudo real time) 1950s Real Time Analytics to Fight Fraud 1990s Traffic Management Dynamic Pricing Shopping & Movie Recommendations 1970s Real Time Roulette Wheel Prediction With A Computer In A Shoe
- 4. H2 O.ai The Speed of Information Factors to consider: ● Speed of Light ○ 3x108 m/s ● Infrastructure ○ Line-of-sight relays ○ Submarine Cables ○ Where is the information coming from? ○ Where is it going? ○ Lossless? ● Power Consumption ○ Efficiency ● Amount of Information ○ Bandwidth considerations (impacts infrastructure) ○ How quickly can you schlepp around 1TB? 1PB? ■ How quickly do you _need_ to do that? ■ I.e., are you making efficient use of resources?
- 5. H2 O.ai The Shannon Limit: Sup({ Bounds on bits/s }) - C = Channel Capacity (bits/s) - B = Bandwidth (Hz) - S = Signal in Joules/s (Watts) - N = Noise in Joules/s (Watts) The Speed of Information
- 6. H2 O.ai The Speed of Information Consider: The Warning Beacons of Gondor 7 beacons (13 in the movie) Probably 1 cord of wood (~3.6 m3 ) 1 bit of information (@ Shannon Limit) optical transmission Compare to the current World Record: 1 Petabit / second Fiber Transmission over 50-km (~5,000 HDTV Videos/Second over single fiber) About 25 orders of magnitude difference! (source: http://www.ntt.co.jp/news2012/1209e/120920a.html)
- 7. H2 O.ai The Speed of Information AT&T “Long Lines”: ● 838 mile route connecting Chicago to New York ● 4GHz microwave line-of-sight radio relays ● ~25 miles separation (due to curvature of the Earth) ● 34 hops in all High Frequency Trading (HFT): ● Light propagation delays between distant points are relevant sources: - Relativistic Statistical Arbitrage (http://www.alexwg.org/publications/PhysRevE_82-056104.pdf) - Information Transmission Between Financial Markets in Chicago and New York (http://arxiv.org/pdf/1302. 5966v1.pdf)
- 8. H2 O.ai The Speed of Information Observations: ● Moving bits around is a big deal! ● ∃ insurmountable physical and theoretical limitations ○ Shannon Limit ○ Speed of Light ○ Landauer’s Principle ○ Relativistic Effects ○ Curvature of the Earth ● Other limitations or complications? ○ Hairpinning: Non-optimal routing to far flung nodes ■ Geographic locality ≠ Internet locality ○ Bad hardware ○ Bad software
- 9. H2 O.ai (n.d.). Retrieved from http://www.us.ntt.net/support/looking-glass/ (n.d.). Retrieved from http://www.submarinecablemap.com/ The Speed of Information
- 10. H2 O.ai The Analytics Workflow The Analytics Process: 1. Define your problem 2. Gather data and explore 3. Prepare your data for modeling 4. Modeling 5. Model Validation 6. Implementation & Tracking
- 11. H2 O.ai The Analytics Workflow The Analytics Process: 1. Define your problem 2. Gather data and explore 3. Prepare your data for modeling 4. Modeling 5. Model Validation 6. Implementation & Tracking } Here’s where H2O fits into the analytics process http://learn.h2o.ai/content/
- 12. H2 O.ai The Analytics Workflow :::Prep::: Data Preparation: ● A sequence of transformations applied to your data ● This step will define your Storm topology ● Take raw information and give it structure
- 13. H2 O.ai The Analytics Workflow :::Modeling::: Questions to ask yourself: ● How fast must a scoring engine classify incoming tuples? ● How do I optimize between scoring latency and predictive power? ● E.g.What are the trade-offs between a GLM and a GBM? Science!
- 14. H2 O.ai The Analytics Workflow :::Validation::: Types of Validation: ● N-fold cross validation ● Train/Validate/Test -- What Features are Important? ● Model Comparison -- Does your model optimize all needs? ○ Business needs ○ Resource needs ● Repeat steps 3 - 5 until satisfied
- 15. H2 O.ai The Analytics Workflow :::Validation::: Types of Validation: ● N-fold cross validation ● Train/Validate/Test -- What Features are Important? ● Model Comparison -- Does your model optimize all needs? ○ Business needs ○ Resource needs ● Repeat steps 3 - 5 until satisfied WRONG: You should never be satisfied! Your model will go out of date (if it hasn’t already)!
- 16. H2 O.ai The Analytics Workflow :::Tracking::: An Extension of Validation: ● Do not open the fire-hose and blast your model with 100% of your data ○ Expect the unexpected ○ Your topology might will break (oops forgot about unicode… derp) ○ Start off with 10% and ramp up; course-correct along the way ● Perform batch modeling in off-peak hours (Jenkins never sleeps) ● Models should be replaced “gradually”
- 18. H2 O.ai H2O // Storm For a complete tutorial please visit: http://learn.h2o.ai/content/demos/streaming_data.html