Distributed computing with Hazelcast - JavaOne 2014
2 likes1,598 views
Today’s amounts of collected data are showing nearly exponential growth. More than 75 percent of all collected data has been collected in the past five years. To store that data and process it within an appropriate time, you need to partition the data and parallelize the processing of reports and analytics. This session demonstrates how to quickly and easily parallelize data processing with Hazelcast and its underlying distributed data structures. By giving a few quick introductions to different terms and some short live coding sessions, the presentation takes you on a journey through distributed computing.
![SOME PSEUDO CODE (2/3)
COMBINING
Multiple values are combined to an
intermediate result to preserve traffic
combine( word:Word, counts:List[Int] ):Void ->
emit( word, sum( counts ) )
www.hazelcast.com](https://image.slidesharecdn.com/distributedcomputingwithhazelcastjavaone-141010133459-conversion-gate02/85/Distributed-computing-with-Hazelcast-JavaOne-2014-34-320.jpg)
![SOME PSEUDO CODE (3/3)
REDUCING
Values are reduced / aggregated to the requested result
reduce( word:String, counts:List[Int] ):Int ->
return sum( counts )
www.hazelcast.com](https://image.slidesharecdn.com/distributedcomputingwithhazelcastjavaone-141010133459-conversion-gate02/85/Distributed-computing-with-Hazelcast-JavaOne-2014-35-320.jpg)
![FOR MATHEMATICIANS
Process: (K x V)* → (L x W)* ¶ [(l1, w1), …, (lm, wm)]
Mapping: (K x V) → (L x W)* ¶ (k, v) → [(l1, w1), …, (ln, wn)]
Reducing: L x W* → X* ¶ (l, [w1, …, wn]) → [x1, …,xn]
www.hazelcast.com](https://image.slidesharecdn.com/distributedcomputingwithhazelcastjavaone-141010133459-conversion-gate02/85/Distributed-computing-with-Hazelcast-JavaOne-2014-36-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (18)
![[OracleCode SF] In memory analytics with apache spark and hazelcast](https://cdn.slidesharecdn.com/ss_thumbnails/in-memoryanalyticswithapachesparkandhazelcast-oraclecode-03-01-2017-170302180618-thumbnail.jpg?width=560&fit=bounds)
Ad
Similar to Distributed computing with Hazelcast - JavaOne 2014 (20)
![[Jfokus] Riding the Jet Streams](https://cdn.slidesharecdn.com/ss_thumbnails/ridingthejetstreams-jfokus-02-08-2017-170208120631-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Christoph Engelbert (20)
Recently uploaded (20)
Distributed computing with Hazelcast - JavaOne 2014
- 1. DISTRIBUTED COMPUTING WITH HAZELCAST www.hazelcast.com
- 2. DISCLAIMER This presentation may include certain forward-looking statements and projections provided by the Company. Any such statements and projections reflect various estimates and assumptions by the Company concerning anticipated results. These statements involve risks, uncertainties and assumptions and are based on the current estimates and assumptions of the management of the Company as of the date of this presentation and are subject to uncertainty and changes. No representations or warranties are made by the Company as to the accuracy of any such statements or projections. Whether or not any such forward-looking statements or projections are in fact achieved will depend upon future events some of which are not within the control of the Company. Accordingly, actual results may vary from the projected results and such variations may be material. Statements contained herein describing documents and agreements are summaries only and such summaries are qualified in their entirety by reference to such documents and agreements. www.hazelcast.com
- 3. THAT'S ME Chris(toph) Engelbert Twitter: @noctarius2k 8+ years of Java Weirdoness Performance, GC, traffic topics Apache Committer Gaming, Travel Management, ... www.hazelcast.com
- 4. OUR SPACE TRIP ROADMAP Hazelcast Distributed Computing Distributed ExecutorService EntryProcessor Map & Reduce Questions www.hazelcast.com
- 5. HAZELCAST PICKIN' DIAMONDS www.hazelcast.com
- 6. WHAT IS HAZELCAST? In-Memory Data-Grid Data Partioning (Sharding) Java Collections Implementation Distributed Computing Platform www.hazelcast.com
- 7. WHY HAZELCAST? Automatic Partitioning Fault Tolerance Sync / Async Backups Fully Distributed In-Memory for Highest Speed www.hazelcast.com
- 10. WHY IN-MEMORY COMPUTING? www.hazelcast.com
- 11. TREND OF PRICES Data Source: http://www.jcmit.com/memoryprice.htm www.hazelcast.com
- 12. SPEED DIFFERENCE Data Source: http://i.imgur.com/ykOjTVw.png www.hazelcast.com
- 13. DISTRIBUTED COMPUTING OR MULTICORE CPU ON STEROIDS www.hazelcast.com
- 14. THE IDEA OF DISTRIBUTED COMPUTING Source: https://www.flickr.com/photos/stefan_ledwina/1853508040 www.hazelcast.com
- 15. THE BEGINNING Source: http://en.wikipedia.org/wiki/File:KL_Advanced_Micro_Devices_AM9080.jpg www.hazelcast.com
- 16. MULTICORE IS NOT NEW Source: http://en.wikipedia.org/wiki/File:80386with387.JPG www.hazelcast.com
- 17. CLUSTER IT Source: http://rarecpus.com/images2/cpu_cluster.jpg www.hazelcast.com
- 18. SUPER COMPUTER Source: http://www.dkrz.de/about/aufgaben/dkrz-geschichte/rechnerhistorie-1 www.hazelcast.com
- 19. CLOUD COMPUTING Source: https://farm6.staticflickr.com/5523/11407118963_e0e0870846_b_d.jpg www.hazelcast.com
- 20. DISTRIBUTED EXECUTORSERVICE THE WHOLE CLUSTER IN YOUR HANDS www.hazelcast.com
- 21. WHY A DISTRIBUTED EXECUTORSERVICE? j.l.Runnable / j.u.c.Callable Only needs to be serializable Same Task all / multiple Nodes Should not work on Data www.hazelcast.com
- 22. QUICK EXAMPLE Talk to me from all nodes Runnable runnable = () -> println( "Hello from my Cluster-World" ); IExecutorService es = hzInstance.getExecutorService( "default" ); es.executeOnAllMembers( runnable ); www.hazelcast.com
- 24. ENTRYPROCESSOR LOCKFREE DATA OPERATIONS www.hazelcast.com
- 25. WHY ENTRYPROCESSOR? Prevents external Locking Guarantees Atomicity Kind of a "Cluster-wide Thread-Safe" www.hazelcast.com
- 26. QUICK EXAMPLE Incrementing a counter atomically private int increment( Map.Entry entry ) { val newValue = entry.getValue() + 1; entry.setValue( newValue ); return newValue; } IMap map = hazelcastInstance.getMap( "default" ); int newId = map.executeOnKey( "idgen", this::increment ); www.hazelcast.com
- 28. MAP & REDUCE THE BLACK MAGIC FROM PLANET GOOGLE www.hazelcast.com
- 29. USE CASES Log Analysis Data Querying Aggregation and summing Distributed Sort ETL (Extract Transform Load) and more... www.hazelcast.com
- 30. SIMPLE STEPS Read Map / Transform Reduce www.hazelcast.com
- 31. FULL STEPS Read Map / Transform Combining Grouping / Shuffling Reduce Collating www.hazelcast.com
- 33. SOME PSEUDO CODE (1/3) MAPPING Data are mapped / transformed in a set of key-value pairs map( key:String, document:String ):Void -> for each w:Word in document: emit( w, 1 ) www.hazelcast.com
- 34. SOME PSEUDO CODE (2/3) COMBINING Multiple values are combined to an intermediate result to preserve traffic combine( word:Word, counts:List[Int] ):Void -> emit( word, sum( counts ) ) www.hazelcast.com
- 35. SOME PSEUDO CODE (3/3) REDUCING Values are reduced / aggregated to the requested result reduce( word:String, counts:List[Int] ):Int -> return sum( counts ) www.hazelcast.com
- 36. FOR MATHEMATICIANS Process: (K x V)* → (L x W)* ¶ [(l1, w1), …, (lm, wm)] Mapping: (K x V) → (L x W)* ¶ (k, v) → [(l1, w1), …, (ln, wn)] Reducing: L x W* → X* ¶ (l, [w1, …, wn]) → [x1, …,xn] www.hazelcast.com
- 37. MAPREDUCE PROGRAMS IN GOOGLE SOURCE TREE Source: http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html www.hazelcast.com
- 39. THANK YOU! ANY QUESTIONS? @noctarius2k @hazelcast http://www.sourceprojects.com http://github.com/noctarius www.hazelcast.com