Storm: The Real-Time Layer - GlueCon 2012
43 likes•11,838 views
The document discusses Storm, a distributed and fault-tolerant real-time computation system designed to handle big data efficiently. It emphasizes the importance of real-time data processing, key components like tuples, streams, spouts, and bolts, as well as how to set up a streaming word count topology. The content outlines how Storm provides a solution to outdated batch processing methods through streaming computations.
Storm: The Real-Time Layer - GlueCon 2012
- 1. Storm The Real-Time Layer Your Big Data’s Been Missing Dan Lynn [email protected] @danklynn
- 2. Keeps Contact Information Current and Complete Based in Denver, Colorado CTO & Co-Founder [email protected] @danklynn
- 3. Turn Partial Contacts Into Full Contacts
- 4. Your data is old
- 5. Old data is confusing
- 6. First, there was the spreadsheet
- 7. Next, there was SQL
- 8. Then, there wasn’t SQL
- 9. Batch Processing = Stale Data
- 11. Queues / Workers
- 12. Queues / Workers Messages Messages Messages Messages Messages Messages Queue Messages Workers
- 13. Queues / Workers Me ssage rou ting can be com ple x Messages Messages Messages Messages Messages Messages Queue Messages Workers
- 14. Queues / Workers
- 15. Brittle
- 16. Hard to scale
- 17. Queues / Workers
- 18. Queues / Workers
- 19. Queues / Workers
- 20. Queues / Workers Rou ting mus t be reco nfig ured whe n sca ling out
- 21. Storm
- 22. Storm Distributed and fault-tolerant real-time computation
- 23. Storm Distributed and fault-tolerant real-time computation
- 24. Storm Distributed and fault-tolerant real-time computation
- 25. Storm Distributed and fault-tolerant real-time computation
- 26. Key Concepts
- 27. Tuples Ordered list of elements
- 28. Tuples Ordered list of elements ("search-01384", "e:[email protected]")
- 29. Streams Unbounded sequence of tuples
- 30. Streams Unbounded sequence of tuples Tuple Tuple Tuple Tuple Tuple Tuple
- 31. Spouts Source of streams
- 32. Spouts Source of streams
- 33. Spouts Source of streams Tuple Tuple Tuple Tuple Tuple Tuple
- 34. Spouts can talk with some images from http://commons.wikimedia.org
- 35. Spouts can talk with •Queues some images from http://commons.wikimedia.org
- 36. Spouts can talk with •Queues •Web logs some images from http://commons.wikimedia.org
- 37. Spouts can talk with •Queues •Web logs •API calls some images from http://commons.wikimedia.org
- 38. Spouts can talk with •Queues •Web logs •API calls •Event data some images from http://commons.wikimedia.org
- 39. Bolts Process tuples and create new streams
- 40. Bolts Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple some images from http://commons.wikimedia.org
- 41. Bolts some images from http://commons.wikimedia.org
- 42. Bolts •Apply functions / transforms some images from http://commons.wikimedia.org
- 43. Bolts •Apply functions / transforms •Filter some images from http://commons.wikimedia.org
- 44. Bolts •Apply functions / transforms •Filter •Aggregation some images from http://commons.wikimedia.org
- 45. Bolts •Apply functions / transforms •Filter •Aggregation •Streaming joins some images from http://commons.wikimedia.org
- 46. Bolts •Apply functions / transforms •Filter •Aggregation •Streaming joins •Access DBs, APIs, etc... some images from http://commons.wikimedia.org
- 47. Topologies A directed graph of Spouts and Bolts
- 48. This is a Topology some images from http://commons.wikimedia.org
- 49. This is also a topology some images from http://commons.wikimedia.org
- 50. Tasks Processes which execute Streams or Bolts
- 51. Running a Topology $ storm jar my-code.jar com.example.MyTopology arg1 arg2
- 52. Storm Cluster Nathan Marz
- 53. Storm Cluster If thi s we re Hadoo p... Nathan Marz
- 54. Storm Cluster If thi s we re Hadoo p... Job Tracke r Nathan Marz
- 55. Storm Cluster If thi s we re Hadoo p... Tas k Tracke rs Nathan Marz
- 56. Storm Cluster But it’s not Hado op Coo rdi nates eve ry thi ng Nathan Marz
- 58. Streaming Word Count TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("sentences", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("sentences"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields("word"));
- 59. Streaming Word Count TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("sentences", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("sentences"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields("word"));
- 60. Streaming Word Count public static class SplitSentence extends ShellBolt implements IRichBolt { public SplitSentence() { super("python", "splitsentence.py"); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } @Override public Map<String, Object> getComponentConfiguration() { return null; } } SplitSentence.java
- 61. Streaming Word Count public static class SplitSentence extends ShellBolt implements IRichBolt { public SplitSentence() { super("python", "splitsentence.py"); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } @Override public Map<String, Object> getComponentConfiguration() { return null; } } SplitSentence.java splitsentence.py
- 62. Streaming Word Count public static class SplitSentence extends ShellBolt implements IRichBolt { public SplitSentence() { super("python", "splitsentence.py"); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } @Override public Map<String, Object> getComponentConfiguration() { return null; } } SplitSentence.java
- 63. Streaming Word Count TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("sentences", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("sentences"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields("word")); java
- 64. Streaming Word Count public static class WordCount extends BaseBasicBolt { Map<String, Integer> counts = new HashMap<String, Integer>(); @Override public void execute(Tuple tuple, BasicOutputCollector collector) { String word = tuple.getString(0); Integer count = counts.get(word); if(count==null) count = 0; count++; counts.put(word, count); collector.emit(new Values(word, count)); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word", "count")); } } WordCount.java
- 65. Streaming Word Count TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("sentences", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("sentences"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields("word")); java Gro upings con tro l how tup les are rou ted
- 66. Shuffle grouping Tuples are randomly distributed across all of the tasks running the bolt
- 67. Fields grouping Groups tuples by specific named fields and routes them to the same task
- 68. o p’s ado r t o H av i o o us b e h An a lo g i ng rt i t io n pa Fields grouping Groups tuples by specific named fields and routes them to the same task
- 69. Distributed RPC
- 70. Before Distributed RPC, time-sensitive queries relied on a pre-computed index
- 71. What if you didn’t need an index?
- 72. Distributed RPC
- 73. Try it out! Huge thanks to Nathan Marz - @nathanmarz http://github.com/nathanmarz/storm https://github.com/nathanmarz/storm-starter @stormprocessor