Lambda architecture
2 likes1,321 views
The Lambda Architecture is a data processing architecture designed to handle large volumes of data by separating the data flow into batch, serving and speed layers. The batch layer computes views over all available data but has high latency. The serving layer serves queries using pre-computed batch views but cannot answer queries in real-time. The speed layer computes real-time views incrementally from new data and answers queries with low latency. Together these layers are able to provide robust, scalable and low-latency query capabilities over massive datasets.
Ad
More Related Content
What's hot (20)
Similar to Lambda architecture (20)
![[Heap con19] designing data intensive applications in serverless architecture](https://cdn.slidesharecdn.com/ss_thumbnails/heapcon19designingdataintensiveapplications-191004142134-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Szilveszter Molnár (7)
Ad
Recently uploaded (20)
Lambda architecture
- 2. LAMBDA ARCHITECTURE A NEW PARADIGM ▸ 30,000 GB of data / second ▸ Traditional database systems at limit, failed ▸ New breed of technologies ▸ MapReduce ▸ Distributed Key/Value Stores
- 4. LAMBDA ARCHITECTURE WEB ANALYTICS APPLICATION ▸ Number of page views for URL
- 5. LAMBDA ARCHITECTURE TRADITIONAL DATABASE Column Name id user_id url pageviews
- 6. LAMBDA ARCHITECTURE PROBLEMS START "Timeout error on inserting to the database"
- 8. LAMBDA ARCHITECTURE NEXT STEP - SHARDING THE DATABASE ▸ How to scale write-heavy relational database? ▸ horizontal partitioning ▸ sharding ▸ You migrate data to 4 shards ▸ Your app is getting more & more popular ▸ more & more painful
- 9. LAMBDA ARCHITECTURE EVEN MORE PROBLEMS ▸ Fault-tolerance issues ▸ disks fail often ▸ add read replica ▸ Corruption issues ▸ deploy a bug, notice one day later
- 10. LAMBDA ARCHITECTURE PROBLEMS OF "FULLY INCREMENTAL" ARCHITECTURES APPLICATION DATABASE
- 11. LAMBDA ARCHITECTURE PROBLEMS OF "FULLY INCREMENTAL" ARCHITECTURES ▸ Operational Complexity ▸ Eventual Consistency ▸ Lack of human-fault tolerance
- 12. SURE WE CAN DO BETTER
- 13. LAMBDA ARCHITECTURE PRINCIPLES "A data system answers questions based on information that was acquired in the past up to the present." Big Data, Nathan Marz
- 14. LAMBDA ARCHITECTURE PRINCIPLES ▸ What is this person's name? ▸ How many friends does this person have? ▸ What is my current balance?
- 15. LAMBDA ARCHITECTURE PRINCIPLES ▸ Not all bits of information are equal ▸ Some are derived from other piece of information
- 16. LAMBDA ARCHITECTURE PRINCIPLES ▸ How many friends does this person have? ▸ friend list changes (add / remove friends) ▸ What is my current balance? ▸ transaction history
- 18. LAMBDA ARCHITECTURE PRINCIPLES query result = function(all data)
- 19. LAMBDA ARCHITECTURE DESIRED PROPERTIES OF A BIG DATA SYSTEM ▸ Robustness & Fault tolerance ▸ Low latency reads & updates ▸ Scalability
- 21. LAMBDA ARCHITECTURE BATCH LAYER SERVING LAYER SPEED LAYER
- 22. LAMBDA ARCHITECTURE BATCH LAYER query result = function(all data) BATCH LAYER SERVING LAYER SPEED LAYER
- 23. LAMBDA ARCHITECTURE BATCH LAYER batch computed view = function(all data) query result = function(batch computed view) BATCH LAYER SERVING LAYER SPEED LAYER
- 24. LAMBDA ARCHITECTURE BATCH LAYER ALL DATA BATCH LAYER BATCH VIEW BATCH VIEW BATCH VIEW BATCH LAYER SERVING LAYER SPEED LAYER
- 25. LAMBDA ARCHITECTURE SERVING LAYER BATCH LAYER SERVING LAYER SPEED LAYER BATCH LAYER MASTER DATASET SERVING LAYER BATCH VIEW BATCH VIEW BATCH VIEW QUERY
- 26. LAMBDA ARCHITECTURE DESIRED PROPERTIES OF A BIG DATA SYSTEM ▸ Satisfied ▸ Robustness & Fault tolerance ▸ Scalability ▸ Not Satisfied ▸ Low latency reads & updates BATCH LAYER SERVING LAYER SPEED LAYER
- 27. LAMBDA ARCHITECTURE SPEED LAYER ‣ Batch layer runs for several hours ‣ Looks only at new data BATCH LAYER SERVING LAYER SPEED LAYER
- 28. LAMBDA ARCHITECTURE SPEED LAYER realtime views = function(realtime views, new data) BATCH LAYER SERVING LAYER SPEED LAYER
- 29. LAMBDA ARCHITECTURE BATCH LAYER, SERVING LAYER, SPEED LAYER batch view = function(all data) realtime views = function(realtime views, new data) query result = function(batch view, realtime view) BATCH LAYER SERVING LAYER SPEED LAYER
- 30. LAMBDA ARCHITECTURE SPEED LAYER NEW DATA BATCH LAYER MASTER DATASET SERVING LAYER BATCH VIEW BATCH VIEW BATCH VIEW REALTIME VIEW REALTIME VIEW REALTIME VIEW QUERY