Lightbend Fast Data Platform
3 likes666 views
The Lightbend Fast Data Platform provides a framework for building streaming data systems and microservices, emphasizing asynchronous operation and continuous service offerings. It includes best practices, sample applications, and machine learning-based monitoring to ensure resiliency and scalability. The platform supports various data storage solutions and provides detailed consideration of performance trade-offs in streaming applications.
![•Each [stream app or μservice]:
•does one responsibility
•ingests unending [data or messages]
Synergies
22
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink](https://image.slidesharecdn.com/fastdataplatformfeb-2017b-170216225910/85/Lightbend-Fast-Data-Platform-22-320.jpg)
![•Each [stream app or μservice] must:
•operate asynchronously
•offer never-ending service
Synergies
23
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Streaming
Streams
SQL
Batch
…
Streaming
Flink](https://image.slidesharecdn.com/fastdataplatformfeb-2017b-170216225910/85/Lightbend-Fast-Data-Platform-23-320.jpg)
Lightbend Fast Data Platform
- 1. Lightbend Fast Data Platform A Technical Overview
- 2. A Bit of History: IT Services vs. Big Data
- 3. Big DataServices Some Overlap: Concerns, Architecture The Recent Past…
- 4. Microservices & Fast Data Much More Overlap The Future?
- 5. Drilling Down: Monoliths to Microservices
- 7. • Tangled responsibilities, lead to infrequent, “big-bang” deployments • App lifetimes months to forever! Monoliths
- 9. • Each does one thing, so must be message driven & asynchronous • Updates easier, deployments frequent • App lifetimes: minutes! to forever Microservices
- 10. • But be careful: • Message overhead is much better than function calls! Microservices
- 15. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Hadoop 2013: Embrace Spark Spark job #1 Spark job #2
- 16. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Go beyond batch? Spark job #1 Spark job #2
- 17. The Emergence of Fast Data: (Time Is Money)
- 19. IoT
- 21. Fast Data and Microservices; Are they Converging?
- 22. •Each [stream app or μservice]: •does one responsibility •ingests unending [data or messages] Synergies 22 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
- 23. •Each [stream app or μservice] must: •operate asynchronously •offer never-ending service Synergies 23 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
- 24. •These architectures are converging: 1.Similar design problems 2.Data becomes dominant problem Thesis 24 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
- 26. 1. An accelerated on ramp for building streaming data systems, data applications, and other microservices. Value Three Ways
- 27. 2. Best practices guidance for solving specific design problems: - Sample apps - Documentation - Enablement services Value Three Ways
- 28. 3. Machine learning-based monitoring and management: -Keep your systems resilient, scalable, and responsive with minimal user intervention. Value Three Ways
- 29. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 30. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 31. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 32. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 33. Service 1 Log & Other Files Internet Services Service 2 Service 3 Services Services N * M links ConsumersProducers
- 34. Service 1 Log & Other Files Internet Services Service 2 Service 3 Services Services N + M links ConsumersProducers
- 35. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 36. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 37. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 38. •Low latency? How low? •High volume? How high? Streaming Tradeoffs (1/3) 38
- 39. •Which kinds of data processing & analytics are required? •How will this processing be done? •Individual processing of events? •Bulk processing of records? Streaming Tradeoffs (2/3) 39
- 40. •Which tools and data sources/sinks must interoperate with your streaming tool? Streaming Tradeoffs (3/3) 40
- 41. 41
- 42. 42 •Low latency •Low volume •Complex flows •Complex Event Processing
- 43. 43 •Med. latency •High volume •Data flows, SQL •En masse processing
- 44. 44 •Low latency •High volume •Data flows, correctness •En masse processing
- 45. 45 •Low latency •Med. volume •ETL, “tables” •Data flow or per event
- 46. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 47. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 48. Kafka & Spark Metrics
- 50. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 51. • Only metrics are uploaded, not sensitive domain data. • One less service for you to manage. • … Why Only Hosted Services?
- 52. • … • We can rapidly evolve this service. without impacting your environment. • You benefit from aggregated knowledge from all FDP clusters. Why Only Hosted Services?
- 53. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 54. •Alpha: Jan 9 •Beta 1: April •Beta 2: July •V1.0: September Timeline 54
- 55. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS, Marathon On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted Graviton Cluster Analysis Machine LearningMicroservices ProducCon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles For More Information: lightbend.com/fast-data-platform
- 58. • FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data