HUG_Ireland_Streaming_Ted_Dunning
0 likes171 views
The document discusses the evolution of software architecture from monolithic systems to micro-services, highlighting the challenges of message-based services and the necessity for durable, high-performance message queues. It critiques traditional architectures such as n-tier and ESB, advocating for the use of scalable systems to handle large volumes of messages efficiently. Additionally, it emphasizes the importance of decoupling components and the need for robust infrastructure to support modern applications.
HUG_Ireland_Streaming_Ted_Dunning
- 1. ® © 2014 MapR Technologies 1 ® © 2014 MapR Technologies
- 2. ® © 2014 MapR Technologies 2 Evolution Beyond Massive Monolithic Systems • In monoliths, complexity of mainframe systems led to specialization – Storage – DB – Systems analysis – Programmers – Operations – Data entry • This made n-tier architectures a natural next step
- 3. ® © 2014 MapR Technologies 3 3-tier Architecture Web tier Middle tier Data tier
- 4. ® © 2014 MapR Technologies 4 3-tier Architecture (essence) Web tier Middle tier Data tier
- 5. ® © 2014 MapR Technologies 5 3-tier, in Practice, Multiple Monoliths Web tier Middle tier Data tier Web tier Middle tier Data tier Web tier Middle tier Data tier Web tier Middle tier Data tier
- 6. ® © 2014 MapR Technologies 6 3-tier, in Practice, Multiple Monoliths Web tier Middle tier Data tier Web tier Middle tier Data tier Web tier Middle tier Data tier Web tier Middle tier Data tier
- 7. ® © 2014 MapR Technologies 7 Summary 1 • Tiering leads to tic-tac-toe architectures • ESB leads to control-heavy balls of string – Better than balls of mud, but only just barely – Massive amounts of coupling via shared schemas, brittle protocols • ESB inverts implementation hiding • This isn’t the answer
- 8. ® © 2014 MapR Technologies 8 RPC layer Logic Disk RPC layer Logic Disk RPC layer Logic Disk Start with Service Partitioning
- 9. ® © 2014 MapR Technologies 9 RPC layer Logic Disk RPC layer Logic Disk RPC layer Logic Disk Start with Service Partitioning
- 10. ® © 2014 MapR Technologies 10 RPC layer Logic Disk RPC layer Logic Disk RPC layer Logic Disk Make Services Opaque
- 11. ® © 2014 MapR Technologies 11 Give Them a Job, and a Way to Communicate Keep it very light-weight! Coupling minimized by not knowing details Flexible protocols are required
- 12. ® © 2014 MapR Technologies 12 This is called micro-services
- 13. ® © 2014 MapR Technologies 13 A micro-service is loosely coupled with bounded context
- 14. ® © 2014 MapR Technologies 14 But … • Much of the discussion talks about RPC (call/response) services • This fine, but limiting • Key idiom is deferred processing – Do something urgently – Queue message to complete later
- 15. ® © 2014 MapR Technologies 15 For Message Based Services • The message receiver may not even be running right now – Required by decoupling goal – So we need a persistent queue • The number of messages is plausibly very high – Total number of external requests (x 5-10) – Total number of persistence ops (x 2-3) • Millions of messages, GB/s of traffic quite plausible • Moving this to enterprise from startups adds challenges
- 16. ® © 2014 MapR Technologies 16 Summary 2 • Micro-services requires durable, high-performance message queues • These systems don’t just like durable, high performance queues • These systems require durability. And high performance. • Old school queues need not apply
- 17. ® © 2014 MapR Technologies 17 What does this mean?
- 18. ® © 2014 MapR Technologies 18 Real World Implications • Messaging must be durable and infrastructural – Can’t depend on sender or receiver actually running • Messages aren’t great for everything – 1TB message? • We need (scalable) files • We need (scalable) tables • We need (scalable) streams • We still should isolate persistence if possible
- 19. ® © 2014 MapR Technologies 19 What is Convergence? Files tokyo Streams User profiles Tables
- 20. ® © 2014 MapR Technologies 20 Thank You!
- 21. ® © 2014 MapR Technologies 21 Streaming Architecture by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly) http://bit.ly/mapr-ebook-streams
- 22. ® © 2014 MapR Technologies 22 Short Books by Ted Dunning & Ellen Friedman • Published by O’Reilly in 2014 - 2016 • For sale from Amazon or O’Reilly • Free e-books currently available courtesy of MapR http://bit.ly/ebook-real- world-hadoop http://bit.ly/mapr-tsdb- ebook http://bit.ly/ebook- anomaly http://bit.ly/ recommendation- ebook
- 23. ® © 2014 MapR Technologies 23 Q&A @mapr maprtech [email protected] Engage with us! MapR maprtech mapr-technologies
- 24. ® © 2014 MapR Technologies 24
- 25. ® © 2014 MapR Technologies 25 Streaming Example
- 26. ® © 2014 MapR Technologies 26 mySQL mySQL files Web-site Auth service Upload service Image extractor Transcoder User profiles Search User action logging Recommendation analysis mySQL mySQL mySQL Oracle Solr Elastic
- 27. ® © 2014 MapR Technologies 27 mySQL mySQL files Web-site Auth service Upload service Image extractor Transcoder User profiles Search User action logging Recommendation analysis mySQL mySQL mySQL Oracle Solr Elastic
- 28. ® © 2014 MapR Technologies 28 Upload service Image extractor Transcoder Video metadata Transition to Micro-service
- 29. ® © 2014 MapR Technologies 29 Micro-service Diagram File upload web service Raw files Thumbnail extraction Transcoding Video metadata Video files uploads thumbs recodes Image files
- 30. ® © 2014 MapR Technologies 30 Some Omitted Details File upload web service Files Thumbnail extraction Transcoding uploads thumbs recodes Files
- 31. ® © 2014 MapR Technologies 31 More Omitted details Thumbnail extraction uploads thumbs metrics exceptions checkpoints Input Output Monitoring Restart
- 32. ® © 2014 MapR Technologies 32 Q&A @mapr maprtech [email protected] Engage with us! MapR maprtech mapr-technologies
- 33. ® © 2014 MapR Technologies 33
- 34. ® © 2014 MapR Technologies 34 Migration to Streams
- 35. ® © 2014 MapR Technologies 35 Shared state Application Application Application Migration from Medieval Architectures
- 36. ® © 2014 MapR Technologies 36 Shared state Application Application Application Migration from Medieval Architectures
- 37. ® © 2014 MapR Technologies 37 Shared state Application Application Application events Private state Migration from Medieval Architectures
- 38. ® © 2014 MapR Technologies 38 Q&A @mapr maprtech [email protected] Engage with us! MapR maprtech mapr-technologies