IoT Architecture - are traditional architectures good enough or do we need new approaches?
10 likes•3,449 views
The document discusses IoT architecture, emphasizing the need for new approaches to integrate IoT with existing systems. It highlights the essential requirements for an effective IoT architecture, such as scalability and recoverability, and outlines various solutions and platforms, including Talend's unified platform for IoT and big data. Additionally, it stresses that understanding specific use cases and requirements is crucial for selecting the right architecture.
IoT Architecture - are traditional architectures good enough or do we need new approaches?
- 1. 2014 © Trivadis 19.11.2014 IoT Architektur 1 IoT Architektur Sind traditionelle Architekturen gut genug, oder braucht es neue Ansätze? Bernd Trops (Talend) Guido Schmutz (Trivadis) 24.03.2015 Open Source Integration
- 2. 2014 © Trivadis Bernd Trops • Working for Talend for more than 7 years • Co-Author of different books • Consultant, Trainer and Software Architect for Java, SOA • Masons of SOA: http://www.soacommunity.com/index.php/institutional/masons-of-soa • Director Professional Services DACH at Talend • More than 25 years of software development experience • Contact: [email protected] 19.11.2014 IoT Architektur 2
- 3. 2014 © Trivadis Guido Schmutz • Working for Trivadis for more than 17 years • Oracle ACE Director for Fusion Middleware and SOA • Co-Author of different books • Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data • Member of Trivadis Architecture Board • Technology Manager @ Trivadis • More than 25 years of software development experience • Contact: [email protected] • Blog: http://guidoschmutz.wordpress.com • Twitter: gschmutz 19.11.2014 IoT Architektur 3
- 4. 2014 © Trivadis Agenda 1. Introduction / Motivation 2. IoT Reference Architecture 3. Components supporting the IoT Reference Architecture 4. Talend – Unified Platform for IoT and Big Data 5. Summary 19.11.2014 IoT Architektur 4
- 5. 2014 © Trivadis Data, Data, Data – Predicted Growth from IoT 19.11.2014 IoT Architektur 5 Source: Brideworks http://bit.ly/19Msx5N
- 6. 2014 © Trivadis Data as an Asset - Store Anything? But then data is just too valuable to delete! We must store anything! Nonsense! Just store the data you know you need today! It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today! 6 19.11.2014 IoT Architektur
- 7. 2014 © Trivadis Requirements for an IoT Architecture • Support for failure • systems should gracefully degrade into lower performing states • If a failed component recovers, it should be able to rejoin the cluster • Recoverability • In the event of a failure, no data should be lost • Consistency • Failure of one task should not affect the result of the entire job • Scalability • Adding load leads to a decline of performance, not failure • Increasing resources results in proportional increase in capacity • Scale-up but also scale-down => elasticity 7 19.11.2014 IoT Architektur
- 8. 2014 © Trivadis Agenda 1. Introduction / Motivation 2. IoT Reference Architecture 3. Components supporting the IoT Reference Architecture 4. Talend – Unified Platform for IoT and Big Data 5. Summary 19.11.2014 IoT Architektur 8
- 9. 2014 © Trivadis IoT Solution Types 19.11.2014 IoT Architektur 9 Device On-Device Logic Gateway Processing Backend Enterprise Apps Enterprise IoT M2M Local Hub Device to Device (P2P) Device to Backend Adapted from: http://enterprise-iot.org
- 10. 2014 © Trivadis EnterpriseServiceBus(ESB) Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device Services WS Event Processes Gateway Event Procesing IoT Smart Device Viszualization Analytics DB 19.11.2014 IoT Architektur 10 APIGateway = one way = request/response 1) IoT with Simple Event Processing 10 Mobile Apps DB Rich (Web) Client Apps DB
- 11. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP ESP/CEP IoT Smart Device Viszualization Analytics DB DB DB 19.11.2014 IoT Architektur 11 APIGateway = one way = request/response 11 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams 2) IoT with Event Stream/Complex Event Processing
- 12. 2014 © Trivadis EnterpriseServiceBus(ESB) 3) IoT with Big Data Analytics Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/ CEP Hadoop Big Data Infrastructure HDFS Batch Processing IoT Smart Device Viszualization Analytics DBDB 19.11.2014 IoT Architektur 12 APIGateway = one way = request/response 12 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams
- 13. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Batch Processing IoT Smart Device Viszualization Analytics DB DB DB DB 19.11.2014 IoT Architektur 13 APIGateway 4) IoT with “Lambda” Architecture = one way = request/response 13 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams
- 14. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Replay to reprocess IoT Smart Device Viszualization Analytics DB DB DB 19.11.2014 IoT Architektur 14 APIGateway 5) IoT with “Kappa” Architecture = one way = request/response 14 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams
- 15. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Batch Processing IoT Smart Device Viszualization Analytics DB DB DB DB 19.11.2014 IoT Architektur 15 APIGateway 6) IoT with Big Data Advanced Analytics = one way = request/response 15 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams
- 16. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure IoT Reference Architecture Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device 16 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Batch Processing IoT Smart Device Viszualization Analytics DB DB DB DB 19.11.2014 IoT Architektur 16 APIGateway = one way = request/response
- 17. 2014 © Trivadis IoT Reference Architecture 19.11.2014 IoT Architektur 17 EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device 17 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Batch Processing IoT Smart Device Viszualization Analytics DB DB DB DB APIGateway = one way = request/response
- 18. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device 18 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Batch Processing IoT Smart Device Viszualization Analytics DB DB DB DB APIGateway 19.11.2014 IoT Architektur 18 IoT Reference Architecture – Supporting Components
- 19. 2014 © Trivadis Agenda 1. Introduction / Motivation 2. IoT Reference Architecture 3. Components supporting the IoT Reference Architecture 4. Talend – Unified Platform for IoT and Big Data 5. Summary 19.11.2014 IoT Architektur 19
- 20. 2014 © Trivadis Enterprise Event Bus - Apache Kafka • A distributed publish-subscribe messaging system • Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …) • Initially developed at LinkedIn, now part of Apache • Does not follow JMS Standards and does not use JMS API • Kafka maintains feeds of messages in topics 19.11.2014 IoT Architektur 20 Kafka Cluster Consumer Consumer Consumer Producer Producer Producer 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 Anatomy of a topic: Partition 0 Partition 1 Partition 2 Writes old new
- 21. 2014 © Trivadis Enterprise Event Bus - Apache Kafka Kafka at LinkedIn Up to 2 million writes/sec on 3 cheap machines § Using 3 producers on 3 different machines 19.11.2014 IoT Architektur 21 10+ billion writes per day 172k messages per second (average) 55+ billion messages per day to real-time consumers http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
- 22. 2014 © Trivadis Management / Monitoring Infrastructure Batch (Big) Data Processing - Hadoop Ecosystem 22 Core Analytics Workflow/JobUnstructured Data Sources Structured Data Sources SQL on Hadoop SerializationData Storage Security 19.11.2014 IoT Architektur
- 23. 2014 © Trivadis Batch (Big) Data Processing - Classical Hadoop vs. Spark SparkHadoop Map Reduce 19.11.2014 IoT Architektur 23 HDFS Map Map Reduce Reduce Data Data Map Join Transform Cache Filter Join Data
- 24. 2014 © Trivadis Stream Processing – Apache Storm Tuple • Immutable Set of Key/value pairs Stream • an unbounded sequence of tuples that can be processed in parallel by Storm Topology • Wires data and functions via a DAG (directed acyclic graph) • Executes on many machines similar to a MR job in Hadoop Spout • Source of data streams (tuples) • can be run in “reliable” and “unreliable” mode Bolt • Consumes 1+ streams and produces new streams • Complex operations often require multiple steps and thus multiple bolts 19.11.2014 IoT Architektur 24 Spout Spout Bolt Bolt Bolt Bolt Source of Stream B Subscribes: A Emits: C Subscribes: A Emits: D Subscribes: A & B Emits: - Subscribes: C & D Emits: - T T T T T T T T
- 25. 2014 © Trivadis Data Storage (Batch) – Cloudera Impala High-performance SQL engine for vast amounts of data • Massively-parallel processing (MPP) • Inspired by Google's Dremel project • Query latency measured in milliseconds Impala runs on Hadoop clusters • Can query data stored in HDFS or HBase tables • Reads and writes data in common Hadoop file formats Developed by Cloudera • 100% open source, release under the Apache software license 19.11.2014 IoT Architektur 25
- 26. 2014 © Trivadis Data Storage (Speed) – Apache Cassandra • Developed at Facebook • Open source distributed database management system • Professional grade support from company called DataStax • Main Features § Real-Time § Highly Distributed § Support for Multiple Data Center § Highly Scalable § No Single Point of Failure § Fault Tolerant § Tunable Consistency § Cassandra Query Language (CQL) 19.11.2014 IoT Architektur 26
- 27. 2014 © Trivadis Data Storage (Speed) – Apache Cassandra • Don’t think of relational table => more of a sorted map • Know your application => model around the queries • De-normalize and duplicate for read performance • Index is not an afterthought, anymore=> index upfront • Think of physical storage structure 19.11.2014 IoT Architektur 27 2013-08 Day-1, keyword-1=>100 Day-2, keyword-1=>150 Day-3, keyword-1=>170 …. Day-31, keyword-1 =>170 2013-08-31 Hour-1, keyword-1 =>10 Hour-2, keyword-1 =>15 Hour-3, keyword-1 =>17 …. Hour-24, keyword-1 =>17 2013-08-31-10 Minute-1, keyword-1=>2 Minute-2, keyword-1=>3 Minute-3, keyword-1 =>5 …. Minute-60, keyword-1=>2 Row-key Columns à
- 28. 2014 © Trivadis Agenda 1. Introduction / Motivation 2. IoT Reference Architecture 3. Components supporting the IoT Reference Architecture 4. Talend – Unified Platform for IoT and Big Data 5. Summary 19.11.2014 IoT Architektur 28
- 29. 2014 © Trivadis EnterpriseServiceBus(ESB) Stream Processing Infrastructure Enterprise Applications WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence IoT Device 29 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams EnterpriseEventBus(Ingress) Enterprise EventBus Services WS Event Processes Gateway ESP/CEP Hadoop Big Data Infrastructure HDFS ESP/CEP Batch Processing IoT Smart Device Viszualization Analytics DB DB DB DB APIGateway Talend – Unified Platform for IoT and Big Data 19.11.2014 IoT Architektur 29
- 30. 2014 © Trivadis Talend Kafka Support 19.11.2014 IoT Architektur 30
- 31. 2014 © Trivadis Talend Spark Support 19.11.2014 IoT Architektur 31
- 32. 2014 © Trivadis Agenda 1. Introduction / Motivation 2. IoT Reference Architecture 3. Components supporting the IoT Reference Architecture 4. Talend – Unified Platform for IoT and Big Data 5. Summary 19.11.2014 IoT Architektur 32
- 33. 2014 © Trivadis Summary • IoT – The technologies are ready to use • IoT tends to make Big Data infrastructures necessary • There are architectures to deal with IoT -> Backend • Know your use case/requirements to choose the right architecture • Can my existing backend landscape handle the new IoT load? • Do I have to handle huge amount of events in “real-time”? • Do I need to filter/aggregate data before invoking existing backend systems? • Do I want to do Advanced Analytics (predictive analytics) where large amounts of historical information is necessary? • Business cases starting to increase 19.11.2014 IoT Architektur 33
- 34. 2014 © Trivadis Weitere Informationen... 19.11.2014 IoT Architektur 34
- 35. 2014 © Trivadis BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN Fragen und Antworten... 2013 © Trivadis Guido Schmutz Technology Manager [email protected] 19.11.2014 IoT Architektur