Best Practices for Streaming IoT Data with MQTT and Apache KafkaKai Wähner
Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges. In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.
We use HiveMQ as open source MQTT broker to ingest data from IoT devices, ingest the data in real time into an Apache Kafka cluster for preprocessing (using Kafka Streams / KSQL), and model training + inference (using TensorFlow 2.0 and its TensorFlow I/O Kafka plugin).
We leverage additional enterprise components from HiveMQ and Confluent to allow easy operations, scalability and monitoring.
Building Cloud-Native App Series - Part 2 of 11
Microservices Architecture Series
Event Sourcing & CQRS,
Kafka, Rabbit MQ
Case Studies (E-Commerce App, Movie Streaming, Ticket Booking, Restaurant, Hospital Management)
Build distributed transactions using MassTransit, Routing Slips, and Automatonymous state machines using C# and .NET with RabbitMQ or Azure Service Bus
This document provides tips for integrating and monitoring AWS billing metrics using Datadog. It discusses using IAM roles for integration, exploring the aws.billing metric in the Metric Explorer, debugging the Datadog agent, using tags in monitor messages, configuring the agent collection interval, and alternatives to DogStatsD for submitting metrics. The tips are presented in 6 sections and conclude by wishing happy monitoring.
How Uber scaled its Real Time Infrastructure to Trillion events per dayDataWorks Summit
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.
Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.
Building Event Driven Services with Kafka StreamsBen Stopford
The Kafka Summit version of this talk is more practical and includes code examples which walk though how to build a streaming application with Kafka Streams.
Monitoring and Scaling Redis at DataDog - Ilan Rabinovitch, DataDogRedis Labs
Think you have big data? What about high availability
requirements? At DataDog we process billions of data points every day including metrics and events, as we help the world
monitor the their applications and infrastructure. Being the world’s monitoring system is a big responsibility, and thanks to
Redis we are up to the task. Join us as we discuss how the DataDog team monitors and scales Redis to power our SaaS based monitoring offering. We will discuss our usage and deployment patterns, as well as dive into monitoring best practices for production Redis workloads
Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uday Sagar Si...confluent
Cloud providers like AWS allow free data transfers within an Availability Zone (AZ), but bill users when data moves between AZs. When the data volume streamed through Kafka reaches big data scale, (e.g. numeric data points or user activity tracking), the costs incurred by cross-AZ traffic can add significantly to your monthly cloud spend. Since Kafka serves reads and writes only from leader partitions, for a topic with a replication factor of 3, a message sent through Kafka can cross AZs up to 4 times. Once when a producer produces a message onto broker in a different AZ, two times during Kafka replication, and once more during message consumption. With careful design, we can eliminate the first and last part of the cross AZ traffic. We can also use message compression strategies provided by Kafka to reduce costs during replication. In this talk, we will discuss the architectural choices that allow us to ensure a Kafka message is produced and consumed within a single AZ, as well as an algorithm that lets consumers intelligently subscribe to partitions with leaders in the same AZ. We will also cover use cases in which cross-AZ message streaming is unavoidable due to design limitations. Talk outline: 1) A review of Kafka replication, 2) Cross-AZ traffic implications, 3) Architectural choices for AZ-aware message streaming, 4) Algorithms for AZ-aware producers and consumers, 5) Results, 6) Limitations, 7) Takeaways.
Preparing for a future Microservices journey using DDD & Wardley MapsSusanne Kaiser
The journey to Microservices can be very challenging. Identifying proper boundaries, integrating services, and handling infrastructure and operational complexities that Microservices come with can be very overwhelming.
How not to loose sight and to cope with those challenges and still delivering user and business value? One approach could be to focus on that part of your business that gives most competitive advantage - your core domain - and outsource undifferentiating commodities to utility suppliers.
Domain Driven Design combined with Wardley Maps can help us to understand the problem domain and to focus on the core domain.
In this talk Susanne will show how Domain Driven Design and Wardley Maps can be used together to visualise how a value chain can evolve during a Microservices journey and keeping focus on your core domain.
codecentric AG: CQRS and Event Sourcing Applications with CassandraDataStax Academy
CQRS (Command Query Responsibility Segregation) is a pattern, which separates the process of querying and updating data. As a query only returns data without any side effects, a command is designed to change data. CQRS is often combined with Event Sourcing. This is an architecture in which all changes to an application state are stored as a sequence of events.
Because of its great capability to store time series data Cassandra is the perfect fit for implementing the event store. But there a still a lot of open questions: What about the data modeling? What techniques will be used to process and store data in the Cassandra database? How to access the current state of the application, without replaying every event? And what about failure handling?
In this talk, I will give a brief introduction to CQRS and the Event Sourcing pattern and will then answer the questions above using a real life example of a data store for customer data.
Cassandra sharding and consistency (lightning talk)Federico Razzoli
If you are only familiar with relational databases, Cassandra can be confusing. It is designed to shard, and it guarantees consistency in an interesting (and frustrating) way.
Throughout the years, the Concentric Architectures (Onion, Hexagonal, Clean-..) have grown into the undisputed leader among backend systems architectures. With the rise of Domain-Driven Design, keeping your Domain ring 'agnostic' to the outside world has become the norm today. But history proved that any 'norm' in software architectures will cause overengineering if applied without criticism.
After a brief recap of these architectures, their pitfalls, and weaknesses, we'll see two alternatives that segregate code not in 'layers' or 'rings' but in vertical slices: Feature Slicing and Modular Monolith.
[Feature Slicing](vertical Slice Architecture) (aka *UseCase) has its own pitfalls and weaknesses, that we'll briefly review. But this will just warm us up for the next style.
Modular Monolith (aka Modulith) is an architecture style that helped many companies break their legacy codebases, and smoothly move to microservices. Most of the techniques discussed here can also come handy when one single microservice grew big and needs to be broken down.
Even more, greenfield projects today opt for this architecture instead of microservices, to avoid paying the high cost of distributability. Imagine cohesive but decoupled modules living in the same code base & deployment, but on which different teams work in harmony, delivering more value much faster than an equivalent microservice ecosystem.🦄
On the agenda:
- patterns to break data structures
- how to protect Domains inside modules
- communication patterns between modules
- breaking cyclic dependencies
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...NETWAYS
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
In this session, Neil Avery covers the planning and operation of your KSQL deployment, including under-the-hood architectural details. You will learn about the various deployment models, how to track and monitor your KSQL applications, how to scale in and out and how to think about capacity planning. This is part 3 out of 3 in the Empowering Streams through KSQL series.
This document discusses Domain-Driven Design (DDD) and provides guidance on its implementation. It explains that DDD focuses on understanding and modeling the core domain of a business to manage complexity. It recommends dividing a domain into subdomains and identifying the core domain. It also discusses building models through collaboration and ensuring the code model aligns with the conceptual model. The document provides an overview of key DDD concepts like bounded contexts, aggregates, domain events, and tactical patterns to structure the solution space.
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative
Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data.
Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic.
Takeaways:
1) Get a general idea about what is a vector database and its real-world use cases.
2) Understand the major design principles of Milvus 2.0.
3) Learn how to build a complex system with the help of a modern log system like Pulsar.
This document summarizes a presentation about monitoring challenges with Kubernetes and containers. The presentation covered:
- Monitoring 101 concepts like categorizing issues and focusing on symptoms to find root causes.
- Challenges of monitoring dynamic container infrastructure like increased operational complexity from many ephemeral containers and scaling to monitor thousands of metrics.
- Approaches for container monitoring including taking a service-centric rather than host-centric view and using tags to query metrics flexibly across infrastructure attributes.
- Features of Datadog for container monitoring like automatic discovery of containers and metadata, off-the-shelf integrations, and dashboards for visualizing metrics and alerts.
If you understand the rule engine, especially how works RETE algorithm, You may use this for Machine Learning. This slide used at Red Hat Forum Tokyo 2018 session.
What is observability and how is it different from traditional monitoring? How do we effectively monitor and debug complex, elastic microservice architectures? In this interactive discussion, we’ll answer these questions. We’ll also introduce the idea of an “observability pipeline” as a way to empower teams following DevOps practices. Lastly, we’ll demo cloud-native observability tools that fit this “observability pipeline” model, including Fluentd, OpenTracing, and Jaeger.
Building and deploying microservices with event sourcing, CQRS and Docker (Be...Chris Richardson
This document discusses building and deploying microservices using event sourcing, CQRS and Docker. It covers an overview of event sourcing and how it solves data consistency issues in microservice architectures. CQRS is used to implement materialized views for queries. Spring Boot is recommended for building microservices and Docker is used to package the microservices. A continuous integration pipeline is used to build, test, publish Docker images and deploy updates.
This document summarizes a presentation about log forwarding at scale. It discusses how logging works internally and requires understanding the logging pipeline of parsing, filtering, buffering and routing logs. It then introduces Fluent Bit as a lightweight log forwarder that can be used to cheaply forward logs from edge nodes to log aggregators in a scalable way, especially in cloud native environments like Kubernetes. Hands-on demos show how Fluent Bit can parse and add metadata to Kubernetes logs.
Domain Driven Design (DDD) is a software design approach that focuses on modeling a domain accurately. It uses ubiquitous language, bounded contexts, and explicit domain models. The key aspects of DDD include developing a shared model with domain experts, separating concerns into bounded contexts, and iteratively refining domain models through close collaboration between technical and domain teams. DDD aims to produce software designs that are more aligned with the mental models of users and stakeholders in a complex domain.
This document outlines the concepts and techniques of Domain-Driven Design (DDD). It begins with basic concepts like the ubiquitous language and domain model. It then covers strategic design patterns such as bounded contexts and context mapping. Next, it discusses tactical design building blocks like entities, aggregates, and repositories. Finally, it briefly introduces related patterns like CQRS, event sourcing, and event-driven architectures. The document is intended to provide an overview of DDD from basic concepts to advanced patterns in both the strategic and tactical spheres.
An introduction to the concepts behind Continuous Delivery as well as an introduction to some of the tools available for implementing continuous delivery practices on a new project. This presentation is geared towards Java developers, but is applicable to all.
Dopo una breve introduzione dei concetti di base legati all'Internet of Things, durante questa sessione si fornirà una panoramica degli strumenti che Microsoft mette a diposizione degli sviluppatori per creare le proprie soluzioni IoT: Windows 10 for IoT e alcuni servizi di Azure quali Event Hubs e Stream Analytics. Si utilizzerà un semplice esempio di telemetria per mostrare la realizzazione pratica di uno scenario end-to-end per la trasformazione dei dati provenienti da un sensore in informazioni utili per effettuare analisi e/o prendere decisioni.
This document discusses Microsoft's perspective on the Internet of Things (IoT). It outlines an end-to-end IoT scenario from sensors and devices to analytics on Microsoft Azure. The agenda includes discussing how to gain value from connected devices through connectivity, data collection, and analytics. Predictive maintenance and remote monitoring are highlighted as examples of IoT applications. The document then demonstrates connecting sensors to a Raspberry Pi gateway and sending the data to Azure Event Hubs for analysis using Stream Analytics.
Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uday Sagar Si...confluent
Cloud providers like AWS allow free data transfers within an Availability Zone (AZ), but bill users when data moves between AZs. When the data volume streamed through Kafka reaches big data scale, (e.g. numeric data points or user activity tracking), the costs incurred by cross-AZ traffic can add significantly to your monthly cloud spend. Since Kafka serves reads and writes only from leader partitions, for a topic with a replication factor of 3, a message sent through Kafka can cross AZs up to 4 times. Once when a producer produces a message onto broker in a different AZ, two times during Kafka replication, and once more during message consumption. With careful design, we can eliminate the first and last part of the cross AZ traffic. We can also use message compression strategies provided by Kafka to reduce costs during replication. In this talk, we will discuss the architectural choices that allow us to ensure a Kafka message is produced and consumed within a single AZ, as well as an algorithm that lets consumers intelligently subscribe to partitions with leaders in the same AZ. We will also cover use cases in which cross-AZ message streaming is unavoidable due to design limitations. Talk outline: 1) A review of Kafka replication, 2) Cross-AZ traffic implications, 3) Architectural choices for AZ-aware message streaming, 4) Algorithms for AZ-aware producers and consumers, 5) Results, 6) Limitations, 7) Takeaways.
Preparing for a future Microservices journey using DDD & Wardley MapsSusanne Kaiser
The journey to Microservices can be very challenging. Identifying proper boundaries, integrating services, and handling infrastructure and operational complexities that Microservices come with can be very overwhelming.
How not to loose sight and to cope with those challenges and still delivering user and business value? One approach could be to focus on that part of your business that gives most competitive advantage - your core domain - and outsource undifferentiating commodities to utility suppliers.
Domain Driven Design combined with Wardley Maps can help us to understand the problem domain and to focus on the core domain.
In this talk Susanne will show how Domain Driven Design and Wardley Maps can be used together to visualise how a value chain can evolve during a Microservices journey and keeping focus on your core domain.
codecentric AG: CQRS and Event Sourcing Applications with CassandraDataStax Academy
CQRS (Command Query Responsibility Segregation) is a pattern, which separates the process of querying and updating data. As a query only returns data without any side effects, a command is designed to change data. CQRS is often combined with Event Sourcing. This is an architecture in which all changes to an application state are stored as a sequence of events.
Because of its great capability to store time series data Cassandra is the perfect fit for implementing the event store. But there a still a lot of open questions: What about the data modeling? What techniques will be used to process and store data in the Cassandra database? How to access the current state of the application, without replaying every event? And what about failure handling?
In this talk, I will give a brief introduction to CQRS and the Event Sourcing pattern and will then answer the questions above using a real life example of a data store for customer data.
Cassandra sharding and consistency (lightning talk)Federico Razzoli
If you are only familiar with relational databases, Cassandra can be confusing. It is designed to shard, and it guarantees consistency in an interesting (and frustrating) way.
Throughout the years, the Concentric Architectures (Onion, Hexagonal, Clean-..) have grown into the undisputed leader among backend systems architectures. With the rise of Domain-Driven Design, keeping your Domain ring 'agnostic' to the outside world has become the norm today. But history proved that any 'norm' in software architectures will cause overengineering if applied without criticism.
After a brief recap of these architectures, their pitfalls, and weaknesses, we'll see two alternatives that segregate code not in 'layers' or 'rings' but in vertical slices: Feature Slicing and Modular Monolith.
[Feature Slicing](vertical Slice Architecture) (aka *UseCase) has its own pitfalls and weaknesses, that we'll briefly review. But this will just warm us up for the next style.
Modular Monolith (aka Modulith) is an architecture style that helped many companies break their legacy codebases, and smoothly move to microservices. Most of the techniques discussed here can also come handy when one single microservice grew big and needs to be broken down.
Even more, greenfield projects today opt for this architecture instead of microservices, to avoid paying the high cost of distributability. Imagine cohesive but decoupled modules living in the same code base & deployment, but on which different teams work in harmony, delivering more value much faster than an equivalent microservice ecosystem.🦄
On the agenda:
- patterns to break data structures
- how to protect Domains inside modules
- communication patterns between modules
- breaking cyclic dependencies
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...NETWAYS
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
In this session, Neil Avery covers the planning and operation of your KSQL deployment, including under-the-hood architectural details. You will learn about the various deployment models, how to track and monitor your KSQL applications, how to scale in and out and how to think about capacity planning. This is part 3 out of 3 in the Empowering Streams through KSQL series.
This document discusses Domain-Driven Design (DDD) and provides guidance on its implementation. It explains that DDD focuses on understanding and modeling the core domain of a business to manage complexity. It recommends dividing a domain into subdomains and identifying the core domain. It also discusses building models through collaboration and ensuring the code model aligns with the conceptual model. The document provides an overview of key DDD concepts like bounded contexts, aggregates, domain events, and tactical patterns to structure the solution space.
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative
Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data.
Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic.
Takeaways:
1) Get a general idea about what is a vector database and its real-world use cases.
2) Understand the major design principles of Milvus 2.0.
3) Learn how to build a complex system with the help of a modern log system like Pulsar.
This document summarizes a presentation about monitoring challenges with Kubernetes and containers. The presentation covered:
- Monitoring 101 concepts like categorizing issues and focusing on symptoms to find root causes.
- Challenges of monitoring dynamic container infrastructure like increased operational complexity from many ephemeral containers and scaling to monitor thousands of metrics.
- Approaches for container monitoring including taking a service-centric rather than host-centric view and using tags to query metrics flexibly across infrastructure attributes.
- Features of Datadog for container monitoring like automatic discovery of containers and metadata, off-the-shelf integrations, and dashboards for visualizing metrics and alerts.
If you understand the rule engine, especially how works RETE algorithm, You may use this for Machine Learning. This slide used at Red Hat Forum Tokyo 2018 session.
What is observability and how is it different from traditional monitoring? How do we effectively monitor and debug complex, elastic microservice architectures? In this interactive discussion, we’ll answer these questions. We’ll also introduce the idea of an “observability pipeline” as a way to empower teams following DevOps practices. Lastly, we’ll demo cloud-native observability tools that fit this “observability pipeline” model, including Fluentd, OpenTracing, and Jaeger.
Building and deploying microservices with event sourcing, CQRS and Docker (Be...Chris Richardson
This document discusses building and deploying microservices using event sourcing, CQRS and Docker. It covers an overview of event sourcing and how it solves data consistency issues in microservice architectures. CQRS is used to implement materialized views for queries. Spring Boot is recommended for building microservices and Docker is used to package the microservices. A continuous integration pipeline is used to build, test, publish Docker images and deploy updates.
This document summarizes a presentation about log forwarding at scale. It discusses how logging works internally and requires understanding the logging pipeline of parsing, filtering, buffering and routing logs. It then introduces Fluent Bit as a lightweight log forwarder that can be used to cheaply forward logs from edge nodes to log aggregators in a scalable way, especially in cloud native environments like Kubernetes. Hands-on demos show how Fluent Bit can parse and add metadata to Kubernetes logs.
Domain Driven Design (DDD) is a software design approach that focuses on modeling a domain accurately. It uses ubiquitous language, bounded contexts, and explicit domain models. The key aspects of DDD include developing a shared model with domain experts, separating concerns into bounded contexts, and iteratively refining domain models through close collaboration between technical and domain teams. DDD aims to produce software designs that are more aligned with the mental models of users and stakeholders in a complex domain.
This document outlines the concepts and techniques of Domain-Driven Design (DDD). It begins with basic concepts like the ubiquitous language and domain model. It then covers strategic design patterns such as bounded contexts and context mapping. Next, it discusses tactical design building blocks like entities, aggregates, and repositories. Finally, it briefly introduces related patterns like CQRS, event sourcing, and event-driven architectures. The document is intended to provide an overview of DDD from basic concepts to advanced patterns in both the strategic and tactical spheres.
An introduction to the concepts behind Continuous Delivery as well as an introduction to some of the tools available for implementing continuous delivery practices on a new project. This presentation is geared towards Java developers, but is applicable to all.
Dopo una breve introduzione dei concetti di base legati all'Internet of Things, durante questa sessione si fornirà una panoramica degli strumenti che Microsoft mette a diposizione degli sviluppatori per creare le proprie soluzioni IoT: Windows 10 for IoT e alcuni servizi di Azure quali Event Hubs e Stream Analytics. Si utilizzerà un semplice esempio di telemetria per mostrare la realizzazione pratica di uno scenario end-to-end per la trasformazione dei dati provenienti da un sensore in informazioni utili per effettuare analisi e/o prendere decisioni.
This document discusses Microsoft's perspective on the Internet of Things (IoT). It outlines an end-to-end IoT scenario from sensors and devices to analytics on Microsoft Azure. The agenda includes discussing how to gain value from connected devices through connectivity, data collection, and analytics. Predictive maintenance and remote monitoring are highlighted as examples of IoT applications. The document then demonstrates connecting sensors to a Raspberry Pi gateway and sending the data to Azure Event Hubs for analysis using Stream Analytics.
QNAP Systems, Inc., headquartered in Taipei, Taiwan, provides a comprehensive range of cutting-edge Network-attached Storage (NAS) and video surveillance solutions based on the principles of usability, high security, and flexible scalability. QNAP offers quality NAS products for home and business users, providing solutions for storage, backup/snapshot, virtualization, teamwork, multimedia, and more. QNAP envisions NAS as being more than "simple storage", and has created many NAS-based innovations to encourage users to host and develop Internet of Things, artificial intelligence, and machine learning solutions on their QNAP NAS.
IoT end-to-end: porta i tuoi dati dal sensore al cloudCodemotion
"IoT end-to-end: porta i tuoi dati dal sensore al cloud" by Erica Barone.
In questa sessione mostreremo come è possibile raccogliere i dati da un semplice sensore connesso a una Raspberry e trasformarli in informazioni utili, sfruttando diversi servizi cloud per collezionarli, analizzarli, memorizzarli e infine visualizzarli in modo da ottenere valore dal dato di partenza. Tutto ciò che serve è una RPI, un sensore e un account Azure: si sfrutterà una semplice demo come esempio base e si mostreranno alcune possibili estensioni e modifiche, in grado di coprire diversi scenari end to end.
This document provides an overview and summary of an IoT solutions architecture presentation. It discusses various AWS IoT services like the device SDK, rules engine, shadows, and registry. It also covers how to connect devices, send and collect data, and addresses common patterns like telemetry and analytics. Examples are given around connecting a smart farm and implementing features like automated sprinklers. Best practices around lifecycle management and debugging architectures are also presented. The document concludes with an overview of labs that will demonstrate patterns using various AWS IoT and related services.
This is a presentation given at the Capital Saratoga Region AWS User Group on May 18, 2017 at The Troy Innovation Garage in Troy, NY.
Before diving into AWS IoT, we take a step back and talk about IoT from a high level overview. We discuss some of the common problems and challenges with IoT projects and then take a walk through AWS IoT and discuss its approach to solve some of those common challenges. We then connect an IoT Button to an AWS IoT project and demonstrate the basic components of building AWS IoT solutions.
Programma
16.30 Ontvangst
17.00 Robbrecht van Amerongen - Head of IoT, Conclusion Connect
• Welkom en opvallendste zaken van het congres
• Stand van zaken PaaS/SaaS, Digital Twin, Context IoT, Big Data en Machine Learning
• IoT en Security
• IoT-projecten: stand van zaken, opvallende voorbeelden en best practices
• Waar staan de vendors/ platformleveranciers
18.00 Diner
18.45 Gertjan van het Hof - IoT Solutions Architect, Conclusion Connect
• Platformen: Microsoft Azure IoT Reference Architecture en Google Cloud IoT
• CrateDB
• EdgeX Foundry en NetFoundry
• Intel Video processing
• Beacons
• Security Maturity Model
• Industrial Internet Consortium (IIC)
19:45 Pauze
20:00 Henk Jan van Wijk - IoT Solution Engineer, Conclusion Connect
• Microsoft Azure Sphere (connected devices)
• Low Power Connectivity (Thingstream)
• Digital Twin
• Anomaly detection at Intel (predict robots failure)
21.00 Afsluiting en borrel
This document provides an overview of IoT and the AWS IoT platform. It discusses key IoT concepts like MQTT and smart home devices. It then details various aspects of the AWS IoT architecture including AWS IoT Core services, security, and pricing. Device SDKs and protocols are covered, as well as how AWS IoT integrates with other AWS services like Lambda, S3, DynamoDB through the AWS IoT rules engine. Device shadowing and registry services are also summarized. Finally, AWS IoT is compared to the Azure IoT Suite in terms of features.
This document summarizes an IoT and Cortana Analytics presentation. It introduces IoT and the opportunity it presents, describes a typical Azure IoT architecture including IoT Hub, Event Hubs, HDInsight and storage, and provides examples of IoT use cases. It also discusses the Cortana Analytics Suite and its ability to transform data into intelligent action. The presentation includes a demo of a smart building application and a healthcare use case.
Internet of Things and Azure - DevTeach 2016Guy Barrette
This document discusses how Azure services can be used to build Internet of Things (IoT) solutions. It provides an overview of IoT and Azure capabilities, including Azure IoT Hub which allows devices to connect and send data to the cloud. It also describes the Azure IoT Suite, which provides preconfigured solutions for common IoT scenarios to help get started quickly with remote monitoring, predictive maintenance, and other solutions.
This document compares and contrasts the cloud platforms AWS, Azure, and GCP. It provides information on each platform's pillars of cloud services, regions and availability zones, instance types, databases, serverless computing options, networking, analytics and machine learning services, development tools, security features, and pricing models. Speakers then provide more details on their experience with each platform, highlighting key products, differences between the platforms, and positives and negatives of each from their perspective.
The explosive growth of the “Internet of Things” is changing our world and today the Internet of Things knows almost as many applications as there are types of devices connected.
From consumer equipment, to innovate new designs and products at home, to industrial machinery… Everything is connected to the cloud.
In this session Glenn will guide you through the Azure IoT Ecosystem and show you some of the key components of the Azure IoT Platform.
The document discusses why Node.js is well-suited for Internet of Things (IoT) applications. Node.js can handle a large volume of data and requests from IoT devices like sensors through its streaming capabilities. It is also suitable for key IoT features like collecting, communicating, and analyzing data in real-time. Protocols like sockets and MQTT that are commonly used for continuous data transmission in IoT work well with Node.js. Additionally, Node.js is fast, scalable, and efficient, making it well-suited for data-intensive real-time applications like IoT.
The Internet of Things (IoT) involves the internet-connected devices we use to perform the processes and services that support our way of life. ... The worker can use a cloud computing service to finish their work because the data is managed remotely by a server
The document discusses using Lambda architecture for big data analytics in IoT applications. It describes the traditional challenges of device, server, and orchestration architecture and proposes using DeviceHive as an open source framework. DeviceHive uses a Lambda architecture approach with Kafka for streaming data to Cassandra and a batch layer. It also discusses deploying such a system using containers and services like Juju, Mesos, and Docker. Visualization tools like Freeboard, ELK, Spark Streaming, and Zeppelin are proposed for analysis and visualization of the IoT data.
This document provides an overview of key concepts in Internet of Things (IoT) including applications, devices, sensors, networks, protocols, challenges, and analytics. It describes common IoT applications, hardware components like microcontrollers and single-board computers, sensors and actuators, network topologies and protocols, security and design challenges, and approaches to managing and analyzing IoT data at large scales.
This document provides an introduction to IoT presented by Sinan Ozmen. It discusses IoT concepts like connected devices and platforms. It outlines key IoT components, stakeholders, and capabilities of IoT platforms. Challenges and examples of using IoT for power generators and cold chain management are presented. The presentation concludes with contact information for Sinan Ozmen.
The document discusses reference architectures for building big data applications with Internet of Things (IoT) technologies. It describes an IoT reference architecture that includes components for device connectivity, data processing/analytics, and business connectivity. It provides examples of device types, connectivity options, and how to use Azure services for device identity/registry, stream processing, analytics, and presentation. Guiding principles are also outlined for building scalable, secure, and flexible IoT solutions.
Chicago AWS Architectural Resilience Day 2024AWS Chicago
September 2024 the first-ever community Resilience Day in Chicago.
See the video recording on the AWSChicago Youtube: https://youtu.be/z4camus_96c
Thank you, presenters from AWS and PWC.
Angelo Mandato: Learn about the benefits with examples how to create and main...AWS Chicago
AWS Community Day Midwest 2024 |
Angelo Mandato |
AWS Columbus OH |
Learn about the benefits with examples how to create and maintain DEV, QA, and UAT environments using unique domain names
HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/
HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client.
Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience.
In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including
- Accessing the console
- Locating and interpreting log files
- Accessing the data folder within the browser’s cache (using OPFS)
- Understand the difference between single- and multi-user scenarios
- Utilizing Client Clocking
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies
In late April 2025, a significant portion of Europe, particularly Spain, Portugal, and parts of southern France, experienced widespread, rolling power outages that continue to affect millions of residents, businesses, and infrastructure systems.
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada
Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
AI and Data Privacy in 2025: Global TrendsInData Labs
In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy.
This infographic contains:
-AI and data privacy: Key findings
-Statistics on AI data privacy in the today’s world
-Tips on how to overcome data privacy challenges
-Benefits of AI data security investments.
Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB
I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.
Semantic Cultivators : The Critical Future Role to Enable AIartmondano
By 2026, AI agents will consume 10x more enterprise data than humans, but with none of the contextual understanding that prevents catastrophic misinterpretations.
This is the keynote of the Into the Box conference, highlighting the release of the BoxLang JVM language, its key enhancements, and its vision for the future.
Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul
Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc
Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency.
This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data.
Attendees will learn:
- Consumer awareness around data brokers and what consumers are doing to limit data collection
- How businesses assess third-party vendors and their consent management operations
- Where business preparedness needs improvement
- What these trends mean for the future of privacy governance and public trust
This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.
Procurement Insights Cost To Value Guide.pptxJon Hansen
Procurement Insights integrated Historic Procurement Industry Archives, serves as a powerful complement — not a competitor — to other procurement industry firms. It fills critical gaps in depth, agility, and contextual insight that most traditional analyst and association models overlook.
Learn more about this value- driven proprietary service offering here.
How Can I use the AI Hype in my Business Context?Daniel Lehner
𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨?
Everyone’s talking about AI but is anyone really using it to create real value?
Most companies want to leverage AI. Few know 𝗵𝗼𝘄.
✅ What exactly should you ask to find real AI opportunities?
✅ Which AI techniques actually fit your business?
✅ Is your data even ready for AI?
If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.
5. What is IoT?
● Network of physical objects (devices, vehicles, home appliances)
● Embedded with sensors, software, connectivity
● Accessible via the cloud through the Internet infrastructure
6. ● “Finger on the pulse” of traditionally hard to measure objects
● Collect and analyze data automatically, at scale
○ Reveal unseen patterns
○ Faster reaction to events
● Economical
○ Low cost hardware and sensors
○ Ubiquitous connectivity
● Growing Industry
○ 26B IoT devices by 2020
○ $3.7B total market size by 2020
Why IoT
12. Gateway
● Edge system to connect Things to the Cloud
● Localized connectivity
○ WiFi, BLE, Private LTE, LoRaWAN, Sigfox, ZigBee, more
● Robust external connectivity
○ High bandwidth, caching
● Additional processing
○ Encrypting, transformation, managing and filtering data
13. Cloud
● Broker
○ Message management (routing, QoS)
○ Device management
○ Edge security
● Data Storage
○ Per message storage
○ Highly structured or low structure (data warehouse vs. data lake)
● Processing
○ Transform messages
○ Trigger workflows
● Analytics
○ Aggregate data for visualization (QuickSight, Power BI, Kibana)
○ Feed machine learning models
14. Cloud
● Broker
○ Message management (routing, QoS)
○ Device management
○ Edge security
● Data Storage
○ Per message storage
○ Highly structured or low structure (data warehouse vs. data lake)
● Processing
○ Transform messages
○ Trigger workflows
● Analytics
○ Aggregate data for visualization (QuickSight, Power BI, Kibana)
○ Feed machine learning models
29. MQTT
● MQ Telemetry Transport (ISO/IEC PRF 20922)
● Publish-subscribe-based messaging protocol
● Designed to have a “small code footprint” and
"limited network bandwidth"
○ Low energy and bandwidth cost
● Three Quality of Service levels
○ At most once delivery (fire and forget)
○ At least once delivery (acknowledged delivery)
○ Exactly once delivery (assured delivery)
31. AWS IoT Core
● Fully managed service
● Security and identity
● MQTT Broker
● Message routing
● Status
32. AWS IoT Core
● Fully managed service
● Security and identity
● MQTT Broker
● Message routing
● Status
33. Message Processing
AWS Lambda Functions
● Serverless node.js, .NET Core, Python, and Java
● Executes in response to an event, then shuts off
Amazon Kinesis
● Real-time message processing
More
34. Message Storage
AWS DynamoDB
● NoSQL database
● Direct integration with AWS IoT
AWS S3
● Object storage
● Save messages as JSON files
35. Visualization
● Amazon QuickSight
● Business analytics service
● Connect to DynamoDB, Redshift, RDS, etc.
● Advanced processing via SPICE
● Kibana
● Open source near real-time data visualization
● Plugin to ElasticSearch
● Built-in to AWS Amazon Elasticsearch service
40. IoT Analytics
Pipeline
● Select/remove fields
● Add/rename fields
● Calculate fields
● Transform with Lambda function
● Enrich with metadata
● Filter message
41. IoT Analytics
Clean and Store Data
● Select/remove fields
● Add/rename fields
● Calculate fields
● Transform with Lambda function
● Enrich with metadata
● Filter message
42. IoT Analytics
Clean and Store Data
● Select/remove fields
● Add/rename fields
● Calculate fields
● Transform with Lambda function
● Enrich with metadata
● Filter message
43. IoT Analytics
Clean and Store Data
● Select/remove fields
● Add/rename fields
● Calculate fields
● Transform with Lambda function
● Enrich with metadata
● Filter message
44. IoT Analytics
Clean and Store Data
● Select/remove fields
● Add/rename fields
● Calculate fields
● Transform with Lambda function
● Enrich with metadata
● Filter message
47. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
48. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
49. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
AWS Greengrass
50. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
51. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
62
22%
20%
19%
13%
13%
11%
8%
5%
5%
5%
5%
4%
4%
4%
3%
3%
3%
Embedded Linux
FreeRTOS
In-house/custom
Android
Debian (Linux)
Ubuntu
Microsoft (Windows Embedded 7/Standard)
Texas Instruments RTOS
Texas Instruments (DSP/BIOS)
Micrium (uC/OS-III)
Microsoft (Windows 7 Compact or earlier)
Keil (RTX)
Micrium (uC/OS-II)
Wind River (VxWorks)
AnalogDevices (VDK)
Express Logic (ThreadX)
Freescale MQX
2017 (N=619)
Only Operating Systems with
Please select ALL of the operating systems
you are currently using.
Top 10 In Use Operating Systems
EE Times 2017 Embedded Markets Study
52. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
Top 10 Operating Systems Under Consideration in Next 12 Months
EE Times 2017 Embedded Markets Study
63
28%
27%
19%
17%
12%
11%
9%
8%
6%
6%
5%
5%
5%
5%
5%
4%
4%
4%
4%
3%
FreeRTOS
Embedded Linux
In-house/custom
Android
Debian (Linux)
Ubuntu
Micrium (uC/OS-III)
Texas Instruments RTOS
Micrium (uC/OS-II)
Microsoft Windows Embedded 7/Standard
Express Logic (ThreadX)
Keil (RTX)
Texas Instruments (DSP/BIOS)
Freescale MQX
Wind River (Linux)
Microsoft (Windows 7 Compact or earlier)
Wind River (VxWorks)
Red Hat (IX Lunix)
AnalogDevices (VDK)
Green Hills (INTEGRITY)
2017 (N = 568)
Please select ALL of the operating systems you are
considering using in the next 12 months.
53. AWS IoT
Operating system for
microcontrollers
Real-time
Open source
Amazon Free RTOS
Bulk Onboarding
Device organization
(grouping, metadata, etc)
Searching
Remote management
AWS IoT Device
ManagementLocal compute
Messaging
Data caching
Sync
Machine learning
AWS Greengrass
54. AWS IoT Core
● Fully managed service
● Security and identity
● MQTT Broker
● Message routing
● Status