IoT Event Processing and Analytics with InfluxDB in Google Cloud | Christoph Bussler | Google, Inc

Oct 7, 20191 like325 views

The presentation introduces a Google Cloud native architecture for collecting, processing, analyzing and archiving of events from IoT devices, vehicles as well as upstream software systems. InfluxDB and its connection to global native Google Cloud services like BigQuery or Cloud Machine Learning Engine as well as Kubernetes is at the center of the architecture. The architecture demonstrates how access to global scaling cloud services addresses use cases from the Energy Sector.

Christoph Bussler
October 3rd, 2019
IoT Event Processing and
Analytics with InfluxDB in
Google Cloud

Agenda
Energy
Sector
Energy sector
use case
analysis
Life cycle
of an
IoT event
Google Cloud
1 2 3 4 5 6
TICK
architecture
Cloud native
IoT event
processing
architecture

● Production
○ Energy production systems monitoring and anomaly detection
■ Oil, gas, wind, hydro, solar and others
● Distribution
○ Smart grid: maintaining an equilibrium across energy supply and demand
■ Renewable and non-renewable energy production/demand forecasting
■ https://towardsdatascience.com/how-machine-learning-can-transform-the-ener
gy-industry-caaa965e282a (devices: synchrophasers)
● Consumption
○ Fleet performance optimization (cars, trucks, planes, etc.)
○ Commercial manufacturing sites, oﬃce buildings
○ Public infrastructure
○ Private households
Use Cases: Energy Sector

● Event collection
○ From production systems
○ From consumption locations
○ Absence check: requires static
equipment inventory data
● Event monitoring
○ Outages
○ Trends
○ Anomalies
■ Was there actually an outage?
Use Cases: Analysis
● Forecasting / prediction
○ Combination of
■ Current events
■ Historic events
■ Non-event data like models,
weather data, road conditions,
etc.
● Off-line analysis
○ Combination with non-event data
● Archiving
○ Long duration analysis

Life Cycle of an IoT Event
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Oﬄine
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

Google Cloud: Regions and Zones
https://cloud.google.com/about/locations/#regions-tab

Google Cloud: Global Network
https://cloud.google.com/about/locations/#network-tab

Google Cloud: Cloud Services
https://cloud.google.com/-select“Products”

Google Cloud: Select Services
BigQuery (https://cloud.google.com/bigquery/)
● Cloud native analytics database
● Columnar - SQL - large scale (PB) - multi-regional service
Cloud Spanner (https://cloud.google.com/spanner/)
● Cloud native global relational database
● Relational - SQL - consistent - linear scaling - multi-regional and intercontinental service
Coldline Storage (https://cloud.google.com/storage/docs/storage-classes)
● Cloud native storage - long term storage
Cloud Machine Learning Engine (https://cloud.google.com/ml-engine/)
● Cloud native ML engine - online and batch predictions

InfluxDB - TICK Architecture
https://www.inﬂuxdata.com/time-series-platform/

Cloud Native IoT Event Processing Architecture
Gateway

Cloud Native IoT Event Processing Architecture
Gateway
Cloud
IoT Core

Cloud Native IoT Event Processing Architecture
Gateway
Cloud
IoT Core
TICK
Stack

Cloud Native IoT Event Processing Architecture
Gateway
Cloud
IoT Core
TICK
Stack
Cloud
Pub/Sub
BigQuery
Cloud
Dataﬂow
Cloud
SQL
Cloud
Dataﬂow

Cloud Native IoT Event Processing Architecture
Gateway
Cloud
IoT Core
TICK
Stack
Cloud
Pub/Sub
BigQuery
Cloud
Dataﬂow
Cloud
SQL
Cloud
Pub/Sub
Cloud
Dataﬂow
Cloud
Pub/Sub
Cloud
Spanner
Kubernetes
Engine
Cloud Run
Cloud
Storage
AI Platform
Cloud
Dataﬂow

Cloud Native IoT Event Processing Architecture
Gateway
Cloud
IoT Core
TICK
Stack
Cloud
Pub/Sub
BigQuery
Cloud
Dataﬂow
Cloud
SQL
Cloud
Run
Cloud
Storage
Cloud
Pub/Sub
Cloud
Dataﬂow
Cloud
Pub/Sub
Cloud
Spanner
Kubernetes
Engine
Cloud
Scheduler
Cloud Run
Cloud
Storage
AI Platform
Cloud
Dataﬂow

Summary
Gateway
Ingest
Time Series
Database
Real-time
Analytics
Oﬄine
Analytics
Static data
sources
Dynamic data
sources
Application
Systems
Short / Long
term archive

InfluxDB IOx Tech Talks Placing a Parquet file into an object store serves as a simple data persistence format. However, storing data into multiple files enabling upserts, deletions, format upgrades, metadata management, and consistency checks at scale requires some form of a catalog that manages these files. In this talk we will explore the requirements for a catalog for InfluxDB IOx, prior art from the Parquet ecosystem, and the proposed solution.

Creating and Using the Flux SQL Datasource | Katy Farmer | InfluxData InfluxData

InfluxDB Cloud Product Update InfluxData

Learn more about InfluxData’s time series platform. InfluxDB Cloud is a fast, elastic, serverless real-time monitoring platform, dashboarding engine, analytics service and event and metrics processor. It is available on AWS, Azure and Google Cloud. Since its launch, we have been busy making updates to the product! Join Balaji Palani, Director of Product Management, as he demonstrates the the latest features of InfluxDB Cloud. This one-hour webinar will feature a product update and Q&A time.

Kapacitor Stream ProcessingInfluxData

Sam Dillard [InfluxData] | Performance Optimization in InfluxDB | InfluxDays...InfluxData

How Sensor Data Can Help Manufacturers Gain Insight to Reduce Waste, Energy C...InfluxData

In this webinar, learn how a long-time Industrial IT Consultant helps his customer make the leap into providing visibility of their processes to everyone in the plant. This journey led to the discovery of untapped opportunity to improve operations, reduce energy consumption, and minimize plant downtime. The collection of data from the individual sensors has led to powerful Grafana dashboards shared across the organization.

Lessons Learned Running InfluxDB Cloud and Other Cloud Services at Scale by T...InfluxData

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...InfluxData

The document discusses Mist.io's use of Telegraf and InfluxDB for infrastructure monitoring. Mist is an open source multi-cloud management platform that provides visibility into containers, VMs and multi-cloud environments. It collects metrics using Telegraf and stores them in InfluxDB for monitoring, alerting and cost optimization. The integration provides flexibility, a large ecosystem and helps Mist users gain insights into their infrastructure.

Setting up InfluxData for IoTInfluxData

Bernard Paques & Kevin Polossat [AWS] | Combining the Power of InfluxDB and A...InfluxData

InfluxDB IOx Tech Talks: A Rusty Introduction to Apache Arrow and How it App...InfluxData

InfluxDB IOx Tech Talks - December 2020 A Rusty Introduction to Apache Arrow and How it Applies to a Time Series Database This session will start with a tech talk from an InfluxDB IOx team member. This is your chance to interact directly with Influxers who are available to answer your questions about all things InfluxDB IOx and time series — including Paul Dix, Founder and CTO of InfluxData. This event will last about an hour and there will be time for live Q&A.

Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...InfluxData

Mist is an open source multicloud management platform. Mist.io's goal is to make multicloud simple and offer a single interface from where you can manage everything. To help users make informed decisions about their infrastructure, Mist is integrating with Telegraf and InfluxDB for collecting and storing monitoring metrics. This is an evolution of a previous stack based on collectd and Graphite. This session will go over why Mist.io moved and the implementation details of its current stack. The session will also analyze the challenges faced and the solutions built. This session is a follow-up to the webinar presentation at https://www.influxdata.com/resources/how-to-gain-visibility-into-containers-vms-and-multi-cloud-environments-using-telegraf-influxdb-and-mist/ which focuses more on the technical details.

How to Deliver a Critical and Actionable Customer-Facing Metrics Product with...InfluxData

In IoT, understanding the health of thousands of devices is critical for deployment at scale especially when troubleshooting an issue. Particle’s customer base needed visibility into their devices with actionable data to reference in real time. Join Cullen Murphy, Site Reliability Engineer at Particle, to learn how the team built a metrics system on Telegraf, Kubernetes, and Prometheus to deploy a customer-facing product that provides critical and relevant data to their IoT product creators.

InfluxDB 101 - Concepts and Architecture | Michael DeSa | InfluxDataInfluxData

How EnerKey Using InfluxDB Saves Customers Millions by Detecting Energy Usage...InfluxData

How to Create a Modern IIoT Monitoring Solution On iOS Using Swift, MQTT and ...InfluxData

MOXIE IoT uses industry standards and cutting-edge technology to create a monitoring solution which provides their manufacturing customers with a single source of truth. Their iOS iPad app equips their customers to visualize and analyze industrial factory data in real time. Discover how they use a time series database, MQTT and Ultra-Wideband to accurately measure and track any object, device, personnel or machinery. In this webinar, Dr. Austin Gurley will dive into: MOXIE’s approach to app-centric asset monitoring Their ability to help customers streamline operations while bridging the gap between hardware and software How a time series database improves customer experience by enabling them to query, filter and parse their data Techniques for performing Flux queries in iOS Swift

How to Store and Visualize CAN Bus Telematic Data with InfluxDB Cloud and Gra...InfluxData

The document discusses how CSS Electronics stores and visualizes telematic data collected from CAN bus networks using InfluxDB and Grafana. Specifically, it summarizes: 1) CSS Electronics develops CAN bus data loggers and sensors to collect data from vehicles and industrial equipment. They implemented InfluxDB to store the decoded telematic data and Grafana to visualize it in customizable dashboards. 2) The process involves logging raw CAN bus data, decoding it using Python scripts, and writing it to InfluxDB. Grafana then pulls the data from InfluxDB to generate intuitive dashboards without coding. 3) Examples are shown of heavy vehicle, automotive, maritime, and sensor fusion dashboards. The solution is open source

Michael DeSa [InfluxData] | Monitoring Methodologies | InfluxDays Virtual Exp...InfluxData

How a Time Series Database Contributes to a Decentralized Cloud Object Storag...InfluxData

Evan Kaplan [InfluxData] | InfluxDays Opening Remarks | InfluxDays EMEA 2021InfluxData

This document appears to be an agenda for InfluxDays EMEA 2021, an event hosted by InfluxData. It includes opening remarks by the CEO, keynotes on InfluxDB IOx and managing dashboards/alerts. There is a networking break and sessions on integrating Telegraf/InfluxDB/Mist and an InfluxDB roadmap update. Day 2 includes sessions on Grafana, InfluxDB Cloud, IIoT monitoring and an InfluxDB community update. The document thanks InfluxData commercial customers and contributors for their support.

Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry...InfluxData

The document discusses observability with InfluxDB/IOx and OpenTelemetry. It provides definitions and descriptions of InfluxDB/TSM and InfluxDB/IOx storage engines, highlighting their strengths. It also defines OpenTelemetry as a standard for observability signals including metrics, logs and traces. The document outlines how OpenTelemetry aims to provide a common instrumentation standard to unify the observability stack, and describes how InfluxData supports and integrates with OpenTelemetry.

IoT Architectural Overview - 3 use case studies from InfluxData InfluxData

This SlideShare reviews how an IoT Data platform fits in with any IoT Architecture to manage the data requirements of every IoT implementation. It is based on the learnings from existing IoT practitioners that have adopted an IoT Data platform using InfluxData. These clients have a range of solutions–from home automation (thermostat monitoring & management), to infrastructure management (solar panel monitoring and control) to manufacturing (equipment monitoring & control) as well as environmental management (green wall monitoring & control). These learnings will help IoT adopters avoid the common pitfalls current clients faced on their journey to developing their IoT solution.

InfluxDB Enterprise Architectural Patterns | Craig Hobbs | InfluxDataInfluxData

A TRUE STORY ABOUT DATABASE ORCHESTRATIONInfluxData

Inside the InfluxDB storage engineInfluxData

Christoph Bussler [Google Cloud] | IoT Event Processing and Analytics with In...InfluxData

The presentation introduces a Google Cloud native architecture for collecting, processing, analyzing and archiving events from IoT devices, vehicles as well as upstream software systems. InfluxDB and its connection to global native Google Cloud services like BigQuery or Cloud Machine Learning Engine as well as Kubernetes is at the center of the architecture. The architecture demonstrates how access to global scaling cloud services address use cases from the Energy Sector.

Cloudera - IoT & Smart CitiesCloudera, Inc.

The document discusses how Cloudera provides a data management platform for IoT data. It handles massive volumes of data from diverse sources in real-time and batch. The platform includes capabilities for data storage, processing, machine learning, analytics and management. Example use cases show how customers use the platform for predictive maintenance, smart cities, connected vehicles and other IoT applications.

More Related Content

What's hot (20)