Continus sql with sql stream builder
2 likes574 views
This document discusses Continuous SQL with SQL Stream Builder. It provides an agenda for a meetup covering an introduction, overview of Apache Flink and SQL Stream Builder, demos and Q&A. SQL Stream Builder allows anyone familiar with SQL to create powerful stream processors for real-time analytics by leveraging Apache Flink's scalable and high performance processing. It provides an interactive interface and deep integration features beyond just UI capabilities.
Continus sql with sql stream builder
- 1. Continuous SQL with SQL Stream Builder Kenny Gorman - Product Owner Timothy Spann - Principal DataFlow Field Engineer John Kuchmek - Senior Solutions Engineer 06-May-2021 https://www.meetup.com/futureofdata-newyork/ @PaasDev
- 2. © 2021 Cloudera, Inc. All rights reserved. 2 Welcome to Future of Data - Virtual Princeton Future of Data Meetup New York Future of Data Meetup Philadelphia Future of Data Meetup From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ...
- 3. © 2021 Cloudera, Inc. All rights reserved. 3 AGENDA ● Introductions with Kenny, John and Tim ● Flink Quick Overview ● SQL Stream Builder Overview ● Q&A ● Demos ● Q&A - Interactive Panel Session ● Next Meetups ● Raffle
- 4. © 2021 Cloudera, Inc. All rights reserved. 4 Cloudera DataFlow Use Cases Data Movement Optimize resource utilization by moving data between data centers or between on-premises and cloud infrastructures e.g. intercontinental data exchange Logging Modernization Optimize log analytics solutions by with CDF in simplifying log ingestion from the edge, reducing costs and gaining key analytics e.g. Splunk / Logstash offload Streaming analytics insights Make key business decisions by analyzing streaming data for complex patterns, gaining actionable intelligence etc. e.g. Fraud detection, Network threat analysis, app monitoring, Clickstream analysis 360° view of customer Ingest, transform and combine customer data from multiple sources into a single data view / lake e.g. Real-time customer offers, Loan approvals IoT & Edge use cases e.g. Predictive Maintenance, Asset Tracking / Monitoring, Patient Monitoring, Quality Processes, Fleet Management, Connected Cars and more Enterprise data management Managing massive volumes of high-velocity data to/from legacy systems, ETL tools and other data stores e.g. Flume offload, ETL replacement, payment data processing, integration with Oracle
- 5. © 2021 Cloudera, Inc. All rights reserved. 5 Simplifying the User Experience
- 6. © 2021 Cloudera, Inc. All rights reserved. 6
- 7. © 2021 Cloudera, Inc. All rights reserved. 7 APACHE FLINK Streaming real-time data pipelines that need to handle complex stream or batch data event processing, analytics, and/or support event-driven applications USE CASE TECHNOLOGY APPLICATION Comcast a global media uses Flink for operationalizing machine learning models and near-real-time event stream processing Flink helps deliver a personalized, contextual interaction reducing time to support resolutions saving millions of dollars per year Flink performs compute at in-memory speed at any scale Flink parses SQL using Apache Calcite, which supports standard ANSI SQL Flink runs standalone, on YARN, and has a K8s Operator Data Freshness SLAs Flink can read and write from Hive data Review requirements for fault tolerance, resilience, and HA Other technologies play in this space like Hive storage handler to connect to Kafka CONSIDERATION 3B+ data points daily streaming in from 25 million customers running real time machine learning prediction Flink
- 8. © 2020 Cloudera, Inc. All rights reserved. 8 FLINK FEATURES • Distributed processing engine for stateful computations • 10s TBs of managed state • Flexible and expressive APIs • Guaranteed correctness & Exactly-once state consistency • Event-time semantics • Flexible deployment & large ecosystem (K8s, YARN, S3, HDFS..) • Support for Flink SQL API
- 9. © 2021 Cloudera, Inc. All rights reserved. 9 DELIVERING STREAMING ANALYTICS 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. (second) SQL Parsing and Blending Data Streaming Analytics Both offline and streaming data Data Analysts Can Write Queries Across the Line of Businesses Capture Events that Matter Low-latency analytics use cases Events Processing
- 10. © 2021 Cloudera, Inc. All rights reserved. 10 MAINTAINS & CHECKPOINTS STATE ● Flink maintains state locally per task (in-mem / on-disk) ○ Fast access! ● State is periodically checkpointed to durable storage ○ A checkpoint is a consistent snapshot of the state of all tasks
- 11. 11 Integrated Governance Unified Governance & Lineage Flow Management Streams Messaging Stream Processing Reports Entity and Lineage information about NiFi Flows Connects with existing Lineage information Topic access centrally managed supporting granular CRUD operations Manage permissions on dedicated clusters or manage multiple clusters at once Manage schemas centrally and make them available to consumers/producers Reports Flink Apps as an operation Lineage through integration with existing Lineage information like Kafka topics, HBase tables etc. Integrated SQL and materialized view engine via SQL Stream Builder.
- 12. © 2021 Cloudera, Inc. All rights reserved. 12 SQL Stream Builder ● Democratize data access across enterprise - anyone who knows SQL can create powerful stream processors. ● Iterative interface - Just like SQL on databases, run queries and reason about the data with an interactive UI. ● Leverages Apache Flink for running of SQL jobs - production grade, scalable and high performance ● Deep integration and features above and beyond just UI features - UDF’s, input transforms, Kafka key and time integration, CEP framework and more. ● Create Materialized Views to integrate with downstream components like notebooks, visualizations and applications.
- 13. © 2021 Cloudera, Inc. All rights reserved. 13 Streaming SQL Democratizing access to streams of data via structured query language
- 14. © 2021 Cloudera, Inc. All rights reserved. 14 Download these assets today
- 15. © 2021 Cloudera, Inc. All rights reserved. 15 TH N Y U