SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Yassine Faihe, EMEA Head of Solutions Consulting
Download as PPTX, PDF0 likes540 views
To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.
![[Giovanni Galloro] How to use machine learning on Google Cloud Platform](https://cdn.slidesharecdn.com/ss_thumbnails/mlcapabilitiesongcp-190115085455-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Yassine Faihe, EMEA Head of Solutions Consulting (20)
Ad
More from HostedbyConfluent (20)
Ad
Recently uploaded (20)
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Yassine Faihe, EMEA Head of Solutions Consulting
- 1. — Kafka & SingleStore: Better Together to Power Modern Real-Time Data Architecture YassineFaihe
- 2. 2 Increasing Focus on Cloud and Real-Time Analytics 90% 75% 30% By 2022, public cloud services will be essential for 90% of data and analytics innovation By 2022, 75% of all workloads will move to Hybrid-Cloud By 2025, nearly 30% of all data generated will be real-time Gartner Top 10 Trends in Data and Analytics for 2020 McKinsey - Unlocking business acceleration in a hybrid cloud world, Aug 2019 IDC - Data Age 2025
- 3. 3 — Traditional Data Architecture System of Record System of Engagement Data Warehouse ODS ETL CDC Reporting Visualization Batch Engine Real-Time Engine Data Lake
- 4. 4 — Modern Real-Time Data Architecture System of Record System of Engagement Smart Apps System of Insight System of Intelligence CDC
- 5. 5 — Digital Integration Hub System of Record System of Engagement Microservices System of Insight CDC Smart Apps System of Intelligence
- 6. 6 The Unified Database for Fast Analytics Data Warehouse Operational Database Transactional Workloads Fast Queries | Large Data Size Aggregation Fast Lookup | High Concurrency Simplifiesthe support of diverseworkloadsby reducingoperationalcomplexity Analytical Workloads
- 7. 7 Ultra-Fast Ingest Parallel, high-scale streamingdataIngest Super-Low Latency High Concurrency BlazingFast Queries Unparalleled Scalability Billions of events/sec for immediate availability Sub-second latencies with immediate consistency Millions of real-time queries across tens of thousands of users Fast Analytics Fastanalyticson dynamicdata for complexanalytical queries SingleStore - Key Capabilities for Fast Analytics
- 8. 8 — Kafka & SingleStore Meet Demands of Operational Analytics Real-Time Millions of records per second Consistent Exactly-once semantic Distributed, Fault Tolerant Parallel ingest Developer-Friendly Pub-Sub & SQL
- 9. 9 — Anatomy of SingleStore Pipelines Sequence MemSQL pulls for changes from a data source system. MemSQL pulls the data into its memory space (no commit) where a transform can be applied. The data is committed in a transaction (and in parallel) Pipelines Kafka SingleStore Data can be directly inserted into tables or pre-processed by a stored procedure Write to Kafka
- 10. 10 SingleStore Pipelines Creation 10M UPSERTS PER SECOND WITHKAFKA + SINGLESTORE CREATE OR REPLACE PIPELINE load_trade_data AS LOAD DATA KAFKA 'hostname:9092/trades' WITH TRANSFORM ('score_data.py','','') -- optional INTO TABLE live_predictions -- directly into tables INTO PROCEDURE trade_proc -- via a stored procedure FIELDS TERMINATED BY ','; START PIPELINE load_trade_data; ; —
- 11. 11 — SingleStore Kafka Pipelines SingleStore Cluster
- 12. 12 — SingleStore Transforms ● Build transforms using any language! ● Transforms are an optional user-defined program that receives data from a pipeline’s extractor and outputs modified data (JSON, Avro, CSV) ○ Examples: Data modification, aggregation, feature engineering, model execution, and more! ● Linux distribution must have the required dependencies to execute the transform ● Data streamed to the transform is byte- length encoded Stream Transform Load
- 13. 13 — SingleStore Stored Procedures DELIMITER // CREATE OR REPLACE PROCEDURE tweets_proc(batch QUERY(tweet JSON)) AS BEGIN INSERT IGNORE INTO tweets(tweet_id, tweet_user_id, tweet_text) SELECT tweet::tweet_id, tweet::tweet_user_id, tweet::tweet_text FROM batch; INSERT INTO retweets_counter(user_id, num_retweets) SELECT tweet::retweet_user_id, 1 FROM batch WHERE tweet::retweet_user_id IS NOT NULL ON DUPLICATE KEY UPDATE num_retweets = num_retweets + 1; END // DELIMITER ; ; ● Preprocess incoming data: cleansing, aggregation, filtering… ● Dispatch to multiple tables ● Cross-reference with dimension tables ● Integrity check ● Push to Kafka
- 14. 14 — SingleStore Push to Kafka ● Allows users to leverage SingleStore as a true Operational Data Hub with downstream decisioning ● “SELECT … INTO KAFKA …” runs a SELECT query, constructs Kafka message for each row in the result set, and publishes the messages to a Kafka topic ● Includes every column value in the result set’s row and separates the column values by a delimiter ● Configure security credentials within the statement easily SELECT col1, col2, col3 FROM t ORDER BY col1 INTO KAFKA 'host.example.com:9092/test-topic' FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY "t" LINES TERMINATED BY '}' STARTING BY '{';
- 15. 15 — SingleStore Confluent Kafka Connector ● SingleStore Kafka Connect Connector on the Confluent Hub ● Integration with Confluent Kafka Connect to stream data into SingleStore ● Management and deployment capabilities of Confluent make this incredibly easy to get started ● Cloud-first: Kafka Connector sits Kafka-side, eliminating many potential security constraints 5X THAN JDBC CONNECTOR FASTER
- 16. Real-time fraud analytics for Credit card swipes in less than 50ms. Real-time geospatial insights with massive concurrency to manage 24/7 operations 300K Events per second Streaming analytics to drive proactive care and real-time recommendations IoT Analytics ingesting and analyzing data from over 1.2 Million smart meters 13x data growth moving from batch to near-real time visibility and analytics 3500+ Users 1.2M Smart meters analyzed 10M Upserts per second Tier-1 US Bank 50ms Real-time Fraud Detection Top Energy Company
- 17. SingleStore is the Unified Database for Fast Analytics on Any Data, Anywhere
- 18. 18 Learn Your Way — Get Started with $250 in Free Credits Today Go to singlestore.com/managed-service-trial ● Learn by Reading ○ docs.singlestore.com ● Learn by Engaging Peers ○ singlestore.com/forum ● Learn by Watching ○ youtube.com/singlestore ● Learn through Training ○ training.singlestore.com
- 19. — 19 Thank You Sales Please fill out the form if you need to learn more. For immediate sales help, call us at 1-855- 463-6775 or email us at [email protected]. Enterprise Edition Support Are you encountering an issue and have an enterprise support? Submit a support request. U.S. OfficeLocations San Francisco (HQ) 534 Fourth Street San Francisco, CA 94107 Seattle 96 Union Street Seattle, WA 98101 Portland 700 SW Fifth Ave Portland, OR 97204
Editor's Notes
- #2: Agenda
- #3: Enterprises looking to provide superior customer experience and grow share must have data infrastructures that can meet this growth in real-time data. - IDC
- #7: Running fast analytical queries across large, constantly changing datasets with consistently high performance.
- #8: FAST: Fast Ingest + Fast Queries Blazing-fast platform to power all your real-time analytics and use cases Easy: Easy to get started on your projects. And not just Easy, it’s EASY & SIMPLE - dramatically simplifies your architecture with lesser moving parts instead of complex Lambda architectures (cache, data mart, ODS, DW augment) - Blog/ Whitepaper on Lambda. Cost Effective - Built Cloud-native and efficient and runs on industry-standard hardware. Predictable cost-management (More performance for the cost). Exceptional performance for the same cost, as compared to other Cloud options
- #9: What do we mean by Operational Analytics? In the moment. Answers in real-time. Kafka is perfect for this. Why…
- #10: Hone in on ingest. Inherently a polling system. Remember all of those times Robin listened to Batman? A lot of times this can replace an ETL process.
- #11: Let’s talk about how easy it is to get some pretty impressive results with pipelines. 5 lines of SQL. 10M upserts per second. Akamai.
- #12: But how?! MemSQL is agg-leaf architecture. Starts with metadata query. Each Kafka partition gets assigned a memSQL partition and they are off to the races.
- #13: So we’ve talked about extracting data. How do we transform it? Any language, user defined. You can even call APIs! Think about the possibilities… I’m sure you’re thinking… with all of this talk about data streaming, manipulation, persistence… What about schemas? Well, we’ve been listening. We understand that there’s much more to streaming data than velocity.
- #14: So we’ve talked about extracting data. How do we transform it? Any language, user defined. You can even call APIs! Think about the possibilities… I’m sure you’re thinking… with all of this talk about data streaming, manipulation, persistence… What about schemas? Well, we’ve been listening. We understand that there’s much more to streaming data than velocity.
- #15: So we’ve talked about extracting data. How do we transform it? Any language, user defined. You can even call APIs! Think about the possibilities… I’m sure you’re thinking… with all of this talk about data streaming, manipulation, persistence… What about schemas? Well, we’ve been listening. We understand that there’s much more to streaming data than velocity.
- #16: Today we announce the MemSQL Kafka Connector. The next phase in the epic partnership between Confluent and MemSQL. The key here: it just works. Available on Confluent Hub today. Push vs. pull. Still fast.
- #18: From the Fortune 500s to the Industry disruptors and tech start ups
- #19: Resources, Blogs