Real time analytics at uber @ strata data 2019
13 likes3,022 views
This document summarizes Uber's use of Presto, an open source distributed SQL query engine, for real-time analytics and business intelligence. Presto allows Uber to query petabytes of data across different data sources like HDFS, Elasticsearch, Pinot and databases in seconds. Uber has optimized Presto for its scale with contributions like geospatial support, security features and connectors. Presto is critical for Uber's data scientists, analysts and operations to power applications, machine learning and business decisions.
![[KAIST ì±ì©ì€ëȘ
í] ë°ìŽí° ìì§ëìŽë ëŹŽìš ìŒì íëì?](https://cdn.slidesharecdn.com/ss_thumbnails/temp-180502031907-thumbnail.jpg?width=560&fit=bounds)
![Naverìëì, ìëì ìí, ìëë„Œ ìí ëȘœêł DB (ë€ìŽëČ ì»ší
ìž êČìêłŒ ëȘœêł DB) [Naver]](https://cdn.slidesharecdn.com/ss_thumbnails/naver-190916181334-thumbnail.jpg?width=560&fit=bounds)
![[pgday.Seoul 2022] PostgreSQL with Google Cloud](https://cdn.slidesharecdn.com/ss_thumbnails/pgday-postgresqlwithgooglecloud-221114013605-5def484f-thumbnail.jpg?width=560&fit=bounds)
![[NDC18] ìŒìì ë
ëëêł ì ë°ìŽí° ìì§ëìŽë§ ìŽìŒêž°: ëĄê·ž ìì€í
ê”Źì¶ êČœí êł”ì (2ë¶)](https://cdn.slidesharecdn.com/ss_thumbnails/ndc2018-2-180430180517-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Real time analytics at uber @ strata data 2019 (20)
Ad
More from Zhenxiao Luo (10)
Ad
Recently uploaded (19)
Real time analytics at uber @ strata data 2019
- 1. Real Time Analytics at Uber: Bring SQL into Everything Zhenxiao Luo
- 2. NYC Uberâs mission is to ignite opportunity by setting the world in motion. 15M Trips/Day 600+ Cities 75M Monthly Riders
- 3. Data informs every decision at the company
- 4. Overview of Uberâs Data Platform DATA SOURCES RAW DATA MODELED TABLES MINING BUSINESS INSIGHTS CONSUMING BUSINESS INSIGHTS EXPERIMENTATION DATA SCIENCE MACHINE LEARNING CUSTOM DATA SETS Dashboarding Alerting Monitoring Data Exploration Knowledge Bases Storage Infrastructure ETL Frameworks Data Integrity Query Engines
- 5. Kafka Uber Data Infrastructure Schemaless MySQL, Postgres Vertica Streamio Raw Data Raw Tables Sqoop Reports Hadoop Hive Presto Spark Notebook Ad Hoc Queries Real Time Applications Machine Learning Jobs Business Intelligence Jobs Cluster Management All-Active Observability Security Vertica Samza Pinot Flink AresDB Modeled Tables Streaming Warehouse Real-time
- 6. Presto @ Uber-scale 5KWeekly Active Users 160KQueries/day 3Data Centers 2KNodes 700MHDFS ïŹles read/day 10PBHDFS ïŹles processed/day
- 7. Presto use cases at Uber Growth Marketing Data Science Marketplace Pricing Community Operations Data Quality Ad-hoc Querying
- 8. The people who rely on us Technical Skills Data Scientists Software Engineers ML/AI Researchers Advanced SQL Advanced Statistics Scala/Spark, Python/R Data Modeling Inventor Ivan Marketing Managers Entry-level Analysts General Managers Product Managers Limited SQL Spreadsheets Reliant Rebecca City Operations Regional Managers Intermediate SQL Spreadsheets Dashboarding Monitoring Matt Operations Managers Data Analysts Product Analysts Advanced SQL Spreadsheets Limited Statistics Limited Python/R Analyst Anna
- 9. Exploratory ML & model-training Data Scientists ML ResearchersEngineers Using ML to ensure data security and compliance
- 10. Advanced data science & complex analytics Data Scientists Ops Analysts Support Agents Surfacing hidden insights to empower restaurants
- 11. Business process automation S&P AnalystsOps Managers Contractors Using technology to make transportation safer
- 12. What is Presto: Interactive SQL Engine for Big Data Interactive query speeds Horizontally scalable ANSI SQL Battle-tested by Facebook, Uber, Linkedin, Twitter, NetïŹix, Airbnb, etc Completely open source Access to petabytes of data in the Hadoop, Elasticsearch, Pinot, etc.
- 13. How Presto Works
- 14. Why Presto is Fast â Data in memory during execution â Pipelining and streaming â Columnar storage & execution â Bytecode generation
- 15. Resource Management â Presto has its own resource manager â Not on YARN â Not on Mesos â CPU Management â Priority queues â Short running queries higher priority â Memory Management â Max memory per query per node â If query exceeds max memory limit, query fails â No OutOfMemory in Presto process
- 16. Presto Connectors: No Need to Copy Data
- 18. Contributions New Features â Geospatial indexing and operations - 10x or more speedup â Pinot connector enhancements (in-house) Optimizations â Elasticsearch connector â New Parquet reader - 4x speedup â Nested column pushdowns (project, predicate) - 10x speedup Security â Metastore authentication support for Kerberos deployments â Dispatch Proxy using HTTP redirect for multi-cluster operation
- 19. Presto Connector Interface â ConnectorMetadata â Schema, Table, Column â ConnectorSplitManager â Divide data into splits â ConnectorSplit â Split data range â Predicate/JsonFunction/Limit pushdown â ConnectorRecordCursor â Transform underlying storage data into Presto internal page/block
- 21. Data Model â each Elasticsearch index is a table partition â each ïŹeld of an index is a column â all Elasticsearch indexes sharing the same preïŹx consist a logical table â Es-vehicles-sjc1, es-vehicles-dca1, es-vehicles
- 22. Describe Table
- 23. Query
- 24. Optimizations â Parallel Reads â Get all indices and search nodes â For each search node, send request for one speciïŹc index â Cap Max Hits â Predicate Pushdown â Json Function Pushdown â Limit Pushdown â Nested Fields
- 25. How many Uber trip requests did we serve in Chicago yesterday?
- 26. Fetch daily trip count in seconds SELECT T.base.city_id AS cid, Count(CASE WHEN T.base.status = 'completed' THEN 1 END) AS completed_trips, Count(CASE WHEN T.base.status = 'canceled' THEN 1 END) AS rider_canceled_trips FROM trips AS T WHERE T.datestr = '2019-03-11' GROUP BY 1
- 27. Column Chunk base.client_uuid Column Chunk base.driver_uuid Column Chunk base.status Column Chunk base.vehicle_id Column Chunk base.city_idRow Group Column Chunk base.client_uuid Column Chunk base.driver_uuid Column Chunk base.city_id Column Chunk base.vehicle_id Column Chunk base.statusRow Group Parquet Parquet Footer: File Metadata, Row Group Metadata Step 1: Read all Parquet nested ïŹelds from disk base.driver_uuid base.client_uuid base.city_id âŠ... base.vehicle_id base.status base.driver_uuid base.driver_uuid base.driver_uuid base.driver_uuid base.client_uuid base.city_id âŠ... base.vehicle_id base.status Presto Columnar Engine Step 2: Transform Parquet rows into Presto columnar blocks Step 3: Evaluate predicates on columnar blocks base.client_uuid base.client_uuid base.client_uuid base.city_id base.city_id base.city_id base.vehicle_id base.vehicle_id base.vehicle_id base.status base.status base.status âŠ. Default Apache Parquet Reader
- 28. Column Chunk base.client_uuid Column Chunk base.driver_uuid Column Chunk base.status Column Chunk base.vehicle_id Column Chunk base.city_id Row Group Column Chunk base.client_uuid Column Chunk base.driver_uuid Column Chunk base.city_id Column Chunk base.vehicle_id Column Chunk base.status Row Group Parquet Footer: File Metadata, Row Group Metadata Step 1: Read ONLY Required nested ïŹelds from disk Presto Columnar Engine Apache Parquet Reader Optimization base.driver_uuid base.driver_uuid base.driver_uuid base.city_id base.city_id base.city_id Step 1: Read ONLY Required nested ïŹelds from disk Evaluate predicates on the ïŹy: Skip reading row group; predicate: base.city_id = 12 dictionary: base.city_id: {3, 5, 9, 14, 21} Build columnar blocks only for predicate matches Step 2. Build columnar blocks on the ïŹy base.driver_uuid base.driver_uuid base.driver_uuid Step 3: Evaluate predicates on columnar blocks Parquet
- 29. Results
- 30. Looking forward
- 31. Federated SQL Layer Vision HDFS VerticaElasticsearch Apache Pinot MySQL Machines Reports Users Presto RealTime Presto Proxy layer Management Universal Metadata Service
- 32. Focus areas Connectors â Apache Hive, Apache Pinot, Elasticsearch, Apache Cassandra, Vertica, MySQL, etc â Aggregation / Join pushdown â Cross-connector optimizations (hybrid connectors) Real-time â Real-time mode with low latency pass through â Query plan / result / data cache â Time-series joins and stitching Universal Metadata Service (UMS) â Logical deïŹnitions / physical schemas â Column stitching and joins â Table and partition caching
- 33. Thank you Proprietary © 2018 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed. All recipients of this document are notiïŹed that the information contained herein includes proprietary information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.