Data pipelines observability: OpenLineage & Marquez

Feb 5, 20210 likes331 views

This document discusses OpenLineage and Marquez, which aim to provide standardized metadata and data lineage collection for data pipelines. OpenLineage defines an open standard for collecting metadata as data moves through pipelines, similar to metadata collected by EXIF for images. Marquez is an open source implementation of this standard, which can collect metadata from various data tools and store it in a graph database for querying lineage and understanding dependencies. This collected metadata helps with tasks like troubleshooting, impact analysis, and understanding how data flows through complex pipelines over time.

Data Pipelines Observability
OpenLineage & Marquez
Julien Le Dem
CTO & Co-Founder Datakin
@J_

AGENDA
● The need for metadata
● OpenLineage: open standard for metadata and
lineage collection
● Marquez: a reference implementation

Building a healthy data ecosystem
Team A Team B
Team C

● What is the data source?
● What is the schema?
● Who is the owner?
● How often is it updated?
● Where is it coming from?
● Who is using the data?
● What has changed?
Today: Limited context

Maslow’s Data hierarchy of needs
New Business Opportunities
Business Optimization
Data Quality
Data Freshness
Data Availability

OpenLineage contributors
Creators and contributors from major open source projects involved

Purpose
Deﬁne an Open standard for metadata and lineage
collection by instrumenting data pipelines as they are
running.

Problem
Before:
● Duplication of effort: Each project
has to instrument all jobs
● Integrations are external and can
break with new versions
● Effort of integration is shared
● Integration can be pushed in
each project: no need to play
catch up
With Open Lineage

Open Lineage scope Not in scope
Backend
Integrations
Metadata
and
lineage
collection
standard
Warehouse
Schedulers
...
Kafka
topic
Graph
db
HTTP
client
Consumers
Kafka
client
GraphDB
client
...

Core Model
● JSONSchema spec
● Consistent naming:
○ Jobs:
Example: scheduler.job.task
○ Datasets:
Example: instance.schema.table

Protocol
● Asynchronous events: unique run id for identifying a run and correlate events
○ Run Start event
■ source code version
■ run parameters
○ Run Complete event
■ input dataset
■ output dataset version and schema
● Conﬁgurable backend
○ Kafka
○ Http
○ ...

Facets
● Extensible:
Facets are atomic pieces of metadata identiﬁed by a unique name that can be
attached to the core entities.
● Decentralized:
Preﬁxes in facet names allow the deﬁnition of Custom facets that can be
promoted to the spec at a later point.

Data pipelines observability: OpenLineage & Marquez

Metadata:
Ingest Storage Compute
Streaming
Batch/ML
● Data Platform
built around
Marquez
● Integrations
○ Ingest
○ Storage
○ Compute
Flink
Airflow
Kafka
Iceberg / S3
BI
OpenLineage

Marquez: Data model
Job
Dataset Job Version
Run
*
1
*
1
*
1
1
*
1
*
Source
1 *
●
●
●
●
●
●
●
●
●
●
●
Dataset Version

API
● Open Lineage and Marquez standardize
metadata collection
○ Job runs
○ Parameters
○ Version
○ Inputs / outputs
● Datakin enables
○ Understanding operational dependencies
○ Impact analysis
○ Troubleshooting: What has changed
since the last time it worked?
Datakin leverages Marquez metadata
Lineage analysis
Graph
Integrations

Join the conversation
OpenLineage:
Github: github.com/OpenLineage
Slack: OpenLineage.slack.com
Twitter: @OpenLineage
Email: groups.google.com/g/openlineage
Marquez:
Github: github.com/MarquezProject/marquez
Slack: MarquezProject.slack.com
Twitter: @MarquezProject

Thank You
*we’re hiring! jobs@datakin.com

This document discusses Marquez, an open source metadata management system. It provides an overview of Marquez and how it can be used to track metadata in data pipelines. Specifically: - Marquez collects and stores metadata about data sources, datasets, jobs, and runs to provide data lineage and observability. - It has a modular framework to support data governance, data lineage, and data discovery. Metadata can be collected via REST APIs or language SDKs. - Marquez integrates with Apache Airflow to collect task-level metadata, dependencies between DAGs, and link tasks to code versions. This enables understanding of operational dependencies and troubleshooting. - The Marquez community aims to build an open

Data Pipline Observability meetup Omid Vahdaty

This document discusses the need for observability in data pipelines. It notes that real data pipelines often fail or take a long time to rerun without providing any insight into what went wrong. This is because of frequent code, data, dependency, and infrastructure changes. The document recommends taking a production engineering approach to observability using metrics, logging, and alerting tools. It also suggests experiment management and encapsulating reporting in notebooks. Most importantly, it stresses measuring everything through metrics at all stages of data ingestion and processing to better understand where issues occur.

Data and AI summit: data pipelines observability with open lineageJulien Le Dem

Observability for Data Pipelines With OpenLineageDatabricks

Data is increasingly becoming core to many products. Whether to provide recommendations for users, getting insights on how they use the product, or using machine learning to improve the experience. This creates a critical need for reliable data operations and understanding how data is flowing through our systems. Data pipelines must be auditable, reliable, and run on time. This proves particularly difficult in a constantly changing, fast-paced environment. Collecting this lineage metadata as data pipelines are running provides an understanding of dependencies between many teams consuming and producing data and how constant changes impact them. It is the underlying foundation that enables the many use cases related to data operations. The OpenLineage project is an API standardizing this metadata across the ecosystem, reducing complexity and duplicate work in collecting lineage information. It enables many projects, consumers of lineage in the ecosystem whether they focus on operations, governance or security. Marquez is an open source project part of the LF AI & Data foundation which instruments data pipelines to collect lineage and metadata and enable those use cases. It implements the OpenLineage API and provides context by making visible dependencies across organizations and technologies as they change over time.

Data QualityVijaya K

This document discusses data quality testing. It begins by defining data quality and listing its key dimensions such as accuracy, consistency, completeness and timeliness. It then notes common business problems caused by poor data quality and the benefits of improving data quality. Key aspects of data quality testing covered include planning, design, execution, monitoring and challenges. Best practices emphasized include understanding the business, planning for data quality early, being proactive about data growth and thoroughly understanding the data.

Knowledge Graphs and Generative AI_GraphSummit Minneapolis Sept 20.pptxNeo4j

This document discusses using knowledge graphs to ground large language models (LLMs) and improve their abilities. It begins with an overview of generative AI and LLMs, noting their opportunities but also challenges like lack of knowledge and inability to verify sources. The document then proposes using a knowledge graph like Neo4j to provide context and ground LLMs, describing how graphs can be enriched with algorithms, embeddings and other data. Finally, it demonstrates how contextual searches and responses can be improved by retrieving relevant information from the knowledge graph to augment LLM responses.

Databricks Platform.pptxAlex Ivy

The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.

bigquery.pptxHarissh16

BigQuery is Google Cloud Platform's interactive big data service that allows users to analyze massive datasets in seconds using SQL-like queries. It offers a scalable and fast way to query terabytes of data without the expense of maintaining servers or databases. BigQuery organizes data into a project-dataset-table hierarchy and uses a distributed architecture to efficiently process queries across servers.

Graph Databases – Benefits and RisksDATAVERSITY

Graph databases provide the ability to quickly discover and integrate key relationships between enterprise data sets. Business use cases such as recommendation engines, social networks, enterprise knowledge graphs, and more provide valuable ways to leverage graph databases in your organization. This webinar will provide an overview of graph database technologies, and how they can be used for practical applications to drive business value.

Data Quality & Data GovernanceTuba Yaman Him

This document discusses the importance of data quality and data governance. It states that poor data quality can lead to wrong decisions, bad reputation, and wasted money. It then provides examples of different dimensions of data quality like accuracy, completeness, currency, and uniqueness. It also discusses methods and tools for ensuring data quality, such as validation, data merging, and minimizing human errors. Finally, it defines data governance as a set of policies and standards to maintain data quality and provides examples of data governance team missions and a sample data quality scorecard.

Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker

Data GovernanceRob Lux

Postgre sql vs oracleJacques Kostic

This document summarizes the results of an OLTP performance benchmark test comparing PostgreSQL and Oracle databases. The test used HammerDB to run the same workload against each database on a server with 2x8 core CPUs and 192GB RAM. With 8 vCPUs, Oracle was 2.6% faster, used 16% less CPU, and had 9.3% more transactions per minute than PostgreSQL. When scaled to 16 vCPUs, Oracle was 3.4% faster, used 12.3% less CPU and had 22.43% more transactions per minute.

Building an Enterprise Knowledge Graph @Uber: Lessons from RealityJoshua Shinavier

This document summarizes Uber's experience building an enterprise knowledge graph. It notes that Uber has over 200,000 managed datasets and billions of trips served, making it an ideal testbed for a knowledge graph. However, it also outlines several lessons learned, including that real-world data is messy, an RDF-based approach is difficult, and property graphs alone are insufficient. The document advocates standardizing on shared vocabularies, fitting tools and data models to existing infrastructure, and collaborating across teams.

LDM Slides: Conceptual Data Models - How to Get the Attention of Business Use...DATAVERSITY

Achieving a ‘single version of the truth’ is critical to any MDM, DW, or data integration initiative. But have you ever tried to get people to agree on a single definition of “customer”? Or to get Sales, Marketing, and IT to agree on a target audience? This webinar will discuss how a conceptual data model can be used as a powerful communication tool for data-intensive initiatives. It will cover how to build a high-level data model, how the core concepts in a data model can have significant business impact on an organization, and will provide some easy-to-use templates and guidelines for a step-by-step approach to implementing a conceptual data model in your organization.

Demystifying data engineeringThang Bui (Bob)

Slides for the talk at AI in Production meetup: https://www.meetup.com/LearnDataScience/events/255723555/ Abstract: Demystifying Data Engineering With recent progress in the fields of big data analytics and machine learning, Data Engineering is an emerging discipline which is not well-defined and often poorly understood. In this talk, we aim to explain Data Engineering, its role in Data Science, the difference between a Data Scientist and a Data Engineer, the role of a Data Engineer and common concepts as well as commonly misunderstood ones found in Data Engineering. Toward the end of the talk, we will examine a typical Data Analytics system architecture.

MDM Strategy & Roadmapvictorlbrown

The document discusses strategies for managing master data through a Master Data Management (MDM) solution. It outlines challenges with current data management practices and goals for an improved MDM approach. Key considerations for implementing an effective MDM strategy include identifying initial data domains, use cases, source systems, consumers, and the appropriate MDM patterns to address business needs.

https://www.slideshare.net/neo4j/a-fusion-of-machine-learning-and-graph-analy...Neo4j

Beyond Retrieval Augmented Generation (RAG): Vector DatabasesZilliz

Modern Metadata StrategiesDATAVERSITY

Metadata is hotter than ever, according a number of recent DATAVERSITY surveys. More and more organizations are realizing that in order to drive business value from data, robust metadata is needed to gain the necessary context and lineage around key data assets. At the same time, industry regulations are driving the need for better transparency and understanding of information. While metadata has been managed for decades, new strategies & approaches have been developed to support the ever-evolving data landscape, and provide more innovative ways to drive business value from metadata. This webinar will provide an overview of metadata strategies & technologies available to today’s organization, and provide insights into building successful business strategies for metadata adoption & use.

Mapping Graph Queries to PostgreSQLGábor Szárnyas

Data Observability Best PracicesAndy Petrella

Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...HostedbyConfluent

Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures, such as data warehouse and data lake, through a cambrian explosion of tools and a large amount of investments to build their next data platform. Despite the intention and the investments the results have been middling. In this keynote, Zhamak shares her observations on the failure modes of a centralized paradigm of a data lake, and its predecessor data warehouse. She introduces Data Mesh, a paradigm shift in big data management that draws from modern distributed architecture: considering domains as the first class concern, applying self-sovereignty to distribute the ownership of data, applying platform thinking to create self-serve data infrastructure, and treating data as a product. This talk introduces the principles underpinning data mesh and Zhamak's recent learnings in creating a path to bring data mesh to life in your organization.

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY

A solid data architecture is critical to the success of any data initiative. But what is meant by “data architecture”? Throughout the industry, there are many different “flavors” of data architecture, each with its own unique value and use cases for describing key aspects of the data landscape. Join this webinar to demystify the various architecture styles and understand how they can add value to your organization.

Data Mesh 101ChrisFord803185

Data Mesh is a new socio-technical approach to data architecture, first described by Zhamak Dehghani and popularised through a guest blog post on Martin Fowler's site. Since then, community interest has grown, due to Data Mesh's ability to explain and address the frustrations that many organisations are experiencing as they try to get value from their data. The 2022 publication of Zhamak's book on Data Mesh further provoked conversation, as have the growing number of experience reports from companies that have put Data Mesh into practice. So what's all the fuss about? On one hand, Data Mesh is a new approach in the field of big data. On the other hand, Data Mesh is application of the lessons we have learned from domain-driven design and microservices to a data context. In this talk, Chris and Pablo will explain how Data Mesh relates to current thinking in software architecture and the historical development of data architecture philosophies. They will outline what benefits Data Mesh brings, what trade-offs it comes with and when organisations should and should not consider adopting it.

Data Management, Metadata Management, and Data Governance – Working TogetherDATAVERSITY

The data disciplines listed in the title must work together. The key to success requires understanding the boundaries and overlaps between the disciplines. Wouldn’t it be great to be able to present the relationships between the disciplines in a simple all-in diagram? At the end of this webinar, you will be able to do just that. This new RWDG webinar with Bob Seiner will outline how Data Management, Metadata Management, and Data Governance can be optimized to work together. Bob will share a diagram that has successfully communicated the relationship between these disciplines to leadership resulting in the disciplines working in harmony and delivering success. Bob will share the following in this webinar: - Categories of disciplines focused on managing data as an asset - A definition of Data Management that embraces numerous data disciplines - The importance of Metadata -Management to all data disciplines - Why data and metadata require formal governance - A graphic that effectively exhibits the relationship between the disciplines

Python for Data ScienceGabriel Moreira

In this presentation its given an introduction about Data Science, Data Scientist role and features, and how Python ecosystem provides great tools for Data Science process (Obtain, Scrub, Explore, Model, Interpret). For that, an attached IPython Notebook ( http://bit.ly/python4datascience_nb ) exemplifies the full process of a corporate network analysis, using Pandas, Matplotlib, Scikit-learn, Numpy and Scipy.

Open core summit: Observability for data pipelines with OpenLineageJulien Le Dem

This document discusses Open Lineage and the Marquez project for collecting metadata and data lineage information from data pipelines. It describes how Open Lineage defines a standard model and protocol for instrumentation to collect metadata on jobs, datasets, and runs in a consistent way. This metadata can then provide context on the data source, schema, owners, usage, and changes. The document outlines how Marquez implements the Open Lineage standard by defining entities, relationships, and facets to store this metadata and enable use cases like data governance, discovery, and debugging. It also positions Marquez as a centralized but modular framework to integrate various data platforms and extensions like Datakin's lineage analysis tools.

Structured Streaming in SparkDigital Vidya

Apache Spark has been gaining steam, with rapidity, both in the headlines and in real-world adoption. Spark was developed in 2009, and open sourced in 2010. Since then, it has grown to become one of the largest open source communities in big data with over 200 contributors from more than 50 organizations. This open source analytics engine stands out for its ability to process large volumes of data significantly faster than contemporaries such as MapReduce, primarily owing to in-memory storage of data on its own processing framework. That being said, one of the top real-world industry use cases for Apache Spark is its ability to process ‘streaming data‘.

More Related Content

What's hot (20)

Graph Databases – Benefits and RisksDATAVERSITY

Data Quality & Data GovernanceTuba Yaman Him

Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker

Data GovernanceRob Lux

Postgre sql vs oracleJacques Kostic

Building an Enterprise Knowledge Graph @Uber: Lessons from RealityJoshua Shinavier

LDM Slides: Conceptual Data Models - How to Get the Attention of Business Use...DATAVERSITY

Demystifying data engineeringThang Bui (Bob)

MDM Strategy & Roadmapvictorlbrown

https://www.slideshare.net/neo4j/a-fusion-of-machine-learning-and-graph-analy...Neo4j

Beyond Retrieval Augmented Generation (RAG): Vector DatabasesZilliz

Modern Metadata StrategiesDATAVERSITY

Mapping Graph Queries to PostgreSQLGábor Szárnyas

Data Observability Best PracicesAndy Petrella

Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...HostedbyConfluent

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY

Data Mesh 101ChrisFord803185

Data Management, Metadata Management, and Data Governance – Working TogetherDATAVERSITY

Python for Data ScienceGabriel Moreira

Graph Databases – Benefits and RisksDATAVERSITY

Data Quality & Data GovernanceTuba Yaman Him

Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker

Data GovernanceRob Lux

Postgre sql vs oracleJacques Kostic

Building an Enterprise Knowledge Graph @Uber: Lessons from RealityJoshua Shinavier

LDM Slides: Conceptual Data Models - How to Get the Attention of Business Use...DATAVERSITY

Demystifying data engineeringThang Bui (Bob)

MDM Strategy & Roadmapvictorlbrown

https://www.slideshare.net/neo4j/a-fusion-of-machine-learning-and-graph-analy...Neo4j

Beyond Retrieval Augmented Generation (RAG): Vector DatabasesZilliz

Modern Metadata StrategiesDATAVERSITY

Mapping Graph Queries to PostgreSQLGábor Szárnyas

Data Observability Best PracicesAndy Petrella

Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...HostedbyConfluent

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...DATAVERSITY

Data Mesh 101ChrisFord803185

Data Management, Metadata Management, and Data Governance – Working TogetherDATAVERSITY

Python for Data ScienceGabriel Moreira

Similar to Data pipelines observability: OpenLineage & Marquez (20)

Open core summit: Observability for data pipelines with OpenLineageJulien Le Dem

Structured Streaming in SparkDigital Vidya

Introduction to Structured Data Processing with Spark SQLdatamantra

Deploying Data Science Engines to ProductionMostafa Majidpour

Machine learning and big data @ uber a tale of two systemsZhenxiao Luo

This document discusses machine learning and big data at Uber. It describes Uber's mission to provide reliable transportation everywhere. It then discusses Uber's use of Kafka for data infrastructure and Hadoop/Spark for batch processing and machine learning. The document outlines pain points with the current system and proposes addressing them by building machine learning capabilities directly into Presto, an interactive SQL query engine, to improve efficiency and allow machine learning and data analytics to leverage shared resources. Key benefits of this approach are reducing complexity for data scientists and improving performance of machine learning pipelines.

Anatomy of Data Frame API : A deep dive into Spark Data Frame APIdatamantra

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...Databricks

Deploying machine learning models seems like it should be a relatively easy task. Take your model and pass it some features in production. The reality is that the code written during the prototyping phase of model development doesn’t always work when applied at scale or on “real” data. This talk will explore 1) common problems at the intersection of data science and data engineering 2) how you can structure your code so there is minimal friction between prototyping and production, and 3) how you can use Apache Spark to run predictions on your models in batch or streaming contexts. You will take away how to address some of productionizing issues that data scientists and data engineers face while deploying machine learning models at scale and a better understanding of how to work collaboratively to minimize disparity between prototyping and productizing.

WhereHows: Taming Metadata for 150K Datasets Over 9 Data PlatformsMars Lan

This document summarizes a presentation given by Mars Lan on LinkedIn's metadata management system called WhereHows. WhereHows collects metadata from LinkedIn's various data platforms and provides a unified view. It extracts metadata from sources using ETL jobs, transforms the data into a standardized format, and loads it into a database. WhereHows tracks over 150,000 datasets across 9 data platforms and visualizes the lineage and relationships between datasets to help users understand how data is processed. The system continues to be improved with better metadata collection, compliance features, and a simplified user experience.

Real time analytics on deep learning @ strata data 2019Zhenxiao Luo

This presentation discusses Uber's use of real-time analytics on deep learning models by combining TensorFlow and Presto. It provides an overview of deep learning and distributed deep learning using TensorFlow and Horovod at Uber. It then discusses Uber's use of Presto, an interactive SQL query engine, to enable real-time querying of deep learning models and datasets. Specific optimizations for the Presto Elasticsearch and Tensorflow connectors are also outlined to improve query performance.

A compute infrastructure for data scientistsStitch Fix Algorithms

Stitch Fix aspires to help you find the style that you will love. Data, the backbone of the business, is used to help with styling recommendations, demand modeling, user acquisition, and merchandise planning and also to influence business decisions throughout the organization. These decisions are backed by algorithms and data collected and interpreted based on client preferences. This talk offers an overview of the compute infrastructure used by the data science team at Stitch Fix, covering the architecture, tools within the larger ecosystem, and the challenges that the team overcame along the way. Apache Spark plays an important role in Stitch Fix’s data platform, and the company’s data scientists use Spark for their ETL and Presto for their ad hoc queries. The goal for the team running the compute infrastructure is to understand and make the data scientists’ lives easier, particularly in terms of usability of Spark, by building tools that make it easier to get started with Spark and transition themselves to a daily workflow. The compute infrastructure is a part of the data platform that is responsible for all the needs of data scientists as Stitch Fix. In this talk, we look at Stitch Fix’s journey, exploring its Spark setup, in-house tools and how they work in synergy with open source frameworks in a cloud environment. There are additional improvements to the infrastructure that help persist information for future use and optimization and we look at how the implementation of Amazon’s EMR FS has helped make it easier for us to read from the S3 source.

Open Data Inside - Why Internal Data Portals are Key to Successful Data Gover...Joel Natividad

Enterprise guide to building a Data MeshSion Smith

RESTful Machine Learning with Flask and TensorFlow Serving - Carlo MazzaferroPyData

The Enterprise Guide to Building a Data Mesh - Introducing SpecMeshIanFurlong4

For organisations to successfully adopt data mesh, setting up and maintaining infrastructure needs to be easy. We believe the best way to achieve this is to leverage the learnings from building a ‘central nervous system‘, commonly used in modern data-streaming ecosystems. This approach formalises and automates of the manual parts of building a data mesh. This presentation introduces SpecMesh; a methodology and supporting developer toolkit to enable business to build the foundations of their data mesh.

Data Discovery and Metadatamarkgrover

Gobblin @ NerdWallet (Nov 2015)NerdWalletHQ

C2_W1---.pdfHumayun Kabir

MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB

Graph Data Science at ScaleNeo4j

The document discusses graph data science and Neo4j's Graph Data Science (GDS) framework. GDS allows running graph algorithms and machine learning models at scale on large graph datasets. It discusses key aspects of GDS including architecture, data import, algorithm selection, and case studies of customers using GDS on graphs with billions of nodes and relationships. GDS runs on dedicated instances and supports features like enterprise graph compression, unlimited parallelization, and named graphs to optimize performance on large datasets.

Data Platform in the CloudAmihay Zer-Kavod

This document discusses building a data platform in the cloud. It covers the evolution of data platforms from monolithic architectures to distributed event-driven architectures using a data lake. Key aspects of a cloud data platform include collecting and persisting all data in a data lake for standardized access, near real-time processing using streaming technologies, and building the platform using either fully managed or DIY/hybrid approaches on AWS. Design principles focus on event-driven separation of data producers and consumers and choosing the right technology for the problem.

Open core summit: Observability for data pipelines with OpenLineageJulien Le Dem

Structured Streaming in SparkDigital Vidya

Introduction to Structured Data Processing with Spark SQLdatamantra

Deploying Data Science Engines to ProductionMostafa Majidpour

Machine learning and big data @ uber a tale of two systemsZhenxiao Luo

Anatomy of Data Frame API : A deep dive into Spark Data Frame APIdatamantra

Deploying Python Machine Learning Models with Apache Spark with Brandon Hamri...Databricks

WhereHows: Taming Metadata for 150K Datasets Over 9 Data PlatformsMars Lan

Real time analytics on deep learning @ strata data 2019Zhenxiao Luo

A compute infrastructure for data scientistsStitch Fix Algorithms

Open Data Inside - Why Internal Data Portals are Key to Successful Data Gover...Joel Natividad

Enterprise guide to building a Data MeshSion Smith

RESTful Machine Learning with Flask and TensorFlow Serving - Carlo MazzaferroPyData

The Enterprise Guide to Building a Data Mesh - Introducing SpecMeshIanFurlong4

Data Discovery and Metadatamarkgrover

Gobblin @ NerdWallet (Nov 2015)NerdWalletHQ

C2_W1---.pdfHumayun Kabir

MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB

Graph Data Science at ScaleNeo4j

Data Platform in the CloudAmihay Zer-Kavod

More from Julien Le Dem (20)

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

This document discusses principles for building a healthy data platform, including: 1. Establishing explicit contracts between teams to define dependencies and service level agreements. 2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest. 3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.

Strata NY 2018: The deconstructed databaseJulien Le Dem

The document discusses the evolution of big data systems from flat files in Hadoop to deconstructed databases. Originally, Hadoop used flat files with no schema and tied data storage to execution. This was flexible but inefficient. SQL databases provided schemas and separated storage from execution but were inflexible. Modern systems decompose databases into independent components like storage, processing, and machine learning that can be mixed and matched. Standards are emerging for columnar storage, querying, schemas, and data exchange. Future improvements include better interoperability, data abstractions, and governance.

From flat files to deconstructed databaseJulien Le Dem

From flat files to deconstructed databases: - Originally, Hadoop used flat files and MapReduce which was flexible but inefficient for queries. - The database world used SQL and relational models with optimizations but were inflexible. - Now components like storage, processing, and machine learning can be mixed and matched more efficiently with standards like Apache Calcite, Parquet, Avro and Arrow.

Strata NY 2017 Parquet Arrow roadmapJulien Le Dem

This document summarizes Apache Parquet and Apache Arrow, two open source projects for columnar data formats. It discusses how Parquet provides an on-disk columnar format for storage while Arrow provides an in-memory columnar format. The document outlines how Arrow builds on the success of Parquet by providing a common in-memory format that avoids serialization overhead and allows systems to share functionality. It provides examples of performance gains from the vertical integration of Parquet and Arrow.

The columnar roadmap: Apache Parquet and Apache ArrowJulien Le Dem

This document discusses Apache Parquet and Apache Arrow, open source projects for columnar data formats. Parquet is an on-disk columnar format that optimizes I/O performance through compression and projection pushdown. Arrow is an in-memory columnar format that maximizes CPU efficiency through vectorized processing and SIMD. It aims to serve as a standard in-memory format between systems. The document outlines how Arrow builds on Parquet's success and provides benefits like reduced serialization overhead and ability to share functionality through its ecosystem. It also describes how Parquet and Arrow representations are integrated through techniques like vectorized reading and predicate pushdown.

Improving Python and Spark Performance and Interoperability with Apache ArrowJulien Le Dem

This document discusses improving Python and Spark performance and interoperability with Apache Arrow. It begins with an overview of current limitations of PySpark UDFs, such as inefficient data movement and scalar computation. It then introduces Apache Arrow, an open source in-memory columnar data format, and how it can help by allowing more efficient data sharing and vectorized computation. The document shows how Arrow improved PySpark UDF performance by 53x through vectorization and reduced serialization. It outlines future plans to further optimize UDFs and integration with Spark and other projects.

Mule soft mar 2017 Parquet ArrowJulien Le Dem

This document discusses the future of column-oriented data processing with Apache Arrow and Apache Parquet. Arrow provides an open standard for in-memory columnar data, while Parquet provides an open standard for on-disk columnar data storage. Together they provide interoperability across systems and high performance by avoiding data copying and format conversions. The document outlines the goals and benefits of Arrow and Parquet, how they improve CPU and I/O efficiency, and examples of performance gains from integrating systems like Spark and Pandas with Arrow.

Data Eng Conf NY Nov 2016 Parquet ArrowJulien Le Dem

- Arrow and Parquet are open source projects focused on column-oriented data formats for efficient in-memory (Arrow) and on-disk (Parquet) analytics. - They allow for interoperability across systems by eliminating the overhead of data serialization and enabling common data representations. - Column-oriented formats improve performance by reducing storage needs, enabling projection of only needed columns, and better utilization of CPU/memory through cache locality and vectorized processing.

Strata NY 2016: The future of column-oriented data processing with Arrow and ...Julien Le Dem

Strata London 2016: The future of column oriented data processing with Arrow ...Julien Le Dem

Sql on everything with drillJulien Le Dem

If you have your own Columnar format, stop now and use Parquet 😛Julien Le Dem

Lightning talk presented at HPTS 2015: http://hpts.ws/ Apache Parquet is the de facto standard columnar storage for big data. Open source and proprietary SQL engines already integrate with it as their users don’t want to load and duplicate their data in every tool. Users want an open, interoperable, efficient format to experiment with the many options they have. The format is defined by the open source community integrating feedback from many teams working on query engines (including but not limited to Impala, Drill, Hawq, SparkSQL, Presto, Hive, etc) or on infrastructure at scale (Twitter, Netflix, Stripe, Criteo, ...). Building on its initial success, the Parquet community is defining new features for the next iteration of the format. For example: improved metadata layout, type system completude or mergeable statistics used for planning.

How to use Parquet as a basis for ETL and analyticsJulien Le Dem

Parquet is a columnar format designed to be extremely efficient and interoperable across the hadoop ecosystem. Its integration in most of the Hadoop processing frameworks (Impala, Hive, Pig, Cascading, Crunch, Scalding, Spark, …) and serialization models (Thrift, Avro, Protocol Buffers, …) makes it easy to use in existing ETL and processing pipelines, while giving flexibility of choice on the query engine (whether in Java or C++). In this talk, we will describe how one can us Parquet with a wide variety of data analysis tools like Spark, Impala, Pig, Hive, and Cascading to create powerful, efficient data analysis pipelines. Data management is simplified as the format is self describing and handles schema evolution. Support for nested structures enables more natural modeling of data for Hadoop compared to flat representations that create the need for often costly joins.

Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014Julien Le Dem

Apache Parquet is an open-source columnar storage format for efficient data storage and analytics. It provides efficient compression and encoding techniques that enable fast scans and queries of large datasets. Parquet 2.0 improves on these efficiencies through techniques like delta encoding, dictionary encoding, run-length encoding and binary packing designed for CPU and cache optimizations. Benchmark results show Parquet provides much better compression and faster query performance than other formats like text, Avro and RCFile. The project is developed as an open source community with contributions from many organizations.

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Parquet is a columnar storage format for Hadoop data. It was developed collaboratively by Twitter and Cloudera to address the need for efficient analytics on large datasets. Parquet provides more efficient compression and I/O compared to row-based formats by only reading and decompressing the columns needed by a query. It has been adopted by many companies for analytics workloads involving terabytes to petabytes of data. Parquet is language-independent and supports integration with frameworks like Hive, Pig, and Impala. It provides significant performance improvements and storage savings compared to traditional row-based formats.

Parquet Hadoop Summit 2013Julien Le Dem

Parquet is a columnar storage format for Hadoop data. It was developed by Twitter and Cloudera to optimize storage and querying of large datasets. Parquet provides more efficient compression and I/O compared to traditional row-based formats by storing data by column. Early results show a 28% reduction in storage size and up to a 114% improvement in query performance versus the original Thrift format. Parquet supports complex nested schemas and can be used with Hadoop tools like Hive, Pig, and Impala.

Parquet Twitter Seattle open houseJulien Le Dem

This document discusses Parquet, an open-source columnar file format for Hadoop. It was created at Twitter to optimize their analytics infrastructure, which includes several large Hadoop clusters processing data from their 200M+ users. Parquet aims to improve on existing storage formats by organizing data column-wise for better compression and scanning capabilities. It uses a row group structure and supports efficient reads of individual columns. Initial results at Twitter found a 28% space savings over existing formats and scan performance improvements. The project is open source and aims to continue optimizing column storage and enabling new execution engines for Hadoop.

Parquet overviewJulien Le Dem

Parquet is a column-oriented storage format for Hadoop that supports efficient compression and encoding techniques. It uses a row group structure to store data in columns in a compressed and encoded column chunk format. The schema and metadata are stored in the file footer to allow for efficient reads and scans of selected columns. The format is designed to be extensible through pluggable components for schema conversion, record materialization, and encodings.

Poster Hadoop summit 2011: pig embedding in scripting languagesJulien Le Dem

Embedding Pig in scripting languagesJulien Le Dem

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

Strata NY 2018: The deconstructed databaseJulien Le Dem

From flat files to deconstructed databaseJulien Le Dem

Strata NY 2017 Parquet Arrow roadmapJulien Le Dem

The columnar roadmap: Apache Parquet and Apache ArrowJulien Le Dem

Improving Python and Spark Performance and Interoperability with Apache ArrowJulien Le Dem

Mule soft mar 2017 Parquet ArrowJulien Le Dem

Data Eng Conf NY Nov 2016 Parquet ArrowJulien Le Dem

Strata NY 2016: The future of column-oriented data processing with Arrow and ...Julien Le Dem

Strata London 2016: The future of column oriented data processing with Arrow ...Julien Le Dem

Sql on everything with drillJulien Le Dem

If you have your own Columnar format, stop now and use Parquet 😛Julien Le Dem

How to use Parquet as a basis for ETL and analyticsJulien Le Dem

Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014Julien Le Dem

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Parquet Hadoop Summit 2013Julien Le Dem

Parquet Twitter Seattle open houseJulien Le Dem

Parquet overviewJulien Le Dem

Poster Hadoop summit 2011: pig embedding in scripting languagesJulien Le Dem

Embedding Pig in scripting languagesJulien Le Dem

Recently uploaded (20)

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how: • Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules. • Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance. • Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity. • Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications. • Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market. With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications. Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Cyber Awareness overview for 2025 month of securityriccardosl1

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts! 📕 Agenda Welcome & Introductions Orchestrator API Overview Exploring the Swagger Interface Test Manager API Highlights Streamlining Automation & Testing with APIs (Demo) Q&A and Open Discussion Perfect for developers, testers, and automation enthusiasts! 👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/ This session streamed live on April 29, 2025, 18:00 CET. Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

AI and Data Privacy in 2025: Global TrendsInData Labs

In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy. This infographic contains: -AI and data privacy: Key findings -Statistics on AI data privacy in the today’s world -Tips on how to overcome data privacy challenges -Benefits of AI data security investments. Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Cyber Awareness overview for 2025 month of securityriccardosl1

TrsLabs - Fintech Product & Business ConsultingTrs Labs

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

AI and Data Privacy in 2025: Global TrendsInData Labs

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Data pipelines observability: OpenLineage & Marquez

1. Data Pipelines Observability OpenLineage & Marquez Julien Le Dem CTO & Co-Founder Datakin @J_

2. AGENDA ● The need for metadata ● OpenLineage: open standard for metadata and lineage collection ● Marquez: a reference implementation

3. The need for Metadata

4. Building a healthy data ecosystem Team A Team B Team C

5. ● What is the data source? ● What is the schema? ● Who is the owner? ● How often is it updated? ● Where is it coming from? ● Who is using the data? ● What has changed? Today: Limited context

6. Maslow’s Data hierarchy of needs New Business Opportunities Business Optimization Data Quality Data Freshness Data Availability

7. OpenLineage

8. OpenLineage contributors Creators and contributors from major open source projects involved

9. Purpose Deﬁne an Open standard for metadata and lineage collection by instrumenting data pipelines as they are running.

10. Purpose: EXIF for data pipelines

11. Problem Before: ● Duplication of effort: Each project has to instrument all jobs ● Integrations are external and can break with new versions ● Effort of integration is shared ● Integration can be pushed in each project: no need to play catch up With Open Lineage

12. Open Lineage scope Not in scope Backend Integrations Metadata and lineage collection standard Warehouse Schedulers ... Kafka topic Graph db HTTP client Consumers Kafka client GraphDB client ...

13. Core Model ● JSONSchema spec ● Consistent naming: ○ Jobs: Example: scheduler.job.task ○ Datasets: Example: instance.schema.table

14. Protocol ● Asynchronous events: unique run id for identifying a run and correlate events ○ Run Start event ■ source code version ■ run parameters ○ Run Complete event ■ input dataset ■ output dataset version and schema ● Conﬁgurable backend ○ Kafka ○ Http ○ ...

15. Facets ● Extensible: Facets are atomic pieces of metadata identified by a unique name that can be attached to the core entities. ● Decentralized: Prefixes in facet names allow the definition of Custom facets that can be promoted to the spec at a later point.

16. Facet examples Dataset: - Stats - Schema - Version - Column level lineage Job: - Source code - Dependencies - params - Source control - Query plan - Query profile Run: - Schedule time - Batch id

18. Metadata: Ingest Storage Compute Streaming Batch/ML ● Data Platform built around Marquez ● Integrations ○ Ingest ○ Storage ○ Compute Flink Airflow Kafka Iceberg / S3 BI OpenLineage

19. Marquez: Data model Job Dataset Job Version Run * 1 * 1 * 1 1 * 1 * Source 1 * ● ● ● ● ● ● ● ● ● ● ● Dataset Version

20. API ● Open Lineage and Marquez standardize metadata collection ○ Job runs ○ Parameters ○ Version ○ Inputs / outputs ● Datakin enables ○ Understanding operational dependencies ○ Impact analysis ○ Troubleshooting: What has changed since the last time it worked? Datakin leverages Marquez metadata Lineage analysis Graph Integrations

21. Join the conversation OpenLineage: Github: github.com/OpenLineage Slack: OpenLineage.slack.com Twitter: @OpenLineage Email: groups.google.com/g/openlineage Marquez: Github: github.com/MarquezProject/marquez Slack: MarquezProject.slack.com Twitter: @MarquezProject

22. Thank You *we’re hiring! [email protected]

Data pipelines observability: OpenLineage & Marquez

Recommended

More Related Content

What's hot (20)

Similar to Data pipelines observability: OpenLineage & Marquez (20)

More from Julien Le Dem (20)

Recently uploaded (20)

Data pipelines observability: OpenLineage & Marquez