Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn

Jul 2, 201339 likes11,318 views

Amy W. Tang

This talk was given by Joel Koshy (Senior Software Engineer at LinkedIn) at the Hadoop Summit (June 2013).

Building a Real-Time Data Pipeline:
Apache Kafka at Linkedin
Hadoop Summit 2013
Joel Koshy
June 2013
LinkedIn Corporation ©2013 All Rights Reserved

HADOOP SUMMIT 2013
Network update stream

LinkedIn Corporation ©2013 All Rights Reserved
We have a lot of data.
We want to leverage this data to build products.
Data pipeline

HADOOP SUMMIT 2013
System and application metrics/logging
LinkedIn Corporation ©2013 All Rights Reserved 5

How do we integrate this variety of data
and make it available to all these systems?
LinkedIn Confidential ©2013 All Rights Reserved

HADOOP SUMMIT 2013
Point-to-point pipelines

HADOOP SUMMIT 2013
LinkedIn’s user activity data pipeline (circa 2010)

HADOOP SUMMIT 2013
Four key ideas
1. Central data pipeline
2. Push data cleanliness upstream
3. O(1) ETL
4. Evidence-based correctness
LinkedIn Corporation ©2013 All Rights Reserved 10

HADOOP SUMMIT 2013
Central data pipeline

First attempt: don’t re-invent the wheel
LinkedIn Confidential ©2013 All Rights Reserved

Second attempt: re-invent the wheel!
LinkedIn Confidential ©2013 All Rights Reserved

Use a central commit log
LinkedIn Confidential ©2013 All Rights Reserved

HADOOP SUMMIT 2013
What is a commit log?

HADOOP SUMMIT 2013
The log as a messaging system
LinkedIn Corporation ©2013 All Rights Reserved 17

HADOOP SUMMIT 2013
Apache Kafka
LinkedIn Corporation ©2013 All Rights Reserved 18

HADOOP SUMMIT 2013
Usage at LinkedIn
 16 brokers in each cluster
 28 billion messages/day
 Peak rates
– Writes: 460,000 messages/second
– Reads: 2,300,000 messages/second
 ~ 700 topics
 40-50 live services consuming user-activity data
 Many ad hoc consumers
 Every production service is a producer (for metrics)
 10k connections/colo
LinkedIn Corporation ©2013 All Rights Reserved 19

HADOOP SUMMIT 2013
Usage at LinkedIn
LinkedIn Corporation ©2013 All Rights Reserved 20

HADOOP SUMMIT 2013
Standardize on Avro in data pipeline
LinkedIn Corporation ©2013 All Rights Reserved 22
{
"type": "record",
"name": "URIValidationRequestEvent",
"namespace": "com.linkedin.event.usv",
"fields": [
{
"name": "header",
"type": {
"type": "record",
"name": ”TrackingEventHeader",
"namespace": "com.linkedin.event",
"fields": [
{
"name": "memberId",
"type": "int",
"doc": "The member id of the user initiating the action"
},
{
"name": ”timeMs",
"type": "long",
"doc": "The time of the event"
},
{
"name": ”host",
"type": "string",
...
...

HADOOP SUMMIT 2013
Hadoop data load (Camus)
 Open sourced:
– https://github.com/linkedin/camus
 One job loads all events
 ~10 minute ETA on average from producer to HDFS
 Hive registration done automatically
 Schema evolution handled transparently

Does it work?
“All published messages must be delivered to all consumers (quickly)”
LinkedIn Confidential ©2013 All Rights Reserved

HADOOP SUMMIT 2013
Kafka replication (0.8)
 Intra-cluster replication feature
– Facilitates high availability and durability
 Beta release available
https://dist.apache.org/repos/dist/release/kafka/
 Rolled out in production at LinkedIn last week
LinkedIn Corporation ©2013 All Rights Reserved 28

HADOOP SUMMIT 2013
Join us at our user-group meeting tonight @ LinkedIn!
– Thursday, June 27, 7.30pm to 9.30pm
– 2025 Stierlin Ct., Mountain View, CA
– http://www.meetup.com/http-kafka-apache-org/events/125887332/
– Presentations (replication overview and use-case studies) from:
 RichRelevance
 Netflix
 Square
 LinkedIn
LinkedIn Corporation ©2013 All Rights Reserved 29

HADOOP SUMMIT 2013LinkedIn Corporation ©2013 All Rights Reserved 30

The document discusses using Apache Kafka's batch message listener capabilities with MuleSoft to load data into Teradata Vantage in parallel. It covers configuring the Kafka consumer, extracting payload data, inserting it into the database table in parallel batches while handling errors, and logging problematic messages. A live demo shows performance of single vs batch loading and handling irreproducible database errors.

아파치 카프카 입문과 활용 강의자료원영 최

https://github.com/AndersonChoi/tacademy-kafka Apache Kafka는 대용량 실시간 로그처리에 특화된 분산 메시징 시스템으로, 대용량/대규모의 스트리밍 메시지 데이터를 빠르게 처리하도록 개발되었습니다. 이번 강의에서는 Kafka Producer/Consumer 등 기본 개념을 익히고, 설치부터 기본적인 애플리케이션 개발 실습을 통해 빅데이터 파이프라인에서 Kafka가 어떤 핵심적인 역할을 수행하는 지 알아보는 시간을 갖습니다.

Introduction to REST API with Node.jsYoann Gotthilf

This document provides an introduction to Node.js and Mongoose. It discusses that Node.js is a JavaScript runtime built on Chrome's V8 engine for building fast and scalable network applications. It then summarizes key aspects of Node.js like its architecture, core modules, use of packages, and creating simple modules. It also introduces Express as a web framework and Mongoose as an ORM for MongoDB, summarizing their basic usage and schemas.

Building an Event Streaming Architecture with Apache PulsarScyllaDB

Sync async-blocking-nonblocking-ioCheoloh Bae

Linux-HA with PacemakerKris Buytaert

Linux High Availability provides concise summaries of key concepts: - High availability (HA) clustering allows services to take over work from others that go down, through IP and service takeover. It is designed for uptime, not performance or load balancing. - Downtime is expensive for businesses due to lost revenue and customer dissatisfaction. Statistics show significant drops in availability even at 99.9% uprates. - To achieve high availability, systems must be designed with simplicity, failure preparation, and reliability testing in mind. Complexity often undermines reliability. - Myths exist around technologies like virtualization and live migration providing complete high availability solutions. True HA requires eliminating all single points of

MySQL on AWS RDSMydbops

When apache pulsar meets apache flinkStreamNative

Both Apache Pulsar and Apache Flink share a similar view on how the data and the computation level of an application can be “streaming-first” with batch as a special case streaming. With Apache Pulsar’s Segmented-Stream storage and Apache Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies to provide elastic data processing at massive scale, and build a real streaming warehouse. In this talk, Sijie Guo from Apache Pulsar community will given an overview of Apache Pulsar and how it provides the unified data view to fully leverage Apache Flink unified computation runtime for elastic data processing. He will share the latest integrations between Apache Pulsar and Apache Flink, especially around effectively-once processing and schema integration.

Application Continuity with Oracle DB 12c Léopold Gault

SignalR for ASP.NET DevelopersShivanand Arur

This document provides an overview of ASP.NET SignalR, a library for building real-time web functionality. It discusses traditional web application approaches using request-response, defines what "real-time" means in terms of pushing data from server to client. It introduces SignalR as a library that uses push technology to provide persistent connections and real-time functionality. It also covers SignalR's transport techniques including websockets, server-sent events, forever frames, and long polling, as well as the types of connections in SignalR including persistent connections and hubs.

Introduction to Kafka and ZookeeperRahul Jain

Impala Architecture presentationhadooparchbook

Apache Web Server Architecture Chaitanya Kulkarniwebhostingguy

Apache Web Server is an open-source web server software widely used on the internet. It has a modular architecture with core components that handle basic functions and additional modules that extend functionality. Apache supports concurrency through persistent processes that handle requests independently in separate address spaces to improve performance on busy websites. The Apache license allows derived open-source and closed-source software.

Apache Kafka Architecture & Fundamentals Explainedconfluent

Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput. This talk provides a comprehensive overview of Kafka architecture and internal functions, including: -Topics, partitions and segments -The commit log and streams -Brokers and broker replication -Producer basics -Consumers, consumer groups and offsets This session is part 2 of 4 in our Fundamentals for Apache Kafka series.

Understanding RESTNitin Pande

- REST (Representational State Transfer) uses HTTP requests to transfer representations of resources between clients and servers. The format of the representation is determined by the content-type header and the interaction with the resource is determined by the HTTP verb used. - The four main HTTP verbs are GET, PUT, DELETE, and POST. GET retrieves a representation of the resource and is safe, while PUT, DELETE, and POST can modify the resource's state in atomic operations. - Resources are abstract concepts acted upon by HTTP requests, while representations are the actual data transmitted in responses. The representation may or may not accurately reflect the resource's current state.

스타트업을 위한 Confluent 세미나confluent

Introduction to GrafanaKnoldus Inc.

Web hdfs and httpfswchevreuil

Reverse proxytim4911

A reverse proxy sits in front of web servers and forwards client requests to those servers. It helps increase security, performance, and reliability compared to a forward proxy. A reverse proxy hides the existence and characteristics of origin servers, distributing load balancing to prevent overload and speed up loading through content compression and caching. It provides benefits like protecting against attacks, increasing speed and improving security by hiding server IP addresses.

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

Flink Forward San Francisco 2022. With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi. by Ethan Guo & Kyle Weller

Distributed Counters in Cassandra (Cassandra Summit 2010)kakugawa

The document discusses the design and implementation of distributed counters in Cassandra. It aims to provide low latency and high availability counters. The design relaxes consistency constraints by using commutative operations, partitioning work across replicas, and allowing idempotent repairs. Counters are implemented using a context format to track counts across replicas, with writes incrementing local counts and repairs retaining the highest counts seen. Eventual consistency is achieved through read repair and anti-entropy repairs that propagate counter states.

Maria db 이중화구성_고민하기NeoClova

Spring Boot+Kafka: the New Enterprise PlatformVMware Tanzu

This document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform focused on continuous delivery, event-driven architectures, and streaming data. It provides examples of companies that have successfully adopted this approach, such as Netflix transitioning to Spring Boot and a banking brand building a new core banking system using Spring Streams and Kafka. The document advocates an "event-first" and microservices-oriented mindset enabled by a streaming data platform and suggests that Spring Boot, Kafka, and related technologies provide a turnkey solution for implementing this new application development approach at large enterprises.

Apache Kafka Fundamentals for Architects, Admins and Developersconfluent

This document summarizes a presentation about Apache Kafka. It introduces Apache Kafka as a modern, distributed platform for data streams made up of distributed, immutable, append-only commit logs. It describes Kafka's scalability similar to a filesystem and guarantees similar to a database, with the ability to rewind and replay data. The document discusses Kafka topics and partitions, partition leadership and replication, and provides resources for further information.

SplunkLive! Getting Started with Splunk EnterpriseSplunk

The document provides an agenda and overview for a Splunk getting started user training workshop. The summary covers the key topics: - Getting started with Splunk including downloading, installing, and starting Splunk - Core Splunk functions like searching, field extraction, saved searches, alerts, reporting, dashboards - Deployment options including universal forwarders, distributed search, and high availability - Integrations with other systems for data input, user authentication, and data output - Support resources like the Splunk community, documentation, and technical support

HBase Operations and Best PracticesVenu Anuganti

This document provides an overview and best practices for operating HBase clusters. It discusses HBase and Hadoop architecture, how to set up an HBase cluster including Zookeeper and region servers, high availability considerations, scaling the cluster, backup and restore processes, and operational best practices around hardware, disks, OS, automation, load balancing, upgrades, monitoring and alerting. It also includes a case study of a 110 node HBase cluster.

Beautiful Monitoring With Grafana and InfluxDBleesjensen

How to use Impala query plan and profile to fix performance issuesCloudera, Inc.

Architecture of a Kafka camus infrastructuremattlieber

This document summarizes the results of a performance evaluation of Kafka and Camus to ingest streaming data into Hadoop. It finds that Kafka can ingest data at rates from 15,000-50,000 messages per second depending on data format (Avro is fastest). Camus can move the data to HDFS at rates from 54,000-662,000 records per second. Once in HDFS, queries on Avro-formatted data are fastest, with count and max aggregation queries completing in under 100 seconds for 20 million records. The customer's goal of 5000 events per second can be easily achieved with this architecture.

Data Infrastructure at LinkedInAmy W. Tang

More Related Content

What's hot (20)

Application Continuity with Oracle DB 12c Léopold Gault

SignalR for ASP.NET DevelopersShivanand Arur

Introduction to Kafka and ZookeeperRahul Jain

Impala Architecture presentationhadooparchbook

Apache Web Server Architecture Chaitanya Kulkarniwebhostingguy

Apache Kafka Architecture & Fundamentals Explainedconfluent

Understanding RESTNitin Pande

스타트업을 위한 Confluent 세미나confluent

Introduction to GrafanaKnoldus Inc.

Web hdfs and httpfswchevreuil

Reverse proxytim4911

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

Distributed Counters in Cassandra (Cassandra Summit 2010)kakugawa

Maria db 이중화구성_고민하기NeoClova

Spring Boot+Kafka: the New Enterprise PlatformVMware Tanzu

Apache Kafka Fundamentals for Architects, Admins and Developersconfluent

SplunkLive! Getting Started with Splunk EnterpriseSplunk

HBase Operations and Best PracticesVenu Anuganti

Beautiful Monitoring With Grafana and InfluxDBleesjensen

How to use Impala query plan and profile to fix performance issuesCloudera, Inc.

Application Continuity with Oracle DB 12c Léopold Gault

SignalR for ASP.NET DevelopersShivanand Arur

Introduction to Kafka and ZookeeperRahul Jain

Impala Architecture presentationhadooparchbook

Apache Web Server Architecture Chaitanya Kulkarniwebhostingguy

Apache Kafka Architecture & Fundamentals Explainedconfluent

Understanding RESTNitin Pande

스타트업을 위한 Confluent 세미나confluent

Introduction to GrafanaKnoldus Inc.

Web hdfs and httpfswchevreuil

Reverse proxytim4911

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

Distributed Counters in Cassandra (Cassandra Summit 2010)kakugawa

Maria db 이중화구성_고민하기NeoClova

Spring Boot+Kafka: the New Enterprise PlatformVMware Tanzu

Apache Kafka Fundamentals for Architects, Admins and Developersconfluent

SplunkLive! Getting Started with Splunk EnterpriseSplunk

HBase Operations and Best PracticesVenu Anuganti

Beautiful Monitoring With Grafana and InfluxDBleesjensen

How to use Impala query plan and profile to fix performance issuesCloudera, Inc.

Viewers also liked (20)

Architecture of a Kafka camus infrastructuremattlieber

Data Infrastructure at LinkedInAmy W. Tang

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.

Data Applications and Infrastructure at LinkedIn__HadoopSummit2010Yahoo Developer Network

LinkedIn Segmentation & Targeting Platform: A Big Data ApplicationAmy W. Tang

Espresso: LinkedIn's Distributed Data Serving Platform (Talk)Amy W. Tang

Data Infrastructure at LinkedIn Amy W. Tang

A Small Overview of Big Data Products, Analytics, and Infrastructure at LinkedInAmy W. Tang

Introduction to Apache KafkaJeff Holoman

The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.

LinkedIn Communication ArchitectureLinkedIn

The document discusses LinkedIn's communication architecture and network updates service. It describes how LinkedIn built scalable communication platforms to support its large professional network. The system evolved from handling 0 to 22 million members. It uses Java, databases like Oracle and MySQL, application servers like Tomcat and Jetty, and technologies like ActiveMQ, Lucene and Spring. The communication service handles messages and email delivery while the network updates service distributes short-lived notifications across LinkedIn's various clients and services.

Introduction to DatabusAmy W. Tang

This document describes Databus, a system used at LinkedIn for distributed data replication and change data capture. Some key points: - Databus provides timeline consistency across distributed data systems by applying a logical clock to data changes and using a pull-based model for replication. - It addresses the challenges of specialization in distributed data systems through standardization, isolation of consumers from sources, and handling slow consumers without impacting fast ones. - The architecture includes fetchers that extract changes from databases, a relay for buffering changes, log and snapshot stores, and client libraries that allow applications to consume changes. - Performance is optimized through partitioning, filtering, and scaling of consumers independently of sources. Databus

Building Distributed Systems Using HelixAmy W. Tang

Building a Data Pipeline from Scratch - Joe CrobakHakka Labs

A data pipeline is a unified system for capturing events for analysis and building products. It involves capturing user events from various sources, storing them in a centralized data warehouse, and performing analysis and building products using tools like Hadoop. Key components of a data pipeline include an event framework, message bus, data serialization, data persistence, workflow management, and batch processing. A Lambda architecture allows for both batch and real-time processing of data captured by the pipeline.

What is a distributed data science pipeline. how with apache spark and friends.Andy Petrella

Rakuten LeoFs - distributed file systemRakuten Group, Inc.

Introduction to apache kafkaSamuel Kerrien

Apache KafkaMaher TEBOURBI

Apache Kafka is a distributed streaming platform that allows building event-driven architectures. It provides high throughput and low latency for processing streaming data. Key features include event logging, publish-subscribe messaging, and stream processing capabilities. Some advantages are eventual consistency, scalability, fault tolerance and being more agile to maintain compared to traditional databases. It requires Zookeeper and the Java client API has undergone changes. Performance can be very high, with examples of LinkedIn processing 1.1 trillion messages per day and 2 million writes per second on modest hardware.

Realtime streaming architecture in INFINARIOJozo Kovac

IMCSummit 2015 - Day 2 IT Business Track - Real-time Interactive Big Data Ana...In-Memory Computing Summit

Non-interactive big-data analysis prohibits experimentation and can interrupt the analyst’s train of thoughts but analyzing and drawing insights in real time is no easy task with jobs often taking minutes/hours to complete. What if you want to put a interactive interface in front of that data that allows iterative insights? What if you need that interactive experience to be sub second? Traditional SQL and most MPP/NoSQL databases cannot run complex calculations over large data in a performant manner. Popular distributed systems such as Hadoop or Spark can execute jobs but their job overhead prohibits sub second response times. Learn how an in-memory computing framework enabled us to perform complex analysis jobs on massive data points with sub second response times — allowing us to plug it into a simple, drag-and-drop web 2.0 interface.

Intro to SnappyData WebinarSnappyData

The slides for the first ever SnappyData webinar. Covers SnappyData core concepts, programming models, benchmarks and more. SnappyData is open sourced here: https://github.com/SnappyDataInc/snappydata We also have a deep technical paper here: http://www.snappydata.io/snappy-industrial We can be easily contacted on Slack, Gitter and more: http://www.snappydata.io/about#contactus

Architecture of a Kafka camus infrastructuremattlieber

Data Infrastructure at LinkedInAmy W. Tang

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

Data Applications and Infrastructure at LinkedIn__HadoopSummit2010Yahoo Developer Network

LinkedIn Segmentation & Targeting Platform: A Big Data ApplicationAmy W. Tang

Espresso: LinkedIn's Distributed Data Serving Platform (Talk)Amy W. Tang

Data Infrastructure at LinkedIn Amy W. Tang

A Small Overview of Big Data Products, Analytics, and Infrastructure at LinkedInAmy W. Tang

Introduction to Apache KafkaJeff Holoman

LinkedIn Communication ArchitectureLinkedIn

Introduction to DatabusAmy W. Tang

Building Distributed Systems Using HelixAmy W. Tang

Building a Data Pipeline from Scratch - Joe CrobakHakka Labs

What is a distributed data science pipeline. how with apache spark and friends.Andy Petrella

Rakuten LeoFs - distributed file systemRakuten Group, Inc.

Introduction to apache kafkaSamuel Kerrien

Apache KafkaMaher TEBOURBI

Realtime streaming architecture in INFINARIOJozo Kovac

IMCSummit 2015 - Day 2 IT Business Track - Real-time Interactive Big Data Ana...In-Memory Computing Summit

Intro to SnappyData WebinarSnappyData

Similar to Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn (20)

All data accessible to all my organization - Presentation at OW2con'19, June...OW2

This document discusses how Dremio provides a unified access point for data across an entire organization. It summarizes how Dremio allows various users, including data engineers, scientists, analysts and business users, to access all kinds of data sources through SQL or REST APIs. Dremio also enables features like data catalogs, collaborative workspaces, and workload monitoring that help organizations better manage and govern their data.

Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Shirshanka Das

So, you finally have a data ecosystem with Kafka and Hadoop both deployed and operating correctly at scale. Congratulations. Are you done? Far from it. As the birthplace of Kafka and an early adopter of Hadoop, LinkedIn has 13 years of combined experience using Kafka and Hadoop at scale to run a data-driven company. Both Kafka and Hadoop are flexible, scalable infrastructure pieces, but using these technologies without a clear idea of what the higher-level data ecosystem should be is perilous. Shirshanka Das and Yael Garten share best practices around data models and formats, choosing the right level of granularity of Kafka topics and Hadoop tables, and moving data efficiently and correctly between Kafka and Hadoop and explore a data abstraction layer, Dali, that can help you to process data seamlessly across Kafka and Hadoop. Beyond pure technology, Shirshanka and Yael outline the three components of a great data culture and ecosystem and explain how to create maintainable data contracts between data producers and data consumers (like data scientists and data analysts) and how to standardize data effectively in a growing organization to enable (and not slow down) innovation and agility. They then look to the future, envisioning a world where you can successfully deploy a data abstraction of views on Hadoop data, like a data API as a protective and enabling shield. Along the way, Shirshanka and Yael discuss observations on how to enable teams to be good data citizens in producing, consuming, and owning datasets and offer an overview of LinkedIn’s governance model: the tools, process and teams that ensure that its data ecosystem can handle change and sustain #datasciencehappiness.

Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Yael Garten

2017 StrataHadoop SJC conference talk. https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/56047 Description: So, you finally have a data ecosystem with Kafka and Hadoop both deployed and operating correctly at scale. Congratulations. Are you done? Far from it. As the birthplace of Kafka and an early adopter of Hadoop, LinkedIn has 13 years of combined experience using Kafka and Hadoop at scale to run a data-driven company. Both Kafka and Hadoop are flexible, scalable infrastructure pieces, but using these technologies without a clear idea of what the higher-level data ecosystem should be is perilous. Shirshanka Das and Yael Garten share best practices around data models and formats, choosing the right level of granularity of Kafka topics and Hadoop tables, and moving data efficiently and correctly between Kafka and Hadoop and explore a data abstraction layer, Dali, that can help you to process data seamlessly across Kafka and Hadoop. Beyond pure technology, Shirshanka and Yael outline the three components of a great data culture and ecosystem and explain how to create maintainable data contracts between data producers and data consumers (like data scientists and data analysts) and how to standardize data effectively in a growing organization to enable (and not slow down) innovation and agility. They then look to the future, envisioning a world where you can successfully deploy a data abstraction of views on Hadoop data, like a data API as a protective and enabling shield. Along the way, Shirshanka and Yael discuss observations on how to enable teams to be good data citizens in producing, consuming, and owning datasets and offer an overview of LinkedIn’s governance model: the tools, process and teams that ensure that its data ecosystem can handle change and sustain #DataScienceHappiness.

The Enterprise Guide to Building a Data Mesh - Introducing SpecMeshIanFurlong4

For organisations to successfully adopt data mesh, setting up and maintaining infrastructure needs to be easy. We believe the best way to achieve this is to leverage the learnings from building a ‘central nervous system‘, commonly used in modern data-streaming ecosystems. This approach formalises and automates of the manual parts of building a data mesh. This presentation introduces SpecMesh; a methodology and supporting developer toolkit to enable business to build the foundations of their data mesh.

Sparkling Water Webinar October 29th, 2014Sri Ambati

Sparkling Water is the newest application on the Apache Spark in-memory platform to extend Machine Learning for better predictions and to quickly deploy models into production. H2O is proud to partner with Cloudera and Databricks to bring this capability to a wide audience. H2O is for data scientists and business analysts who need scalable and fast machine learning. H2O is an open source predictive analytics platform. Unlike traditional analytics tools, H2O provides a combination of extraordinary math and high performance parallel processing with unrivaled ease of use. H2O speaks the language of data science with support for R, Python, Scala, Java and a robust REST API. Smart business applications are powered by H2O’s NanoFast¬TM Scoring Engine. Learn more by going to http://www.h2o.ai and contact us for more information. - Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai - To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata

Advanced Analytics and Machine Learning with Data VirtualizationDenodo

Watch: https://bit.ly/2DYsUhD Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Attend this webinar and learn: - How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo - How you can use the Denodo Platform with large data volumes in an efficient way - How Prologis accelerated their use of Machine Learning with data virtualization

Interactive Analytics at Scale in Apache Hive Using DruidDataWorks Summit/Hadoop Summit

Druid is an open-source analytics data store specially designed to execute OLAP queries on event data. Its speed, scalability and efficiency have made it a popular choice to power user-facing analytic applications, including multiple BI tools and dashboards. However, Druid does not provide important features requested by many of these applications, such as a SQL interface or support for complex operations such as joins. This talk presents our work on extending Druid indexing and querying capabilities using Apache Hive. In particular, our solution allows to index complex query results in Druid using Hive, query Druid data sources from Hive using SQL, and execute complex Hive queries on top of Druid data sources. We describe how we built an extension that brings benefits to both systems alike, leveraging Apache Calcite to overcome the challenge of transparently generating Druid JSON queries from the input Hive SQL queries. We conclude with a demo highlighting the performant and powerful integration of these projects.

Bg linkedin bigdata_martinschultz_symposium_yale_oct2012Bhaskar Ghosh

Breaking down data silos with ODataWoodruff Solutions LLC

- OData (Open Data Protocol) is a standard protocol for building and consuming RESTful APIs in a simple and standard way. - It allows data from various sources to be unified and consumed using common HTTP methods like GET, PUT, POST, DELETE and common formats like JSON and AtomPub. - Major companies like Microsoft use OData to expose data sources like reporting services through standardized APIs that can then be consumed by various clients and applications in a consistent manner.

The LOD Gateway: Open Source Infrastructure for Linked DataDavid Newbury

The oecd delta project – providing easier access to data through api'sJonathan Challener

The document discusses the OECD's efforts to make its data more open, accessible and reusable through various standards and formats over time. It details the OECD's experience with SDMX from 2007-2012, its work to simplify SDMX and develop JSON-based formats from 2012 onward, and the convergence of proposed formats into a single Simplified SDMX JSON format. The OECD launched an API in early 2013 to provide data in the new proposed JSON standards ahead of a workshop on statistical data dissemination.

Microsoft Graph: Connect to essential data every app needsMicrosoft Tech Community

Big Data, Bigger BrainsDenny Lee

Interactive Analytics at Scale in Apache Hive Using DruidDataWorks Summit

Druid is an open-source analytics data store specially designed to execute OLAP queries on event data. Its speed, scalability and efficiency have made it a popular choice to power user-facing analytic applications, including multiple BI tools and dashboards. However, Druid does not provide important features requested by many of these applications, such as a SQL interface or support for complex operations such as joins. This talk presents our work on extending Druid indexing and querying capabilities using Apache Hive. In particular, our solution allows to index complex query results in Druid using Hive, query Druid data sources from Hive using SQL, and execute complex Hive queries on top of Druid data sources. We describe how we built an extension that brings benefits to both systems alike, leveraging Apache Calcite to overcome the challenge of transparently generating Druid JSON queries from the input Hive SQL queries. We conclude with an experimental evaluation highlighting the performant and powerful integration of these projects. Speaker Jesus Camancho Rodriquez, Hortonworks

Advanced Analytics and Machine Learning with Data VirtualizationDenodo

Watch full webinar here: https://bit.ly/32c6TnG Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Attend this webinar and learn: - How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo - How you can use the Denodo Platform with large data volumes in an efficient way - About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization

Graphs & Big Data - Philip Rathle and Andreas Kollegger @ Big Data Science Me...Neo4j

Opensocial Haifa Seminar - 2008.04.08Ari Leichtberg

Better integrations through open interfacesSteve Speicher

Steve Speicher presented on better integrations through open interfaces. He discussed how using open standards like OSLC and linked data allows tools to integrate using open protocols instead of tight coupling. This provides benefits like increased adoption rates, a focus on important aspects rather than integration details, and opportunities for innovation. Speicher also mentioned additional resources on the OSLC website and related projects.

Test trend analysis: Towards robust reliable and timely testsHugh McCamphill

All data accessible to all my organization - Presentation at OW2con'19, June...OW2

Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Shirshanka Das

Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Yael Garten

The Enterprise Guide to Building a Data Mesh - Introducing SpecMeshIanFurlong4

Sparkling Water Webinar October 29th, 2014Sri Ambati

Advanced Analytics and Machine Learning with Data VirtualizationDenodo

Interactive Analytics at Scale in Apache Hive Using DruidDataWorks Summit/Hadoop Summit

Bg linkedin bigdata_martinschultz_symposium_yale_oct2012Bhaskar Ghosh

Breaking down data silos with ODataWoodruff Solutions LLC

The LOD Gateway: Open Source Infrastructure for Linked DataDavid Newbury

The oecd delta project – providing easier access to data through api'sJonathan Challener

Microsoft Graph: Connect to essential data every app needsMicrosoft Tech Community

Big Data, Bigger BrainsDenny Lee

Interactive Analytics at Scale in Apache Hive Using DruidDataWorks Summit

Advanced Analytics and Machine Learning with Data VirtualizationDenodo

Graphs & Big Data - Philip Rathle and Andreas Kollegger @ Big Data Science Me...Neo4j

Opensocial Haifa Seminar - 2008.04.08Ari Leichtberg

Better integrations through open interfacesSteve Speicher

Test trend analysis: Towards robust reliable and timely testsHugh McCamphill

More from Amy W. Tang (6)

Espresso: LinkedIn's Distributed Data Serving Platform (Paper)Amy W. Tang

LinkedIn Graph PresentationAmy W. Tang

Data Infrastructure at LinkedInAmy W. Tang

This document provides an overview of LinkedIn's data infrastructure. It discusses LinkedIn's large user base and data needs for products like profiles, communications, and recommendations. It describes LinkedIn's data ecosystem with three paradigms for online, nearline and offline data. It then summarizes key parts of LinkedIn's data infrastructure, including Databus for change data capture, Voldemort for distributed key-value storage, Kafka for messaging, and Espresso for distributed data storage. Overall, the document outlines how LinkedIn builds scalable data solutions to power its products and services for its large user base.

Voldemort on Solid State DrivesAmy W. Tang

Untangling Cluster Management with HelixAmy W. Tang

All Aboard the DatabusAmy W. Tang

Espresso: LinkedIn's Distributed Data Serving Platform (Paper)Amy W. Tang

LinkedIn Graph PresentationAmy W. Tang

Data Infrastructure at LinkedInAmy W. Tang

Voldemort on Solid State DrivesAmy W. Tang

Untangling Cluster Management with HelixAmy W. Tang

All Aboard the DatabusAmy W. Tang

Recently uploaded (20)

Splunk Security Update | Public Sector Summit Germany 2025Splunk

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

AI and Data Privacy in 2025: Global TrendsInData Labs

In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy. This infographic contains: -AI and data privacy: Key findings -Statistics on AI data privacy in the today’s world -Tips on how to overcome data privacy challenges -Benefits of AI data security investments. Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

Splunk Security Update | Public Sector Summit Germany 2025Splunk

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

AI and Data Privacy in 2025: Global TrendsInData Labs

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

How analogue intelligence complements AIPaul Rowe

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn

2. HADOOP SUMMIT 2013 Network update stream

4. HADOOP SUMMIT 2013 People you may know

7. HADOOP SUMMIT 2013 Point-to-point pipelines

8. HADOOP SUMMIT 2013 LinkedIn’s user activity data pipeline (circa 2010)

9. HADOOP SUMMIT 2013 Point-to-point pipelines

11. HADOOP SUMMIT 2013 Central data pipeline

13. HADOOP SUMMIT 2013

16. HADOOP SUMMIT 2013 What is a commit log?

19. HADOOP SUMMIT 2013 Usage at LinkedIn  16 brokers in each cluster  28 billion messages/day  Peak rates – Writes: 460,000 messages/second – Reads: 2,300,000 messages/second  ~ 700 topics  40-50 live services consuming user-activity data  Many ad hoc consumers  Every production service is a producer (for metrics)  10k connections/colo LinkedIn Corporation ©2013 All Rights Reserved 19

22. HADOOP SUMMIT 2013 Standardize on Avro in data pipeline LinkedIn Corporation ©2013 All Rights Reserved 22 { "type": "record", "name": "URIValidationRequestEvent", "namespace": "com.linkedin.event.usv", "fields": [ { "name": "header", "type": { "type": "record", "name": ”TrackingEventHeader", "namespace": "com.linkedin.event", "fields": [ { "name": "memberId", "type": "int", "doc": "The member id of the user initiating the action" }, { "name": ”timeMs", "type": "long", "doc": "The time of the event" }, { "name": ”host", "type": "string", ... ...

24. HADOOP SUMMIT 2013 Hadoop data load (Camus)  Open sourced: – https://github.com/linkedin/camus  One job loads all events  ~10 minute ETA on average from producer to HDFS  Hive registration done automatically  Schema evolution handled transparently

27. HADOOP SUMMIT 2013 Audit Trail

28. HADOOP SUMMIT 2013 Kafka replication (0.8)  Intra-cluster replication feature – Facilitates high availability and durability  Beta release available https://dist.apache.org/repos/dist/release/kafka/  Rolled out in production at LinkedIn last week LinkedIn Corporation ©2013 All Rights Reserved 28

29. HADOOP SUMMIT 2013 Join us at our user-group meeting tonight @ LinkedIn! – Thursday, June 27, 7.30pm to 9.30pm – 2025 Stierlin Ct., Mountain View, CA – http://www.meetup.com/http-kafka-apache-org/events/125887332/ – Presentations (replication overview and use-case studies) from:  RichRelevance  Netflix  Square  LinkedIn LinkedIn Corporation ©2013 All Rights Reserved 29

Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn (20)

More from Amy W. Tang (6)

Recently uploaded (20)

Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn