Hw09 Next Steps For Hadoop

Oct 25, 20092 likes1,153 views

The document discusses next steps for the Hadoop project, focusing on adopting the Avro data format and RPC protocol. Key points include: 1) Avro would provide a more expressive, efficient, and dynamic data format for Hadoop that allows browsing and working with arbitrary data without code generation. 2) Avro includes capabilities for data storage, encoding, versioning, and RPC that could help Hadoop provide cross-language access and compatibility between versions. 3) The author plans to incorporate Avro starting with using it for Hadoop job history and then providing full MapReduce support, with the possibility of using it for RPC in future Hadoop versions.

Next Steps for Hadoop

Doug Cutting
Cloudera

Proviso
● Linus Torvalds:
● “Whatever they contribute.”
● diverse set of contributors
● central planning impossible

The Dream
● faster, more reliable, available
● of course
● spreadsheet-like interfaces
● provide non-programmers
● with powerful, interactive tools
● easier sharing
● of data & hardware resources

Requirements
● security
● facilitate sharing of resources
● stable cross-language APIs
● facilitate diverse tools & apps
● expressive, inter-operable data
● facilitates sharing of datasets
● facilitates dynamic analyses

Data Formats
● today in Hadoop:
● text
– pro: inter-operable
– con: not expressive, inefficient
● - Java Writable
– pro: expressive, efficient
– con: platform-specific, fragile

Protocol Buffers & Thrift
● expressive
● efficient (small & fast)
● but not very dynamic
● cannot browse arbitrary data
● no DESCRIBE or SHOW
● viewing a new dataset
– requires code generation & load
● writing a new dataset
– requires generating schema text
– plus code generation & load

Avro Data
● as expressive
● smaller and faster
● dynamic
● schema stored with data
– but factored out of instances
● API permits reading & creating
– arbitrary datatypes
– without generating & loading code

Avro Data
● includes a file format
● includes a textual encoding
● handles versioning
● if schema changes
● can still process data
● Hadoop apps can
● upgrade from text
● and standardize on Avro for data

Avro RPC
● leverage versioning support
● to permit different versions of services to
interoperate
● for Hadoop services, will
● provide cross-language access
● let apps talk to clusters running different versions

Avro Status
● 1.1 release out
● added JSON and comparators
● 1.2 soon
● adds HTTP & UDP-based RPC
● will first appear in Hadoop 0.21
● as format for job history
● in sequence files

Avro Near Future
● full mapreduce support
● used for RPC in Hadoop 0.22 (1.0)?

This document discusses strategies for building interactive streaming applications in Spark Streaming. It describes using Zookeeper as a dynamic configuration source to allow modifying a Spark Streaming application's behavior at runtime. The key points are: - Zookeeper can be used to track configuration changes and trigger Spark Streaming context restarts through its watch mechanism and Curator library. - This allows building interactive applications that can adapt to configuration updates without needing to restart the whole streaming job. - Examples are provided of using Curator caches like node and path caches to monitor Zookeeper for changes and restart Spark Streaming contexts in response.

Build real time stream processing applications using Apache KafkaHotstar

This talk was presented at the Hotstar Scale Meetup in Bangalore by Jayesh Sidhwani In this talk, the presenter introduces Apache Kafka and the Apache Kafka Streams library. Starting from the need for building streaming applications to thinking the use-cases as a streaming job - this talk covers all the technicalities. It ends with a short description of how Kafka is deployed and used at Hotstar

Evolution of apache sparkdatamantra

1) Spark 1.0 was released in 2014 as the first production-ready version containing Spark batch, streaming, Shark, and machine learning libraries. 2) By 2014, most big data processing used higher-level tools like Hive and Pig on structured data rather than the original MapReduce assumption of only unstructured data. 3) Spark evolved to support structured data through the DataFrame API in versions 1.2-1.3, providing a unified way to read from structured sources.

Structured Streaming with Kafkadatamantra

This document provides an overview of structured streaming with Kafka in Spark. It discusses data collection vs ingestion and why they are key. It also covers Kafka architecture and terminology. It describes how Spark integrates with Kafka for streaming data sources. It explains checkpointing in structured streaming and using Kafka as a sink. The document discusses delivery semantics and how Spark supports exactly-once semantics with certain output stores. Finally, it outlines new features in Kafka for exactly-once guarantees and the future of structured streaming.

Build intelligent, real-time applications using Machine LearningHotstar

Building end to end streaming application on Sparkdatamantra

This document discusses building a real-time streaming application on Spark to analyze sensor data. It describes collecting data from servers through Flume into Kafka and processing it using Spark Streaming to generate analytics stored in Cassandra. The stages involve using files, then Kafka, and finally Cassandra for input/output. Testing streaming applications and redesigning for testability is also covered.

Scaling ELK Stack - DevOpsDays SingaporeAngad Singh

This document summarizes a presentation about using the ELK stack to process logs at scale. It discusses using Logstash for log ingestion and filtering, Elasticsearch for indexing and searching logs, and Kibana for visualizing logs. The document provides details on using Logstash forwarders to ship logs to Logstash from application containers, scaling Logstash and Elasticsearch horizontally, hardware recommendations for Elasticsearch, and configuration techniques for optimizing Elasticsearch performance and reliability.

Building microservices with grpcSathiyaseelan Muthu kumar

Understanding transactional writes in datasource v2datamantra

This document discusses the new Transactional Writes in Datasource V2 API introduced in Spark 2.3. It outlines the shortcomings of the previous V1 write API, specifically the lack of transaction support. It then describes the anatomy of the new V2 write API, including interfaces like DataSourceWriter, DataWriterFactory, and DataWriter that provide transactional capabilities at the partition and job level. It also covers how the V2 API addresses partition awareness through preferred location hints to improve performance.

Automatic Scaling Iterative ComputationsGuozhang Wang

This document discusses iterative graph computations and limitations of MapReduce for such computations. It proposes GRACE, a graph processing framework that separates the vertex-centric computation logic from execution policies to allow both synchronous and asynchronous execution. As an example, it shows how belief propagation can be implemented in a vertex-centric manner and executed asynchronously using GRACE. This provides easier programming while enabling performance benefits of asynchronous execution.

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

This document describes a Parallel Streaming Transformation Loader (PSTL) that uses Kafka, Spark, and Vertica for real-time data ingestion and analytics. It summarizes the PSTL as follows: 1. The PSTL ingests streaming data from Kafka into Spark RDDs in parallel. 2. Spark is used to transform the data, including assigning IDs and hashing records to partitions. 3. The transformed data is written in parallel from the Spark partitions directly to Vertica for analytics and querying. 4. Vertica demonstrated impressive parallel copy performance of 2.42 billion rows in under 8 minutes using this approach.

Cassandra Summit 2015 - Building a multi-tenant API PaaS with DataStax Enterp...Restlet

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, VectorizedHostedbyConfluent

Enforcing format, changing schema, introducing privacy filters have always been a challenge with the classical Kafka-API. In this talk we'll cover how to extend existing applications with webassembly, allowing developers to change the shape of data at runtime, per application without creating additional topics. By leveraging WebAssembly, we can extend the capabilities of the Kafka-API beyond what it was initially imagined. Come and learn about the future of the Kafka-API

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.

Exploratory Data Analysis in Sparkdatamantra

This document discusses exploratory data analysis (EDA) techniques that can be performed on large datasets using Spark and notebooks. It covers generating a five number summary, detecting outliers, creating histograms, and visualizing EDA results. EDA is an interactive process for understanding data distributions and relationships before modeling. Spark enables interactive EDA on large datasets using notebooks for visualizations and Pandas for local analysis.

Bootstrap SaaS startup using Open Source Toolsbotsplash.com

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, ShopifyHostedbyConfluent

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.

CDC to the Max!Bronco Oostermeyer

Kafka Tiered Storage | Satish Duggana and Sriharsha Chintalapani, UberHostedbyConfluent

Kafka is a vital part of data infrastructure in many organizations. When the Kafka cluster grows and more data is stored in Kafka for a longer duration, several issues related to scalability, efficiency, and operations become important to address. Kafka cluster storage is typically scaled by adding more broker nodes to the cluster. But this also adds needless memory and CPUs to the cluster making overall storage cost less efficient compared to storing the older data in external storage. Tiered storage is introduced to extend Kafka's storage beyond the local storage available on the Kafka cluster by retaining the older data in cheaper stores, such as HDFS, S3, Azure or GCS with minimal impact on the internals of Kafka. We will talk about - How tiered storage addresses the above problems and also brings several other advantages. - High level architecture of tiered storage - Future work planned as part of tiered storage.

State management in Structured Streamingdatamantra

This document discusses state management in Apache Spark Structured Streaming. It begins by introducing Structured Streaming and differentiating between stateless and stateful stream processing. It then explains the need for state stores to manage intermediate data in stateful processing. It describes how state was managed inefficiently in old Spark Streaming using RDDs and snapshots, and how Structured Streaming improved on this with its decoupled, asynchronous, and incremental state persistence approach. The document outlines Apache Spark's implementation of storing state to HDFS and the involved code entities. It closes by discussing potential issues with this approach and how embedded stores like RocksDB may help address them in production stream processing systems.

Introduction to Structured streamingdatamantra

This document provides an introduction to Structured Streaming in Apache Spark. It discusses the evolution of stream processing, drawbacks of the DStream API, and advantages of Structured Streaming. Key points include: Structured Streaming models streams as infinite tables/datasets, allowing stream transformations to be expressed using SQL and Dataset APIs; it supports features like event time processing, state management, and checkpointing for fault tolerance; and it allows stream processing to be combined more easily with batch processing using the common Dataset abstraction. The document also provides examples of reading from and writing to different streaming sources and sinks using Structured Streaming.

Productionalizing a spark applicationdatamantra

1. The document discusses the process of productionalizing a financial analytics application built on Spark over multiple iterations. It started with data scientists using Python and data engineers porting code to Scala RDDs. They then moved to using DataFrames and deployed on EMR. 2. Issues with code quality and testing led to adding ScalaTest, PR reviews, and daily Jenkins builds. Architectural challenges were addressed by moving to Databricks Cloud which provided notebooks, jobs, and throwaway clusters. 3. Future work includes using Spark SQL windows and Dataset API for stronger typing and schema support. The iterations improved the code, testing, deployment, and use of latest Spark features.

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...confluent

Interactive Queries in Apache Kafka's Streams API allows users to query the local state of a Kafka Streams application without accessing an external database. It treats the Kafka Streams application as an embedded, lightweight database. The local state is fault-tolerant and can be sharded across tasks to scale horizontally. Users can discover other application instances and their state to perform queries on remote state if needed. Interactive Queries simplifies stateful stream processing by reducing moving parts compared to using an external database.

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

Docker for mac & local developer environment optimizationRadek Baczynski

Docker can be used to optimize a local development environment by providing the same environment as production. Issues with performance on Docker for Mac can be addressed through techniques like using delegated volume mounts, removing xdebug, and using a solution like mutagen that syncs files without mounted volumes for faster performance. Mutagen provides near native performance, easy setup and monitoring, and works with any dockerized application.

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...Redis Labs

The document summarizes a presentation about using Protocol Buffers and Redis together. It discusses how Protocol Buffers provide strict data types, versioning, and serialization/deserialization benefits. It then outlines Redis key patterns using namespaces, versions, data categories and identifiers. Examples are provided to show how Protocol Buffers messages can be stored in Redis using these key patterns, including storing connections data in sets, sorted sets and individual messages. Benefits discussed include structure, readability, testing and abstraction.

Case Study: Stream Processing on AWS using Kappa ArchitectureJoey Bolduc-Gilbert

Prashant_Agrawal_CVPrashant Agrawal

- Prashant Agrawal has over 5 years of experience as a Big Data Analyst with expertise in log analytics, search engine solutions, and ETL using tools like Spark, Elasticsearch, Logstash, and Kibana. - He has strong skills in distributed computing systems like Hadoop, Spark, and working with Hortonworks Data Platform clusters. - His projects include log analytics and visualization using ELK, data lake modules in Spark, Spark ETL, and developing a big data platform for predictive analysis of system logs.

Hw09 Counting And Clustering And Other Data TricksCloudera, Inc.

This document summarizes the early use of Hadoop at The New York Times to generate PDFs of archived newspaper articles and analyze web traffic data. It describes how over 100 Amazon EC2 instances were used with Hadoop to pre-generate over 11 million PDFs from 4.3TB of source data in under 24 hours for a total cost of $240. The document then discusses how the Times began using Hadoop to perform web analytics, counting page views and unique users, and merging this data with demographic and article metadata to better understand user traffic and behavior.

Hw09 Map Reduce Over Tahoe A Least Authority Encrypted Distributed Filesy...Cloudera, Inc.

This document discusses integrating the Tahoe distributed storage system with Hadoop's MapReduce framework to enable secure large-scale data analysis across organizational boundaries. Tahoe allows flexible access control and encrypts data across storage nodes, addressing security concerns. The author details how Tahoe works, how to configure Hadoop to use Tahoe storage, and deployment scenarios for large organizations, communities, and public clouds. Performance tests show Tahoe write speeds are slower but read-intensive jobs perform similarly to HDFS. Source code is available under open source licenses.

More Related Content

What's hot (20)

Understanding transactional writes in datasource v2datamantra

Automatic Scaling Iterative ComputationsGuozhang Wang

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

Cassandra Summit 2015 - Building a multi-tenant API PaaS with DataStax Enterp...Restlet

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, VectorizedHostedbyConfluent

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

Exploratory Data Analysis in Sparkdatamantra

Bootstrap SaaS startup using Open Source Toolsbotsplash.com

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, ShopifyHostedbyConfluent

CDC to the Max!Bronco Oostermeyer

Kafka Tiered Storage | Satish Duggana and Sriharsha Chintalapani, UberHostedbyConfluent

State management in Structured Streamingdatamantra

Introduction to Structured streamingdatamantra

Productionalizing a spark applicationdatamantra

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...confluent

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative

Docker for mac & local developer environment optimizationRadek Baczynski

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...Redis Labs

Case Study: Stream Processing on AWS using Kappa ArchitectureJoey Bolduc-Gilbert

Prashant_Agrawal_CVPrashant Agrawal

Understanding transactional writes in datasource v2datamantra

Automatic Scaling Iterative ComputationsGuozhang Wang

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

Cassandra Summit 2015 - Building a multi-tenant API PaaS with DataStax Enterp...Restlet

Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, VectorizedHostedbyConfluent

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

Exploratory Data Analysis in Sparkdatamantra

Bootstrap SaaS startup using Open Source Toolsbotsplash.com

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, ShopifyHostedbyConfluent

CDC to the Max!Bronco Oostermeyer

Kafka Tiered Storage | Satish Duggana and Sriharsha Chintalapani, UberHostedbyConfluent

State management in Structured Streamingdatamantra

Introduction to Structured streamingdatamantra

Productionalizing a spark applicationdatamantra

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...confluent

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative

Docker for mac & local developer environment optimizationRadek Baczynski

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...Redis Labs

Case Study: Stream Processing on AWS using Kappa ArchitectureJoey Bolduc-Gilbert

Prashant_Agrawal_CVPrashant Agrawal

Viewers also liked (7)

Hw09 Counting And Clustering And Other Data TricksCloudera, Inc.

Hw09 Map Reduce Over Tahoe A Least Authority Encrypted Distributed Filesy...Cloudera, Inc.

Hw09 Protein AlignmentCloudera, Inc.

1. Hadoop provides a scalable platform for storing and processing large datasets in a distributed manner at low cost, enabling computational applications that were previously impossible. 2. This talk explores how Hadoop can have a transformative impact on bioinformatics by enabling the comparison and analysis of very large DNA and protein sequences. 3. The document demonstrates implementations of common bioinformatics algorithms like sequence alignment on Hadoop, showing it can scale to analyze much larger datasets than on a single computer.

Hadoop Summit 2012 | HDFS High AvailabilityCloudera, Inc.

Strata + Hadoop World 2012: Apache HBase Features for the EnterpriseCloudera, Inc.

Apache HBase is a distributed data store that is in production today at many enterprises and sites serving large volumes of near-real-time random-accesses. As Apache HBase matures, the community has augmented the system with new features that many enterprise consider to be hard requirements. We will discuss how the upcoming HBase 0.96 release addresses many of these shortcomings by introducing new features that will help the administrator monitor and control access to the system, and new mechanisms to minimize downtime due to expected and unexpected outages.

Hadoop Lecture for Harvard's CS 264 -- October 19, 2009Cloudera, Inc.

Hw09 Sqoop Database Import For HadoopCloudera, Inc.

This document discusses Sqoop, a tool for importing structured data from databases into Hadoop. Sqoop automates the process of importing data from relational databases into HDFS and generating code to work with the imported data. It supports importing from popular databases into various data formats like text files or SequenceFiles. Sqoop also integrates with Hive, allowing SQL-like queries over imported data. The tool aims to simplify common ETL tasks and make database data accessible for MapReduce jobs and analytics on Hadoop.

Hw09 Counting And Clustering And Other Data TricksCloudera, Inc.

Hw09 Map Reduce Over Tahoe A Least Authority Encrypted Distributed Filesy...Cloudera, Inc.

Hw09 Protein AlignmentCloudera, Inc.

Hadoop Summit 2012 | HDFS High AvailabilityCloudera, Inc.

Strata + Hadoop World 2012: Apache HBase Features for the EnterpriseCloudera, Inc.

Hadoop Lecture for Harvard's CS 264 -- October 19, 2009Cloudera, Inc.

Hw09 Sqoop Database Import For HadoopCloudera, Inc.

Similar to Hw09 Next Steps For Hadoop (20)

ApacheCon09: AvroCloudera, Inc.

The document discusses the state of Hadoop in 2009 and goals for 2010 and beyond. It focuses on improving data formats by proposing the use of Apache Avro, which provides an expressive, compact, and dynamic data serialization system with built-in capabilities for data sharing, remote procedure calls, and schema evolution. The document outlines how Avro could address current issues with data formats in Hadoop and its potential inclusion in future Hadoop releases.

Chicago Data Summit: Keynote - Data Processing with Hadoop: Scalable and Cost...Cloudera, Inc.

Hadoop is a new paradigm for data processing that scales near linearly to petabytes of data. Commodity hardware running open source software provides unprecedented cost effectiveness. It is affordable to save large, raw datasets, unfiltered, in Hadoop's file system. Together with Hadoop's computational power, this facilitates operations such as ad hoc analysis and retroactive schema changes. An extensive open source tool-set is being built around these capabilities, making it easy to integrate Hadoop into many new application areas.

Introduction to Apache Sparkdatamantra

Apache Spark is a fast, general engine for large-scale data processing. It provides unified analytics engine for batch, interactive, and stream processing using an in-memory abstraction called resilient distributed datasets (RDDs). Spark's speed comes from its ability to run computations directly on data stored in cluster memory and optimize performance through caching. It also integrates well with other big data technologies like HDFS, Hive, and HBase. Many large companies are using Spark for its speed, ease of use, and support for multiple workloads and languages.

Savanna - Elastic Hadoop on OpenStackSergey Lukjanov

Savanna is an OpenStack component that allows elastic provisioning of Hadoop clusters in OpenStack. It has a 3 phase roadmap - phase 1 allows basic cluster provisioning which is complete, phase 2 will add advanced configuration and tool integration currently in progress, and phase 3 will enable analytics as a service with a job execution framework. Savanna uses an extensible plugin architecture to provision Hadoop VMs and configure the clusters, integrating with other OpenStack components like Nova, Glance, and Swift.

Hadoop Introductionsheetal sharma

Intro to Apache HadoopSufi Nawaz

Apache Hadoop is an open-source software framework that supports distributed applications and processing of large data sets across clusters of commodity hardware. It is highly scalable, fault-tolerant and allows processing of data in parallel. Hadoop consists of Hadoop Common, HDFS for storage, YARN for resource management and MapReduce for distributed processing. HDFS stores large files across clusters and provides high throughput access to application data. MapReduce allows distributed processing of large datasets across clusters using a simple programming model.

Plugging the Holes: Security and Compatability in HadoopOwen O'Malley

Hw09 Security And Api CompatibilityCloudera, Inc.

The document discusses security and compatibility issues in Hadoop. It outlines plans to add authentication using Kerberos to secure access to HDFS and MapReduce. It also discusses maintaining backwards compatibility for APIs, protocols, file formats and configurations when rolling out new versions of Hadoop to avoid requiring clients to change and recompile code. The goal is to enable placing sensitive customer and financial data on shared Hadoop clusters while ensuring only authorized users have access.

What is Apache Hadoop and its ecosystem?tommychauhan

Blackray @ SAPO CodeBits 2009fschupp

Change data captureRon Barabash

This document discusses change data capture (CDC) and its components. CDC is an approach that identifies, captures, and delivers changes made to enterprise data sources. It feeds these changes into a central data stream that can be combined with other data sources in real-time. The document outlines Kafka Connect, Debezium, Schema Registry, and Apache Avro which are key parts of the CDC architecture. It also discusses future steps like supporting additional databases and improving deployment, as well as open issues around performance and compatibility with certain databases.

Present and future of unified, portable, and efficient data processing with A...DataWorks Summit

The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere." This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem. Speaker Davor Bonaci, Apache Software Foundation; Simbly, V.P. of Apache Beam; Founder/CEO at Operiant

Getting started big dataKibrom Gebrehiwot

Streamsets and spark at SF Hadoop User GroupHari Shreedharan

BIGDATA pptsKrisshhna Daasaarii

Business intelligence analyzes data to provide actionable information for decision making. Big data is a $50 billion market by 2017, referring to technologies that capture, store, manage and analyze large variable data collections. Hadoop is an open source framework for distributed storage and processing of large data sets on commodity hardware, enabling businesses to gain insight from massive amounts of structured and unstructured data. It involves components like HDFS for data storage, MapReduce for processing, and others for accessing, storing, integrating, and managing data.

Go at uberRob Skillington

This document discusses Uber's transition from a monolithic architecture to a microservices architecture and the adoption of Go as a primary programming language. It provides examples of some key Go services at Uber including Geofences, an early service, and Geobase, a more recent service. It also discusses Uber's development of open source Go libraries and tools like Ringpop, TChannel, go-torch, and others to help establish Go as a first-class language at Uber.

Cloud Native API Design and ManagementAllBits BVBA (freelancer)

Big Data Hoopla Simplified - TDWI Memphis 2014Rajan Kanitkar

The document provides an overview and quick reference guide to big data concepts including Hadoop, MapReduce, HDFS, YARN, Spark, Storm, Hive, Pig, HBase and NoSQL databases. It discusses the evolution of Hadoop from versions 1 to 2, and new frameworks like Tez and YARN that allow different types of processing beyond MapReduce. The document also summarizes common big data challenges around skills, integration and analytics.

Apache frameworks for Big and Fast DataNaveen Korakoppa

Apache frameworks provide solutions for processing big and fast data. Traditional APIs use a request/response model with pull-based interactions, while modern data streaming uses a publish/subscribe model. Key concepts for big data architectures include batch processing frameworks like Hadoop, stream processing tools like Storm, and hybrid options like Spark and Flink. Popular data ingestion tools include Kafka for messaging, Flume for log data, and Sqoop for structured data. The best solution depends on requirements like latency, data volume, and workload type.

Apache Tez -- A modern processing enginebigdatagurus_meetup

The document discusses Apache Tez, a framework for accelerating Hadoop query processing. Some key points: - Tez is a dataflow framework that expresses computations as directed acyclic graphs (DAGs) of tasks. This allows optimizations like container reuse and locality-aware scheduling. - It is built on YARN and provides a customizable execution engine as well as runtime and DAG APIs for applications to define computations. - Compared to MapReduce, Tez can provide better performance, predictability, and resource utilization through its DAG execution model and optimizations like reducing intermediate data writes. - It has been used to improve performance for workloads like Hive, Pig, and large TPC-DS queries

ApacheCon09: AvroCloudera, Inc.

Chicago Data Summit: Keynote - Data Processing with Hadoop: Scalable and Cost...Cloudera, Inc.

Introduction to Apache Sparkdatamantra

Savanna - Elastic Hadoop on OpenStackSergey Lukjanov

Hadoop Introductionsheetal sharma

Intro to Apache HadoopSufi Nawaz

Plugging the Holes: Security and Compatability in HadoopOwen O'Malley

Hw09 Security And Api CompatibilityCloudera, Inc.

What is Apache Hadoop and its ecosystem?tommychauhan

Blackray @ SAPO CodeBits 2009fschupp

Change data captureRon Barabash

Present and future of unified, portable, and efficient data processing with A...DataWorks Summit

Getting started big dataKibrom Gebrehiwot

Streamsets and spark at SF Hadoop User GroupHari Shreedharan

BIGDATA pptsKrisshhna Daasaarii

Go at uberRob Skillington

Cloud Native API Design and ManagementAllBits BVBA (freelancer)

Big Data Hoopla Simplified - TDWI Memphis 2014Rajan Kanitkar

Apache frameworks for Big and Fast DataNaveen Korakoppa

Apache Tez -- A modern processing enginebigdatagurus_meetup

More from Cloudera, Inc. (20)

Partner Briefing_January 25 (FINAL).pptxCloudera, Inc.

The document discusses using Cloudera DataFlow to address challenges with collecting, processing, and analyzing log data across many systems and devices. It provides an example use case of logging modernization to reduce costs and enable security solutions by filtering noise from logs. The presentation shows how DataFlow can extract relevant events from large volumes of raw log data and normalize the data to make security threats and anomalies easier to detect across many machines.

Cloudera Data Impact Awards 2021 - Finalists Cloudera, Inc.

The document outlines the 2021 finalists for the annual Data Impact Awards program, which recognizes organizations using Cloudera's platform and the impactful applications they have developed. It provides details on the challenges, solutions, and outcomes for each finalist project in the categories of Data Lifecycle Connection, Cloud Innovation, Data for Enterprise AI, Security & Governance Leadership, Industry Transformation, People First, and Data for Good. There are multiple finalists highlighted in each category demonstrating innovative uses of data and analytics.

2020 Cloudera Data Impact Awards FinalistsCloudera, Inc.

Cloudera is proud to present the 2020 Data Impact Awards Finalists. This annual program recognizes organizations running the Cloudera platform for the applications they've built and the impact their data projects have on their organizations, their industries, and the world. Nominations were evaluated by a panel of independent thought-leaders and expert industry analysts, who then selected the finalists and winners. Winners exemplify the most-cutting edge data projects and represent innovation and leadership in their respective industries.

Edc event vienna presentation 1 oct 2019Cloudera, Inc.

The document outlines the agenda for Cloudera's Enterprise Data Cloud event in Vienna. It includes welcome remarks, keynotes on Cloudera's vision and customer success stories. There will be presentations on the new Cloudera Data Platform and customer case studies, followed by closing remarks. The schedule includes sessions on Cloudera's approach to data warehousing, machine learning, streaming and multi-cloud capabilities.

Machine Learning with Limited Labeled Data 4/3/19Cloudera, Inc.

Data Driven With the Cloudera Modern Data Warehouse 3.19.19Cloudera, Inc.

Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.

Introducing Cloudera Data Science Workbench for HDP 2.12.19Cloudera, Inc.

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Cloudera, Inc.

Leveraging the cloud for analytics and machine learning 1.29.19Cloudera, Inc.

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Cloudera, Inc.

Leveraging the Cloud for Big Data Analytics 12.11.18Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 3Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 2Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 1Cloudera, Inc.

The document discusses the benefits and trends of modernizing a data warehouse. It outlines how a modern data warehouse can provide deeper business insights at extreme speed and scale while controlling resources and costs. Examples are provided of companies that have improved fraud detection, customer retention, and machine performance by implementing a modern data warehouse that can handle large volumes and varieties of data from many sources.

Extending Cloudera SDX beyond the PlatformCloudera, Inc.

Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.

Analyst Webinar: Doing a 180 on Customer 360Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.

Introducing the data science sandbox as a service 8.30.18Cloudera, Inc.

Partner Briefing_January 25 (FINAL).pptxCloudera, Inc.

Cloudera Data Impact Awards 2021 - Finalists Cloudera, Inc.

2020 Cloudera Data Impact Awards FinalistsCloudera, Inc.

Edc event vienna presentation 1 oct 2019Cloudera, Inc.

Machine Learning with Limited Labeled Data 4/3/19Cloudera, Inc.

Data Driven With the Cloudera Modern Data Warehouse 3.19.19Cloudera, Inc.

Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.

Introducing Cloudera Data Science Workbench for HDP 2.12.19Cloudera, Inc.

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Cloudera, Inc.

Leveraging the cloud for analytics and machine learning 1.29.19Cloudera, Inc.

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Cloudera, Inc.

Leveraging the Cloud for Big Data Analytics 12.11.18Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 3Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 2Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 1Cloudera, Inc.

Extending Cloudera SDX beyond the PlatformCloudera, Inc.

Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.

Analyst Webinar: Doing a 180 on Customer 360Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.

Introducing the data science sandbox as a service 8.30.18Cloudera, Inc.

Recently uploaded (20)

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Cyber Awareness overview for 2025 month of securityriccardosl1

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Build Your Own Copilot & Agents For DevsBrian McKeiver

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

How analogue intelligence complements AIPaul Rowe

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Cyber Awareness overview for 2025 month of securityriccardosl1

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

How Can I use the AI Hype in my Business Context?Daniel Lehner

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Build Your Own Copilot & Agents For DevsBrian McKeiver

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Hw09 Next Steps For Hadoop

1. Next Steps for Hadoop Doug Cutting Cloudera

2. Proviso ● Linus Torvalds: ● “Whatever they contribute.” ● diverse set of contributors ● central planning impossible

3. The Dream ● faster, more reliable, available ● of course ● spreadsheet-like interfaces ● provide non-programmers ● with powerful, interactive tools ● easier sharing ● of data & hardware resources

4. Requirements ● security ● facilitate sharing of resources ● stable cross-language APIs ● facilitate diverse tools & apps ● expressive, inter-operable data ● facilitates sharing of datasets ● facilitates dynamic analyses

5. Data Formats ● today in Hadoop: ● text – pro: inter-operable – con: not expressive, inefficient ● - Java Writable – pro: expressive, efficient – con: platform-specific, fragile

6. Protocol Buffers & Thrift ● expressive ● efficient (small & fast) ● but not very dynamic ● cannot browse arbitrary data ● no DESCRIBE or SHOW ● viewing a new dataset – requires code generation & load ● writing a new dataset – requires generating schema text – plus code generation & load

7. Avro Data ● as expressive ● smaller and faster ● dynamic ● schema stored with data – but factored out of instances ● API permits reading & creating – arbitrary datatypes – without generating & loading code

8. Avro Data ● includes a file format ● includes a textual encoding ● handles versioning ● if schema changes ● can still process data ● Hadoop apps can ● upgrade from text ● and standardize on Avro for data

9. Avro RPC ● leverage versioning support ● to permit different versions of services to interoperate ● for Hadoop services, will ● provide cross-language access ● let apps talk to clusters running different versions

10. Avro Status ● 1.1 release out ● added JSON and comparators ● 1.2 soon ● adds HTTP & UDP-based RPC ● will first appear in Hadoop 0.21 ● as format for job history ● in sequence files

11. Avro Near Future ● full mapreduce support ● used for RPC in Hadoop 0.22 (1.0)?

12. Thanks! What are your next steps?

Hw09 Next Steps For Hadoop

Recommended

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Hw09 Next Steps For Hadoop (20)

More from Cloudera, Inc. (20)

Recently uploaded (20)

Hw09 Next Steps For Hadoop