Big Data Pipeline for Analytics at Scale @ FIT CVUT 2014

Nov 21, 20145 likes3,020 views

The recent boom in big data processing and democratization of the big data space has been enabled by the fact that most of the concepts originated in the research labs of companies such as Google, Amazon, Yahoo and Facebook are now available as open source. Technologies such as Hadoop, Cassandra let businesses around the world to become more data driven and tap into their massive data feeds to mine valuable insights. At the same time, we are still at a certain stage of the maturity curve of these new big data technologies and of the entire big data technology stack. Many of the technologies originated from a particular use case and attempts to apply them in a more generic fashion are hitting the limits of their technological foundations. In some areas, there are several competing technologies for the same set of use cases, which increases risks and costs of big data implementations. We will show how GoodData solves the entire big data pipeline today, starting from raw data feeds all the way up to actionable business insights. All this provided as a hosted multi-tenant environment letting its customers to solve their particular analytical use case or many analytical use cases for thousands of their customers all using the same platform and tools while abstracting them away from the technological details of the big data stack.

GoodData – the Case Study #2:
Big Data Pipeline for Analytics at Scale
DB Technologies for Big Data @ FIT CVUT
2014 GoodData Corporation. All Rights Reserved.
November 19 2014

GoodData Corporation. All Rights Reserved.
GoodData Corporation

GoodData Corporation. All Rights Reserved.
End to End, Analytics
Platform as a Service
Traditional BI
Data Visualization
Tableau, Qlikview, Spotfire, etc.
Analytics Engine
Cognos, Oracle, Business Objects, etc.
Data Marts
MySQL, PostgreSQL, etc.
Data Warehouse
Oracle, Teradata, Netezza, Microsoft, etc.
ETL
Informatica, DataStage, Boomi, Snaplogic, etc.
Infrastructure
Servers, Storage, Networking, etc.
Data Collaboration
Data Visualization
Analytics Engine
Data Marts
Data Warehouse
ELT / ETL
Infrastructure

One Platform. Two Markets.
For Your Customers
Powered By GoodData Partner Program
for disruptive ISVs including Zendesk,
Switchfly, and Phizzle
GoodData Corporation. All Rights Reserved.
For Your Business
Drive your business with your data.
Experience and accelerators for
Social, Sales, Marketing, Yammer

GoodData Corporation. All Rights Reserved.
Our Focus

GoodData Corporation. All Rights Reserved.
What The End Users See...

GoodData Corporation. All Rights Reserved.
What Is In The Box...

GoodData Platform Zoom-In
End to End, Analytics
Platform as a Service
Data Collaboration
Data Visualization
Analytics Engine
Data Marts
Data Warehouse
ELT / ETL
Infrastructure

GoodData Analytics Platform - The Data Pipeline

GoodData Corporation. All Rights Reserved.
Let’s Start With The Outcome - The Insights

GoodData Corporation. All Rights Reserved.
Let’s Start With The Outcome - The Insights
• User Experience
○ Visual Appeal
○ Ease of Use
○ Performance
• Analytical Power
• Many Data Sources
○ Need to cross analyze all of them
○ Need to add/remove sources as needed
• Cost Efficiency
○ Computational density allowed by multi-tenancy

GoodData Corporation. All Rights Reserved.
Let’s Start With The Outcome - The Insights
● Analytical Engine / MAQL
● Exploration, Visualization
and Distribution Layer
● Pluggable Database
Backends
● 10s of GB up to TBs

GoodData Corporation. All Rights Reserved.
Behind The Scenes - The Big Data Pipeline
• Large Data Throughput
○ Close to Real-time Updates
• Many Data Sources
○ Need to cross analyze all of them
○ Need to add/remove sources as needed
• Agility
○ Capture all data without knowing the analytical use case in advance
• Cost Efficiency
○ Computational density allowed by multi-tenancy

GoodData Corporation. All Rights Reserved.
Behind The Scenes - The Big Data Pipeline
• Big Data Store
○ 100s of TBs per customer
○ Persist All Incoming Data
○ CSV, XML, JSON, ...
• Immutable
○ Append Only
○ Keep Ingestion History
• Technologies
○ Amazon S3
○ Cloud Files

GoodData Corporation. All Rights Reserved.
Behind The Scenes - The Big Data Pipeline
• Agile Data Warehouse
○ 10s of TBs per customer
○ Relational Model
○ Semi-Cleansed
○ Complete History Captured
• Technologies
○ HP Vertica
○ GoodData BI Integration Services

GoodData Corporation. All Rights Reserved.
Behind The Scenes - The Big Data Pipeline
• Combine Input Stage Data Sets
○ Mapping, Cleansing
• Perform Data Transformations in Data Warehouse
○ Benchmarking, Snapshotting, Sampling
• Generate Data Mart Input Data
○ Data Warehouse : Data Mart relation is typically 1 : N
○ 10s of thousands Data Marts in PbG (OEM) use case!

GoodData Corporation. All Rights Reserved.
Behind The Scenes - The Big Data Pipeline
• GoodData BI Integration Services
○ CloudConnect Runtime
○ Ruby Runtime
○ Data Integration Console
Over 2M ETL jobs per week!

GoodData Corporation. All Rights Reserved.
The Wrap-Up - The Big Data Pipeline
Progression Through:
• Big Data Store
• Data Warehouse
• Data Marts
As a means to satisfy the end user:
• User Experience
• Analytical Power
• Many Data Sources
• Cost Efficiency

GoodData Corporation. All Rights Reserved.
Questions?

GoodData Corporation. All Rights Reserved.
Thank you!

Abstract: Cloud computing is accelerating the pace of migration from on-premise behind-the-firewall software deployment to Software as a Service (SaaS) subscription based delivery model. While accelerated innovation cycle and reduced time to value enabled by the above paradigm shift are well recognized due to their direct business impact, it is often overlooked how the new delivery model is affecting software engineering itself in terms of tools, processes as well as of its theoretical arsenal needed to develop cloud applications. Updated GoodData platform statistics as of Q2 2013!

GoodData: The DevOps Story @ FIT CVUT October 16 2013Jaroslav Gergic

GoodData case study at "Nápad roku 2013" - "Jak vybudovat úspěšný globální st...Jaroslav Gergic

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...Databricks

Zalando transitioned from a centralized data platform to a data mesh architecture. This decentralized their data infrastructure by having individual domains own datasets and pipelines rather than a central team. It provided self-service data infrastructure tools and governance to enable domains to operate independently while maintaining global interoperability. This improved data quality by making domains responsible for their data and empowering them through the data mesh approach.

End to End Supply Chain Control TowerDatabricks

How a Media Data Platform Drives Real-time Insights & Analytics using Apache ...Databricks

Roularta is a leading publishing company in Belgium. As digital news and channels move at a rapid pace and contain massive volumes of data, Roularta decided in 2019 to invest in a Spark-based data platform to drive true real-time website analytics and unlock insights on previously untouched (big) data sources. In this talk we’ll first explain why and how Roularta embarked from a classical data warehouse to a Spark-based Lakehouse using Delta. We’ll outline the series of publishing & marketing use-cases done in the last 12 months and highlight for each use-case the advantages of Spark and how the team further tuned performance to truly deliver insights with high velocity.

VoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big DataVoltDB

Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...Big Data Spain

How to Realize an Additional 270% ROI on SnowflakeAtScale

Companies of all sizes have embraced the power, scale and ease of use of Snowflake’s cloud data platform, along with the promise of cost-savings. But if you aren’t careful, cloud compute usage can sneak up on you and leave you with runaway costs no matter what BI tool you are using. The presentation from experts from Rakuten Rewards and AtScale shows practical techniques on how you can reduce unnecessary compute and boost BI performance to realize an additional 270% ROI on Snowflake. For the on-demand webinar, go to: https://www.atscale.com/resource/wbr-cloud-compute-cost-snowflake-tableau/

Cloud and Big Data trendsSebastien Goasguen

A look at clouds and big data trends and history. While Big Data arrived first on the scene -looking at google file system, hadoop, dynamo- Cloud was first in the hyper cycle. Google trends show this clearly. Amazon AWS however has already deployed analytics services on the their cloud while open source IaaS solutions are still struggling to deliver a EC2 clone. Cloud and Big data has three common points: 1-use an EC2 clone and a S3 clone (riakCS, glusterfs etc) to build a cloud 2-Use a big data solutions as a backend to your cloud to provide EBS or large scale image catalogue 3-deploy big data solutions on your cloud with tools like apache whirr, pallet, and newer devops tool chains with vagrant and co.

Next generation Polyglot Architectures using Neo4j by Stefan KolmarBig Data Spain

Google BigtableGirdhareeSaran

Bigtable is Google's proprietary large-scale database system that handles structured data. It uses Google File System, Chubby Lock Service, SSTable and other Google technologies. Bigtable allows storing terabytes or petabytes of time-series, financial, IoT, graph and other data. It is used prominently in Google Search, App Engine, Google Analytics, Google Earth and other applications. Quickstarts are provided to write and read data from Bigtable tables using the cbt tool or HBase shell. Key features include large storage capacity, integration with machine learning tools, high throughput, availability and scalability.

Memory Database Technology is Driving a New Cycle of Business InnovationVoltDB

In-memory database technology enables a new wave of fast data use cases that are extremely challenging and in some cases not possible with older technologies. In this webinar, Noel Yuhanna, Principal Analyst of Forrester Research, and VoltDB CMO, Peter Vescuso will discuss the latest market and data access technology trends, the new use cases these trends enable, and the implications for business and IT leaders.

Webinar: BI in the Sky - The New Rules of Cloud AnalyticsSnapLogic

In this webinar, we talk about the shift in data gravity as more and more business applications are moving to the cloud, and how the ability to deliver analytics in the cloud has evolved from idea to enterprise reality with new solutions being announced constantly that appeal to the need for speed, simplicity and access to insight on demand. Joining us in this webinar is David Glueck, Sr. Director of Data Science and Engineering at Bonobos. To learn more, visit: www.SnapLogic.com/salesforce-analytics

QCon 2018 | Gimel | PayPal's Analytic PlatformDeepak Chandramouli

Site | https://www.infoq.com/qconai2018/ Youtube | https://www.youtube.com/watch?v=2h0biIli2F4&t=19s At PayPal, data engineers, analysts and data scientists work with a variety of datasources (Messaging, NoSQL, RDBMS, Documents, TSDB), compute engines (Spark, Flink, Beam, Hive), languages (Scala, Python, SQL) and execution models (stream, batch, interactive). Due to this complex matrix of technologies and thousands of datasets, engineers spend considerable time learning about different data sources, formats, programming models, APIs, optimizations, etc. which impacts time-to-market (TTM). To solve this problem and to make product development more effective, PayPal Data Platform developed "Gimel", a unified analytics data platform which provides access to any storage through a single unified data API and SQL, that are powered by a centralized data catalog. In this session, we will introduce you to the various components of Gimel - Compute Platform, Data API, PCatalog, GSQL and Notebooks. We will provide a demo depicting how Gimel reduces TTM by helping our engineers write a single line of code to access any storage without knowing the complexity behind the scenes.

Unified Data Catalog - Recommendations powered by Apache Spark & Neo4jDeepak Chandramouli

This document discusses using recommendation systems in Unified Data Catalog (UDC) to help users discover relevant datasets. It outlines how recommendation engines have benefited Amazon and Netflix by generating personalized suggestions. The architecture uses Neo4J and Spark to build a graph of user, dataset and metadata relationships to power recommendations. Future plans include expanding the graph with additional data sources to improve recommendations and enable new use cases around privacy, compliance and data lineage.

The Big Data Ecosystem for Financial ServicesDataStax

1) The document discusses the use of big data and modern data architectures in the financial services industry. 2) It notes challenges such as aging IT systems, high costs, and lack of resources that are barriers to using big data. 3) Reference architectures are presented from IBM, Oracle, and DataStax that incorporate technologies like Hadoop, Spark, and Cassandra to power applications like customer 360, fraud detection, and real-time analytics.

Munich Re: Driving a Big Data TransformationDataWorks Summit

Data and analytics are at the heart of the digital transformation. Implementing a modern data platform can be challenging; moreover, success requires a shift in culture. Andreas will discuss the ways Munich Re drives cultural and technological change within their company, focusing on three key elements: people, processes, and technology. What does it mean to be a data-driven organization? How can we provide self-service analytics to our internal and external customers in an agile way? How do we get the most value out of our big data lake? How does Munich Re balance technology and culture to meet the data demands of their business? Speaker Andreas Kohlmaier, Head of Data Engineering, Munich Re

Migration and Coexistence between Relational and NoSQL Databases by Manuel H...Big Data Spain

Securing and governing a multi-tenant data lake within the financial industryDataWorks Summit

Standard Bank South Africa is a Hortonworks client, with several multi-node clusters hosting Hortonworks Data Platform (HDP) and Hortonworks Data Flow (HDF). This presentation will discuss the technical detail of implementing security, governance and multi-tenancy on a "Data Lake" within the finance industry. The talk will address the team's experiences, challenges, failures and learnings that we took away from this behemoth of an adventure. After introducing Standard Bank and the Hadoop admin team, the presentation will describe the security and governance journey Standard Bank has undergone since the project's inception in 2015, as well as the roadmap for the future ahead. Presentation structure: 1. Team introduction with background information 2. Environment overview (Where we are - Current) -----Security ---------Authentication through Kerberos and LDAP/ AD ---------Authorization through Ranger and Centrify ---------Transparent Data Encryption (TDE) at rest -----Governance ---------Centralized auditing ---------Ranger policies and data steward ownership -----Multi-Tenancy ---------Data lake Vs. data analytics platform ---------Edge nodes Vs. API framework through Knox 3. How did we get to this stage? (Past) -----Challenges faced (Kerberos, AD integration, SSL) -----How we overcame these challenges 4. Future challenges we foresee (Future) -----How we are planning to prepare for them Speakers Ian Pillay, Hadoop Administrator, Standard Bank Brad Smith, Hadoop Administrator, Standard Bank

Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo

Watch the presentation on-demand now: https://goo.gl/kceFTe Today’s digital economy demands a new way of running business. Flexible access to information and responses in real time are essential for outpacing competition. Watch this Denodo DataFest 2017 session to discover: • Data access challenges faced by organizations today. • How data virtualization facilitates real-time analytics. • Key use cases and customer success stories.

How Yellowbrick Data Integrates to Existing Environments WebcastYellowbrick Data

This document discusses how Yellowbrick can integrate into existing data environments. It describes Yellowbrick's data warehouse capabilities and how it compares to other solutions. The document recommends upgrading from single server databases or traditional MPP systems to Yellowbrick when data outgrows a single server or there are too many disparate systems. It also recommends moving from pre-configured or cloud-only systems to Yellowbrick to significantly reduce costs while improving query performance. The document concludes with a security demonstration using a netflow dataset.

Snaplogic Live: Big Data in MotionSnapLogic

Postgres Vision 2018: Your Migration Path - BinckBank Case StudyEDB

How to Build Fast Data Applications: Evaluating the Top ContendersVoltDB

Massive amounts of data generated from mobile devices, M2M communications, sensors and other IoT devices is redefining the world. What kind of applications will you build to take advantage of this data and provide value to your customers? What technologies are out there to help you? This deck will illustrate the difference between fast OLAP, stream-processing, and OLTP database solutions. You will also learn the importance of state, real-time analytics and real-time decisions when building applications on streaming data, and how streaming applications deliver more value when built on a super-fast in-memory, SQL database. To view the webinar in its entirety, click here: http://learn.voltdb.com/WRFastDataAppsTopContenders.html

VoltDB and Flytxt Present: Building a Single Technology Platform for Real-Tim...VoltDB

Webinar presented by Prateek Kapadia, CTO, Flytxt and Ryan Betts, CTO, VoltDB. Prateek and Ryan highlight the technology stack for Fast + Big data and iterative analytics using Hadoop and VoltDB. In addition, they will outline use cases for Big Data analytics with Hadoop, iterative analytics and machine learning using Spark and real-time analytics using VoltDB supplemented by Big Data + iterative analytics.

Data Warehouse Like a Tech Startup with Oracle Autonomous Data WarehouseRittman Analytics

“Tech startups can't afford DBAs, and they don't have time to provision servers and scale them up and down or deal with patches or downtime. They've never heard of indexes and they need data loaded and ready for analysis in days, not months. In this session learn how Oracle Database developers can build data warehouses as a hip startup data engineer would—but using a proper database built on Oracle technology. Oracle Data Visualization Desktop provides analytics and data exploration with techniques explained in this session. Hear real-world development experiences from working on data and analytics projects at a tech startup in the UK.”

Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...Denodo

This document discusses connected data and edge computing. It summarizes that connected devices, customers, vehicles, and assets are fueling new business models powered by streaming data, artificial intelligence, cloud computing, and the internet of things. It then describes Hortonworks' data platforms for managing both data at rest and in motion across cloud, on-premises and hybrid environments to enable analytics and power the modern data architecture.

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.

Building a Data Pipeline from Scratch - Joe CrobakHakka Labs

A data pipeline is a unified system for capturing events for analysis and building products. It involves capturing user events from various sources, storing them in a centralized data warehouse, and performing analysis and building products using tools like Hadoop. Key components of a data pipeline include an event framework, message bus, data serialization, data persistence, workflow management, and batch processing. A Lambda architecture allows for both batch and real-time processing of data captured by the pipeline.

More Related Content

What's hot (20)

How to Realize an Additional 270% ROI on SnowflakeAtScale

Cloud and Big Data trendsSebastien Goasguen

Next generation Polyglot Architectures using Neo4j by Stefan KolmarBig Data Spain

Google BigtableGirdhareeSaran

Memory Database Technology is Driving a New Cycle of Business InnovationVoltDB

Webinar: BI in the Sky - The New Rules of Cloud AnalyticsSnapLogic

QCon 2018 | Gimel | PayPal's Analytic PlatformDeepak Chandramouli

Unified Data Catalog - Recommendations powered by Apache Spark & Neo4jDeepak Chandramouli

The Big Data Ecosystem for Financial ServicesDataStax

Munich Re: Driving a Big Data TransformationDataWorks Summit

Migration and Coexistence between Relational and NoSQL Databases by Manuel H...Big Data Spain

Securing and governing a multi-tenant data lake within the financial industryDataWorks Summit

Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo

How Yellowbrick Data Integrates to Existing Environments WebcastYellowbrick Data

Snaplogic Live: Big Data in MotionSnapLogic

Postgres Vision 2018: Your Migration Path - BinckBank Case StudyEDB

How to Build Fast Data Applications: Evaluating the Top ContendersVoltDB

VoltDB and Flytxt Present: Building a Single Technology Platform for Real-Tim...VoltDB

Data Warehouse Like a Tech Startup with Oracle Autonomous Data WarehouseRittman Analytics

Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...Denodo

How to Realize an Additional 270% ROI on SnowflakeAtScale

Cloud and Big Data trendsSebastien Goasguen

Next generation Polyglot Architectures using Neo4j by Stefan KolmarBig Data Spain

Google BigtableGirdhareeSaran

Memory Database Technology is Driving a New Cycle of Business InnovationVoltDB

Webinar: BI in the Sky - The New Rules of Cloud AnalyticsSnapLogic

QCon 2018 | Gimel | PayPal's Analytic PlatformDeepak Chandramouli

Unified Data Catalog - Recommendations powered by Apache Spark & Neo4jDeepak Chandramouli

The Big Data Ecosystem for Financial ServicesDataStax

Munich Re: Driving a Big Data TransformationDataWorks Summit

Migration and Coexistence between Relational and NoSQL Databases by Manuel H...Big Data Spain

Securing and governing a multi-tenant data lake within the financial industryDataWorks Summit

Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo

How Yellowbrick Data Integrates to Existing Environments WebcastYellowbrick Data

Snaplogic Live: Big Data in MotionSnapLogic

Postgres Vision 2018: Your Migration Path - BinckBank Case StudyEDB

How to Build Fast Data Applications: Evaluating the Top ContendersVoltDB

VoltDB and Flytxt Present: Building a Single Technology Platform for Real-Tim...VoltDB

Data Warehouse Like a Tech Startup with Oracle Autonomous Data WarehouseRittman Analytics

Denodo DataFest 2017: Edge Computing: Collecting vs. Connecting to Streaming ...Denodo

Viewers also liked (16)

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

Building a Data Pipeline from Scratch - Joe CrobakHakka Labs

Big data prototyping in AWS cloudSamuel Yee

This document discusses using AWS cloud for rapid prototyping of big data minimum viable products (MVPs). Key benefits of cloud computing include no upfront costs, ability to scale quickly, and pay only for resources used. Typical big data use cases involve retrospective analysis, real-time processing, and machine learning. The document demonstrates building a basic Hadoop cluster on AWS in 15 minutes for word count analysis. It recommends measuring MVP outcomes and considering ongoing cloud use based on needs for 24/7 access, internal resources, and security requirements.

The Big Data Analytics Ecosystem at LinkedInrajappaiyer

LinkedIn has several data driven products that improve the experience of its users -- whether they are professionals or enterprises. Supporting this is a large ecosystem of systems and processes that provide data and insights in a timely manner to the products that are driven by it. This talk provides an overview of the various components of this ecosystem which are: - Hadoop - Teradata - Kafka - Databus - Camus - Lumos etc.

Application Lifecycle Transformation...a DevOps Discussion - By David Miller ...Melissa Luongo

This document discusses DevOps and its key practices for transforming an organization's application lifecycle. It provides the following key points: 1) DevOps addresses all aspects of the software delivery lifecycle by promoting continuous delivery, collaboration between development and operations, and incorporating customer feedback at every step. 2) There are 12 key practices that organizations leverage for DevOps transformation, including agile development, continuous integration, automated testing and deployment, and cross-functional teams. 3) A DevOps application delivery pipeline diagram shows the continuous flow from development to production with feedback at each stage.

Designing Data Pipelines Using HadoopDataWorks Summit

This presentation will cover the design principles and techniques used to build data pipelines taking into consideration the following aspects: architecture evolution, capacity, data quality, performance, flexibility and alignment with business objectives. The discussions will be based on the context of managing a pipeline with multi-petabyte data sets; a code-base composed of Java map/reduce jobs with HBase integration; Hive scripts and Kafka/Storm inputs. We?ll talk about how to make sure that data pipelines have the following features: 1) Assurance that the input data is ready at each step. 2) Workflows are easy to maintain. 3) Data quality and validation comes included in the architecture. Part of presentation will be dedicated to show how to organize the warehouse using layers of data sets. A suggested starting point for these layers are: 1) Raw Input (Logs, Messages, etc.), 2) Logical Input (Scrubbed data), 3) Foundational Warehouse Data (Most relevant joins), 4) Departmental/Project Data Sets and 5) Report Data Sets. (Used by Traditional Report engines) The final part will discuss the design of a rule-based system to perform validation and trending reporting.

Building Scalable Big Data PipelinesChristian Gügi

Real Time Data Processing using Spark Streaming | Data Day Texas 2015Cloudera, Inc.

Speaker: Hari Shreedharan Data Day Texas 2015 Apache Spark has emerged over the past year as the imminent successor to Hadoop MapReduce. Spark can process data in memory at very high speed, while still be able to spill to disk if required. Spark’s powerful, yet flexible API allows users to write complex applications very easily without worrying about the internal workings and how the data gets processed on the cluster. Spark comes with an extremely powerful Streaming API to process data as it is ingested. Spark Streaming integrates with popular data ingest systems like Apache Flume, Apache Kafka, Amazon Kinesis etc. allowing users to process data as it comes in. In this talk, Hari will discuss the basics of Spark Streaming, its API and its integration with Flume, Kafka and Kinesis. Hari will also discuss a real-world example of a Spark Streaming application, and how code can be shared between a Spark application and a Spark Streaming application. Each stage of the application execution will be presented, which can help understand practices while writing such an application. Hari will finally discuss how to write a custom application and a custom receiver to receive data from other systems.

Google Cloud and Data Pipeline PatternsLynn Langit

1. The document discusses various Google Cloud Platform products and patterns for data pipelines, including virtual machines, storage, data warehousing, streaming analytics, machine learning, internet of things, and bioinformatics. 2. Demos and examples are provided of storage, virtual machines, BigQuery, Cloud Spanner, and machine learning on the Google Cloud Platform. 3. The core Google Cloud Platform products discussed for various data and analytics use cases include Cloud Storage, BigQuery, Cloud Dataflow, Compute Engine, Cloud Pub/Sub, and Bigtable.

Spark+flume seattleHari Shreedharan

This document provides an overview of Flume and Spark Streaming. It describes how Flume is used to reliably ingest streaming data into Hadoop using an agent-based architecture. Events are collected by sources, stored reliably in channels, and sent to sinks. The Flume connector allows ingested data to be processed in real-time using Spark Streaming's micro-batch architecture, where streams of data are processed through RDD transformations. This combined Flume + Spark Streaming approach provides a scalable and fault-tolerant way to reliably ingest and process streaming data.

Data Engineering 101: Building your first data product by Jonathan Dinu PyDat...PyData

Often times there exists a divide between data teams, engineering, and product managers in organizations, but with the dawn of data driven companies/applications, it is more prescient now than ever to be able to automate your analyses to personalize your users experiences. LinkedIn's People you May Know, Netflix and Pandora's recommenders, and Amazon's eerily custom shopping experience have all shown us why it is essential to leverage data if you want to stay relevant as a company. As data analyses turn into products, it is essential that your tech/data stack be flexible enough to run models in production, integrate with web applications, and provide users with immediate and valuable feedback. I believe Python is becoming the lingua franca of data science due to its flexibility as a general purpose performant programming language, rich scientific ecosystem (numpy, scipy, scikit-learn, pandas, etc.), web frameworks/community, and utilities/libraries for handling data at scale. In this talk I will walk through a fictional company bringing it's first data product to market. Along the way I will cover Python and data science best practices for such a pipeline, cover some of the pitfalls of what happens when you put models into production, and how to make sure your users (and engineers) are as happy as they can be. https://github.com/Jay-Oh-eN/pydatasv2014

NoSQL databases, the CAP theorem, and the theory of relativityLars Marius Garshol

The document discusses NoSQL databases and the CAP theorem. It begins by providing an overview of NoSQL databases, their key features like being schemaless and supporting eventual consistency over ACID transactions. It then explains the CAP theorem - that a distributed system can only provide two of consistency, availability, and partition tolerance. It also discusses how Google's Spanner database achieves consistency and scalability using ideas from Lamport's Paxos algorithm and a new time service called TrueTime.

Implementing the Lambda Architecture efficiently with Apache SparkDataWorks Summit

This document discusses implementing the Lambda Architecture efficiently using Apache Spark. It provides an overview of the Lambda Architecture concept, which aims to provide low latency querying while supporting batch updates. The Lambda Architecture separates processing into batch and speed layers, with a serving layer that merges the results. Apache Spark is presented as an efficient way to implement the Lambda Architecture due to its unified processing engine, support for streaming and batch data, and ability to easily scale out. The document recommends resources for learning more about Spark and the Lambda Architecture.

Big Data pipeline with Scala by Rohit Rai, Tuplejump - presented at Pune Scal...Thoughtworks

At Tuplejump we have a built a big data platform powered by Scala everywhere. Using Akka for message/event processing, spark for streaming and batch processing, Shark for adhoc querying and Play to power our web based IDE. This talk will walk through what various components of the platform and how Scala, the concepts like Reactive programming, event driven architecture and ecosystem components like Akka actors framework, Spark RDDs and Play Web framework guided and inspired our decisions and also provided the kickstart to attempt this huge challenge of building a complete integrated end-to-end Big Data Application Framework.

Big Data and Fast Data - Lambda Architecture in ActionGuido Schmutz

Big Data (volume) and real-time information processing (velocity) are two important aspects of Big Data systems. At first sight, these two aspects seem to be incompatible. Are traditional software architectures still the right choice? Do we need new, revolutionary architectures to tackle the requirements of Big Data? This presentation discusses the idea of the so-called lambda architecture for Big Data, which acts on the assumption of a bisection of the data-processing: in a batch-phase a temporally bounded, large dataset is processed either through traditional ETL or MapReduce. In parallel, a real-time, online processing is constantly calculating the values of the new data coming in during the batch phase. The combination of the two results, batch and online processing is giving the constantly up-to-date view. This talk presents how such an architecture can be implemented using Oracle products such as Oracle NoSQL, Hadoop and Oracle Event Processing as well as some selected products from the Open Source Software community. While this session mostly focuses on the software architecture of BigData and FastData systems, some lessons learned in the implementation of such a system are presented as well.

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...Helena Edelson

Regardless of the meaning we are searching for over our vast amounts of data, whether we are in science, finance, technology, energy, health care…, we all share the same problems that must be solved: How do we achieve that? What technologies best support the requirements? This talk is about how to leverage fast access to historical data with real time streaming data for predictive modeling for lambda architecture with Spark Streaming, Kafka, Cassandra, Akka and Scala. Efficient Stream Computation, Composable Data Pipelines, Data Locality, Cassandra data model and low latency, Kafka producers and HTTP endpoints as akka actors...

Netflix Data Pipeline With KafkaAllen (Xiaozhong) Wang

Building a Data Pipeline from Scratch - Joe CrobakHakka Labs

Big data prototyping in AWS cloudSamuel Yee

The Big Data Analytics Ecosystem at LinkedInrajappaiyer

Application Lifecycle Transformation...a DevOps Discussion - By David Miller ...Melissa Luongo

Designing Data Pipelines Using HadoopDataWorks Summit

Building Scalable Big Data PipelinesChristian Gügi

Real Time Data Processing using Spark Streaming | Data Day Texas 2015Cloudera, Inc.

Google Cloud and Data Pipeline PatternsLynn Langit

Spark+flume seattleHari Shreedharan

Data Engineering 101: Building your first data product by Jonathan Dinu PyDat...PyData

NoSQL databases, the CAP theorem, and the theory of relativityLars Marius Garshol

Implementing the Lambda Architecture efficiently with Apache SparkDataWorks Summit

Big Data pipeline with Scala by Rohit Rai, Tuplejump - presented at Pune Scal...Thoughtworks

Big Data and Fast Data - Lambda Architecture in ActionGuido Schmutz

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...Helena Edelson

Similar to Big Data Pipeline for Analytics at Scale @ FIT CVUT 2014 (20)

Data Platform in the CloudAmihay Zer-Kavod

This document discusses building a data platform in the cloud. It covers the evolution of data platforms from monolithic architectures to distributed event-driven architectures using a data lake. Key aspects of a cloud data platform include collecting and persisting all data in a data lake for standardized access, near real-time processing using streaming technologies, and building the platform using either fully managed or DIY/hybrid approaches on AWS. Design principles focus on event-driven separation of data producers and consumers and choosing the right technology for the problem.

Self-Service Analytics with Guard RailsDenodo

Watch this webinar in full here: https://buff.ly/2MVTKqL Self-Service BI promises to remove the bottleneck that exists between IT and business users. The truth is, if data is handed over to a wide range of data consumers without proper guardrails in place, it can result in data anarchy. Attend this session to learn why data virtualization: • Is a must for implementing the right self-service BI • Makes self-service BI useful for every business user • Accelerates any self-service BI initiative

Modernizing Global Shared Data Analytics Platform and our Alluxio JourneyAlluxio, Inc.

Building a Single Logical Data Lake: For Advanced Analytics, Data Science, an...Denodo

Achieving Business Value by Fusing Hadoop and Corporate DataInside Analysis

The Briefing Room with Richard Hackathorn and Teradata Live Webcast March 25, 2015 Watch the Archive: https://bloorgroup.webex.com/bloorgroup/onstage/g.php?MTID=e7254708146d056339a0974f097f569b2 Hadoop data lakes are emerging as peers to corporate data warehouses. However, successful analytic solutions require a fusion of all relevant data, big and small, which has proven challenging for many companies. By allowing business analysts to quickly access data wherever it rests, success factors shift to focus on three key aspects: 1) business objectives, 2) organizational workflow, and 3) data placement. Register for this Special Edition of The Briefing Room to hear veteran Analyst Richard Hackathorn as he provides details from his recent research report focused on success stories using Teradata QueryGrid. Examples of use cases described will include: Joining sensor data in Hadoop with data warehouse labor schedules in seconds How bridging corporate cultures and systems creates new business opportunities The 360 view of customer journeys using weblogs in Hadoop via BI tools How can you put the data where you want and query it however you want Virtualizing Hadoop data with Teradata QueryGrid Visit InsideAnalysis.com for more information.

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)Denodo

Watch full webinar here: https://bit.ly/34iCruM Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Data Platform Architecture Principles and Evaluation CriteriaScyllaDB

Open Data Inside - Why Internal Data Portals are Key to Successful Data Gover...Joel Natividad

The Maturity Model: Taking the Growing Pains Out of HadoopInside Analysis

The Briefing Room with Rick van der Lans and Think Big, a Teradata Company Live Webcast on June 16, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=197f8106531874cc5c14081ca214eaff Hadoop is arguably one of the most disruptive technologies of the last decade. Once lauded solely for its ability to transform the speed of batch processing, it has marched steadily forward and promulgated an array of performance-enhancing accessories, notably Spark and YARN. Hadoop has evolved into much more than a file system and batch processor, and it now promises to stand as the data management and analytics backbone for enterprises. Register for this episode of The Briefing Room to learn from veteran Analyst Rick van der Lans, as he discusses the emerging roles of Hadoop within the analytics ecosystem. He’ll be briefed by Ron Bodkin of Think Big, a Teradata Company, who will explore Hadoop’s maturity spectrum, from typical entry use cases all the way up the value chain. He’ll show how enterprises that already use Hadoop in production are finding new ways to exploit its power and build creative, dynamic analytics environments. Visit InsideAnalysis.com for more information.

ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY

Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020. Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms. Data lakes will be built in cloud object storage. We’ll discuss the options there as well. Get this data point for your data lake journey.

Make your data fly - Building data platform in AWSKimmo Kantojärvi

Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...Seeling Cheung

Citizens Bank was implementing a BigInsights Hadoop Data Lake with PureData System for Analytics to support all internal data initiatives and improve the customer experience. Testing BigInsights on the ViON Hadoop Appliance yielded the productivity, maintenance, and performance Citizens was looking for. Citizens Bank moved some analytics processing from Teradata to Netezza for better cost and performance, implemented BigInsights Hadoop for a data lake, and avoided large capital expenditures for additional Teradata capacity.

Big data4businessusersBob Hardaway

Powering Real-Time Big Data Analytics with a Next-Gen GPU DatabaseKinetica

Freed from the constraints of storage, network and memory, many big data analytics systems now are routinely revealing themselves to be compute bound. To compensate, big data analytic systems often result in wide horizontal sprawl (300-node Spark or NoSQL clusters are not unusual!)— to bring in enough compute for the task at hand. High system complexity and crushing operational costs often result. As the world shifts from physical to virtual assets and methods of engagement, there is an increasing need for systems of intelligence to live alongside the more traditional systems of record and systems of analysis. New approaches to data processing are required to support the real-time processing of data required to drive these systems of intelligence. Join 451 Research and Kinetica to learn: •An overview of the business and technical trends driving widespread interest in real-time analytics •Why systems of analysis need to be transformed and augmented with systems of intelligence bringing new approaches to data processing •How a new class of solution—a GPU-accelerated, scale out, in-memory database–can bring you orders of magnitude more compute power, significantly smaller hardware footprint, and unrivaled analytic capabilities. •Hear how other companies in a variety of industries, such as financial services, entertainment, pharmaceutical, and oil and gas, benefit from augmenting their legacy systems with a modern analytics database.

Bridging the Last Mile: Getting Data to the People Who Need ItDenodo

Watch full webinar here: https://bit.ly/3cUA0Qi Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Lyft data Platform - 2019 slidesKarthik Murugesan

The Lyft data platform: Now and in the futuremarkgrover

- Lyft has grown significantly in recent years, providing over 1 billion rides to 30.7 million riders through 1.9 million drivers in 2018 across North America. - Data is core to Lyft's business decisions, from pricing and driver matching to analyzing performance and informing investments. - Lyft's data platform supports data scientists, analysts, engineers and others through tools like Apache Superset, change data capture from operational stores, and streaming frameworks. - Key focuses for the platform include business metric observability, streaming applications, and machine learning while addressing challenges of reliability, integration and scale.

Metadata Lakes for Next-Gen AI/ML - DatastratoZilliz

As data catalogs evolve to meet the growing and new demands of high-velocity, unstructured data, we see them taking a new shape as an emergent and flexible way to activate metadata for multiple uses. This talk discusses modern uses of metadata at the infrastructure level for AI-enablement in RAG pipelines in response to the new demands of the ecosystem. We will also discuss Apache (incubating) Gravitino and its open source-first approach to data cataloging across multi-cloud and geo-distributed architectures.

The Future of Data Management: The Enterprise Data HubCloudera, Inc.

The document discusses the future of data management through the use of an enterprise data hub (EDH). It notes that an EDH provides a centralized platform for ingesting, storing, exploring, processing, analyzing and serving diverse data from across an organization on a large scale in a cost effective manner. This approach overcomes limitations of traditional data silos and enables new analytic capabilities.

Dirty data? Clean it up! - Datapalooza Denver 2016Dan Lynn