JOSA TechTalk - Lambda architecture and real-time processing

Jul 7, 20170 likes287 views

Although Hadoop might be the first thing to come in your mind when you think of processing large data sets, it is not always the best solution for your Big Data problems. Hadoop might be the right choice for batch-processing big data, but when it comes to real-time data processing there are other architectures and tools to consider. This TechTalk shows the need behind solving real time data problems and explains the Lambda architecture, mentioning Druid as an example, and the simpler and less expensive "event sourcing model".

REAL-TIME AND BIG DATA
Mahmoud M. Jalajel

OUTLINE
• Intro: Real-time with Big Data
• The Lambda Architecture
• The Relay Model

WHY SOLVE FOR REAL-TIME
• Real-time offers more business value
• Live Web Analytics
• Recommendations
• Real-time = (semi-) realtime
• Event to index ~ single digit minutes
• Query duration ~ single digit seconds

REAL-TIME IMPLEMENTATION
• Incremental Implementation
• Stream processing / No full data context
• A real-time implementation is:
• Far more useful
• Faster
• Easily adaptable to batch mode

REAL-TIME IN HADOOP
MongoDb QueryTime
(optimized, single-node)
Hive QueryTime
(5 nodes)
Hangs, crashes, starts begging for
mercy then commits suicide and
weepingly dies
A few hours
2 Seconds 15 Minutes

LAMBDA ARCHITECTURE
Created by: Nathan Marz
lambda-architecture.net

BASIC ASSUMPTIONS
1. Query = Function(All Data)
2. Data are immutable timely facts
3. Append-Only (CRUD becomes CR)
4. Human Fault-Tolerance

THE BATCH LAYER
• Accepts stream of data
• Appends to master dataset
• Uses: HDFS

THE SERVING LAYER
• Precomputes different views
• Works on full dataset
• Refreshes regularly ofﬂine
• Batch views are usually stored
in a key-value store

JOSA TechTalk - Lambda architecture and real-time processing

CHECKPOINT
• Typical Hadoop Setup
• Slow, inefﬁcient
• Outdated. usually lagging by hours or days
• Although accurate for surveyed data
• Costly to re-run. Real-time is not an option

THE SPEED LAYER
• Works with recent data
• Complements results
• Incremental implementation

REVIEWING LA
PROs
• Modular
• Flexible
• Self-Auditing
• Proven components
CONs
• Complex
• Maintainability
• Query Merging

REVIEWING RM
PROs
• Coherent, Simpler
than LA
• Extensible to full LA
• Cheaper
CONs
• Master Data Storage
• Query ﬂexibility

WHY NOT HADOOP NOW?
• Too much time, no capacity
• Too soon or too late
• Too expensive
• Hammer/nail problem

CONCLUSIONS
• Think big data, now!
• No need to invest years of development to perfect a
big data system.
• Start now! gradually grow system requirements and
engineering skill-set
• Select scalable components

The document discusses Tuplejump, a data engineering startup with a vision to simplify data engineering. It summarizes Tuplejump's big data pipeline platform which collects, transforms, predicts, stores, explores and visualizes data using various tools like Hydra, Spark, Cassandra, MinerBot, Shark, UberCube and Pissaro. It advocates using Scala as the primary language due to its object oriented and functional capabilities. It also discusses advantages of Tuplejump's platform and how tools like Akka, Spark, Play, SBT, ScalaTest, Shapeless and Scalaz are leveraged.

Continuous database deploymentMike (Michael) Acord

This presentation discusses continuous database deployments. It begins with an introduction of the presenter and an overview of topics to be covered. It then contrasts manual database change management with continuous deployment. The main methods covered are schema-based, using the database schema in source control; script-based, using change scripts; and code-based, coding database changes. Benefits include reduced errors and faster releases. Best practices discussed include backing up data and deploying breaking changes in steps. The presentation concludes with a call for questions.

Building Robust Pipelines with AirflowErin Shellman

The data science team at Zymergen is applying machine learning techniques to identify genetic targets, work that is supported by extensive analytical automation that systematically identifies outliers, removes process-related bias, and quantifies performance improvements. We’re using Apache Airflow to construct robust data pipelines that allow us to produce clean, reliable inputs to our predictive models. In this talk, I’ll discuss the unique data processing challenges we face in working with high-throughput, biological data and provide an overview of how we’re using Apache Airflow to meet those challenges.

Lessons Learned from Building and Operating ScubaSingleStore

This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.

Introducing Tupilak, Snowplow's unified log fabricAlexander Dean

In this talk at Snowplow London Meetup #3 I introduced Tupilak, Snowplow’s unified log fabric. Putting a real-time event pipeline into production has many challenges: we need the pipeline to scale automatically based on event volumes, we need constant monitoring to prevent data loss and minimise end-to-end lag, and we need the ability to upgrade and extend the pipeline with zero downtime. We call software which does all this a “unified log fabric”, to distinguish it from the unified logs (e.g. Kafka and Kinesis) and stream processing frameworks (e.g. Spark Streaming and Kafka Streams) which such a fabric monitors and orchestrates. As part of incorporating Snowplow’s Kinesis-based event pipeline into our Managed Service, we developed our own unified log fabric, called Tupilak. In this talk, I introduced Tupilak, explaining the core monitoring and scaling functions of Tupilak and showing live real-time pipelines visualised in the Tupilak UI. I dived into the architecture of Tupilak, shared its basic scaling algorithm and also took a look at how Tupilak itself is built on a Snowplow event stream. I also talked about the roadmap for Tupilak, including our plans for introducing lag-based auto-scaling and porting Tupilak to Kubernetes.

Strata+Hadoop 2015 NYC End User Panel on Real-Time Data AnalyticsSingleStore

Reactive streams and components on OSGi - C Schneidermfrancis

OSGi Community Event 2017 Presentation by Christian Schneider [Adobe] Reactive frameworks allow to implement non blocking asynchronous processing in a convenient way. This talk explores how to apply reactive patterns to typical use cases on OSGi like REST services, MQTT Message processing, computations over sliding windows. We combine messaging and reactor.io streams like also used in Spring 5 to create services that are highly scalable while not tying the user to the technologies being used. The presented reactive components framework allows to abstract from the messaging technologies using OSGi services. These offer standard Publishers and Subscribers that work nicely with any reactive framework based on the reactive streams API. This allows to create integrations like with Apache Camel but in a much leaner and OSGi compliant way.

Apache Airflow (incubating) NL HUG Meetup 2016-07-19Bolke de Bruin

AWS Serverless solution for developersMichael Haberman

What Crimean War gunboats teach us about the need for schema registriesAlexander Dean

In 1853 Britain’s workshops built 90 new gunboats for the Royal Navy in just 90 days: an astonishing feat of engineering. Industrial standardization made this possible - and in this talk, my first at Strata, I argued that data-sophisticated corporations need a new standardization of their own, in the form of schema registries like Confluent Schema Registry or Snowplow’s own Iglu. Talk abstract: At the start of the Crimean War in 1853, Britain's Royal Navy needed 90 new gunboats ready to fight in the Baltic in just 90 days. Assembling the boats was straightforward - the challenge was to build all of the engine sets in time. Marine engineer John Penn did an unusual thing: he took a pair of reference engines, disassembled them and distributed the pieces to the best machine shops across Britain. These workshops - latter-day micro-services - each built 90 sets of their allocated parts, which were then assembled into the engines for the new gunboats, ready for battle. This was the nineteenth century - how could the Admiralty be certain that the parts from all these independent workshops would come together to form 90 high-powered engines? The answer lay in a crucial piece of standardization: the Whitworth thread, the world’s first national screw thread standard, devised by Sir Joseph Whitworth in 1841. By the time the Royal Navy came knocking, this standard had been adopted by workshops across Britain; John Penn could be confident that engine parts built by any workshop to the Whitworth standard would fit together. In this talk, Snowplow co-founder Alexander Dean will draw on the story of the Crimean War gunboats to argue that our data processing architectures urgently require a standardization of their own, in the form of schema registries. Like the Whitworth screw thread, a schema registry, such as Confluent Schema Registry or Snowplow’s own Iglu, allows enterprises to standardise on a set of business entities which can be used throughout their batch and stream processing architectures. Like the artisanal workshops in 1850s Britain, micro-services can work on narrowly defined data processing tasks, confident that their inputs and outputs will be compatible with their peers. This talk will start with the rationale for putting a schema registry at the heart of your business, before moving on to the practicalities of an implementation, including: a side-by-side comparison of the available registries; best practises about schema versioning; strategies around schema federation across different companies such as Snowplow’s own Iglu Central.

Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske

Apache Flink features two APIs which are based on relational algebra, a SQL interface and the so-called Table API, which is a LINQ-style API available for Scala and Java. Relational APIs are interesting because they are easy to use and queries can be automatically optimized and translated into efficient runtime code. Flink offers both APIs for streaming and batch data sources. This talk takes a look under the hood of Flink’s relational APIs. The presentation shows the unified architecture to handle streaming and batch queries and explain how Flink translates queries of both APIs into the same representation, leverages Apache Calcite to optimize them, and generates runtime code for efficient execution. Finally, the slides discuss potential improvements and give an outlook for future extensions and features.

Quarterly Technology Briefing, Manchester, UK September 2013Thoughtworks

This document summarizes a tech lab experiment conducted by ThoughtWorks to explore graph databases and cloud deployment. The team modeled Manchester's tram network as a graph database to understand its benefits. They iteratively improved the data and query models. The working app was deployed to AWS and automatically updated. The team successfully met the desired outcomes of the lab by gaining technical knowledge, producing a working demo, collateral materials, and excitement in the community. Next steps discussed expanding the work to clients and open sourcing parts of the project.

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...Flink Forward

The mission of Uber is to make transportation as reliable as running water. The business is fundamentally driven by real-time data -- more than half of the employees in Uber, many of whom are non-technical, use SQL on a regular basis to analyze data and power their business decisions. We are building AthenaX, a stream processing platform built on top of Apache Flink to enable our users to write SQL to process real-time data efficiently and reliably at Uber's scale. Using Apache Calcite as query parser, AthenaX compiles the SQL down to Flink jobs. Leveraging Flink's unique streaming capabilities, AthenaX supports (1) consistent computations reliably thanks to at-least-once guarantees, (2) nontrivial analytics (e.g., windowing and joins) on multiple data sources, and (3) efficient and cost-effective executions in production through code generation and elastic scaling.

Window phone programing pain and how to deal withZalo_app

This document discusses ways to improve performance when working with images and LINQ queries in Windows Phone applications. It recommends caching images locally using a FIFO rule to manage memory usage. When making LINQ queries, it suggests writing queries in batches, using compiled queries for frequent queries, and only selecting the needed entity properties to reduce response sizes. The document provides examples showing how these techniques can significantly improve query write and read times.

Elk meetup boston - logz.iotomerlevy9

This document summarizes an ELK meetup that took place on March 2nd 2015. It discusses using ELK for log processing, in public clouds like AWS, and activities like kite surfing. The document also provides information on Wind Analytics and their next steps, monitoring large AWS environments, implementing ELK with the right architecture, and Logz.io which provides an ELK as a service solution and insights. It includes demos of Logz.io's architecture and log processing. The meetup concluded with information on job opportunities at Logz.io.

Agile Lab_BigData_Meetup_AKKAPaolo Platter

This document discusses reactive and parallel programming using Akka. It introduces Akka, an open-source toolkit for building distributed, concurrent applications on the JVM. Akka supports actors, futures, software transactional memory, and message passing to handle concurrency. Akka cluster allows building fault-tolerant distributed systems. Examples demonstrate using Akka for microservices, streaming engines, authentication platforms, and IoT applications. The document concludes with an example of how Agile Lab uses Akka in their applications.

GraphQL vs. (the) RESTcoliquio GmbH

What comes to you if you want to have some fun with GraphQL? Join us for the meetup in Munich and enjoy the interesting talk about ‚GraphQL vs. (the) REST ‘. In this session our colleagues Tsvetan Nikolov (Senior Developer at coliquio) and Tom Sedlmeier (Senior Developer at coliquio) will discuss the basic concepts and how to use them. We will present you GraphQL on the backend and on the client side with Apollo.

eHarmony in the CloudCraig Dickson

The Journey To Serverless At Home24 - reflections and insights AWS Germany

Promise of a better future by Rahul Goma Phulore and Pooja Akshantal, Thought...Thoughtworks

With the recent, vivid trend towards multicore hardware and the ever growing application requirements, concurrency is no more a niche area it used to be, and is slowly becoming a norm. In this talk, we will talk about promises/futures, one of the concurrency models that has risen to the occasion. We will look at what they are, how they're implemented and used in Java and Javascript. We will see how Scala, with its functional paradigm and greater abstraction capabilities, avoids "callback hell" typically associated with the model, allows writing of concurrent code in "direct style", and thereby greatly reduces the cognitive burden, allowing you to focus on application logic better.

Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

This document compares Java and Scala for web development using the Playframework 2.5.x. It provides an overview of Scala and the Playframework, including how to set up a Play application with Scala. It demonstrates building a sample web application for a speaker database using Play and Scala, including setting up the project structure, models, database integration, JSON handling, testing, and deployment. The document aims to demonstrate how to develop a web application from start to finish using the Playframework and Scala.

Introduction to Graph QLDeepak More

Serverless microservices in the wildRotem Tamir

Managing Serverless Microservices in the Wild Handing off responsibility for your microservice's infrastructure with AWS API Gateway and Lambda is great, but it also means losing visibility and influence over your app in production. In this talk, we'll discuss different strategies to gain insights into what actually is going on inside your serverless app. We will present a tool we've built at atomData to get high quality, actionable logs and metrics about our services.

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent

Whether you are deploying a new application in Microservices or transitioning from a monolithic database application to a cloud-ready architecture, you will inevitably face the decision of either creating a service mesh of API’s – or – using an event bus for better durability, reliability and extensibility of your application. If you choose to go the event bus route, Kafka is an excellent choice for several reasons. One key technology not to overlook is Avro Schemas. They provide a definition for your event payload, just like an API, to ensure all of the event consumers can reliably consume the events. They also handle schema evolution as requirements change and much, much more. In this talk we will discuss all the nuances and considerations around using Avro Schemas for your JSON event payloads. From developer tools, to DevOps approaches, versioning, governance and some “gotchas” we found when working with Avro Schemas and the Confluent Schema Registry.

Running Airflow Workflows as ETL Processes on Hadoopclairvoyantllc

While working with Hadoop, you'll eventually encounter the need to schedule and run workflows to perform various operations like ingesting data or performing ETL. There are a number of tools available to assist you with this type of requirement and one such tool that we at Clairvoyant have been looking to use is Apache Airflow. Apache Airflow is an Apache Incubator project that allows you to programmatically create workflows through a python script. This provides a flexible and effective way to design your workflows with little code and setup. In this talk, we will discuss Apache Airflow and how we at Clairvoyant have utilized it for ETL pipelines on Hadoop.

Running Yarn at Scale DataWorks Summit

At Yahoo! over the past year we have helped migrate hundreds of our grids? users to YARN. Our YARN clusters have in aggregate run over 18 million jobs with more than 3 billion tasks consuming over 10 thousand years of compute time. With one single cluster running 90 thousand jobs a day. From this experience we would like to share what we have learned about running YARN well, how this is different from running a 1.0 based cluster, and what it takes to migrate your jobs to YARN from 1.0.

Prometheus on AWSMitsuhiro Tanda

This document discusses using Prometheus on AWS to monitor infrastructure. Key features of Prometheus discussed include its multi-dimensional data model, flexible query language, and pull-based collection over HTTP. The document outlines challenges of monitoring AWS infrastructure due to short instance lifecycles. It explains how Prometheus' data model and service discovery help monitor metrics aggregated by attributes like availability zone and role. Configuration and deployment of Prometheus on AWS is also covered, including using EC2 service discovery and storing CloudWatch metrics in Prometheus.

Lattice yapc-slideshareGwenn Etourneau

This document discusses Lattice, an open source Platform as a Service (PaaS) that was born from CloudFoundry. Lattice aims to make deploying and running containerized workloads easy through features like easy installation, clustering, scheduling, self-healing, load balancing, and log aggregation. The document provides an overview of Lattice's architecture and components like Diego for scheduling and X-Ray for visualization. It also demonstrates how to deploy Docker images and buildpacks, submit custom workloads, configure routing, and view logs using Lattice.

DrupalCampLA 2014 - Drupal backend performance and scalabilitycherryhillco

This document discusses various techniques for optimizing Drupal backend performance and scalability. It covers diagnosing issues through tools like Apache Benchmark and Munin, optimizing hardware, web and database servers like using Nginx, Varnish, MySQL tuning, and alternative databases like MongoDB. It also discusses PHP optimizations like opcode caching and HHVM. The goal is to provide strategies to handle more traffic, improve page response times, and minimize downtime through infrastructure improvements and code optimizations.

Operational-AnalyticsNiloy Mukherjee

More Related Content

What's hot (20)

AWS Serverless solution for developersMichael Haberman

What Crimean War gunboats teach us about the need for schema registriesAlexander Dean

Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske

Quarterly Technology Briefing, Manchester, UK September 2013Thoughtworks

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...Flink Forward

Window phone programing pain and how to deal withZalo_app

Elk meetup boston - logz.iotomerlevy9

Agile Lab_BigData_Meetup_AKKAPaolo Platter

GraphQL vs. (the) RESTcoliquio GmbH

eHarmony in the CloudCraig Dickson

The Journey To Serverless At Home24 - reflections and insights AWS Germany

Promise of a better future by Rahul Goma Phulore and Pooja Akshantal, Thought...Thoughtworks

Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Introduction to Graph QLDeepak More

Serverless microservices in the wildRotem Tamir

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent

Running Airflow Workflows as ETL Processes on Hadoopclairvoyantllc

Running Yarn at Scale DataWorks Summit

Prometheus on AWSMitsuhiro Tanda

Lattice yapc-slideshareGwenn Etourneau

AWS Serverless solution for developersMichael Haberman

What Crimean War gunboats teach us about the need for schema registriesAlexander Dean

Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske

Quarterly Technology Briefing, Manchester, UK September 2013Thoughtworks

Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...Flink Forward

Window phone programing pain and how to deal withZalo_app

Elk meetup boston - logz.iotomerlevy9

Agile Lab_BigData_Meetup_AKKAPaolo Platter

GraphQL vs. (the) RESTcoliquio GmbH

eHarmony in the CloudCraig Dickson

The Journey To Serverless At Home24 - reflections and insights AWS Germany

Promise of a better future by Rahul Goma Phulore and Pooja Akshantal, Thought...Thoughtworks

Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Introduction to Graph QLDeepak More

Serverless microservices in the wildRotem Tamir

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent

Running Airflow Workflows as ETL Processes on Hadoopclairvoyantllc

Running Yarn at Scale DataWorks Summit

Prometheus on AWSMitsuhiro Tanda

Lattice yapc-slideshareGwenn Etourneau

Similar to JOSA TechTalk - Lambda architecture and real-time processing (20)

DrupalCampLA 2014 - Drupal backend performance and scalabilitycherryhillco

Operational-AnalyticsNiloy Mukherjee

Batch to near-realtime: inspired by a real production incidentShivji Kumar Jha

This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024. This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards. Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.

Innovation in the Data Warehouse - StampedeCon 2016StampedeCon

Enterprise Holding’s first started with Hadoop as a POC in 2013. Today, we have clusters on premises and in the cloud. This talk will explore our experience with Big Data and outline three common big data architectures (batch, lambda, and kappa). Then, we’ll dive into the decision points to necessary for your own cluster, for example: cloud vs on premises, physical vs virtual, workload, and security. These decisions will help you understand what direction to take. Finally, we’ll share some lessons learned with the pieces of our architecture worked well and rant about those which didn’t. No deep Hadoop knowledge is necessary, architect or executive level.

Intro to Big DataZohar Elkayam

Drupal performancePiyuesh Kumar

This document discusses various techniques for optimizing Drupal performance, including: - Defining goals such as faster page loads or handling more traffic - Applying patches and rearchitecting content to optimize at a code level - Using tools like Apache Benchmark and MySQL tuning to analyze performance bottlenecks - Implementing solutions like caching, memcached, and reverse proxies to improve scalability

noSQL choiceslugiamaster4

Cassandra consistently outperforms other NoSQL databases in throughput and scalability according to various benchmark tests, but has higher read latencies. MongoDB typically has the worst performance in terms of latency. The best database depends on application requirements - no single NoSQL database is best for all use cases. Combining database types, such as using Cassandra for analytics and an RDBMS for transactions, can leverage each database's strengths.

Technologies for Data Analytics PlatformN Masahiro

This document discusses building a data analytics platform and summarizes various technologies that can be used. It begins by outlining reasons for analyzing data like reporting, monitoring, and exploratory analysis. It then discusses using relational databases, parallel databases, Hadoop, and columnar storage to store and process large volumes of data. Streaming technologies like Storm, Kafka, and services like Redshift, BigQuery, and Treasure Data are also summarized as options for a complete analytics platform.

Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...Precisely

This document discusses engineering machine learning data pipelines and addresses five big challenges: 1) scattered and difficult to access data, 2) data cleansing at scale, 3) entity resolution, 4) tracking data lineage, and 5) ongoing real-time changed data capture and streaming. It presents DMX Change Data Capture as a solution to capture changes from various data sources and replicate them in real-time to targets like Kafka, HDFS, databases and data lakes to feed machine learning models. Case studies demonstrate how DMX-h has helped customers like a global hotel chain and insurance and healthcare companies build scalable data pipelines.

Chirp 2010: Scaling TwitterJohn Adams

This document summarizes a keynote speech given by John Adams, an early Twitter engineer, about scaling Twitter operations from 2008-2009. Some key points: 1) Twitter saw exponential growth rates from 2008-2009, processing over 55 million tweets per day and 600 million searches per day. 2) Operations focused on improving performance, reducing errors and outages, and using metrics to identify weaknesses and bottlenecks like network latency and database delays. 3) Technologies like Unicorn, memcached, Flock, Cassandra, and daemons were implemented to improve scalability beyond a traditional RDBMS and handle Twitter's data volumes and real-time needs. 4) Caching,

HBaseCon 2013: Near Real Time Indexing for eBay SearchCloudera, Inc.

Near Real Time Indexing for ebay Search The document discusses eBay's near real time indexing pipeline to build indexes for search within minutes of data updates. It outlines the challenges of handling a large volume of updates at scale and describes optimizations made to HBase and the indexing process to reduce indexing time and improve stability. These include improved HBase configuration, major compaction scheduling, and standalone indexing to reduce overhead.

Neotys PAC - Ian MolyneauxNeotys_Partner

Neotys organized its first Performance Advisory Council in Scotland, the 14th & 15th of November. With 15 Load Testing experts from several countries (UK, France, New-Zeland, Germany, USA, Australia, India…) we explored several theme around Load Testing such as DevOps, Shift Right, AI etc. By discussing around their experience, the methods they used, their data analysis and their interpretation, we created a lot of high-value added content that you can use to discover what will be the future of Load Testing. You want to know more about this event ? https://www.neotys.com/performance-advisory-council

20160331 sa introduction to big data pipelining berlin meetup 0.3Simon Ambridge

This document discusses building data pipelines with Apache Spark and DataStax Enterprise (DSE) for both static and real-time data. It describes how DSE provides a scalable, fault-tolerant platform for distributed data storage with Cassandra and real-time analytics with Spark. It also discusses using Kafka as a messaging queue for streaming data and processing it with Spark. The document provides examples of using notebooks, Parquet, and Akka for building pipelines to handle both large static datasets and fast, real-time streaming data sources.

Real-Time Analytics with MemSQL and SparkSingleStore

Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, EuropeFlip Kromer

Project Sherpa: How RightScale Went All in on DockerRightScale

Modern software architectures - PHP UK Conference 2015Ricard Clau

The web has changed. Users demand responsive, real-time interactive applications and companies need to store and analyze tons of data. Some years ago, monolithic code bases with a basic LAMP stack, some caching and perhaps a search engine were enough. These days everybody is talking about micro-services architectures, SOA, Erlang, Golang, message passing, queue systems and many more. PHP seems to not be cool anymore but... is this true? Should we all forget everything we know and just learn these new technologies? Do we really need all these things?

Five Years of EC2 DistilledGrig Gheorghiu

Netflix's Big Leap from Oracle to CassandraRoopa Tangirala

This document summarizes Netflix's migration from Oracle to Cassandra. It discusses how Netflix moved its backend database from Oracle to Cassandra to gain scalability and reduce costs. The migration strategy involved dual writes to both databases, fork lifting the existing Oracle dataset, and a consistency checker. Challenges included security, denormalization, and engineering effort. Real use cases like APIs and viewing history are discussed along with lessons learned around data modeling, performance testing, and thinking of Cassandra as just storage.

John adams talk cloudyJohn Adams

DrupalCampLA 2014 - Drupal backend performance and scalabilitycherryhillco

Operational-AnalyticsNiloy Mukherjee

Batch to near-realtime: inspired by a real production incidentShivji Kumar Jha

Innovation in the Data Warehouse - StampedeCon 2016StampedeCon

Intro to Big DataZohar Elkayam

Drupal performancePiyuesh Kumar

noSQL choiceslugiamaster4

Technologies for Data Analytics PlatformN Masahiro

Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...Precisely

Chirp 2010: Scaling TwitterJohn Adams

HBaseCon 2013: Near Real Time Indexing for eBay SearchCloudera, Inc.

Neotys PAC - Ian MolyneauxNeotys_Partner

20160331 sa introduction to big data pipelining berlin meetup 0.3Simon Ambridge

Real-Time Analytics with MemSQL and SparkSingleStore

Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, EuropeFlip Kromer

Project Sherpa: How RightScale Went All in on DockerRightScale

Modern software architectures - PHP UK Conference 2015Ricard Clau

Five Years of EC2 DistilledGrig Gheorghiu

Netflix's Big Leap from Oracle to CassandraRoopa Tangirala

John adams talk cloudyJohn Adams

More from Mahmoud Jalajel (6)

Data Science Toolkit for Product ManagersMahmoud Jalajel

Case studies on openness and collaborationMahmoud Jalajel

Proposal for open government dataMahmoud Jalajel

The document proposes an open government data system for Jordan with the following key points: - It would make more government data available to the public in open formats like CSV and JSON to enable academic and commercial uses. - Data on the system would include both raw datasets and summarized data and insights from government agencies. Formats would need to follow open standards. - Each dataset would include the raw data files, metadata files describing the data, and checksum files to ensure correctness. Metadata would also provide descriptions, collection methods, and potential uses. - The system would have a centralized agency to manage it, government agencies to upload data, and public users to access and analyze the data through a web interface or API

JOSA Data Science Bootcamp OverviewMahmoud Jalajel

Technology building blocks for innovationMahmoud Jalajel

Big Data in FinTechMahmoud Jalajel

The document discusses how big data is being applied in financial technology (FinTech). It begins with an agenda and introduction to the speaker, Mahmoud Jalajel. It then discusses how tech companies are leading innovations in FinTech through applications like money transfers. The bulk of the document outlines key concepts in big data including ingestion, ETL processes, software, analytics, and data science. It provides examples of how these are applied in FinTech for areas like predictive modeling, personalization, and fraud detection. Finally, it shares two case studies of startups leveraging big data for applications like automated lending and risk management.

Data Science Toolkit for Product ManagersMahmoud Jalajel

Case studies on openness and collaborationMahmoud Jalajel

Proposal for open government dataMahmoud Jalajel

JOSA Data Science Bootcamp OverviewMahmoud Jalajel

Technology building blocks for innovationMahmoud Jalajel

Big Data in FinTechMahmoud Jalajel

Recently uploaded (20)

MINDCTI revenue release Quarter 1 2025 PRMIND CTI

TrsLabs - Leverage the Power of UPI PaymentsTrs Labs

Revolutionize your Fintech growth with UPI Payments "Riding the UPI strategy" refers to leveraging the Unified Payments Interface (UPI) to drive digital payments in India and beyond. This involves understanding UPI's features, benefits, and potential, and developing strategies to maximize its usage and impact. Essentially, it's about strategically utilizing UPI to promote digital payments, financial inclusion, and economic growth.

The Future of Cisco Cloud Security: Innovations and AI IntegrationRe-solution Data Ltd

Play It Safe: Manage Security Risks - Google CertificateVICTOR MAESTRE RAMIREZ

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Build Your Own Copilot & Agents For DevsBrian McKeiver

Hundred applicable AI Cases for oil and Gasbengsoon3

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

tecnologias de las primeras civilizaciones.pdffjgm517

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

GyrusAI - Broadcasting & Streaming Applications Driven by AI and MLGyrus AI

Gyrus AI: AI/ML for Broadcasting & Streaming Gyrus is a Vision Al company developing Neural Network Accelerators and ready to deploy AI/ML Models for Video Processing and Video Analytics. Our Solutions: Intelligent Media Search Semantic & contextual search for faster, smarter content discovery. In-Scene Ad Placement AI-powered ad insertion to maximize monetization and user experience. Video Anonymization Automatically masks sensitive content to ensure privacy compliance. Vision Analytics Real-time object detection and engagement tracking. Why Gyrus AI? We help media companies streamline operations, enhance media discovery, and stay competitive in the rapidly evolving broadcasting & streaming landscape. 🚀 Ready to Transform Your Media Workflow? 🔗 Visit Us: https://gyrus.ai/ 📅 Book a Demo: https://gyrus.ai/contact 📝 Read More: https://gyrus.ai/blog/ 🔗 Follow Us: LinkedIn - https://www.linkedin.com/company/gyrusai/ Twitter/X - https://twitter.com/GyrusAI YouTube - https://www.youtube.com/channel/UCk2GzLj6xp0A6Wqix1GWSkw Facebook - https://www.facebook.com/GyrusAI

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

AI and Data Privacy in 2025: Global TrendsInData Labs

In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy. This infographic contains: -AI and data privacy: Key findings -Statistics on AI data privacy in the today’s world -Tips on how to overcome data privacy challenges -Benefits of AI data security investments. Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

Canadian book publishing: Insights from the latest salary survey - Tech Forum...BookNet Canada

Join us for a presentation in partnership with the Association of Canadian Publishers (ACP) as they share results from the recently conducted Canadian Book Publishing Industry Salary Survey. This comprehensive survey provides key insights into average salaries across departments, roles, and demographic metrics. Members of ACP’s Diversity and Inclusion Committee will join us to unpack what the findings mean in the context of justice, equity, diversity, and inclusion in the industry. Results of the 2024 Canadian Book Publishing Industry Salary Survey: https://publishers.ca/wp-content/uploads/2025/04/ACP_Salary_Survey_FINAL-2.pdf Link to presentation recording and transcript: https://bnctechforum.ca/sessions/canadian-book-publishing-insights-from-the-latest-salary-survey/ Presented by BookNet Canada and the Association of Canadian Publishers on May 1, 2025 with support from the Department of Canadian Heritage.

Procurement Insights Cost To Value Guide.pptxJon Hansen

Vibe Coding_ Develop a web application using AI (1).pdfBaiju Muthukadan

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Hybridize Functions: A Tool for Automatically Refactoring Imperative Deep Lea...Raffi Khatchadourian

Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code—supporting symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, imperative DL frameworks encouraging eager execution have emerged but at the expense of run-time performance. Though hybrid approaches aim for the “best of both worlds,” using them effectively requires subtle considerations to make code amenable to safe, accurate, and efficient graph execution—avoiding performance bottlenecks and semantically inequivalent results. We discuss the engineering aspects of a refactoring tool that automatically determines when it is safe and potentially advantageous to migrate imperative DL code to graph execution and vice-versa.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

MINDCTI revenue release Quarter 1 2025 PRMIND CTI

TrsLabs - Leverage the Power of UPI PaymentsTrs Labs

The Future of Cisco Cloud Security: Innovations and AI IntegrationRe-solution Data Ltd

Play It Safe: Manage Security Risks - Google CertificateVICTOR MAESTRE RAMIREZ

How analogue intelligence complements AIPaul Rowe

Build Your Own Copilot & Agents For DevsBrian McKeiver

Hundred applicable AI Cases for oil and Gasbengsoon3

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

tecnologias de las primeras civilizaciones.pdffjgm517

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

GyrusAI - Broadcasting & Streaming Applications Driven by AI and MLGyrus AI

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

AI and Data Privacy in 2025: Global TrendsInData Labs

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

Canadian book publishing: Insights from the latest salary survey - Tech Forum...BookNet Canada

Procurement Insights Cost To Value Guide.pptxJon Hansen

Vibe Coding_ Develop a web application using AI (1).pdfBaiju Muthukadan

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Hybridize Functions: A Tool for Automatically Refactoring Imperative Deep Lea...Raffi Khatchadourian

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

JOSA TechTalk - Lambda architecture and real-time processing

1. REAL-TIME AND BIG DATA Mahmoud M. Jalajel

2. OUTLINE • Intro: Real-time with Big Data • The Lambda Architecture • The Relay Model

3. WHY SOLVE FOR REAL-TIME • Real-time offers more business value • Live Web Analytics • Recommendations • Real-time = (semi-) realtime • Event to index ~ single digit minutes • Query duration ~ single digit seconds

4. REAL-TIME IMPLEMENTATION • Incremental Implementation • Stream processing / No full data context • A real-time implementation is: • Far more useful • Faster • Easily adaptable to batch mode

5. REAL-TIME IN HADOOP MongoDb QueryTime (optimized, single-node) Hive QueryTime (5 nodes) Hangs, crashes, starts begging for mercy then commits suicide and weepingly dies A few hours 2 Seconds 15 Minutes

6. LAMBDA ARCHITECTURE Created by: Nathan Marz lambda-architecture.net

7. LAMBDA ARCHITECTURE

8. BASIC ASSUMPTIONS 1. Query = Function(All Data) 2. Data are immutable timely facts 3. Append-Only (CRUD becomes CR) 4. Human Fault-Tolerance

9. THE BATCH LAYER • Accepts stream of data • Appends to master dataset • Uses: HDFS

10. THE SERVING LAYER • Precomputes different views • Works on full dataset • Refreshes regularly ofﬂine • Batch views are usually stored in a key-value store

13. CHECKPOINT • Typical Hadoop Setup • Slow, inefﬁcient • Outdated. usually lagging by hours or days • Although accurate for surveyed data • Costly to re-run. Real-time is not an option

14. THE SPEED LAYER • Works with recent data • Complements results • Incremental implementation

16. THE FULL PICTURE Query Merging

17. EXAMPLETECHNOLOGIES

18. DRUID EXAMPLE

19. REVIEWING LA PROs • Modular • Flexible • Self-Auditing • Proven components CONs • Complex • Maintainability • Query Merging

20. THE RELAY MODEL

21. RELAY MODEL Query Merging

22. THE WORKFLOW

23. REVIEWING RM PROs • Coherent, Simpler than LA • Extensible to full LA • Cheaper CONs • Master Data Storage • Query ﬂexibility

24. WHY NOT HADOOP NOW? • Too much time, no capacity • Too soon or too late • Too expensive • Hammer/nail problem

25. CONCLUSIONS • Think big data, now! • No need to invest years of development to perfect a big data system. • Start now! gradually grow system requirements and engineering skill-set • Select scalable components

26. Mahmoud Jalajel – @mjalajel Questions ?

27. APPENDIX

28. Apache Kafka

29. Apache Storm

30. Apache Storm with external systems

JOSA TechTalk - Lambda architecture and real-time processing

Recommended

More Related Content

What's hot (20)

Similar to JOSA TechTalk - Lambda architecture and real-time processing (20)

More from Mahmoud Jalajel (6)

Recently uploaded (20)

JOSA TechTalk - Lambda architecture and real-time processing