Real time analytics with Netty, Storm, Kafka
35 likes12,538 views
This document discusses a real-time analytics architecture using Netty, Apache Kafka, and Storm. It includes an overview of the system architecture with Netty handling HTTP logging and producing to Apache Kafka for stream data storage. Storm analytics clusters then consume from Kafka topics to perform analytics functions like tokenization, parsing, aggregation, and saving results to Redis and a data warehouse.
1 of 13
Downloaded 183 times
![Netty.io
Netty is a non-blocking I/O (NIO) client-server framework
for the development of Java network applications such as
protocol servers and clients.
The asynchronous event-driven network application
framework and tools are used to simplify network
programming such as TCP and UDP socket servers.[2]
Netty includes an implementation of the reactor pattern of
programming.
http://en.wikipedia.org/wiki/Netty_(software)
http://nguyentantrieu.info/blog/backend-system-with-netty-io](https://image.slidesharecdn.com/stormreal-timeanalytics-140806055850-phpapp01/85/Real-time-analytics-with-Netty-Storm-Kafka-7-320.jpg)
Ad
Recommended
Building Microservices with Apache Kafka
Building Microservices with Apache Kafkaconfluent Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
Presentation by Colin McCabe, Confluent, Big Data Day LA
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019confluent Tesla ingests trillions of events every day from hundreds of unique data sources through our streaming data platform. Find out how we developed a set of high-throughput, non-blocking primitives that allow us to transform and ingest data into a variety of data stores with minimal development time. Additionally, we will discuss how these primitives allowed us to completely migrate the streaming platform in just a few months. Finally, we will talk about how we scale team size sub-linearly to data volumes, while continuing to onboard new use cases.
Exactly once with spark streaming
Exactly once with spark streamingQuentin Ambard How to ensure exactly once with spark streaming and Kafka: Dstream & structured streaming with Cassandra
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
Webinar: Deep Dive on Apache Flink State - Seth WiesmanVerverica Apache Flink is a world class stateful stream processor presents a huge variety of optional features and configuration choices to the user. Determining out the optimal choice for any production environment and use-case be challenging. In this talk, we will explore and discuss the universe of Flink configuration with respect to state and state backends.
We will start with a closer look under the hood, at core data structures and algorithms, to build the foundation for understanding the impact of tuning parameters and the costs-benefit-tradeoffs that come with certain features and options. In particular, we will focus on state backend choices (Heap vs RocksDB), tuning checkpointing (incremental checkpoints, ...) and recovery (local recovery), serializers and Apache Flink's new state migration capabilities.
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsGuido Schmutz Independent of the source of data, the integration and analysis of event streams gets more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analyzed, often with many consumers or systems interested in all or part of the events. In this session we compare two popular Streaming Analytics solutions: Spark Streaming and Kafka Streams.
Spark is fast and general engine for large-scale data processing and has been designed to provide a more efficient alternative to Hadoop MapReduce. Spark Streaming brings Spark's language-integrated API to stream processing, letting you write streaming applications the same way you write batch jobs. It supports both Java and Scala.
Kafka Streams is the stream processing solution which is part of Kafka. It is provided as a Java library and by that can be easily integrated with any Java application.
This presentation shows how you can implement stream processing solutions with each of the two frameworks, discusses how they compare and highlights the differences and similarities.
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?Kai Wähner Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...confluent Ever wondered just how many CPU cores of KSQL Server you need to provision to handle your planned stream processing workload ? Or how many GBits of aggregate network bandwidth, spread across some number of processing threads, you'll need to deal with combined peak throughput of multiple queries ? In this talk we'll first explore the basic drivers of KSQL throughput and hardware requirements, building up to more advanced query plan analysis and capacity-planning techniques, and review some real-world testing results along the way. Finally we will recap how and what to monitor to know you got it right!
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022HostedbyConfluent Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs is a hyperscale PaaS event stream broker with protocol support for HTTP, AMQP, and Apache Kafka RPC that accepts and forwards several trillion (!) events per day and is available in all global Azure regions. This session is a look behind the curtain where we dive deep into the architecture of Event Hubs and look at the Event Hubs cluster model, resource isolation, and storage strategies and also review some performance figures.
Facebook Messages & HBase
Facebook Messages & HBase强 王 The document discusses Facebook's use of HBase to store messaging data. It provides an overview of HBase, including its data model, performance characteristics, and how it was a good fit for Facebook's needs due to its ability to handle large volumes of data, high write throughput, and efficient random access. It also describes some enhancements Facebook made to HBase to improve availability, stability, and performance. Finally, it briefly mentions Facebook's migration of messaging data from MySQL to their HBase implementation.
Apache Camel K - Copenhagen v2
Apache Camel K - Copenhagen v2Claus Ibsen 2 hour session where I cover what is Apache Camel, latest news on the upcoming Camel v3, and then the main topic of the talk is the new Camel K sub-project for running integrations natively on the cloud with kubernetes. The last part of the talk is about running Camel with GraalVM / Quarkus to archive native compiled binaries that has impressive startup and footprint.
Query logging with proxysql
Query logging with proxysqlYoungHeon (Roy) Kim This document explains how to set up ProxySQL to log queries from users connecting directly to the database servers. It details installing and configuring ProxySQL to log queries to binary files, using a tool to convert the binary logs to text format, and setting up an ELK stack to index the query logs and make them searchable in Kibana. Filebeat is configured to ship the text query logs to Logstash, which parses them and sends the data to Elasticsearch. Kibana provides a web interface for viewing and analyzing the query logs.
Configuring global infrastructure in terraform
Configuring global infrastructure in terraformSANGGI CHOI Configuring global infrastructure in terraform
Adopting Java for the Serverless world at Serverless Meetup New York and Boston
Adopting Java for the Serverless world at Serverless Meetup New York and BostonVadym Kazulkin Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless Community. Java is known for its high cold start times and high memory footprint. For both you have to pay to the cloud providers of your choice. That's why most developers tried to avoid using Java for such use cases. But the times change: Community and cloud providers improve things steadily for Java developers. In this talk we look at the features and possibilities AWS cloud provider offers for the Java developers and look the most popular Java frameworks, like Micronaut, Quarkus and Spring (Boot) and look how (AOT compiler and GraalVM native images play a huge role) they address Serverless challenges and enable Java for broad usage in the Serverless world.
Developing Microservices with Apache Camel
Developing Microservices with Apache CamelClaus Ibsen Red Hat Microservices Architecture Day - New York, November 2015. Presented by Claus Ibsen.
Apache Camel is a very popular integration library that works very well with microservice architecture. This talk introduces you to Apache Camel and how you can easily get started with Camel on your computer. Then we cover how to create new Camel projects from scratch as microservices, which you can boot using Camel or Spring Boot, or other micro containers such as Jetty or fat JARs. We then take a look at what options you have for monitoring and managing your Camel microservices using tooling such as Jolokia, and hawtio web console.
Kubernetes Networking
Kubernetes NetworkingCJ Cullen The document discusses Kubernetes networking. It describes how Kubernetes networking allows pods to have routable IPs and communicate without NAT, unlike Docker networking which uses NAT. It covers how services provide stable virtual IPs to access pods, and how kube-proxy implements services by configuring iptables on nodes. It also discusses the DNS integration using SkyDNS and Ingress for layer 7 routing of HTTP traffic. Finally, it briefly mentions network plugins and how Kubernetes is designed to be open and customizable.
Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureScyllaDB This document summarizes the key design decisions behind ScyllaDB's shard-per-core database architecture. It discusses how ScyllaDB addresses the challenges of scaling databases across hundreds of CPU cores by utilizing an asynchronous task model with one thread and one data shard per CPU core. This allows for linear scalability. It also overhauls the I/O scheduling to prioritize workloads and maximize throughput from SSDs under mixed read/write workloads. Benchmark results show ScyllaDB's architecture can handle petabyte-scale databases with high performance and low latency even on commodity hardware.
Kafka Streams State Stores Being Persistent
Kafka Streams State Stores Being Persistentconfluent This document discusses Kafka Streams state stores. It provides examples of using different types of windowing (tumbling, hopping, sliding, session) with state stores. It also covers configuring state store logging, caching, and retention policies. The document demonstrates how to define windowed state stores in Kafka Streams applications and discusses concepts like grace periods.
Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.
Join semantics in kafka streams
Join semantics in kafka streamsKnoldus Inc. With the recent adoption of the Confluent and Kafka Streams, organizations have experienced significantly improved system stability with real-time processing framework, as well as improved scalability and lower maintenance costs.
The focus of this webinar is:
~Different join operators in Kafka Streams.
~Exploring different options in Kafka Streams to join semantics, both with and without shared keys.
~How to put Application Owner in control by leveraging simplified app-centric architecture.
If you have any queries, contact Himani over mail [email protected]
Detailed Analysis of AWS Lambda vs EC2
Detailed Analysis of AWS Lambda vs EC2Whizlabs Amazon has proved its might in the field of offering diverse cloud services and has excelled in almost all scenarios to date. Amazon EC2 came into play in 2006 and has gained immense popularity since then. But, along with that, AWS Lambda is also a popular service that came out in 2014 and is now walking side-to-side with EC2 in terms of popularity and adaptation.
To know the major differences between AWS Lambda and CE2 please visit https://www.whizlabs.com/blog/aws-lambda-vs-ec2/
Message Queue 가용성, 신뢰성을 위한 RabbitMQ Server, Client 구성
Message Queue 가용성, 신뢰성을 위한 RabbitMQ Server, Client 구성Yoonjeong Kwon RabbitMQ Concept & How to work
High Availability in RabbitMQ
- Clustering & Mirrored Queue
HAProxy
Spring Cloud Stream Sample & configuration: Spring Boot App with Rabbit
Kafka Retry and DLQ
Kafka Retry and DLQGeorge Teo The document describes Uber's use of Kafka for reliable messaging. Kafka is used for inter-service messaging, stream processing, database changelog transport, data ingestion, and logging. It provides ordered, partitioned streaming and unordered queueing semantics. The summary describes Uber's consumer wrapper that implements features like acknowledgments, redelivery, delays between retries, dead letter queues, competing consumers, and multi-datacenter failover to support reliable messaging on top of Kafka.
Data profiling in Apache Calcite
Data profiling in Apache CalciteDataWorks Summit Query optimizers and people have one thing in common: the better they understand their data, the better they can do their jobs. Optimizing queries is hard if you don't have good estimates for the sizes of the intermediate join and aggregate results. Data profiling is a technique that scans data, looking for patterns within the data such as keys, functional dependencies, and correlated columns. These richer statistics can be used in Apache Calcite's query optimizer, and the projects that use it, such as Apache Hive, Phoenix and Drill. We describe how we built a data profiler as a table function in Apache Calcite, review the recent research and algorithms that made it possible, and show how you can use the profiler to improve the quality of your data.
HDFS Overview
HDFS OverviewJEONGPHIL HAN Apache Hadoop의 HDFS에 대해 정리한 자료입니다.
틀린 점이 있다면 메일로 주시면 감사하겠습니다.
[email protected]
Best practices for MySQL High Availability Tutorial
Best practices for MySQL High Availability TutorialColin Charles Given at O'Reilly Velocity London in October 2017, this tutorial focuses on the current best practices for MySQL High Availability.
Kafka tiered-storage-meetup-2022-final-presented
Kafka tiered-storage-meetup-2022-final-presentedSumant Tambe Kafka Tiered Storage separates compute and data storage in two independently scalable layers. Uber's Kafka Improvement Proposal (KIP) #405 describes two-tiered storage, which is a major step towards cloud-native Kafka. It stores the most recent data locally and offloads older data to a remote storage service. Operationally, the benefit is faster routine cluster maintenance activities. In Linkedin, Kafka tiered storage is strongly desired to reduce the cost of running Kafka in the Azure cloud environment. As KIP-405 does not dictate the implementation of remote storage substrate, Linkedin's choice for tiering Kafka in Azure deployments is the Azure Blob Service. This presentation will begin with the motivation behind Linkedin efforts to adopt Kafka Tiered Storage. Next, the architecture of KIP-405 will be discussed. Finally, the Remote Storage Manager for Azure Blobs, which is a work-in-progress, will be presented.
Video: https://youtu.be/V5gaBE5CMwg?t=1387
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent Speaker: Matthias J. Sax, Software Engineer, Confluent
ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries.
Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems.
https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRed Hat Developers Have you ever wondered what the relative differences are between two of the more popular open source, in-memory data stores and caches? In this session, we will describe those differences and, more importantly, provide live demonstrations of the key capabilities that could have a major impact on your architectural Java application designs.
Experience with Kafka & Storm
Experience with Kafka & StormOtto Mok This document discusses AcuityAds' use of Apache Kafka and Storm for processing over 10 billion daily ad impressions. It describes their architecture with Kafka used to ingest bid request data from multiple sources into partitions. Storm topologies read from Kafka and processed the data to calculate metrics like daily impressions by site. Initial issues included unbalanced Kafka partitions and low Storm uptime due to exceptions. Future improvements involved upgrading versions and adding monitoring capabilities.
Non blocking io with netty
Non blocking io with nettyZauber This document discusses non-blocking I/O and the traditional blocking I/O approach for building servers. The traditional approach uses one thread per connection, blocking I/O, and a simple programming model. However, this can cause issues like shared state between clients, synchronization problems, inability to prioritize clients, difficulty scaling to thousands of connections, and challenges with persistent connections. The document explores using non-blocking I/O with Netty as an alternative.
Ad
More Related Content
What's hot (20)
Facebook Messages & HBase
Facebook Messages & HBase强 王 The document discusses Facebook's use of HBase to store messaging data. It provides an overview of HBase, including its data model, performance characteristics, and how it was a good fit for Facebook's needs due to its ability to handle large volumes of data, high write throughput, and efficient random access. It also describes some enhancements Facebook made to HBase to improve availability, stability, and performance. Finally, it briefly mentions Facebook's migration of messaging data from MySQL to their HBase implementation.
Apache Camel K - Copenhagen v2
Apache Camel K - Copenhagen v2Claus Ibsen 2 hour session where I cover what is Apache Camel, latest news on the upcoming Camel v3, and then the main topic of the talk is the new Camel K sub-project for running integrations natively on the cloud with kubernetes. The last part of the talk is about running Camel with GraalVM / Quarkus to archive native compiled binaries that has impressive startup and footprint.
Query logging with proxysql
Query logging with proxysqlYoungHeon (Roy) Kim This document explains how to set up ProxySQL to log queries from users connecting directly to the database servers. It details installing and configuring ProxySQL to log queries to binary files, using a tool to convert the binary logs to text format, and setting up an ELK stack to index the query logs and make them searchable in Kibana. Filebeat is configured to ship the text query logs to Logstash, which parses them and sends the data to Elasticsearch. Kibana provides a web interface for viewing and analyzing the query logs.
Configuring global infrastructure in terraform
Configuring global infrastructure in terraformSANGGI CHOI Configuring global infrastructure in terraform
Adopting Java for the Serverless world at Serverless Meetup New York and Boston
Adopting Java for the Serverless world at Serverless Meetup New York and BostonVadym Kazulkin Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless Community. Java is known for its high cold start times and high memory footprint. For both you have to pay to the cloud providers of your choice. That's why most developers tried to avoid using Java for such use cases. But the times change: Community and cloud providers improve things steadily for Java developers. In this talk we look at the features and possibilities AWS cloud provider offers for the Java developers and look the most popular Java frameworks, like Micronaut, Quarkus and Spring (Boot) and look how (AOT compiler and GraalVM native images play a huge role) they address Serverless challenges and enable Java for broad usage in the Serverless world.
Developing Microservices with Apache Camel
Developing Microservices with Apache CamelClaus Ibsen Red Hat Microservices Architecture Day - New York, November 2015. Presented by Claus Ibsen.
Apache Camel is a very popular integration library that works very well with microservice architecture. This talk introduces you to Apache Camel and how you can easily get started with Camel on your computer. Then we cover how to create new Camel projects from scratch as microservices, which you can boot using Camel or Spring Boot, or other micro containers such as Jetty or fat JARs. We then take a look at what options you have for monitoring and managing your Camel microservices using tooling such as Jolokia, and hawtio web console.
Kubernetes Networking
Kubernetes NetworkingCJ Cullen The document discusses Kubernetes networking. It describes how Kubernetes networking allows pods to have routable IPs and communicate without NAT, unlike Docker networking which uses NAT. It covers how services provide stable virtual IPs to access pods, and how kube-proxy implements services by configuring iptables on nodes. It also discusses the DNS integration using SkyDNS and Ingress for layer 7 routing of HTTP traffic. Finally, it briefly mentions network plugins and how Kubernetes is designed to be open and customizable.
Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureScyllaDB This document summarizes the key design decisions behind ScyllaDB's shard-per-core database architecture. It discusses how ScyllaDB addresses the challenges of scaling databases across hundreds of CPU cores by utilizing an asynchronous task model with one thread and one data shard per CPU core. This allows for linear scalability. It also overhauls the I/O scheduling to prioritize workloads and maximize throughput from SSDs under mixed read/write workloads. Benchmark results show ScyllaDB's architecture can handle petabyte-scale databases with high performance and low latency even on commodity hardware.
Kafka Streams State Stores Being Persistent
Kafka Streams State Stores Being Persistentconfluent This document discusses Kafka Streams state stores. It provides examples of using different types of windowing (tumbling, hopping, sliding, session) with state stores. It also covers configuring state store logging, caching, and retention policies. The document demonstrates how to define windowed state stores in Kafka Streams applications and discusses concepts like grace periods.
Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.
Join semantics in kafka streams
Join semantics in kafka streamsKnoldus Inc. With the recent adoption of the Confluent and Kafka Streams, organizations have experienced significantly improved system stability with real-time processing framework, as well as improved scalability and lower maintenance costs.
The focus of this webinar is:
~Different join operators in Kafka Streams.
~Exploring different options in Kafka Streams to join semantics, both with and without shared keys.
~How to put Application Owner in control by leveraging simplified app-centric architecture.
If you have any queries, contact Himani over mail [email protected]
Detailed Analysis of AWS Lambda vs EC2
Detailed Analysis of AWS Lambda vs EC2Whizlabs Amazon has proved its might in the field of offering diverse cloud services and has excelled in almost all scenarios to date. Amazon EC2 came into play in 2006 and has gained immense popularity since then. But, along with that, AWS Lambda is also a popular service that came out in 2014 and is now walking side-to-side with EC2 in terms of popularity and adaptation.
To know the major differences between AWS Lambda and CE2 please visit https://www.whizlabs.com/blog/aws-lambda-vs-ec2/
Message Queue 가용성, 신뢰성을 위한 RabbitMQ Server, Client 구성
Message Queue 가용성, 신뢰성을 위한 RabbitMQ Server, Client 구성Yoonjeong Kwon RabbitMQ Concept & How to work
High Availability in RabbitMQ
- Clustering & Mirrored Queue
HAProxy
Spring Cloud Stream Sample & configuration: Spring Boot App with Rabbit
Kafka Retry and DLQ
Kafka Retry and DLQGeorge Teo The document describes Uber's use of Kafka for reliable messaging. Kafka is used for inter-service messaging, stream processing, database changelog transport, data ingestion, and logging. It provides ordered, partitioned streaming and unordered queueing semantics. The summary describes Uber's consumer wrapper that implements features like acknowledgments, redelivery, delays between retries, dead letter queues, competing consumers, and multi-datacenter failover to support reliable messaging on top of Kafka.
Data profiling in Apache Calcite
Data profiling in Apache CalciteDataWorks Summit Query optimizers and people have one thing in common: the better they understand their data, the better they can do their jobs. Optimizing queries is hard if you don't have good estimates for the sizes of the intermediate join and aggregate results. Data profiling is a technique that scans data, looking for patterns within the data such as keys, functional dependencies, and correlated columns. These richer statistics can be used in Apache Calcite's query optimizer, and the projects that use it, such as Apache Hive, Phoenix and Drill. We describe how we built a data profiler as a table function in Apache Calcite, review the recent research and algorithms that made it possible, and show how you can use the profiler to improve the quality of your data.
HDFS Overview
HDFS OverviewJEONGPHIL HAN Apache Hadoop의 HDFS에 대해 정리한 자료입니다.
틀린 점이 있다면 메일로 주시면 감사하겠습니다.
[email protected]
Best practices for MySQL High Availability Tutorial
Best practices for MySQL High Availability TutorialColin Charles Given at O'Reilly Velocity London in October 2017, this tutorial focuses on the current best practices for MySQL High Availability.
Kafka tiered-storage-meetup-2022-final-presented
Kafka tiered-storage-meetup-2022-final-presentedSumant Tambe Kafka Tiered Storage separates compute and data storage in two independently scalable layers. Uber's Kafka Improvement Proposal (KIP) #405 describes two-tiered storage, which is a major step towards cloud-native Kafka. It stores the most recent data locally and offloads older data to a remote storage service. Operationally, the benefit is faster routine cluster maintenance activities. In Linkedin, Kafka tiered storage is strongly desired to reduce the cost of running Kafka in the Azure cloud environment. As KIP-405 does not dictate the implementation of remote storage substrate, Linkedin's choice for tiering Kafka in Azure deployments is the Azure Blob Service. This presentation will begin with the motivation behind Linkedin efforts to adopt Kafka Tiered Storage. Next, the architecture of KIP-405 will be discussed. Finally, the Remote Storage Manager for Azure Blobs, which is a work-in-progress, will be presented.
Video: https://youtu.be/V5gaBE5CMwg?t=1387
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent Speaker: Matthias J. Sax, Software Engineer, Confluent
ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries.
Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems.
https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRed Hat Developers Have you ever wondered what the relative differences are between two of the more popular open source, in-memory data stores and caches? In this session, we will describe those differences and, more importantly, provide live demonstrations of the key capabilities that could have a major impact on your architectural Java application designs.
Viewers also liked (20)
Experience with Kafka & Storm
Experience with Kafka & StormOtto Mok This document discusses AcuityAds' use of Apache Kafka and Storm for processing over 10 billion daily ad impressions. It describes their architecture with Kafka used to ingest bid request data from multiple sources into partitions. Storm topologies read from Kafka and processed the data to calculate metrics like daily impressions by site. Initial issues included unbalanced Kafka partitions and low Storm uptime due to exceptions. Future improvements involved upgrading versions and adding monitoring capabilities.
Non blocking io with netty
Non blocking io with nettyZauber This document discusses non-blocking I/O and the traditional blocking I/O approach for building servers. The traditional approach uses one thread per connection, blocking I/O, and a simple programming model. However, this can cause issues like shared state between clients, synchronization problems, inability to prioritize clients, difficulty scaling to thousands of connections, and challenges with persistent connections. The document explores using non-blocking I/O with Netty as an alternative.
Netty: asynchronous data transfer
Netty: asynchronous data transferVictor Cherkassky Netty is an asynchronous event-driven network application framework for Java. It provides an asynchronous model with non-blocking I/O based on Java NIO. To create an HTTP server with Netty, one sets up thread pools for handling connections and requests, provides a ChannelFactory and PipelineFactory, and puts it all together with a Bootstrap. The server can then be started to listen for connections. Custom handlers can be added to the pipeline to process requests. Channels are gathered and closed to release resources when the server is shut down. TCP options like keepAlive and noDelay can be configured but their effects should be measured for each application.
Netty from the trenches
Netty from the trenchesJordi Gerona Netty is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. AND IT'S TRUE!
In this talk given at JBCNConf 2015 in Barcelona, we will see how we use Netty at Trovit since 2013, what brought to us and how it opened our minds. We will share tips that helped us to learn more about Netty, some performance tricks and all things that worked for us.
Building scalable network applications with Netty (as presented on NLJUG JFal...
Building scalable network applications with Netty (as presented on NLJUG JFal...Jaap ter Woerds The presentation I gave on creating server application with Netty, including an example of how it is used to power XMS the mobile messaging platform of eBuddy.
Example code is on github: https://github.com/jaapterwoerds/jfall-netty4
More information on eBuddy: xms.me and tech.ebuddy.com
Zero-Copy Event-Driven Servers with Netty
Zero-Copy Event-Driven Servers with NettyDaniel Bimschas Netty is a Java framework that provides tools for developing high performance and event-driven network applications. It uses non-blocking I/O and zero-copy techniques to minimize overhead and maximize throughput and scalability. Netty provides buffers, codecs, pipelines and handlers that allow building applications as a stack of processing layers. Example applications include a discard server and an HTTP file server that demonstrate Netty's core features and event-driven architecture.
Scaling Apache Storm - Strata + Hadoop World 2014
Scaling Apache Storm - Strata + Hadoop World 2014P. Taylor Goetz Scaling Apache Storm: Cluster Sizing and Performance Optimization
Slides from my presentation at Strata + Hadoop World 2014
Yahoo compares Storm and Spark
Yahoo compares Storm and SparkChicago Hadoop Users Group Bobby Evans and Tom Graves, the engineering leads for Spark and Storm development at Yahoo will talk about how these technologies are used on Yahoo's grids and reasons why to use one or the other.
Bobby Evans is the low latency data processing architect at Yahoo. He is a PMC member on many Apache projects including Storm, Hadoop, Spark, and Tez. His team is responsible for delivering Storm as a service to all of Yahoo and maintaining Spark on Yarn for Yahoo (Although Tom really does most of that work).
Tom Graves a Senior Software Engineer on the Platform team at Yahoo. He is an Apache PMC member on Hadoop, Spark, and Tez. His team is responsible for delivering and maintaining Spark on Yarn for Yahoo.
Hadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureP. Taylor Goetz Storm is an open-source distributed real-time computation system. It uses a distributed messaging system to reliably process streams of data. The core abstractions in Storm are spouts, which are sources of streams, and bolts, which are basic processing elements. Spouts and bolts are organized into topologies which represent the flow of data. Storm provides fault tolerance through message acknowledgments and guarantees exactly-once processing semantics. Trident is a high-level abstraction built on Storm that supports operations like aggregations, joins, and state management through its micro-batch oriented and stream-based API.
Apache Storm 0.9 basic training - Verisign
Apache Storm 0.9 basic training - VerisignMichael Noll Apache Storm 0.9 basic training (130 slides) covering:
1. Introducing Storm: history, Storm adoption in the industry, why Storm
2. Storm core concepts: topology, data model, spouts and bolts, groupings, parallelism
3. Operating Storm: architecture, hardware specs, deploying, monitoring
4. Developing Storm apps: Hello World, creating a bolt, creating a topology, running a topology, integrating Storm and Kafka, testing, data serialization in Storm, example apps, performance and scalability tuning
5. Playing with Storm using Wirbelsturm
Audience: developers, operations, architects
Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/
Verisign is a global leader in domain names and internet security.
Tools mentioned:
- Wirbelsturm (https://github.com/miguno/wirbelsturm)
- kafka-storm-starter (https://github.com/miguno/kafka-storm-starter)
Blog post at:
http://www.michael-noll.com/blog/2014/09/15/apache-storm-training-deck-and-tutorial/
Many thanks to the Twitter Engineering team (the creators of Storm) and the Apache Storm open source community!
Netty Notes Part 3 - Channel Pipeline and EventLoops
Netty Notes Part 3 - Channel Pipeline and EventLoopsRick Hightower Learning more about Netty helps me understand Vert.x better. Netty in Action is a great book. The threading model of Netty is very important to understanding event loops and reactive programming.
Resource Aware Scheduling in Apache Storm
Resource Aware Scheduling in Apache StormDataWorks Summit/Hadoop Summit This document provides an overview of resource aware scheduling in Apache Storm. It discusses the challenges of scheduling Storm topologies at Yahoo scale, including increasing heterogeneous clusters, low cluster utilization, and unbalanced resource usage. It then introduces the Resource Aware Scheduler (RAS) built for Storm, which allows fine-grained resource control and isolation for topologies through APIs and cgroups. Key features of RAS include pluggable scheduling strategies, per user resource guarantees, and topology priorities. Experimental results from Yahoo Storm clusters show significant improvements to throughput and resource utilization with RAS. The talk concludes with future work on improved scheduling strategies and real-time resource monitoring.
Storm: distributed and fault-tolerant realtime computation
Storm: distributed and fault-tolerant realtime computationnathanmarz Storm is a distributed real-time computation system that provides guaranteed message processing, horizontal scalability, and fault tolerance. It allows users to define data processing topologies and submit them to a Storm cluster for distributed execution. Spouts emit streams of tuples that are processed by bolts. Storm tracks processing to ensure reliability and replays failed tasks. It provides tools for deployment, monitoring, and optimization of real-time data processing.
Realtime Analytics with Storm and Hadoop
Realtime Analytics with Storm and HadoopDataWorks Summit This document discusses how to use Storm and Hadoop together to enable real-time and batch processing of large datasets. It describes using Hadoop to precompute batch views of data, and Storm to incrementally update real-time views as new data streams in. This allows for low-latency queries by combining precomputed batch views with real-time views that compensate for recent data not yet absorbed into the batch views.
Apache storm vs. Spark Streaming
Apache storm vs. Spark StreamingP. Taylor Goetz Slides for an upcoming talk about Apache Storm and Spark Streaming.
This is a draft and is subject to change. Comments welcome.
Nettyらへん
NettyらへんGo Tanaka The document discusses Netty, an asynchronous event-driven network application framework for Java. It provides high-performance network programming by leveraging Java NIO and an event-driven architecture. Netty allows building high levels of concurrency with low latency. It includes features like HTTP servers, protocol buffers, and handlers for building custom server-side applications.
Notes on Netty baics
Notes on Netty baicsRick Hightower Some basic notes about Netty Fundamentals using Java.
Focus on concepts and how they map to top level classes in the Netty API.
55 New Features in JDK 9
55 New Features in JDK 9Simon Ritter A whirlwind tour of some (but not all) of the features being added (and some that are being removed) from JDK 9.
Ad
Similar to Real time analytics with Netty, Storm, Kafka (20)
Java one2013
Java one2013Aleksei Kornev This document discusses distributed and highly available server applications built in Java and Scala. It describes an architecture using lightweight microservices called Talkbits that communicate over the Finagle distributed RPC framework. Key principles for Talkbits include stateless services, service discovery with Zookeeper, and functional composition of RPC calls. The document also covers configuration, deployment, logging, metrics collection and monitoring of the distributed system using tools like Loggly, CodaHale, Jolokia, Datadog, and Fabric.
Io t data streaming
Io t data streamingratthaslip ranokphanuwat - IoT devices generate large streams of data that need to be collected and processed in real-time. MQTT and Kafka are common protocols for collecting IoT data streams. MQTT is lightweight but lacks scalability while Kafka is highly scalable.
- Stream processing platforms like Flink, Storm and Spark can be used to analyze the IoT data streams. Flink supports both batch and stream processing while Storm is best for low-latency streaming. Spark is better for machine learning on streams.
- An example use case is real-time equipment monitoring in a factory where IoT sensors stream data to Kafka which is then processed by Flink to detect abnormalities and enable predictive maintenance. Performance is evaluated based on latency and
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Timothy Spann Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Spark
DevNexus 2022 Atlanta
https://devnexus.com/presentations/7150/
This talk is a quick overview of the How, What and WHY of Apache Pulsar, Apache Flink and Apache NiFi. I will show you how to design event-driven applications that scale the cloud native way.
This talk was done live in person at DevNexus across from the booth in room 311
Tim Spann
Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar, Apache Flink, Flink SQL, Apache NiFi, MiniFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
OpenStack Neutron Havana Overview - Oct 2013
OpenStack Neutron Havana Overview - Oct 2013Edgar Magana Presentation about OpenStack Neutron Overview presented during three meet-ups in NYC, Connecticut and Philadelphia during October 2013 by Edgar Magana from PLUMgrid
MINA2 (Apache Netty)
MINA2 (Apache Netty)ducquoc_vn Apache MINA2 is a Java open-source network application framework that provides an asynchronous, non-blocking infrastructure for developing network applications. It features reusable and extensible components like IoSession for connections, IoService for servers/clients, IoFilters for cross-cutting concerns, and ProtocolCodecFilters for encoding network protocols. MINA supports various network protocols and data types and provides an elegant way to build scalable and manageable network applications.
B.Eng-Final Year Project interim-report
B.Eng-Final Year Project interim-reportAkash Rajguru This interim report summarizes the progress made on a project to develop a Java-based intrusion detection system tool. The tool will monitor network traffic by capturing packets from the host machine interface using the Jpcap library. Work accomplished so far includes researching IDS and honeypot technologies, learning to use third party Java libraries, developing a GUI to list interfaces and select one for packet capture, capturing live packet streams, port scanning the selected interface, and storing captured packets to a file. Future work includes improving the GUI, connecting to a NoSQL database, analyzing log files, and implementing honeypots and testing with JUnit. The goal is to create a complete IDS with functionality like interface selection, packet capture, port blocking rules
Defining the stack for service delivery models and interoperability in the in...
Defining the stack for service delivery models and interoperability in the in...ieeepondy Defining the stack for service delivery models and interoperability in the internet of things a practical case with open iot vdk
+91-9994232214,8144199666, [email protected],
www.projectsieee.com, www.ieee-projects-chennai.com
IEEE PROJECTS 2015-2016
-----------------------------------
Contact:+91-9994232214,+91-8144199666
Email:[email protected]
ieee projects chennai, ieee projects bangalore
Big mountain data and dev conference apache pulsar with mqtt for edge compu...
Big mountain data and dev conference apache pulsar with mqtt for edge compu...Timothy Spann This document provides an overview and summary of Apache Pulsar with MQTT for edge computing. It discusses how Pulsar is an open-source, cloud-native distributed messaging and streaming platform that supports MQTT and other protocols. It also summarizes Pulsar's key capabilities like data durability, scalability, geo-replication, and unified messaging model. The document includes diagrams showcasing Pulsar's publish-subscribe model and different subscription modes. It demonstrates how Pulsar can be used with edge devices via protocols like MQTT and how streams of data from edge can be processed using connectors, functions and SQL.
Cloud computing and OpenStack
Cloud computing and OpenStackEdgar Magana This document provides an overview of cloud computing and OpenStack. It defines cloud computing and its components, service models, and benefits. OpenStack is introduced as an open source cloud management platform that controls compute, storage, and networking resources across a datacenter. Key OpenStack services like Nova, Neutron, Glance, Swift, and Keystone are summarized, along with their roles and basic functionality. The document concludes with information on how to get involved in the OpenStack community through contributions and using DevStack for development.
OpenStack Introduction
OpenStack Introductionopenstackindia OpenStack is an open source cloud computing platform that consists of a series of related projects that control large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface. It is developed as an open source project by an international community of developers and corporate sponsors and supports both private and public cloud deployments. Major components include compute (Nova), object storage (Swift), image service (Glance), networking (Quantum), and an identity service (Keystone).
Open stack nova reverse engineer
Open stack nova reverse engineerVietnam Open Infrastructure User Group This document provides an overview of OpenStack Nova's architecture and code structure. It begins with an introduction to Nova's mission to provide scalable on-demand access to compute resources. It then covers Nova's core components, data flows, and code organization. Key aspects summarized include Nova's use of Cells to partition compute nodes, its facade design pattern for APIs, and its reliance on common OpenStack libraries like Oslo and Stevedore for configuration, logging, and extensibility.
20141111_SOS3_Gallo
20141111_SOS3_GalloAndrea Gallo Linaro has enabled server class workloads for ARM servers by optimizing key open source software. They have contributed patches to projects like the Linux kernel, KVM, Xen, OpenJDK, Hadoop, and OpenStack. This has allowed OpenStack to run on ARMv8 hardware, with all applicable Tempest tests passing. Linaro is also working on optimizations for server workloads like the LAMP stack, HDFS, and HipHop JIT. Their efforts are helping to accelerate ARM's adoption in the server market.
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computingTimothy Spann Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
by Timothy Spann
Developer Advocate
StreamNative
datainmotion.dev
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...Takashi Torii 1) The Okinawa Open Laboratory is a research institute founded in 2013 by NEC, NTT Communications, and IIGA based in Okinawa, Japan that focuses on research and development of advanced ICT technologies like SDN and OpenStack.
2) The Laboratory operates a testbed platform that provides automated provisioning of bare metal servers and SDN switches to support verification of new technologies through use cases and reference models.
3) Key technologies being explored include an SDN application to control hybrid bare metal and virtual network topologies, a federated cloud platform for cloud native applications, and a framework for network functions virtualization and service function chaining.
Introduction to Orchestration and DevOps with OpenStack
Introduction to Orchestration and DevOps with OpenStackAbderrahmane TEKFI I would like to thank all who participates in the webinar, it was a great pleasure to share and contribute,
Below are the links to the record of the Webinar,
All the Webinar:
Just the Demo:
you can also find all the slides the HEAT template file, the CLI and all the materials used in this webinar here:
The OpenStack VM all-in-one: https://www.dropbox.com/s/501ul31o6ilnmv3/coa-aio-newton.ova?dl=0
All the materials: https://drive.google.com/drive/folders/1dTSe4n2m3VoevIHZGT_q8uZIV7_f9ZJt?usp=sharing
Thanks to Racim and to the ELIANIS TECHNOLOGIES team.
Special thanks to our REDHAT ARCHITECT Sir. Djelloul Bouida for attending the webinar and all our group member.
For those who didn't join our Group, here the link to our Group on Facebook: https://www.facebook.com/groups/475301352862998/
Programming IoT Gateways with macchina.io
Programming IoT Gateways with macchina.ioGünter Obiltschnig This article shows how to program IoT Gateway devices in macchina.io, a new open source software platform.
Implementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdfJose Manuel Ortega Candel No production system is complete without a way to monitor it. In software, we define observability as the ability to understand how our system is performing. This talk dives into capabilities and tools that are recommended for implementing observability when running K8s in production as the main platform today for deploying and maintaining containers with cloud-native solutions.
We start by introducing the concept of observability in the context of distributed systems such as K8s and the difference with monitoring. We continue by reviewing the observability stack in K8s and the main functionalities. Finally, we will review the tools K8s provides for monitoring and logging, and get metrics from applications and infrastructure.
Between the points to be discussed we can highlight:
-Introducing the concept of observability
-Observability stack in K8s
-Tools and apps for implementing Kubernetes observability
-Integrating Prometheus with OpenMetrics
SnapLogic- iPaaS (Elastic Integration Cloud and Data Integration) 
SnapLogic- iPaaS (Elastic Integration Cloud and Data Integration) Surendar S Especially this document provide very useful and meaningful concepts about SnapLogic. Also this document will be more useful for beginner/intermediate level SnapLogic learners.
Cloud lunch and learn real-time streaming in azure
Cloud lunch and learn real-time streaming in azureTimothy Spann Cloud lunch and learn real-time streaming in azure
Apache pulsar is an open source, cloud-native distributed messaging and streaming platform.
Distributed & Highly Available server applications in Java and Scala
Distributed & Highly Available server applications in Java and ScalaMax Alexejev This document summarizes a presentation about distributed and highly available server applications in Java and Scala. It discusses the Talkbits architecture, which uses lightweight SOA principles with stateless edge services and specialized systems to manage state. The presentation describes using the Finagle library as a distributed RPC framework with Apache Zookeeper for service discovery. It also covers configuration, deployment, monitoring and logging of services using tools like SLF4J, Logback, CodaHale metrics, Jolokia, Fabric, and Datadog.
Ad
More from Trieu Nguyen (20)
Building Your Customer Data Platform with LEO CDP in Travel Industry.pdf
Building Your Customer Data Platform with LEO CDP in Travel Industry.pdfTrieu Nguyen 1. The document outlines the Chief Platform Engineer's background and introduces LEO CDP, a customer data platform for the travel industry.
2. It discusses 5 challenges companies face related to customer growth, journeys, data platforms, communication and understanding customers with big data.
3. A case study shows how LEO CDP can be used to create a customer journey map for a travel agency, including personalized promotions and offers sent via email.
Building Your Customer Data Platform with LEO CDP - Spa and Hotel Business
Building Your Customer Data Platform with LEO CDP - Spa and Hotel BusinessTrieu Nguyen 1) Five challenges we are tackling today
2) How to build your solutions with LEO CDP
3) Case study in Spa and Hotel Business
Building Your Customer Data Platform with LEO CDP 
Building Your Customer Data Platform with LEO CDP Trieu Nguyen Summary of content:
1) Key problems of data-driven business
2) How to solve problems with USPA mindset
3) How to build your solutions with LEO CDP
How to track and improve Customer Experience with LEO CDP
How to track and improve Customer Experience with LEO CDPTrieu Nguyen This document discusses how to track and improve customer experience using LEO CDP. It begins by explaining why measuring customer experience is important, then introduces four key metrics: Customer Feedback Score, Customer Effort Score, Customer Satisfaction Score, and Net Promoter Score. It describes using journey maps to manage customer experience data and visualize the customer journey. Finally, it presents LEO CDP as a software solution for collecting customer experience data, building surveys, and generating reports to gain insights to improve products, services, and the overall customer experience.
[Notes] Customer 360 Analytics with LEO CDP![[Notes] Customer 360 Analytics with LEO CDP](https://cdn.slidesharecdn.com/ss_thumbnails/notescustomer360analyticswithleocdp-220126053232-thumbnail.jpg?width=560&fit=bounds)
[Notes] Customer 360 Analytics with LEO CDPTrieu Nguyen Part 1: Why should every business need to deploy a CDP ?
1. Big data is the reality of business today
2. What are technologies to manage customer data ?
3. The rise of first-party data and new technologies for Digital Marketing
4. How to apply USPA mindset to build your CDP for data-driven business
Part 2: How to use LEO CDP for your business
1. Core functions of LEO CDP for marketers and IT managers
2. Data Unification for Customer 360 Analytics
3. Data Segmentation
4. Customer Personalization
5. Customer Data Activation
Part 3: Case study in O2O Retail and Ecommerce
1. How to build customer journey map for ecommerce and retail
2. How to do customer analytics to find ideal customer profiles
The ideal customer profile in a B2B context
The ideal customer profile in a B2C context
3. Manage product catalog for customer personalization
4. Monitoring Data of Customer Experience (CX Analytics)
CX Data Flow
CX Rating plugin is embedded in the website, to collect feedback data
An overview of CX Report
A CX Report in a customer profile
5. Monitoring data with real-time event tracking reports
Event Data Flow
Summary Event Data Report
Event Data Report in a Customer Profile
Part 4: How to setup an instance of LEO CDP for free
1. Technical architecture
2. Server infrastructure
3. Setup middlewares: Nginx, ArangoDB, Redis, Java and Python
Network requirements
Software requirements for new server
ArangoDB
Nginx Proxy
SSL for Nginx Server
Java 8 JVM
Redis
Install Notes for Linux Server
Clone binary code for new server
Set DNS hosts for LEO CDP workers
4. Setup data for testing and system verification
Part 5: Summary all key ideas
Leo CDP - Pitch Deck
Leo CDP - Pitch DeckTrieu Nguyen Why should you invest in LEO CDP ?
Purpose: Big data and AI democracy for SMEs companies
Problem: Customer Analytics and Customer Personalization
Solutions: CDP + CX + Personalization Engine
Product demo: LEO CDP for Ecommerce and Fintech
Business model: Freemium → Ecosystem → Subscription
Market size: 20 billion USD in 2026 and CAGR 34.6%
Differentiation: cloud-native software
Go-to-market approach: Community → Free → Paid
Team: 1 full-stack dev, 1 data scientist and 12,000 fans of BigDataVietnam.org Community
Need 150,000 USD for scaling business (you get 20% share)
LEO CDP - What's new in 2022
LEO CDP - What's new in 2022Trieu Nguyen The document outlines new features and updates for 2022 from USPA Technology Company, including a new dedicated dashboard for CMOs, updated UI for Customer 360 Insights, and a focus on data-driven business processes and digital marketing in B2B through standardizing data-driven processes and focusing on customer insights.
Lộ trình triển khai LEO CDP cho ngành bất động sản
Lộ trình triển khai LEO CDP cho ngành bất động sảnTrieu Nguyen 1) Hiểu bài toán số hoá trải nghiệm khách hàng
2) Nghiên cứu giải pháp LEO CDP
3) Lộ trình triển khai
Phát triển / số hoá điểm chạm khách hàng
Xây dựng bản đồ hành trình khách hàng
Định nghĩa các metrics và KPI quan trọng
Xây dựng web portal và mobile data hub
Xây dựng kế hoạch Digital Marketing
Triển khai CDP và Marketing Automation
Xây dựng đội Analytics để phân tích dữ liệu
Why is LEO CDP important for digital business ?
Why is LEO CDP important for digital business ?Trieu Nguyen The slide for my startup to get funding
1) Current problems of digital business
2) My solution to solve problems
3) Demo for key functions of Leo CDP
From Dataism to Customer Data Platform
From Dataism to Customer Data PlatformTrieu Nguyen 1) How to think in the age of Dataism with LEO CDP ?
2) Why is Dataism for human, business and society ?
3) How should LEO Customer Data Platform (LEO CDP) work ?
4) How to use LEO CDP for your business ?
Data collection, processing & organization with USPA framework
Data collection, processing & organization with USPA frameworkTrieu Nguyen 1) How to think in the age of Dataism with USPA framework ?
2) How to collect customer data
3) Data Segmentation Processing for flexibility and scalability
4) Data Organization for personalization and business activation
Part 1: Introduction to digital marketing technology
Part 1: Introduction to digital marketing technologyTrieu Nguyen This document provides an overview of a mini-course on data-driven marketing using the USPA framework presented by Trieu Nguyen. It includes biographical information about Trieu Nguyen's background and experience in big data projects, machine learning, and digital marketing roles. The document also outlines the topics that will be covered in the mini-course, including digital media models, search engine marketing, social media marketing, advertising technology, customer data platforms, and case studies. Key terms like omnichannel strategy, customer experience strategy, and artificial intelligence strategies for marketing are also defined.
Why is Customer Data Platform (CDP) ?
Why is Customer Data Platform (CDP) ?Trieu Nguyen Transform your marketing and sales capabilities with Big Data and A.I
1) Why is Customer Data Platform (CDP) ?
Case study: Enhancing the revenue of your restaurant with CDP and mobile app marketing
Question: Why can CDP disrupt business model for restaurant industry (B2C) ?
2) How would CDP work in practice ?
Introducing USPA.tech as logical framework for implementing CDP in practice
How Can a Customer Data Platform Enhance Your Account-Based Marketing Strategy (B2B) ?
3) How can we implement CDP for business?
Introducing the CDP as customer-first marketing platform for all industries (my key idea in this slide)
How to build a Personalized News Recommendation Platform
How to build a Personalized News Recommendation PlatformTrieu Nguyen This document discusses how to build a personalized news recommendation platform. It explains that recommendation systems are needed to retain users, increase traffic, and improve the content experience. It describes popular techniques like collaborative filtering, content-based filtering, and hybrid systems. Specifically, it outlines a case study using a USPA framework with real social news data. Key factors for a news recommendation system are discussed like novelty, user history, and location. The document also provides a simple example of building a recommendation engine with Apache Spark.
How to grow your business in the age of digital marketing 4.0
How to grow your business in the age of digital marketing 4.0Trieu Nguyen 1. The document discusses how businesses can grow in the digital marketing age using technologies like cloud services, big data, AI, and headless CMS platforms.
2. It introduces LeoCloudCMS as a headless API CMS that is built for digital marketing 4.0 and can run scalably on cloud computing.
3. The key idea is to think of your entire business as a "box" and use LeoCloudCMS to attract internet users into the box and offer valuable services.
Video Ecosystem and some ideas about video big data
Video Ecosystem and some ideas about video big dataTrieu Nguyen Introduction to Video Ecosystem Mind Map
Video Streaming Platform
Video Ad Tech Platform
Video Player Platform
Video Content Distribution Platform
Video Analytics Platform
Summary of key ideas
Q & A
Concepts, use cases and principles to build big data systems (1)
Concepts, use cases and principles to build big data systems (1)Trieu Nguyen 1) Introduction to the key Big Data concepts
1.1 The Origins of Big Data
1.2 What is Big Data ?
1.3 Why is Big Data So Important ?
1.4 How Is Big Data Used In Practice ?
2) Introduction to the key principles of Big Data Systems
2.1 How to design Data Pipeline in 6 steps
2.2 Using Lambda Architecture for big data processing
3) Practical case study : Chat bot with Video Recommendation Engine
4) FAQ for student
Open OTT - Video Content Platform
Open OTT - Video Content PlatformTrieu Nguyen This document discusses open over-the-top (OTT) video content platforms. It defines OTT as streaming media distributed directly over the internet bypassing traditional distribution methods. The document then covers OTT market drivers and business models. It examines the most popular OTT platform in Vietnam and challenges for successful OTT platforms including scalability, content acquisition and management, audience engagement, and business models. Finally, it proposes a modular technical architecture for an open OTT video platform using open source technologies.
Apache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Apache Hadoop and Spark: Introduction and Use Cases for Data AnalysisTrieu Nguyen This document provides an introduction to Apache Hadoop and Spark for data analysis. It discusses the growth of big data from sources like the internet, science, and IoT. Hadoop is introduced as providing scalability on commodity hardware to handle large, diverse data types with fault tolerance. Key Hadoop components are HDFS for storage, MapReduce for processing, and HBase for non-relational databases. Spark is presented as improving on MapReduce by using in-memory computing for iterative jobs like machine learning. Real-world use cases of Spark at companies like Uber, Pinterest, and Netflix are briefly described.
Introduction to Recommendation Systems (Vietnam Web Submit)
Introduction to Recommendation Systems (Vietnam Web Submit)Trieu Nguyen 1) Why do we need recommendation systems ?
2) How can we think with recommendation systems ?
3) How can we implement a recommendation system with open source technologies ?
RFX framework https://github.com/rfxlab
Apache Kafka: https://kafka.apache.org
Apache Spark: https://spark.apache.org
Recently uploaded (20)
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell With expertise in data architecture, performance tracking, and revenue forecasting, Andrew Marnell plays a vital role in aligning business strategies with data insights. Andrew Marnell’s ability to lead cross-functional teams ensures businesses achieve sustainable growth and operational excellence.
Technology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data AnalyticsInData Labs At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including:
-Artificial Intelligence Market Overview
-Strategies for AI Adoption in 2025
-Anticipated drivers of AI adoption and transformative technologies
-Benefits of AI and Big data for your business
-Tips on how to prepare your business for innovation
-AI and data privacy: Strategies for securing data privacy in AI models, etc.
Download your free copy nowand implement the key findings to improve your business.
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan This is a Quick Research Guide (QRG).
QRGs include the following:
- A brief, high-level overview of the QRG topic.
- A milestone timeline for the QRG topic.
- Links to various free online resource materials to provide a deeper dive into the QRG topic.
- Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic.
QRGs planned for the series:
- Artificial Intelligence QRG
- Quantum Computing QRG
- Big Data Analytics QRG
- Spacecraft Guidance, Navigation & Control QRG (coming 2026)
- UK Home Computing & The Birth of ARM QRG (coming 2027)
Any questions or comments?
- Please contact Arthur Morgan at [email protected].
100% human made.
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Splunk Security Update
Sprecher: Marcel Tanuatmadja
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1 Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark?
At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍
Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀
Mastering Advance Window Functions in SQL.pdf
Mastering Advance Window Functions in SQL.pdfSpiral Mantra How well do you really know SQL?📊
.
.
If PARTITION BY and ROW_NUMBER() sound familiar but still confuse you, it’s time to upgrade your knowledge
And you can schedule a 1:1 call with our industry experts: https://spiralmantra.com/contact-us/ or drop us a mail at [email protected]
Vaibhav Gupta BAML: AI work flows without Hallucinations
Vaibhav Gupta BAML: AI work flows without Hallucinationsjohn409870 Shipping Agents
Vaibhav Gupta
Cofounder @ Boundary
in/vaigup
boundaryml/baml
Imagine if every API call you made
failed only 5% of the time
boundaryml/baml
Imagine if every LLM call you made
failed only 5% of the time
boundaryml/baml
Imagine if every LLM call you made
failed only 5% of the time
boundaryml/baml
Fault tolerant systems are hard
but now everything must be
fault tolerant
boundaryml/baml
We need to change how we
think about these systems
Aaron Villalpando
Cofounder @ Boundary
Boundary
Combinator
boundaryml/baml
We used to write websites like this:
boundaryml/baml
But now we do this:
boundaryml/baml
Problems web dev had:
boundaryml/baml
Problems web dev had:
Strings. Strings everywhere.
boundaryml/baml
Problems web dev had:
Strings. Strings everywhere.
State management was impossible.
boundaryml/baml
Problems web dev had:
Strings. Strings everywhere.
State management was impossible.
Dynamic components? forget about it.
boundaryml/baml
Problems web dev had:
Strings. Strings everywhere.
State management was impossible.
Dynamic components? forget about it.
Reuse components? Good luck.
boundaryml/baml
Problems web dev had:
Strings. Strings everywhere.
State management was impossible.
Dynamic components? forget about it.
Reuse components? Good luck.
Iteration loops took minutes.
boundaryml/baml
Problems web dev had:
Strings. Strings everywhere.
State management was impossible.
Dynamic components? forget about it.
Reuse components? Good luck.
Iteration loops took minutes.
Low engineering rigor
boundaryml/baml
React added engineering rigor
boundaryml/baml
The syntax we use changes how we
think about problems
boundaryml/baml
We used to write agents like this:
boundaryml/baml
Problems agents have:
boundaryml/baml
Problems agents have:
Strings. Strings everywhere.
Context management is impossible.
Changing one thing breaks another.
New models come out all the time.
Iteration loops take minutes.
boundaryml/baml
Problems agents have:
Strings. Strings everywhere.
Context management is impossible.
Changing one thing breaks another.
New models come out all the time.
Iteration loops take minutes.
Low engineering rigor
boundaryml/baml
Agents need
the expressiveness of English,
but the structure of code
F*** You, Show Me The Prompt.
boundaryml/baml
<show don’t tell>
Less prompting +
More engineering
=
Reliability +
Maintainability
BAML
Sam
Greg Antonio
Chris
turned down
openai to join
ex-founder, one
of the earliest
BAML users
MIT PhD
20+ years in
compilers
made his own
database, 400k+
youtube views
Vaibhav Gupta
in/vaigup
[email protected]
boundaryml/baml
Thank you!
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john Analyze the growth of meme coins from mere online jokes to potential assets in the digital economy. Explore the community, culture, and utility as they elevate themselves to a new era in cryptocurrency.
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts!
📕 Agenda
Welcome & Introductions
Orchestrator API Overview
Exploring the Swagger Interface
Test Manager API Highlights
Streamlining Automation & Testing with APIs (Demo)
Q&A and Open Discussion
Perfect for developers, testers, and automation enthusiasts!
👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/
This session streamed live on April 29, 2025, 18:00 CET.
Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency.
This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data.
Attendees will learn:
- Consumer awareness around data brokers and what consumers are doing to limit data collection
- How businesses assess third-party vendors and their consent management operations
- Where business preparedness needs improvement
- What these trends mean for the future of privacy governance and public trust
This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.
Cybersecurity Identity and Access Solutions using Azure AD
Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ Cybersecurity Identity and Access Solutions using Azure AD
tecnologias de las primeras civilizaciones.pdf
tecnologias de las primeras civilizaciones.pdffjgm517 descaripcion detallada del avance de las tecnologias en mesopotamia, egipto, roma y grecia.
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityriccardosl1 Cyber awareness training educates employees on risk associated with internet and malicious emails
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/
HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar.
Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten.
In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich
- Zugriff auf die Konsole
- Auffinden und Interpretieren von Protokolldateien
- Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS)
- Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien
- Nutzung der Client Clocking-Funktion
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
Web and Graphics Designing Training in Rajpura
Web and Graphics Designing Training in RajpuraErginous Technology Web & Graphics Designing Training at Erginous Technologies in Rajpura offers practical, hands-on learning for students, graduates, and professionals aiming for a creative career. The 6-week and 6-month industrial training programs blend creativity with technical skills to prepare you for real-world opportunities in design.
The course covers Graphic Designing tools like Photoshop, Illustrator, and CorelDRAW, along with logo, banner, and branding design. In Web Designing, you’ll learn HTML5, CSS3, JavaScript basics, responsive design, Bootstrap, Figma, and Adobe XD.
Erginous emphasizes 100% practical training, live projects, portfolio building, expert guidance, certification, and placement support. Graduates can explore roles like Web Designer, Graphic Designer, UI/UX Designer, or Freelancer.
For more info, visit erginous.co.in , message us on Instagram at erginoustechnologies, or call directly at +91-89684-38190 . Start your journey toward a creative and successful design career today!
Real time analytics with Netty, Storm, Kafka
- 1. Real-time Analytics with Netty, Apache Kafka and Storm Case study with “lambda architecture” http://nguyentantrieu.info Update: 07/06/2013
- 2. Agenda 1. Overview Architecture 2. Log HTTP-Handler and producer: Netty 4 3. Kafka 0.8 (Stream Data Log Storage) 4. Storm Analytics Cluster
- 4. Concept Flow
- 5. Concept Flow JavaScript Tracking Mobile SDK Http Log Server Kafka
- 6. S2 HTTP Log Server netty framework 4
- 7. Netty.io Netty is a non-blocking I/O (NIO) client-server framework for the development of Java network applications such as protocol servers and clients. The asynchronous event-driven network application framework and tools are used to simplify network programming such as TCP and UDP socket servers.[2] Netty includes an implementation of the reactor pattern of programming. http://en.wikipedia.org/wiki/Netty_(software) http://nguyentantrieu.info/blog/backend-system-with-netty-io
- 8. Apache Kafka (version 0.8) https://cwiki.apache.org/confluence/display/KAFKA/Index
- 9. In Production ● Clustering ( 4 nodes) ● Partitions ○ user-activity: 24 partitions ● Producer and consumer are replication aware
- 10. Core Producer Class (S2 HTTP Log Server)
- 12. The Storm Topology Kafka Cluster topic: user-activity Tokenizer Bolt Parser Bolt Aggregate Bolt Redis Statistics Bolt Raw Data Save DWH Bolt Kafka Consumer Spout
- 13. Useful Links ● https://storm.incubator.apache.org ● https://www.youtube.com/watch? v=DRgs8L4AOgc ● https://cwiki.apache. org/confluence/display/KAFKA/Index ● http://nguyentantrieu.info/blog/backend-system- with-netty-io ●