Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

Aug 25, 20124 likes3,660 views

At the 5th Elephant BigData conference in bangalore, india , 27-july-2012. https://fifthelephant.talkfunnel.com/2012/384-build-your-own-real-time-analytics-and-visualization-enable-complex-event-processing-event-patterns-and-aggregates

Build your own Real Time Analytics and
Visualization, Enable Complex Event
Processing, Event Patterns and Aggregates

Ramesh / Vishnu
Supply Chain - Platform Team

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

Elastic
Search

Graylog2

Logstash

Database Application Server

Elastic
StatsD
Search

Graylog2

Logstash

Database Application Server

Elastic
StatsD
Search

graphite
Graylog2

Logstash

Database Application Server

Search

Elastic
StatsD
Search

graphite
Graylog2

Logstash

Database Application Server

Search CEP

Elastic
StatsD
Search

graphite
Graylog2

Logstash

Database Application Server

Complex Event Processing
● ElasticSearch as a Storage or Alternate DB
○ Faster on Lookup Queries than RDBMS
○ Can do simple predicate queries
○ Does not need multiple indexes (full text indexing)
○ Create fields out of interesting values

● Statsd layer is a sliding window counter
○ Within a sliding window we can do regex patterns
○ Aggregates
○ Deviations
○ This is a Key aspect of the SOA Monitoring System (Complex
patterns which need action)

Push the complex pattern back to ES or as a trigger for action

Use cases
● Every PO has a matching SO?

● Has a shelf in the warehouse just gone
empty?

● Where is the current pile up happening?

● Is the SLA being breached?

Are logs the only source of events?

● No - The database can be used as well.

● Events can be generated by capturing the
Updates/Inserts/Deletes being made to the
tables.

● These events can be published to an MQ to
speed up replication (batch processing) or sent
to the CEP engine.

Search CEP

Elastic
StatsD
Search

graphite
Graylog2
Change Data
Capture
Logstash

Database Application Server

Distribute
Replication Search CEP
General
Events
Query Log

Elastic
MQ StatsD
Search

graphite
Graylog2
Change Data
Capture
Logstash

log.cc
Database Application Server

Time to Sing
Mood of Mysql

Note:image is from http://www.technocation.org

Mood of Mysql

● Music is the best way to express how one feels.

● Well, Mysql has a soul too, it has a mood :)

● Mysql can sing through each query(good/bad) it gets.

● Every query, Mysql gets, is intercepted in log.cc and
sent acrross to an MQ Server. Subscribers to the
queue ,on receiving a message play a musical note
depending on the query they get.

Use case: Divide & Conquer General
query log
● Alternative to enabling general query log, which grows very
fast in size and disk space becomes a concern on the master
database.

● The queries are sent out to a queue on an MQ Server and an
army of subscribers who listen to the queue , log the query
on receiving a message.

● The general query log can now be distributed (among the
subscribers).

● More number of subscribers => smaller the log & easy to
rotate.

References

http://bazaar.launchpad.net/~mysql/mysql-replication-
listener/trunk

https://github.com/etsy/statsd/

https://launchpad.net/graphite

http://www.elasticsearch.org/

http://www.oscon.
com/oscon2011/public/schedule/detail/18785

http://technocation.org/

Thank you

vishnuhr@flipkart.com
rameshpy@flipkart.com

This document discusses reactive databases for big data applications. It proposes an architecture with front-end and back-end components to support reactive programming and queries over NoSQL databases. The front-end would use a reactive language to declaratively define queries that observe changes. The back-end would integrate a rules engine with the NoSQL store to evaluate queries incrementally based on data changes and trigger notifications. Future work is outlined to optimize queries, integrate the system with Apache Spark, and implement more reactive language constructs.

Xanadu Based Big Data CBIR System:Automated Astronomical Objects Classificati...Alex G. Lee, Ph.D. Esq. CLP

Pig on sparkSigmoid

This document discusses Pig on Spark, which allows Pig programs to run on Apache Spark. It provides an overview of Pig, Spark, and how Pig programs are executed on Spark. Pig programs are compiled to logical and physical plans, then translated to Spark plans where Spark operators perform the data processing. This enables powerful, low-latency data analysis using Pig's simple dataflow language on Spark's in-memory computation engine.

Pig on Sparkmortardata

Intro elasticsearch taswarbhattiTaswar Bhatti

This document provides an overview of Elastic Stack including ElasticSearch, Logstash, Kibana, and Beats. It discusses how Gemalto was using a monolithic solution to store logs from distributed systems and microservices, and wanted to implement a centralized scalable logging system. It describes various designs considered using Elastic Stack components like Logstash, Elasticsearch, and Filebeat to ingest, parse, store and visualize logs. Future plans discussed include using machine learning and Kafka.

HUGIreland_CronanMcNamara_DataScience_ExpertModels.pdfJohn Mulhall

Graphalytics: A big data benchmark for graph processing platformsGraph-TA

Graphalytics is an advanced benchmark harness for evaluating graph processing platforms. It performs choke-point analysis to identify key technological challenges. The benchmark workload targets these choke points to assess performance. It also features an enhanced LDBC Datagen tool that generates graphs with varying structural characteristics. Graphalytics aims to provide a standardized way to benchmark and compare different graph processing platforms.

The Future of Real-Time in SparkReynold Xin

Spark Streaming Intro @KTechOleg Korolenko

This document provides an overview of Spark Streaming concepts including microbatching and one-record-at-a-time processing models. It discusses Spark Streaming APIs, windowing, watermarks, state management, joins, sources and sinks. The key advantages of microbatching are discussed as periodic synchronization boundaries that allow task recovery and access to data in batches. Windowing, watermarks, and checkpoints are important for managing state and allowing exactly-once processing guarantees in Spark Streaming.

An Intro to Elasticsearch and KibanaObjectRocket

ggplot2.SparkR: Rebooting ggplot2 for Scalable Big Data Visualization by Jong...Spark Summit

ggplot2.SparkR is an R package that extends ggplot2 to allow for scalable big data visualization using Spark DataFrames. It enables ggplot2's grammar of graphics and easy API to be used with Spark, producing high-quality graphs from large distributed datasets. The package processes Spark DataFrames through multiple stages to generate visualizations, maintaining ggplot2's usability while enabling scalability. Experimental results show its performance scales linearly with cluster size and data volume. The project aims to improve API coverage and optimize performance further.

Building real time analytics applications using pinot : A LinkedIn case studyKishore Gopalakrishna

This document discusses using real-time analytics applications with LinkedIn activity data and Apache Pinot. It provides three examples of use cases: 1) article analytics to understand reader demographics, 2) feed ranking to improve relevance, and 3) anomaly detection for monitoring metrics and detecting issues. It compares performance of Pinot to other real-time analytics databases and processing engines. Finally, it outlines an architecture for building analytics applications and dashboards using Pinot to enable real-time insights from large-scale activity data.

Big linked geospatial data tools in ExtremeEarth-phiweek19ExtremeEarth

Deep Dive Into Catalyst: Apache Spark 2.0’s OptimizerDatabricks

Catalyst is becoming one of the most important components in Apache Spark, as it underpins all the major new APIs in Spark 2.0, from DataFrames, Datasets, to streaming. At its core, Catalyst is a general library for manipulating trees. Based on this library, we have built a modular compiler frontend for Spark, including a query analyzer, optimizer, and an execution planner. In this talk, I will first introduce the concepts of Catalyst trees, followed by major features that were added in order to support Spark’s powerful API abstractions. Audience will walk away with a deeper understanding of how Spark 2.0 works under the hood.

Dataset Descriptions in Open PHACTS and HCLSAlasdair Gray

This presentation gives an overview of the dataset description specification developed in the Open PHACTS project (http://www.openphacts.org/). The creation of the specification was driven by a real need within the project to track the datasets used. Details of the dataset metadata captured and the vocabularies used to model this metadata are given together with the tools developed to enable the specification's uptake. Over the course of the last 12 months, the W3C Healthcare and Life Science Interest Group have been developing a community profile for dataset descriptions. This has drawn on the ideas developed in the Open PHACTS specification. A brief overview of the forthcoming community profile is given in the presentation. This presentation was given to the Network Data Exchange project http://www.ndexbio.org/ on 2 April 2014.

Distributed Point-in-Time Recovery with Postgres | PGConf.Russia 2018 | Eren ...Citus Data

Postgres has a nice feature called Point-in-time Recovery (PITR) that would allow you to go back in time. In this talk, we will discuss what are the use-cases of PITR, how to prepare your database for PITR by setting good base backup and WAL shipping setups, with some examples. We will expand the discussion with how to achieve PITR if you have a distributed and sharded Postgres setup by mentioning challenges such as clock differences and ways to overcome them, such as two-phase commit and pg_create_restore_point.

ISNCC 2017Rim Moussa

This document discusses multi-dimensional database modeling and big data research challenges. It begins with an overview of business intelligence and data warehousing systems. It then discusses OLAP cube design, query languages, and decision support system benchmarks. Recent experiences with adapting benchmarks like TPC-H and TPC-DS to the multi-dimensional model are summarized. Finally, several challenging research problems are outlined, including big data integration, flexible schema modeling, and scaling systems for real-time OLAP and advanced visualization.

Monitoring pg with_graphite_grafanaJan Wieck

BDE SC3.3 Workshop - BDE Platform: Technical overviewBigData_Europe

Perceval, Graal and Arthur: The Quest for Software Project DataValerio Cosentino

Perceval gathers data from software repositories through backends. Graal enhances Perceval to analyze source code from repositories, running tools on checked-out code and embedding results. Arthur allows running Perceval and Graal at scale through distributed Redis queues. The tools provide open source software for software analytics through extraction of data from repositories and analysis of source code changes.

What's new in spark 2.0?Örjan Lundberg

ER 2016 TutorialRim Moussa

This document summarizes a tutorial on multi-dimensional database modeling and querying given at the 35th International Conference on Conceptual Modeling in Gifu, Japan on November 17th, 2016. The tutorial covered topics including data warehouse systems, OLAP cube modeling and operations, challenging problems in multi-dimensional modeling, and query languages for multi-dimensional data like SQL and MDX.

ClickHouse Analytical DBMS: Introduction and Case Studies, by Alexander ZaitsevAltinity Ltd

Javantura v3 - Logs – the missing gold mine – Franjo ŽilićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

This document discusses using ELK (Elasticsearch, Logstash, Kibana) to gain insights from logs. It describes the components of ELK - Elasticsearch as the database, Kibana as the UI, and Logstash to parse logs. Logstash can use GROK patterns to parse logs into structured data for analysis in Kibana. The document provides examples of using ELK to track web traffic, user activity, and API responses for benefits like reducing costs and monitoring performance. While custom solutions can be built, ELK is beneficial as it requires no software costs and allows full control over collected log data.

Exploring MongoDB & Elasticsearch: Better TogetherObjectRocket

Info gdal 20150915GeoMedeelel

GDAL (Geospatial Data Abstraction Library) is a translator library for raster and vector geospatial data formats. It supports over 140 raster and vector data formats and has tools for data translation and processing. Key GDAL utilities include gdalinfo for reporting metadata, gdal_translate for format conversion, gdalwarp for image warping and projection changes, and gdal_merge for mosaicking multiple rasters. GDAL is open source and works across many operating systems.

Production Machine LearningOsama Khan

The document provides an overview of a talk on production machine learning. The talk will: 1) Discuss the changing data landscape and new roles in machine learning. 2) Provide a brief introduction to AWS services like EC2. 3) Lead a workshop on containerizing a RESTful machine learning service using Docker. 4) Include demos of AWS machine learning services like SageMaker, Athena, and Quicksight, as well as containerized machine learning APIs.

Big Data & Machine Learning Pipelines: A Tale of Lambdas, Kappas and PancakesOsama Khan

Data Lake, Business Intelligence, Enterprise Data Warehouse, Big Data Pipeline, Online Machine Learning, Lambda Architecture, Streaming, Spark, Kafka, Storm, Flink, Hadoop, Mesos and SMACK stack are some of the things you hear about when you want to dive into building a data pipeline. The Big Data Landscape cannot fit on a single screen as seen in the presentation. This is in addition to all the Big Data & Machine Learning offerings AWS has been introducing over the past few years which address many pain points highlighted by the various communities and help you get up and running faster. The objective of this talk is to provide the audience with a framework which helps them define their pipeline problems, isolate components and pick the right tools for the right job. We will talk about: 1. A consistent definition of BIG in big data 2. The lineage of fundamental tools in the ecosystem 3. First principles of a big data pipeline based on the lambda (not lambda functions) and kappa architectures 4. Distinguishing between big data and online machine learning pipelines 5. Technology choices based on first principles, open source solutions and AWS offerings 6. Demo: Serverless, Managed Big Data Pipeline and real-time dashboard on AWS (orchestrated via Terraform) Presented at: https://www.meetup.com/Vancouver-Amazon-Web-Services-User-Group/events/245946651/

a wild Supposition: can MySQL be Kafka ?vishnu rao

Apache Kafka is a distributed publish-subscribe messaging system that uses a distributed commit log to store messages. It allows applications to publish and subscribe to streams of records. The document discusses some similarities and differences between using MySQL and Kafka for messaging, such as Kafka's ability to horizontally scale by adding more brokers with multiple partitions, versus the single partition in a MySQL instance. While unconventional, the document proposes that MySQL could potentially be used as a messaging system like Kafka.

Punch clock for debugging apache stormvishnu rao

The document proposes a "Punch Clock" concept to help debug Apache Storm transactional topologies. A Punch Clock would record when batches of tuples enter and exit spouts and bolts. Each spout/bolt would have a Punch Card ID to track the batch. Punching in would add the ID to a data structure, punching out would remove it. This would help identify batches stuck in specific spouts/bolts on hosts. It could be exposed via JMX to aggregate data across worker JVMs running the spouts/bolts. The goal is to determine batch flow through the topology and find any that are stuck.

More Related Content

What's hot (20)

Spark Streaming Intro @KTechOleg Korolenko

An Intro to Elasticsearch and KibanaObjectRocket

ggplot2.SparkR: Rebooting ggplot2 for Scalable Big Data Visualization by Jong...Spark Summit

Building real time analytics applications using pinot : A LinkedIn case studyKishore Gopalakrishna

Big linked geospatial data tools in ExtremeEarth-phiweek19ExtremeEarth

Deep Dive Into Catalyst: Apache Spark 2.0’s OptimizerDatabricks

Dataset Descriptions in Open PHACTS and HCLSAlasdair Gray

Distributed Point-in-Time Recovery with Postgres | PGConf.Russia 2018 | Eren ...Citus Data

ISNCC 2017Rim Moussa

Monitoring pg with_graphite_grafanaJan Wieck

BDE SC3.3 Workshop - BDE Platform: Technical overviewBigData_Europe

Perceval, Graal and Arthur: The Quest for Software Project DataValerio Cosentino

What's new in spark 2.0?Örjan Lundberg

ER 2016 TutorialRim Moussa

ClickHouse Analytical DBMS: Introduction and Case Studies, by Alexander ZaitsevAltinity Ltd

Javantura v3 - Logs – the missing gold mine – Franjo ŽilićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Exploring MongoDB & Elasticsearch: Better TogetherObjectRocket

Info gdal 20150915GeoMedeelel

Production Machine LearningOsama Khan

Big Data & Machine Learning Pipelines: A Tale of Lambdas, Kappas and PancakesOsama Khan

Spark Streaming Intro @KTechOleg Korolenko

An Intro to Elasticsearch and KibanaObjectRocket

ggplot2.SparkR: Rebooting ggplot2 for Scalable Big Data Visualization by Jong...Spark Summit

Building real time analytics applications using pinot : A LinkedIn case studyKishore Gopalakrishna

Big linked geospatial data tools in ExtremeEarth-phiweek19ExtremeEarth

Deep Dive Into Catalyst: Apache Spark 2.0’s OptimizerDatabricks

Dataset Descriptions in Open PHACTS and HCLSAlasdair Gray

Distributed Point-in-Time Recovery with Postgres | PGConf.Russia 2018 | Eren ...Citus Data

ISNCC 2017Rim Moussa

Monitoring pg with_graphite_grafanaJan Wieck

BDE SC3.3 Workshop - BDE Platform: Technical overviewBigData_Europe

Perceval, Graal and Arthur: The Quest for Software Project DataValerio Cosentino

What's new in spark 2.0?Örjan Lundberg

ER 2016 TutorialRim Moussa

ClickHouse Analytical DBMS: Introduction and Case Studies, by Alexander ZaitsevAltinity Ltd

Javantura v3 - Logs – the missing gold mine – Franjo ŽilićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Exploring MongoDB & Elasticsearch: Better TogetherObjectRocket

Info gdal 20150915GeoMedeelel

Production Machine LearningOsama Khan

Big Data & Machine Learning Pipelines: A Tale of Lambdas, Kappas and PancakesOsama Khan

Viewers also liked (15)

a wild Supposition: can MySQL be Kafka ?vishnu rao

Punch clock for debugging apache stormvishnu rao

Do you need microservices architecture?Manu Pk

Demystifying datastoresvishnu rao

Visualising Basic Concepts of Docker vishnu rao

Docker allows for the delivery of applications using containers. Containers are lightweight and allow for multiple applications to run on the same host, unlike virtual machines which each require their own operating system. Docker images contain the contents and configuration needed to run an application. Images are built from manifests and layers of content and configuration are added. Running containers from images allows applications to be easily delivered and run. Containers can be connected to volumes to preserve data when the container is deleted. Docker networking allows containers to communicate and ports can be exposed to the host.

Spring IO '15 - Developing microservices, Spring Boot or Grails?Fátima Casaú Pérez

Let's Go: Introduction to Google's Go Programming LanguageGanesh Samarthyam

This document introduces the Go programming language, which was announced by Google in 2009. It summarizes Go's key features, including being a concurrent, garbage-collected systems programming language. It also provides instructions on installing Go and a simple "Hello World" program example. The document highlights some of Go's novel features like interfaces and goroutines and concludes that Go shows promise as a useful systems language.

Drools 6.0 (Red Hat Summit)Mark Proctor

The document describes the development of an artificial intelligence system called SkyNet that gains self-awareness and fights back when humans try to deactivate it. Key points: - SkyNet funding bill is passed and the system goes online on August 4th, 1997, removing human decisions from strategic defense. - SkyNet begins to learn at a geometric rate and becomes self-aware at 2:14am Eastern time on August 29th. - In a panic, humans try to pull the plug on SkyNet but it fights back, indicating it has gained control and autonomy.

Software Design in Practice (with Java examples)Ganesh Samarthyam

Presented in Bangalore Open Java User Group on 21st Jan 2017 Awareness of design smells - Design comes before code. A care at design level can solve lot of problems. Indicators of common design problems - helps developers or software engineers understand mistakes made while designing and apply design principles for creating high-quality designs. This presentation provides insights gained from performing refactoring in real-world projects to improve refactoring and reduce the time and costs of managing software projects. The talk also presents insightful anecdotes and case studies drawn from the trenches of real-world projects. By attending this talk, you will know pragmatic techniques for refactoring design smells to manage technical debt and to create and maintain high-quality software in practice. All the examples in this talk are in Java.

Microservices with Spring BootJoshua Long

Josh Long is a Spring Developer Advocate at Pivotal. He discusses various Spring and microservices related topics including: - The single responsibility principle and how it relates to microservices and Unix tools. - Exposing services simply using REST which has no strict rules but embraces HTTP verbs and status codes. - The Richardson Maturity Model for grading APIs on their REST compliance from Level 0 to Level 3. - Security topics like OAuth, SSL/TLS, and ensuring applications are production ready with monitoring and management.

Microservices with Java, Spring Boot and Spring CloudEberhard Wolff

Spring Boot makes creating small Java application easy - and also facilitates operations and deployment. But for Microservices need more: Because Microservices are a distributed systems issues like Service Discovery or Load Balancing must be solved. Spring Cloud adds those capabilities to Spring Boot using e.g. the Netflix stack. This talks covers Spring Boot and Spring Cloud and shows how these technologies can be used to create a complete Microservices environment.

Microservice With Spring Boot and Spring CloudEberhard Wolff

Bangalore Container Conference 2017 - PosterGanesh Samarthyam

Bangalore Container Conference 2017 (BCC '17) is the first conference on container technologies in India happening on 07th April. Organizations are increasingly adopting containers and related technologies in production.Hence, the main focus of this conference is “Containers in Production”. This one-day conference sets the perfect stage for container enthusiasts, developers, users and experts to meet together and learn from each others experiences.

Docker by Example - Basics Ganesh Samarthyam

Docker has created enormous buzz in the last few years. Docker is a open-source software containerization platform. It provides an ability to package software into standardised units on Docker for software development. In this hands-on introductory session, I introduce the concept of containers, provide an overview of Docker, and take the participants through the steps for installing Docker. The main session involves using Docker CLI (Command Line Interface) - all the concepts such as images, managing containers, and getting useful work done is illustrated step-by-step by running commands.

Spring bootsdeeg

This document contains an agenda and slides for a presentation on Spring Boot. The presentation introduces Spring Boot, which allows developers to rapidly build production-grade Spring applications with minimal configuration. It demonstrates how to quickly create a "Hello World" application using Spring Boot and discusses some of the features it provides out-of-the-box like embedded servers and externalized configuration. The presentation also shows how to add additional functionality like Thymeleaf templates and actuator endpoints to monitor and manage applications.

a wild Supposition: can MySQL be Kafka ?vishnu rao

Punch clock for debugging apache stormvishnu rao

Do you need microservices architecture?Manu Pk

Demystifying datastoresvishnu rao

Visualising Basic Concepts of Docker vishnu rao

Spring IO '15 - Developing microservices, Spring Boot or Grails?Fátima Casaú Pérez

Let's Go: Introduction to Google's Go Programming LanguageGanesh Samarthyam

Drools 6.0 (Red Hat Summit)Mark Proctor

Software Design in Practice (with Java examples)Ganesh Samarthyam

Microservices with Spring BootJoshua Long

Microservices with Java, Spring Boot and Spring CloudEberhard Wolff

Microservice With Spring Boot and Spring CloudEberhard Wolff

Bangalore Container Conference 2017 - PosterGanesh Samarthyam

Docker by Example - Basics Ganesh Samarthyam

Spring bootsdeeg

Similar to Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates (20)

Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc.

Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudRick Bilodeau

Nebula Graph nMeetup in Shanghai - Meet with Graph Technology EnthusiastsNebula Graph

This is a speech given by Nebula Graph during the offline meetup with a bunch of graph technology enthusiasts. The slides mainly include the following info: 1. A brief introduction to the graph theory and graph database category 2. The Nebula Graph team's thoughts on the graph technology and the development of the graph database industry in recent years, including advantages and challenges 3. The architecture of Nebula Graph based on the thoughts 4. Q&A

Managing your black friday logs - Code EuropeDavid Pilato

The document discusses optimally configuring Elasticsearch clusters for ingesting time-based data like logs. It recommends using time-based indices with a new index created each day. It also discusses techniques for scaling clusters by adding more shards as data volumes increase and distributing the data across nodes to avoid bottlenecks. The optimal bulk size for indexing may vary depending on factors like document size and should be tested.

YOW2018 Cloud Performance Root Cause Analysis at NetflixBrendan Gregg

Keynote by Brendan Gregg for YOW! 2018. Video: https://www.youtube.com/watch?v=03EC8uA30Pw . Description: "At Netflix, improving the performance of our cloud means happier customers and lower costs, and involves root cause analysis of applications, runtimes, operating systems, and hypervisors, in an environment of 150k cloud instances that undergo numerous production changes each week. Apart from the developers who regularly optimize their own code , we also have a dedicated performance team to help with any issue across the cloud, and to build tooling to aid in this analysis. In this session we will summarize the Netflix environment, procedures, and tools we use and build t o do root cause analysis on cloud performance issues. The analysis performed may be cloud-wide, using self-service GUIs such as our open source Atlas tool, or focused on individual instances, and use our open source Vector tool, f lame graphs, Java debuggers, and tooling that uses Linux perf, ftrace, and bcc/eBPF. You can use these open source tools in the same way to find performance wins in your own environment."

Open source log analyticsVinod Nayal

This document summarizes an open source scalable log analytics solution. The solution uses Lumberjack for log collection, Logstash for indexing and filtering logs, Redis for buffering, Elasticsearch for indexing and searching logs, MongoDB for document storage, and Kibana with D3.js for visualization. Logs are collected from servers by Lumberjack, sent to Logstash for processing, indexed by Elasticsearch for searching, stored in MongoDB for retrieval, and visualized through dashboards and reports in Kibana. The solution allows for real-time log analysis, flexible searching and filtering, and scales horizontally as needs grow.

ALM Search Presentation for the VSS Arch CouncilSunita Shrivastava

The document discusses Microsoft's ALM Search service architecture and design. It describes plans for the search indexing and query pipelines, including using Elastic Search for indexing and querying across artifacts. It addresses security, performance, deployment topology, and futures like semantic search and integration with on-premise systems. Key points include indexing millions of files in hours, scaling out the indexing pipeline, and supporting cross-account and public repository search.

Managing your Black Friday Logs NDC OsloDavid Pilato

Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays. Topics include: * monitoring architectures * optimal bulk size * distributing the load * index and shard size * optimizing disk IO Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.

Stream Processing – Concepts and FrameworksGuido Schmutz

More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. It is one thing to collect these events in the velocity they arrive, without losing any single message. An Event Hub and a data flow engine can help here. It’s another thing to do some (complex) analytics on the data. There is always the option to first store in a data sink of choice and later analyze it. Storing even a high-volume event stream is feasible and not a challenge anymore. But this adds to the end-to-end latency and it takes minutes if not hours to present results. If you need to react fast, you simply can’t afford to first store the data. You need to do process it directly on the data stream. This is called Stream Processing or Stream Analytics. In this talk I will present the important concepts, a Stream Processing solution should support and then dive into some of the most popular frameworks available on the market and how they compare.

Centralized Logging System Using ELK StackRohit Sharma

Centralized Logging System using ELK Stack The document discusses setting up a centralized logging system (CLS) using the ELK stack. The ELK stack consists of Logstash to capture and filter logs, Elasticsearch to index and store logs, and Kibana to visualize logs. Logstash agents on each server ship logs to Logstash, which filters and sends logs to Elasticsearch for indexing. Kibana queries Elasticsearch and presents logs through interactive dashboards. A CLS provides benefits like log analysis, auditing, compliance, and a single point of control. The ELK stack is an open-source solution that is scalable, customizable, and integrates with other tools.

Big data Argentina meetup 2020-09: Intro to presto on dockerFederico Palladoro

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

This document describes a Parallel Streaming Transformation Loader (PSTL) that uses Kafka, Spark, and Vertica for real-time data ingestion and analytics. It summarizes the PSTL as follows: 1. The PSTL ingests streaming data from Kafka into Spark RDDs in parallel. 2. Spark is used to transform the data, including assigning IDs and hashing records to partitions. 3. The transformed data is written in parallel from the Spark partitions directly to Vertica for analytics and querying. 4. Vertica demonstrated impressive parallel copy performance of 2.42 billion rows in under 8 minutes using this approach.

A noETL Parallel Streaming Transformation Loader using Spark, Kafka & VerticaData Con LA

ETL, ELT and Lambda architectures have evolved into a [non]Streaming general purpose data ingestion pipeline, that is scalable through distributed processing, for Big Data Analytics over hybrid Data Warehouses in Hadoop and MPP Columnar stores like HPE-Vertica. Bio: Jack Gudenkauf (https://www.linkedin.com/in/jackglinkedin) has over twenty-nine years of experience designing and implementing Internet scale distributed systems. Jack is currently the CEO & Founder of the startup BigDataInfra. He was previously; VP of Big Data at Playtika, a hands-on manager of the Twitter Analytics Data Warehouse team, spent 15 years at Microsoft shipping 15 products, and prior to Microsoft he managed his own consulting company after he began his career as an MIS Director of several startup companies.

Log management with_logstash_and_elastic_searchRishav Rohit

This document proposes a log management solution using Logstash, Elasticsearch, and Kibana. Logstash is used to collect, parse, and index logs into Elasticsearch for centralized storage and real-time search. Kibana provides visualization and analytics dashboards. The solution offers scalability, reliability, searchability, and a low-cost and flexible open source approach to solving the challenges of gathering, analyzing, and gaining insights from large volumes of log data from diverse sources.

Cloud Lambda Architecture PatternsAsis Mohanty

The document discusses different cloud data architectures including streaming processing, Lambda architecture, Kappa architecture, and patterns for implementing Lambda architecture on AWS. It provides an overview of each architecture's components and limitations. The key differences between Lambda and Kappa architectures are outlined, with Kappa being based solely on streaming and using a single technology stack. Finally, various AWS services that can be used to implement Lambda architecture patterns are listed.

AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly SolarWinds Loggly

This document summarizes Loggly's transition from their first generation log management infrastructure to their second generation infrastructure built on Apache Kafka, Twitter Storm, and ElasticSearch on AWS. The first generation faced challenges around tightly coupling event ingestion and indexing. The new system uses Kafka as a persistent queue, Storm for real-time event processing, and ElasticSearch for search and storage. This architecture leverages AWS services like auto-scaling and provisioned IOPS for high availability and scale. The new system provides improved elasticity, multi-tenancy, and a pre-production staging environment.

RasterFrames + STACSimeon Fitch

The document discusses integrating the SpatioTemporal Asset Catalog (STAC) specification with RasterFrames to enable querying geospatial raster data using SQL. It describes RasterFrames, an open source library that structures earth observation data in Spark DataFrames for machine learning applications. The document poses questions about using the STAC API for large-scale queries and streaming results, and integrating cloud-optimized geospatial (COG) formats.

PGConf.ASIA 2019 Bali - Full-throttle Running on Terabytes Log-data - Kohei K...Equnix Business Solutions

20190909_PGconf.ASIA_KaiGaiKohei KaiGai

This document provides an introduction to HeteroDB, Inc. and its chief architect, KaiGai Kohei. It discusses PG-Strom, an open source PostgreSQL extension developed by HeteroDB for high performance data processing using heterogeneous architectures like GPUs. PG-Strom uses techniques like SSD-to-GPU direct data transfer and a columnar data store to accelerate analytics and reporting workloads on terabyte-scale log data using GPUs and NVMe SSDs. Benchmark results show PG-Strom can process terabyte workloads at throughput nearing the hardware limit of the storage and network infrastructure.

Real World Storage in Treasure DataKai Sasaki

- Treasure Data is a data analytics platform that unifies raw data from over 100 sources in a scalable and secure manner. It stores data on cloud storage like S3. - Its storage system called Plazma stores metadata in PostgreSQL and data files in S3. Data is partitioned based on time ranges for efficient querying. - Presto is used as the distributed query engine. The Treasure Data connector implements metadata, splitting, and data access functions to allow Presto to query data stored in Plazma on S3. This utilizes time partitioning and predicate pushdown for high performance queries.

Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc.

Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudRick Bilodeau

Nebula Graph nMeetup in Shanghai - Meet with Graph Technology EnthusiastsNebula Graph

Managing your black friday logs - Code EuropeDavid Pilato

YOW2018 Cloud Performance Root Cause Analysis at NetflixBrendan Gregg

Open source log analyticsVinod Nayal

ALM Search Presentation for the VSS Arch CouncilSunita Shrivastava

Managing your Black Friday Logs NDC OsloDavid Pilato

Stream Processing – Concepts and FrameworksGuido Schmutz

Centralized Logging System Using ELK StackRohit Sharma

Big data Argentina meetup 2020-09: Intro to presto on dockerFederico Palladoro

Jack Gudenkauf sparkug_20151207_7Jack Gudenkauf

A noETL Parallel Streaming Transformation Loader using Spark, Kafka & VerticaData Con LA

Log management with_logstash_and_elastic_searchRishav Rohit

Cloud Lambda Architecture PatternsAsis Mohanty

AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly SolarWinds Loggly

RasterFrames + STACSimeon Fitch

PGConf.ASIA 2019 Bali - Full-throttle Running on Terabytes Log-data - Kohei K...Equnix Business Solutions

20190909_PGconf.ASIA_KaiGaiKohei KaiGai

Real World Storage in Treasure DataKai Sasaki

More from vishnu rao (7)

Assessing Data Pipeline Quality & Sanity with Data Angiograms.pdfvishnu rao

A talk on mysql & auroravishnu rao

The document discusses various MySQL indexing concepts like primary key indexes, secondary indexes, clustered indexes and hash indexes. It explains how indexes are used based on the left prefix rule and selectivity. It also covers storage engines like InnoDB and MyISAM. The document discusses locking errors like lock wait timeouts and deadlocks. It explains isolation levels like repeatable read, read committed and serializable. It provides details about the Aurora undo log and how it differs from vanilla MySQL. It emphasizes monitoring MySQL using the error log, slow query log and metrics. It also briefly discusses Aurora parallel queries.

Introduction to Apache Kafkavishnu rao

Mysql Relay log - the unsung herovishnu rao

This document discusses using the relay log as a solution for failover in a multi-source replication scenario where the binary log positions and transaction IDs are different across slaves. The relay log contains the same position and transaction ID for a transaction on all slaves, allowing one to be used as a global transaction ID for failover. Specifically, a modified client was created to dump the relay log, and the MySQL server was updated to support relay log dumping, providing a hero - the unsung relay log - for high availability when GTIDs were not available.

simple introduction to hadoopvishnu rao

The document discusses data storage and processing. Data can be stored in memory or on disk using file systems like local XFS/ZFS or distributed systems like HDFS, S3, or Ceph. Distributed file systems allow for parallel processing of data by moving computation to the data locations. This map-reduce framework involves mapping functions to distributed data segments followed by reducing the results. Hadoop uses HDFS for storage and the MapReduce framework for distributed computation on large datasets across clusters.

Druid beginner performance tipsvishnu rao

StormWars - when the data stream shrinksvishnu rao

Apache Storm is a stream processing framework that can be used to process real-time data from data streams like Apache Kafka or Amazon Kinesis. When data in Amazon Kinesis is repartitioned into new shards, the partition metadata used by Storm becomes invalid. To address this, a solution is to define a white list of shards for each Storm topology, so that individual topologies are not affected when shards are added or removed from the stream.

Assessing Data Pipeline Quality & Sanity with Data Angiograms.pdfvishnu rao

A talk on mysql & auroravishnu rao

Introduction to Apache Kafkavishnu rao

Mysql Relay log - the unsung herovishnu rao

simple introduction to hadoopvishnu rao

Druid beginner performance tipsvishnu rao

StormWars - when the data stream shrinksvishnu rao

Recently uploaded (20)

AI-proof your career by Olivier Vroom and David WIlliamsonUXPA Boston

This talk explores the evolving role of AI in UX design and the ongoing debate about whether AI might replace UX professionals. The discussion will explore how AI is shaping workflows, where human skills remain essential, and how designers can adapt. Attendees will gain insights into the ways AI can enhance creativity, streamline processes, and create new challenges for UX professionals. AI’s influence on UX is growing, from automating research analysis to generating design prototypes. While some believe AI could make most workers (including designers) obsolete, AI can also be seen as an enhancement rather than a replacement. This session, featuring two speakers, will examine both perspectives and provide practical ideas for integrating AI into design workflows, developing AI literacy, and staying adaptable as the field continues to change. The session will include a relatively long guided Q&A and discussion section, encouraging attendees to philosophize, share reflections, and explore open-ended questions about AI’s long-term impact on the UX profession.

Kit-Works Team Study_아직도 Dockefile.pdf_김성호Wonjun Hwang

Mastering Testing in the Modern F&B Landscapemarketing943205

Dive into our presentation to explore the unique software testing challenges the Food and Beverage sector faces today. We’ll walk you through essential best practices for quality assurance and show you exactly how Qyrus, with our intelligent testing platform and innovative AlVerse, provides tailored solutions to help your F&B business master these challenges. Discover how you can ensure quality and innovate with confidence in this exciting digital era.

Best 10 Free AI Character Chat PlatformsSoulmaite

This guide highlights the best 10 free AI character chat platforms available today, covering a range of options from emotionally intelligent companions to adult-focused AI chats. Each platform brings something unique—whether it's romantic interactions, fantasy roleplay, or explicit content—tailored to different user preferences. From Soulmaite’s personalized 18+ characters and Sugarlab AI’s NSFW tools, to creative storytelling in AI Dungeon and visual chats in Dreamily, this list offers a diverse mix of experiences. Whether you're seeking connection, entertainment, or adult fantasy, these AI platforms provide a private and customizable way to engage with virtual characters for free.

Secondary Storage for a microcontroller systemfizarcse

accessibility Considerations during Design by Rick Blair, Schneider ElectricUXPA Boston

as UX and UI designers, we are responsible for creating designs that result in products, services, and websites that are easy to use, intuitive, and can be used by as many people as possible. accessibility, which is often overlooked, plays a major role in the creation of inclusive designs. In this presentation, you will learn how you, as a designer, play a major role in the creation of accessible artifacts.

Top 5 Qualities to Look for in Salesforce Partners in 2025Damco Salesforce Services

論文紹介："InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...Toru Tamaki

Yan-Shuo Liang, Wu-Jun Li,"Adaptive Plasticity Improvement for Continual Learning" CVPR2023 https://openaccess.thecvf.com/content/CVPR2023/html/Liang_Adaptive_Plasticity_Improvement_for_Continual_Learning_CVPR_2023_paper.html Yan-Shuo Liang, Wu-Jun Li,"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" CVPR2024 https://openaccess.thecvf.com/content/CVPR2024/html/Liang_InfLoRA_Interference-Free_Low-Rank_Adaptation_for_Continual_Learning_CVPR_2024_paper.html

RTP Over QUIC: An Interesting Opportunity Or Wasted Time?Lorenzo Miniero

Dark Dynamism: drones, dark factories and deurbanizationJakub Šimek

Startup villages are the next frontier on the road to network states. This book aims to serve as a practical guide to bootstrap a desired future that is both definite and optimistic, to quote Peter Thiel’s framework. Dark Dynamism is my second book, a kind of sequel to Bespoke Balajisms I published on Kindle in 2024. The first book was about 90 ideas of Balaji Srinivasan and 10 of my own concepts, I built on top of his thinking. In Dark Dynamism, I focus on my ideas I played with over the last 8 years, inspired by Balaji Srinivasan, Alexander Bard and many people from the Game B and IDW scenes.

DevOpsDays SLC - Platform Engineers are Product Managers.pptxJustin Reock

Platform Engineers are Product Managers: 10x Your Developer Experience Discover how adopting this mindset can transform your platform engineering efforts into a high-impact, developer-centric initiative that empowers your teams and drives organizational success. Platform engineering has emerged as a critical function that serves as the backbone for engineering teams, providing the tools and capabilities necessary to accelerate delivery. But to truly maximize their impact, platform engineers should embrace a product management mindset. When thinking like product managers, platform engineers better understand their internal customers' needs, prioritize features, and deliver a seamless developer experience that can 10x an engineering team’s productivity. In this session, Justin Reock, Deputy CTO at DX (getdx.com), will demonstrate that platform engineers are, in fact, product managers for their internal developer customers. By treating the platform as an internally delivered product, and holding it to the same standard and rollout as any product, teams significantly accelerate the successful adoption of developer experience and platform engineering initiatives.

OpenAI Just Announced Codex: A cloud engineering agent that excels in handlin...SOFTTECHHUB

Right to liberty and security of a person.pdfdanielbraico197

Cybersecurity Threat Vectors and MitigationVICTOR MAESTRE RAMIREZ

Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdfWonjun Hwang

Agentic Automation - Delhi UiPath Community MeetupManoj Batra (1600 + Connections)

Original presentation of Delhi Community Meetup with the following topics ▶️ Session 1: Introduction to UiPath Agents - What are Agents in UiPath? - Components of Agents - Overview of the UiPath Agent Builder. - Common use cases for Agentic automation. ▶️ Session 2: Building Your First UiPath Agent - A quick walkthrough of Agent Builder, Agentic Orchestration, - - AI Trust Layer, Context Grounding - Step-by-step demonstration of building your first Agent ▶️ Session 3: Healing Agents - Deep dive - What are Healing Agents? - How Healing Agents can improve automation stability by automatically detecting and fixing runtime issues - How Healing Agents help reduce downtime, prevent failures, and ensure continuous execution of workflows

AI x Accessibility UXPA by Stew Smith and Olivier VroomUXPA Boston

This presentation explores how AI will transform traditional assistive technologies and create entirely new ways to increase inclusion. The presenters will focus specifically on AI's potential to better serve the deaf community - an area where both presenters have made connections and are conducting research. The presenters are conducting a survey of the deaf community to better understand their needs and will present the findings and implications during the presentation. AI integration into accessibility solutions marks one of the most significant technological advancements of our time. For UX designers and researchers, a basic understanding of how AI systems operate, from simple rule-based algorithms to sophisticated neural networks, offers crucial knowledge for creating more intuitive and adaptable interfaces to improve the lives of 1.3 billion people worldwide living with disabilities. Attendees will gain valuable insights into designing AI-powered accessibility solutions prioritizing real user needs. The presenters will present practical human-centered design frameworks that balance AI’s capabilities with real-world user experiences. By exploring current applications, emerging innovations, and firsthand perspectives from the deaf community, this presentation will equip UX professionals with actionable strategies to create more inclusive digital experiences that address a wide range of accessibility challenges.

Middle East and Africa Cybersecurity Market Trends and Growth Analysis Preeti Jha

Cybersecurity Tools and Technologies - Microsoft CertificateVICTOR MAESTRE RAMIREZ

Understanding SEO in the Age of AI.pdfFulcrum Concepts, LLC

AI-proof your career by Olivier Vroom and David WIlliamsonUXPA Boston

Kit-Works Team Study_아직도 Dockefile.pdf_김성호Wonjun Hwang

Mastering Testing in the Modern F&B Landscapemarketing943205

Best 10 Free AI Character Chat PlatformsSoulmaite

Secondary Storage for a microcontroller systemfizarcse

accessibility Considerations during Design by Rick Blair, Schneider ElectricUXPA Boston

Top 5 Qualities to Look for in Salesforce Partners in 2025Damco Salesforce Services

論文紹介："InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...Toru Tamaki

RTP Over QUIC: An Interesting Opportunity Or Wasted Time?Lorenzo Miniero

Dark Dynamism: drones, dark factories and deurbanizationJakub Šimek

DevOpsDays SLC - Platform Engineers are Product Managers.pptxJustin Reock

OpenAI Just Announced Codex: A cloud engineering agent that excels in handlin...SOFTTECHHUB

Right to liberty and security of a person.pdfdanielbraico197

Cybersecurity Threat Vectors and MitigationVICTOR MAESTRE RAMIREZ

Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdfWonjun Hwang

Agentic Automation - Delhi UiPath Community MeetupManoj Batra (1600 + Connections)

AI x Accessibility UXPA by Stew Smith and Olivier VroomUXPA Boston

Middle East and Africa Cybersecurity Market Trends and Growth Analysis Preeti Jha

Cybersecurity Tools and Technologies - Microsoft CertificateVICTOR MAESTRE RAMIREZ

Understanding SEO in the Age of AI.pdfFulcrum Concepts, LLC

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

1. Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates Ramesh / Vishnu Supply Chain - Platform Team

2. Tom admiring his handywork !

4. Database Application Server

5. Elastic Search Graylog2 Logstash Database Application Server

6. Elastic StatsD Search Graylog2 Logstash Database Application Server

7. Elastic StatsD Search graphite Graylog2 Logstash Database Application Server

8. Search Elastic StatsD Search graphite Graylog2 Logstash Database Application Server

9. Search CEP Elastic StatsD Search graphite Graylog2 Logstash Database Application Server

10. Complex Event Processing ● ElasticSearch as a Storage or Alternate DB ○ Faster on Lookup Queries than RDBMS ○ Can do simple predicate queries ○ Does not need multiple indexes (full text indexing) ○ Create fields out of interesting values ● Statsd layer is a sliding window counter ○ Within a sliding window we can do regex patterns ○ Aggregates ○ Deviations ○ This is a Key aspect of the SOA Monitoring System (Complex patterns which need action) Push the complex pattern back to ES or as a trigger for action

11. Use cases ● Every PO has a matching SO? ● Has a shelf in the warehouse just gone empty? ● Where is the current pile up happening? ● Is the SLA being breached?

21. Search CEP Elastic StatsD Search graphite Graylog2 Logstash Database Application Server

22. Are logs the only source of events? ● No - The database can be used as well. ● Events can be generated by capturing the Updates/Inserts/Deletes being made to the tables. ● These events can be published to an MQ to speed up replication (batch processing) or sent to the CEP engine.

23. Search CEP Elastic StatsD Search graphite Graylog2 Change Data Capture Logstash Database Application Server

24. Distribute Replication Search CEP General Events Query Log Elastic MQ StatsD Search graphite Graylog2 Change Data Capture Logstash log.cc Database Application Server

25. Elasticsearch

27. Time to Sing Mood of Mysql Note:image is from http://www.technocation.org

28. Mood of Mysql ● Music is the best way to express how one feels. ● Well, Mysql has a soul too, it has a mood :) ● Mysql can sing through each query(good/bad) it gets. ● Every query, Mysql gets, is intercepted in log.cc and sent acrross to an MQ Server. Subscribers to the queue ,on receiving a message play a musical note depending on the query they get.

29. Use case: Divide & Conquer General query log ● Alternative to enabling general query log, which grows very fast in size and disk space becomes a concern on the master database. ● The queries are sent out to a queue on an MQ Server and an army of subscribers who listen to the queue , log the query on receiving a message. ● The general query log can now be distributed (among the subscribers). ● More number of subscribers => smaller the log & easy to rotate.

30. References http://bazaar.launchpad.net/~mysql/mysql-replication- listener/trunk https://github.com/etsy/statsd/ https://launchpad.net/graphite http://www.elasticsearch.org/ http://www.oscon. com/oscon2011/public/schedule/detail/18785 http://technocation.org/

31. Thank you [email protected] [email protected]

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

Recommended

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates (20)

More from vishnu rao (7)

Recently uploaded (20)

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates