Open source log analytics

Jun 2, 20141 like1,370 views

This document summarizes an open source scalable log analytics solution. The solution uses Lumberjack for log collection, Logstash for indexing and filtering logs, Redis for buffering, Elasticsearch for indexing and searching logs, MongoDB for document storage, and Kibana with D3.js for visualization. Logs are collected from servers by Lumberjack, sent to Logstash for processing, indexed by Elasticsearch for searching, stored in MongoDB for retrieval, and visualized through dashboards and reports in Kibana. The solution allows for real-time log analysis, flexible searching and filtering, and scales horizontally as needs grow.

Open source Scalable Log Analytics
Presented by Vinod Nayal

Log Analytics Overview
Collection search and analysis of log collected from
various app servers
Ability to search by attributes within a timeframe and ability
to export related log files
Real time Reports/dash-board like specific events per
hour

Solution Architecture
Redis
Broker
Logstash
-Indexer
Elastic Search
Redis
Broker
Mongodb
writer
Indexer
weserver
lumberjack
webserver
lumberjack
webserver
lumberjack
Elastic Search
Elastic Search
mongodb
mongodb
mongodb
KibanaUI+D3.js
Agent (Web
browser)
 Lumberjack ,Log stash , Redis Log collection
 Elastic Search Indexing
 Mongodb Document Storage for 1 week
 Kibana,D3.js UI

Dashboard
 Ability to
search
and filter
by any
attribute
 Customiz
able Time
series
graphs
 Various
aggregati
on across
time
geographi
es host
etc
H I G H L I G H T S

Solution Highlights
 Log indexing in Elastic search distributed cluster.
 Log collection via lumberjack( logstash-forwarder) on various client
nodes . It has a very low memory footprint . It support compression
and encryption in log transmission .
 Collected logs are sent to logstash –servers which saves to elastic
search for indexing . log file are also sent to mongodb for keeping
original data for export and future integrated view . Documents in
mongodb will have a retention period of 5 -7 days
 Redis is used for buffering log events at server side , it make system
able to take peak loads without failure . It also provides pub sub
architecture for sending logs to multiple processing concurrently
 Log enrichment and filtering capability with logstash filters and
pluggable architecture
 Kibana Integration for Spunk like UI for log searching and analysis
 All technologies used are open source ,scalable ,distributed and
customisable

Solution Details – Why Elastic Search
 Distributed
Elastic search allows you to start small, but will grow with your
business. It is built to scale horizontally out of the box. As you
need more capacity, just add more nodes, and l et the cluster
reorganize itself to take advantage of the extra hardware.
 Multi-tenancy
A cluster can host multiple indices which can be queried
independently or as a group. Index aliases allow you to add
indexes on the fly, while being transparent to your application.
 Schema free
Elastic search allows you to get started easily. Toss it a JSON
document and it will try to detect the data structure, index the
data and make it searchable. Later, apply your domain specific
knowledge of your data to customize how your data is indexed.

Solution Details – Why LogStash
 Configurable and
customizable log
collection that can be
scaled by adding more
nodes at server side
 Inputs specifies where
to watch for logs .
 Filter and grok gives
filtering and regular
expression capability
 Output can be directed
to elastic search /
mongodb Redis/
logstash servers etc

Solution Details – Why Kibana
 Elasticsearch works seamlessly with kibana and gives ability to
interact with your data for visualizing logs and time-stamped data
 Highly scalable and Real-time analysis of streaming data
 Customisable splunk like UI and can integrate with D3.js for
augmenting capability

( ELK Stack Training - https://www.edureka.co/elk-stack-trai... ) This Edureka tutorial on What Is ELK Stack will help you in understanding the fundamentals of Elasticsearch, Logstash, and Kibana together and help you in building a strong foundation in ELK Stack. Below are the topics covered in this ELK tutorial for beginners: 1. Need for Log Analysis 2. Problems with Log Analysis 3. What is ELK Stack? 4. Features of ELK Stack 5. Companies Using ELK Stack

Elk - An introductionHossein Shemshadi

So, what is the ELK Stack? "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Elastic Stack roadmap deep diveElasticsearch

- Elastic provides a search and analytics platform called the Elastic Stack that includes the Elastic Stack, Beats data shippers, and Kibana analytics and visualization tools. - The presentation discussed updates to Elastic's products including performance improvements to search, new features for distributed search across data centers, and enhanced security options for authentication and authorization. - Elastic aims to provide customizable and extensible solutions for users to ingest, store, search, analyze and visualize large volumes of data from various sources.

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...Spark Summit

Moving at the speed of a startup often means rapid iterative development, which can lead to a patchwork of systems and processes. In the early days at Kik (one of the most popular chat apps among U.S. teens), the data team was able to move extremely quickly but often at the expense of scalable data engineering. In this session, Kik’s head of data will share the eight things they did to save time and money. The team took their data stack from a complex combination of systems and processes to a scalable, simple, and robust platform leveraging Apache Spark and Databricks to make data super easy for everyone in the company to use.

Building Data Pipelines with Spark and StreamSetsPat Patterson

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Metadata in upstream sources can ‘drift’ due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an Apache 2.0 licensed open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we’ll look at how SDC’s “intent-driven” approach keeps the data flowing, with a particular focus on clustered deployment with Spark and other exciting Spark integrations in the works.

Elastic Stack RoadmapImma Valls Bernaus

Fighting Cybercrime: A Joint Task Force of Real-Time Data and Human Analytics...Spark Summit

Cybercrime is big business. Gartner reports worldwide security spending at $80B, with annual losses totalling more than $1.2T (in 2015). Small to medium sized businesses now account for more than half of the attacks targeting enterprises today. The threat actors behind these attacks are continually shifting their techniques and toolkits to evade the security defenses that businesses commonly use. Thanks to the growing frequency and complexity of attacks, the task of identifying and mitigating security-related events has become increasingly difficult. At eSentire, we use a combination of data and human analytics to identify, respond to and mitigate cyber threats in real-time. We capture all network traffic on our customers’ networks, hence ingesting a large amount of time-series data. We process the data as it is being streamed into our system to extract relevant threat insights and block attacks in real-time. Furthermore, we enable our cybersecurity analysts to perform in-depth investigations to: i) confirm attacks and ii) identify threats that analytical models miss. Having security experts in the loop provides feedback to our analytics engine, thereby improving the overall threat detection effectiveness. So how exactly can you build an analytics pipeline to handle a large amount of time-series/event-driven data? How do you build the tools that allow people to query this data with the expectation of mission-critical response times? In this presentation, William Callaghan will focus on the challenges faced and lessons learned in building a human-in-the loop cyber threat analytics pipeline. They will discuss the topic of analytics in cybersecurity and highlight the use of technologies such as Spark Streaming/SQL, Cassandra, Kafka and Alluxio in creating an analytics architecture with missions-critical response times.

GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...Yann Cluchey

My talk from GOTO Aarhus, 30th September 2014. Cogenta is a retail intelligence company which tracks ecommerce web sites around the world to provide competitive monitoring and analysis services to retailers. Using its proprietary crawler technology, Lucene and SQL Server, a stream of 20 million raw product data entries is captured and processed each day. This case study looks at how Cogenta uses Elasticsearch to break the shackles imposed by the RDBMS (and a limited budget) to make the data available in real time to its customers. Cogenta uses SQL as its canonical store & for complex reporting, and Elasticsearch for real-time processing & to drive its SaaS web applications. Elasticsearch is easy to use, delivers the powerful features of Lucene and enables the data & platform cost to scale linearly. But… synchronising your existing data in two places presents some interesting challenges such as aggregation and concurrency control. This talk will take a detailed look at how Cogenta how overcame those challenges, with a perpetually changing and asynchronously updated dataset. http://gotocon.com/aarhus-2014/presentation/Cogenta%20-%20Making%20Enterprise%20Data%20Available%20in%20Real%20Time%20with%20Elasticsearch

Elasticsearch + Cascading for Scalable Log ProcessingCascading

Supreet Oberoi's presentation on "Large scale log processing with Cascading & Elastic Search". Elasticsearch is becoming a popular platform for log analysis with its ELK stack: Elasticsearch for search, Logstash for centralized logging, and Kibana for visualization. Complemented with Cascading, the application development platform for building Data applications on Apache Hadoop, developers can correlate at scale multiple log and data streams to perform rich and complex log processing before making it available to the ELK stack.

Redis + Structured Streaming—A Perfect Combination to Scale-Out Your Continuo...Databricks

This presentation discusses how to use Redis and Spark Structured Streaming to process streaming data at scale. The solution breaks down into three functional blocks - data ingest using Redis Streams, data processing using Spark Structured Streaming, and data querying using Spark SQL. Redis Streams are used to ingest streaming click data, Spark Structured Streaming processes the data in micro-batches, and Spark SQL queries the processed data stored as Redis hashes. This combination provides a scalable solution to continuously collect, process, and query data streams in real-time.

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...Lucidworks

This document discusses how Walmart uses Apache Solr as a "not-so-evil twin" to complement their source of truth database and help scale their data infrastructure. It describes how Walmart abstracts the complexity of managing databases, caches, search queries, and messaging to provide scalable querying across database shards. The use of Solr has allowed Walmart to offload queries, recurring reads, analytics

Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...Codit

Technology behind-real-time-log-analytics Data Science Thailand

This document discusses Real Time Log Analytics using the ELK (Elasticsearch, Logstash, Kibana) stack. It provides an overview of each component, including Elasticsearch for indexing and searching logs, Logstash for collecting, parsing, and enriching logs, and Kibana for visualizing and analyzing logs. It describes common use cases for log analytics like issue debugging and security analysis. It also covers challenges like non-consistent log formats and decentralized logs. The document includes examples of log entries from different systems and how ELK addresses issues like scalability and making logs easily searchable and reportable.

Cascalog at May Bay Area Hadoop User Groupnathanmarz

Cascalog is a Clojure-based query language for Hadoop that provides a powerful and easy-to-use tool for data analysis. It allows users to write queries as regular Clojure code, offering features like joins, aggregators, functions, and sorting. Cascalog is unique in that it offers the full power of Clojure at all times by integrating queries directly into the programming language. BackType uses Cascalog for tasks like identifying influencers on social media, determining exposure to URLs, and studying engagement over time.

Rental Cars and Industrialized Learning to Rank with Sean DownesDatabricks

Data can be viewed as the exhaust of online activity. With the rise of cloud-based data platforms, barriers to data storage and transfer have crumbled. The demand for creative applications and learning from those datasets has accelerated. Rapid acceleration can quickly accrue disorder, and disorderly data design can turn the deepest data lake into an impenetrable swamp. In this talk, I will discuss the evolution of the data science workflow at Expedia with a special emphasis on Learning to Rank problems. From the heroic early days of ad-hoc Spark exploration to our first production sort model on the cloud, we will explore the process of industrializing the workflow. Layered over our story, I will share some best practices and suggestions on how to keep your data productive, or even pull your organization out of the data swamp.

Architecture at ScaleElasticsearch

Building a data pipeline to ingest data into Hadoop in minutes using Streamse...Guglielmo Iozzia

Columbia Migrates from Legacy Data Warehouse to an Open Data Platform with De...Databricks

Optimizing Elastic for Search at McQueen SolutionsElasticsearch

Kibana + timelion: time series with the elastic stackSylvain Wallez

The document discusses Kibana and Timelion, which are tools for visualizing and analyzing time series data in the Elastic Stack. It provides an overview of Kibana's evolution and capabilities for creating dashboards. Timelion is introduced as a scripting language that allows users to transform, aggregate, and calculate on time series data from multiple sources to create visualizations. The document demonstrates Timelion's expression language, which includes functions, combinations, filtering, and attributes to process and render time series graphs.

Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armb...Spark Summit

WhereHows: Taming Metadata for 150K Datasets Over 9 Data PlatformsMars Lan

This document summarizes a presentation given by Mars Lan on LinkedIn's metadata management system called WhereHows. WhereHows collects metadata from LinkedIn's various data platforms and provides a unified view. It extracts metadata from sources using ETL jobs, transforms the data into a standardized format, and loads it into a database. WhereHows tracks over 150,000 datasets across 9 data platforms and visualizes the lineage and relationships between datasets to help users understand how data is processed. The system continues to be improved with better metadata collection, compliance features, and a simplified user experience.

Replicate Elasticsearch Data with Cross-Cluster Replication (CCR)Elasticsearch

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...Spark Summit

In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc. Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language. We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.

Big Telco - Yousun JeongSpark Summit

This document provides an overview of SK Telecom's use of big data analytics and Spark. Some key points: - SKT collects around 250 TB of data per day which is stored and analyzed using a Hadoop cluster of over 1400 nodes. - Spark is used for both batch and real-time processing due to its performance benefits over other frameworks. Two main use cases are described: real-time network analytics and a network enterprise data warehouse (DW) built on Spark SQL. - The network DW consolidates data from over 130 legacy databases to enable thorough analysis of the entire network. Spark SQL, dynamic resource allocation in YARN, and integration with BI tools help meet requirements for timely processing and quick

Near Real-Time Analytics with Apache Spark: Ingestion, ETL, and Interactive Q...Databricks

Near real-time analytics has become a common requirement for many data teams as the technology has caught up to the demand. One of the hardest aspects of enabling near-realtime analytics is making sure the source data is ingested and deduplicated often enough to be useful to analysts while writing the data in a format that is usable by your analytics query engine. This is usually the domain of many tools since there are three different aspects of the problem: streaming ingestion of data, deduplication using an ETL process, and interactive analytics. With Spark, this can be done with one tool. This talk with walk you through how to use Spark Streaming to ingest change-log data, use Spark batch jobs to perform major and minor compaction, and query the results with Spark.SQL. At the end of this talk you will know what is required to setup near-realtime analytics at your organization, the common gotchas including file formats and distributed file systems, and how to handle data the unique data integrity issues that arise from near-realtime analytics.

OAP: Optimized Analytics Package for Spark Platform with Daoyuan Wang and Yua...Databricks

Spark SQL is one of the most popular components in big data warehouse for SQL queries in batch mode, and it allows user to process data from various data sources in a highly efficient way. However, Spark SQL is a general purpose SQL engine and not well designed for ad hoc queries. Intel invented an Apache Spark data source plugin called Spinach for fulfilling such requirements, by leveraging user-customized indices and fine-grained data cache mechanisms. To be more specific, Spinach defines a new Parquet-like data storage format, offering a fine-grained hierarchical cache mechanism in the unit of “Fiber” in memory. Even existing Parquet or ORC data files can be loaded using corresponding adaptors. Data can be cached in off-heap memory to boost data loading. What’s more, Spinach has extended the Spark SQL DDL, to allow users to define the customized indices based on relation. Currently, B+ tree and bloom filter are the first two types of indices supported. Last but not least, since Spinach resides in the process of Spark executor, there’s no extra effort in deployment. All you need to do is to pick Spinach from Spark packages when launching the Spark SQL. sing corresponding adaptors. Data can be cached in off-heap memory to boost data loading. What’s more, Spinach has extended the Spark SQL DDL, to allow user to define the customized indices based on relation. Currently B+ tree and bloom filter are the first 2 types of index we’ve supported. Last but not least, since Spinach resides in the process of Spark executor, there’s no extra effort in deployment, all we need to do is to pick Spinach from Spark packages when launch the Spark SQL. Spinach has been imported in Baidu’s production environment since Q4 2016. It helps several teams migrating their regular data analysis tasks from Hive or MR jobs to ad-hoc queries. In Baidu search ads system FengChao, data engineers analyze advertising effectiveness based on several TBs data of display and click logs every day. Spinach brings a 5x boost compared to original Spark SQL (version 2.1), especially in the scenario of complex search and large data volume. It optimizes the average search cost from minutes to seconds, while brings only 3% data size increase for adding a single index.

使用 Elastic Stack 进行端对端安全分析 Elasticsearch

This document discusses security analytics for detection and response. It summarizes collecting data from various sources, normalizing and enriching the data, indexing it, performing automated detection using machine learning, and interactive analysis and response. Key parts of the platform include collecting data using Beats, normalizing it with Elastic Common Schema, enriching it with contextual data, indexing it in Elasticsearch, detecting threats via dynamic correlation and machine learning, and responding by integrating alerts with other systems.

ELK Wrestling (Leeds DevOps)Steve Elliott

Scalable Cloud Solutions with Node.jsmpneuried

More Related Content

What's hot (20)