Introduction to ELK
Download as PPTX, PDF2 likes1,674 views
This document introduces the ELK stack, which consists of Elasticsearch, Logstash, and Kibana. It provides instructions on setting up each component and using them together. Elasticsearch is a search engine that stores and searches data in JSON format. Logstash is an agent that collects logs from various sources, applies filters, and outputs to Elasticsearch. Kibana visualizes and explores the logs stored in Elasticsearch. The document demonstrates setting up each component and running a proof of concept to analyze sample log data.
![Setup Logstash
•Download and un-archive to a local directory
•Create a configuration file
–Needs to have input{}, filter{optional}, output{}
-Sample
input {
file {
path => ["C:/TEMP/server-logs/*.*"]
codec => "json"
}
}
filter{
date {
match => ["eventTime","yyyy-MM-dd HH:mm:ss,SSS"]
add_tag => ["date_matched"]
}
}
output {
stdout{}
elasticsearch {
port => 9200
}
}](https://image.slidesharecdn.com/0061bd2e-74c4-4a8c-8b39-74d6c8f5f4fd-150416092732-conversion-gate02/85/Introduction-to-ELK-4-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Introduction to ELK (20)
Ad
Introduction to ELK
- 1. Introduction to ELK Proof of Concept & Demo
- 2. What is ELK •Elastic Search –Lucene based search engine (Java Stack) –Distributed capability –REST API over Http –Data share using JSON format •Logstash –Ruby agent application –Agent to collect log data in numerous input formats –Filters can be applied –Many Output formats supported •Kibana
- 3. Setup Elastic Search •Download and extract to a local directory •JRE 7 is required •Default configuration is good •Start –C:Programselasticsearch-1.3.2binelasticsearch.bat
- 4. Setup Logstash •Download and un-archive to a local directory •Create a configuration file –Needs to have input{}, filter{optional}, output{} -Sample input { file { path => ["C:/TEMP/server-logs/*.*"] codec => "json" } } filter{ date { match => ["eventTime","yyyy-MM-dd HH:mm:ss,SSS"] add_tag => ["date_matched"] } } output { stdout{} elasticsearch { port => 9200 } }
- 5. Setup Logstash Contd.. •Groking is most common method for parsing log contents •Grok patterns can be configured and passed as an input file •Sample grok pattern –KMMLOG4J %{TIMESTAMP_LOG4J:timestamp} %{THREAD:thread} u:%{LOG4JUSER:user}/d: %{LOG4JCATEGORY:category} %{GREEDYDATA:logmessage} •Master file available here –logstash-1.4.2patternsgrok-patterns
- 6. Setup Logstash Contd.. •Log 4j parsing is tricky when there is multiline parsing required for stack traces. •PatternLayout for log4j that generates logstash json_event formatted data is available –Refer https://github.com/logstash/log4j-jsonevent-layout
- 7. Kibana •Download and unzip to a web server •Default log stash dashboard is built in •Default configuration is good •Edits can be made to app/dashboards/logstash.js •URL is http://localhost:8090/kibana/index.html#/dashboard/file/logstash.json •Or replace default.json with logstash.json
- 8. Proof Of Concept •Download logs files using SCP Tool –SCP program used is pscp, supports non-interactive mode. –SCP should be performed once manually through command line to enable adding the host to trusted list. •Convert log file messages into JSON messages •Place it under log stash monitored directories when ELK stack is up and running. •Perform queries on Kibana •Analyze spikes visually and identify log noises.
- 9. Demo Logstash •Run test.SCPTool •Start elastic search •Start log stash agent •Run test.LogstashFileCopier •Start Kibana server
- 10. Demo – Kibana •Run demo.po.lucene.POJsonBuilder •Build a new dashboard using the data.