Anwendungsfälle für Elasticsearch JavaLand 2015

Anwendungsfälle für
Florian Hopf - @fhopf

Installation
# download archive
wget https://download.elasticsearch.org/
elasticsearch/elasticsearch/elasticsearch-1.4.4.zip
# zip is for windows and linux
unzip elasticsearch-1.4.4.zip
# on windows: elasticsearch.bat
elasticsearch-1.4.4/bin/elasticsearch

Zugriff
curl -XGET http://localhost:9200
{
"status" : 200,
"name" : "Anomalito",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.4",
"build_hash" :
"c88f77ffc81301dfa9dfd81ca2232f09588bd512",
"build_timestamp" : "2015-02-19T13:05:36Z",
"build_snapshot" : false,
"lucene_version" : "4.10.3"
},
"tagline" : "You Know, for Search"
}

Document
{
"title" : "Anwendungsfälle für Elasticsearch",
"speaker" : "Florian Hopf",
"date" : "2015-03-24T11:00:00.000Z",
"tags" : ["Java", "Lucene"],
"conference" : {
"name" : "JavaLand",
"city" : "Brühl"
}
}

Speichern
curl -XPOST http://localhost:9200/conferences/talk/
--data-binary @talk-example.json
{
"_index":"conferences",
"_type":"talk",
"_id":"GqjY7l8sTxa3jLaFx67_aw",
"_version":1,
"created":true
}

Speichern
{
"_type":"talk",
"_version":1,
"created":true
}
Index

Speichern
{
"_type":"talk",
"_version":1,
"created":true
}
Type

Lesen
curl -XGET http://localhost:9200/conferences/talk/
GqjY7l8sTxa3jLaFx67_aw?pretty=true
{
"_index" : "conferences",
[...]
"_source":{
"date" : "2015-03-24T11:00:00.000Z",
"conference" : {
"city" : "Brühl"
}
}
}

Sharding
●
Aufteilen eines Index in mehrere Teile
●
Default: 5 Shards pro Elasticsearch-Index
●
Mehrere ES-Instanzen bilden einen Cluster
●
Automatische Verteilung auf die Knoten

Recap
●
Einfache Speicherung von JSON-Dokumenten
●
Index und Type
●
Sharding für große Datenmengen
●
Verteilung ist First-Class-Citizen

Nutzer
●
Atlassian HipChat
●
http://highscalability.com/blog/2014/1/6/how-hipchat-
●
OpenCrowd
●
https://www.elastic.co/use-cases/open-crowd

Anwendungsfälle für Elasticsearch JavaLand 2015

Suche per Parameter
curl -XGET "http://localhost:9200/conferences/talk/_search
?q=elasticsearch&pretty=true"
{"took" : 73,
[…]
"hits" : {
[…]
"hits" : [ {
[…]
"_score" : 0.076713204,
"_source":{
[…]
} } ]
}
}

Query-DSL
curl -XGET "http://localhost:9200/conference/talk/_search" -d'
{
"query": {
"filtered": {
"query": {
"match": {
"title": "elasticsearch"
}
},
"filter": {
"term": {
"tags": "java"
}
}
}
}
}'

Query-DSL
client.prepareSearch("conferences")
.setQuery(filteredQuery(
matchQuery("title", "elasticsearch"),
termFilter("tag", "java")))
.execute()
.actionGet();

Sprache
curl -XGET "http://localhost:9200/conferences/talk/_search
?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

Mapping
curl -XDELETE "http://localhost:9200/conferences/"
curl -XPUT "http://localhost:9200/conferences/“
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
"title": {
"type": "string",
"analyzer": "german"
}
}
}'

Sprache
curl -XGET
"http://localhost:9200/conferences/talk/_search
?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 1,
[...]
}
}

Was noch?
●
Faceting/Aggregations
●
Suggestions
●
Highlighting
●
Sortierung
●
Pagination
●
...

Recap
●
Ausdrucksstarke Suchen über Query DSL
●
Analyzing und Mapping
●
Lucene bietet viele Search-Features

Users
●
GitHub
●
http://exploringelasticsearch.com/github_interview.html
●
http://www.elasticsearch.org/case-study/github/
●
StackOverflow
●
http://meta.stackexchange.com/questions/160100/a-new-
search-engine-for-stack-exchange
●
http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-
stack-overflow/
●
XING
●
http://www.elasticsearch.org/case-study/xing/

Ergebnislisting
curl -XPOST "http://localhost:9200/conferences/_search
" -d'
{
"filter": {
"term": {
"tags": "java"
}
}
}'

Anwendungen
●
Zeige nächste Filiale
●
Filialsuche
●
Sortierung Kleinanzeigen
●
Filterung auf Nähe
●
Social Media-Analysen

Document
{
"date" : "2014-07-17T15:35:00.000Z",
"conference" : {
"city" : "Brühl",
"coordinates": {
"lon": "6.88125",
"lat": "50.7984642"
}
}
}

Mapping
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
[…],
"conference": {
"type": "object",
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}'

Filterung
curl -XPOST "http://localhost:9200/conferences/_search" -d'
{
"filter": {
"geo_distance": {
"conference.coordinates": {
"lon": 8.403697,
"lat": 49.006616
},
"distance": "300km",
"distance_type": "arc"
}
}
}'

Recap
●
Elasticsearch kann mehr als Volltext
●
Ausgefeilte Geo-Algorithmen
●
Sortierung nach Distanz
●
Filterung nach Distanz oder Bereich
●
Berechnung von Distanz

User
●
FourSquare
●
http://engineering.foursquare.com/2012/08/09/fo
ursquare-now-uses-elastic-search-and-on-a-
related-note-slashem-also-works-with-elastic-
search/
●
Gild
●
http://www.elasticsearch.org/case-study/gild/

Logfile-Analyse
●
Zentralisierung Logs aus Anwendungen
●
Zentralisierung Logs über Maschinen
●
Auch ohne Zugriff
●
Leichte Durchsuchbarkeit
●
Real-Time-Analysis / Visualisierung
●
Daten für alle!

Logfile-Analyse
●
Einlesen
●
Logstash
●
Speicherung
●
Elasticsearch
●
Auswertung
●
Kibana

Logstash-Config
input {
file {
path => "/var/log/apache2/access.log"
}
}
filter {
grok {
match => { message => "%{COMBINEDAPACHELOG}" }
}
}
output {
elasticsearch_http {
host => "localhost"
}
}

Recap
●
Einlesen, Anreichern, Speichern von Logevents
●
Zahlreiche Inputs in Logstash
●
Konsolidierung
●
Zentralisierung
●
Auswertung

User
●
CERN
●
https://medium.com/@ghoranyi/needle-in-a-
haystack-873c97a99983
●
Bloomberg
●
http://www.elasticsearch.org/videos/using-
elasticsearch-logstash-kibana-techologies-
centralized-viewing-logs-bloomberg/

Analytics
●
Auswertung auch großer Datenmengen
●
Social Media
●
Data Warehouse
●
Aggregationen auf Feldern
●
Datenkonsolidierung aus unterschiedlichen
Quellen
●
Visualisierung

Aggregations
curl -XGET "http://localhost:9200/devoxx/tweet/_search" -d'
{
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
}
}
}
}'

Aggregations
"aggregations": {
"hashtags": {
"buckets": [
{
"key": "dartlang",
"doc_count": 229
},
{
"key": "java",
"doc_count": 216
},
[...]

Aggregations
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
},
"aggs" : {
"hashtagusers" : {
"terms" : {
"field" : "user.screen_name"
}
}
}
}
}

Aggregations
"key": "scala",
"doc_count": 130,
"hashtagusers": {
"buckets": [
{
"key": "jaceklaskowski",
"doc_count": 74
},
{
"key": "ManningBooks",
"doc_count": 3
},
[...]

Recap
●
Auswertung großer Datenmengen
●
Visualisierung
●
Zahlreiche Aggregationen
●
Bucket-Aggregations
●
Metric-Aggregations

User
●
Cogenta
●
https://www.elastic.co/use-cases/cogenta
●
The Guardian
●
http://www.elasticsearch.org/blog/using-
elasticsearch-and-logstash-to-serve-billions-of-
searchable-events-for-customers/
●
http://www.infoq.com/presentations/elasticsearch
-guardian

Vielen
Dank!
@fhopf
mail@florian-hopf.de
http://blog.florian-hopf.de

Images
●
http://www.morguefile.com/archive/display/685952
●
●
●
●
●
●

Anwendungsfälle für Elasticsearch JavaLand 2015

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Anwendungsfälle für Elasticsearch JavaLand 2015 (20)

More from Florian Hopf (10)

Recently uploaded (20)

Anwendungsfälle für Elasticsearch JavaLand 2015