ElasticSearch AJUG 2013

Agenda
• Me
• ElasticSearch Basics
• Concepts
• Network / Discovery
• Data Structure
• Inverted Index
• The REST API
• Bulk API
• Percolator
• Java Integration
• Stuff I didn’t cover
2

Me
3
• Roy Russo
• Former JBoss Portal Co-Founder
• LoopFuse Co-Founder
• ElasticHQ Founder
• http://www.elastichq.org

The Basics
• Document - Oriented Search Engine
• JSON, Lucene
• No Schema
• Mapping Types
• Horizontal Scale, Distributed
• REST API
• Vibrant Ecosystem
• Apps, Plugins, Hosting
4

The Basics - Distro
• Download and Run
5
├── bin
│ ├── elasticsearch
│ ├── elasticsearch.in.sh
│ └── plugin
├── config
│ ├── elasticsearch.yml
│ └── logging.yml
├── data
│ └── cluster1
├── lib
│ ├── elasticsearch-x.y.z.jar
│ ├── ...
│ └──
└── logs
├── elasticsearch.log
└── elasticsearch_index_search_slowlog.log
└── elasticsearch_index_indexing_slowlog.log
Executables
Log files
Node Configs
Data Storage

The Basics - Glossary
• Node = One ElasticSearch instance (1 java proc)
• Cluster = 1..N Nodes w/ same Cluster Name
• Index = Similar to a DB
• Named Collection of Documents
• Maps to 1..N Primary shards && 0..N Replica shards
• Type = Similar to a DB Table
• Document Definition
• Shard = One Lucene instance
• Distributed across all nodes in the cluster.
6

The Basics - Document Structure
• Modeled as a JSON object
7
{
"genre": "Crime",
“language": "English",
"country": "USA",
"runtime": 170,
"title": "Scarface",
"year": 1983
}
{
"_index": "imdb",
"_type": "movie",
"_id": "u17o8zy9RcKg6SjQZqQ4Ow",
"_version": 1,
"exists": true,
"_source": {
"genre": "Crime",
"language": "English",
"country": "USA",
"runtime": 170,
"title": "Scarface",
"year": 1983
}
}

• Document Metadata fields
• _id
• _type : mapping type
• _source : enabled/disabled
• _timestamp
• _ttl
• _size : size of uncompressed _source
• _version
8

• Mapping:
• ES will auto-map fields
• You can specify mapping, if needed
• Data Types:
• String
• Analyzers, Tokenizers, Filters
• Number
• Int, long, float, double, short, byte
• Boolean
• Datetime
• formatted
• geo_point, geo_shape, ip
• Attachment (requires plugin)
9

Lucene – Inverted Index
• Which presidential speeches contain the words “fair”
• Go over every speech, word by word, and mark the speeches that
contain it
• Linear to number of words
• Fails at large scale
10

• Inverting Obama
• Take all the speeches
• Break them down by word (tokenize)
• For each word, store the IDs of the speeches
• Sort all words (tokens)
• Searching
• Finding the word is fast
• Iterate over document IDs that are referenced
11
Token Doc Frequency Doc IDs
Jobs 2 4,8
Fair 5 1,2,4,8,42
Bush 300 1,2,3,4,5,6, …

• Not an algorithm
• Implementations vary
12

Cluster Topology
• 4 Node Cluster
• Index Configuration:
• “A”: 2 Shards, 1 Replica
• “B”: 3 Shards, 1 Replica
13
A1 A2
B2 B2 B1
B3
B1 A1 A2
B3

The Basics - Shards
• Paths…
• Primary Shard:
• First time Indexing
• Index has 1..N primary shards (default: 5)
• # Not changeable once index created
• Replica Shard:
• Copy of the primary shard
• Can be changed later
• Each primary has 0..N replicas
• HA:
• Promoted to primary if primary fails
• Get/Search handled by primary||replica
14

The Basics - Shards
• Shard Stages
• UNASSIGNED
• INITIALIZING
• STARTED
• RELOCATING
• Viewed in Cluster State
• Routing table : from indices perspective
• Routing nodes
15

The Basics - Searching
• How it works:
• Search request hits a node
• Node broadcasts to every shard in the index (primary & replica)
• Each shard performs query
• Each shard returns results
• Results merged, sorted, and returned to client.
• Problems:
• ES has no idea where your document is
• Randomly distributed around cluster
• Broadcast query to 100 nodes
• Performance degrades
16

The Basics - Shards
• Shard Allocation Awareness
• cluster.routing.allocation.awareness.attributes: rack_id
• Example:
• 2 Nodes with node.rack_id=rack_one
• Create Index 5 shards / 1 replica (10 shards)
• Add 2 Nodes with node.rack_id=rack_two
• Shards RELOCATE to even distribution
• Primary & Replica will NOT be on the same rack_id value.
• Shard Allocation Filtering
• node.tag=val1
• index.routing.allocation.include.tag:val1,val2
17
curl -XPUT localhost:9200/newIndex/_settings -d '{
"index.routing.allocation.include.tag" : "val1,val2"
}'

The Basics - Routing
18
curl -XPOST localhost:9200/crunchbase/person/1?routing=xerox -d '{
...
}'
curl -XPOST localhost:9200/crunchbase/_search?routing=xerox -d '{
"query" : {
"filtered" : {
"query" : { ... },}
}
}'
Routing can be used to constrain the shards being
searched on, otherwise, all shards will be hit

Discovery
• Nodes discover each other using multicast.
• Unicast is an option
• Each cluster has an elected master node
• Beware of split-brain
• discovery.zen.minimum_master_nodes
• N/2+1, where N>2
• N: # of master nodes in cluster
19
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1", "host2:port", "host3"]

Nodes
• Master node handles cluster-wide (Meta-API) events:
• Node participation
• New indices create/delete
• Re-Allocation of shards
• Data Nodes
• Indexing / Searching operations
• Client Nodes
• REST calls
• Light-weight load balancers
• Beware of Heap Size
• ES_HEAP_SIZE: ~1/2 machine memory
20

Cluster State
• Cluster State
• Node Membership
• Indices Settings and Mappings (Types)
• Shard Allocation Table
• Shard State
• cURL -XGET http://localhost:9200/_cluster/state?pretty=1'
21

Cluster State
• Changes in State published from Master to other nodes
22
PUT /newIndex
2 3
1 (M)
CS1 CS1 CS1
2 3
1 (M)
CS2 CS1 CS1
2 3
1 (M)
CS2 CS2 CS2

REST API
23
• Building IMDB
• Two Indexes

REST API
• Create Index
• action.auto_create_index: 0
• Index Document
• Dynamic type mapping
• Versioning
• ID specification
• Parent / Child (/1122?parent=1111)
• Explicit Refresh (?refresh=1)
• Timeout flag (?timeout=5m)
24

REST API – Versioning
• Every document is Versioned
• Version assigned on creation
• Version number can be assigned
• Re-Index, Update, and Delete update Version
25

REST API - Update
• Update using partial data
• Partial doc merged with existing
• Fails if document doesn’t exist
• “Upsert” data used to create a doc, if doesn’t exist
26
{
“upsert" : {
“title": “Blade Runner”
}
}

REST API
• Exists
• No overhead in loading
• Status Code Result
• Delete
• Get
• Multi-Get
27
{
"docs" : [
{
"_id" : "1"
"_index" : "imdb"
"_type" : "movie"
},
{
"_id" : "5"
"_index" : "oldmovies"
"_type" : "movie"
"_fields" " ["title", "genre"]
}
]
}

REST API - Search
• Free Text Search
• URL Request
• http://localhost:9200/imdb/movie/_search?q=scar*
• Complex Query
• http://localhost:9200/imdb/movie/_search?q=scarface+OR
+star
• http://localhost:9200/imdb/movie/_search?q=(scarface+O
R+star)+AND+year:[1981+TO+1984]
• Term, Boolean, range, fuzzy, etc…
28

REST API - Search
• Search Types:
• http://localhost:9200/imdb/movie/_search?q=(scarface+OR+star)+A
ND+year:[1981+TO+1984]&search_type=count
• http://localhost:9200/imdb/movie/_search?q=(scarface+OR+star)+A
ND+year:[1981+TO+1984]&search_type=query_then_fetch
• Query and Fetch:
• Executes on all shards and return results
• Query then Fetch:
• Executes on all shards. Only some information returned for rank/sort,
only the relevant shards are asked for data
29

REST API – Query DSL
30
http://localhost:9200/imdb/movie/_search?q=(scarface+OR+star)+AND+year:[1981+TO+1984]
curl -XPOST 'localhost:9200/_search?pretty' -d '{
"query" : {
"bool" : {
"must" : [
{
"query_string" : {
"query" : “scarface or star"
}
},
{
"range" : {
“year" : { "gte" : 1981 }
}
}
]
}
}
}'
Becomes…

• Query String Request use Lucene query syntax
• Limited
• Error-prone
• Instead use “match” query
31
curl -XPOST 'localhost:9200/_search?pretty' -d '{
"query" : {
"bool" : {
"must" : [
{
“match" : {
“message" : “scarface star"
}
},
{
"range" : {
“year" : { "gte" : 1981 }
}
}
]
…
Automatically builds
a boolean query

• Match Query
• Boolean Query
• Must: document must match query
• Must_not: document must not match query
• Should: document doesn’t have to match
• If it matches… higher score
• Compound queries
32
{
"bool":{
"must":[
{
"match":{
"color":"blue"
}
},
{
"match":{
"title":"shirt"
}
}
],
"must_not":[
{
"match":{
"size":"xxl"
}
}
],
"should":[
{
"match":{
"textile":"cotton"
}
{
“match”:{
“title”:{
“type”:“phrase”,
“query”:“quick fox”,
“slop”:1
}
}
}

• Range Query
• Numeric / Date Types
• Prefix/Wildcard Query
• Match on partial terms
• Fuzzy Query
• Similar looking text matched.
• RegExp Query
33
{
"range":{
"founded_year":{
"gte":1990,
"lt":2000
}
}
}

• Geo_bbox
• Bounding box filter
• Geo_distance
• Geo_distance_range
34
{
"query":{
"filtered":{
"query":{
"match_all":{
}
},
"filter":{
"geo_bbox":{
"location":{
"top_left":{
"lat":40.73,
"lon":-74.1
},
"bottom_right":{
"lat":40.717,
"lon":-73.99
}
…
{
"query":{
"filtered":{
"query":{
"match_all":{
}
},
"filter":{
"geo_distance":{
"distance":"400km"
"location":{
"lat":40.73,
"lon":-74.1
}
}

REST API – Bulk Operations
• Bulk API
• Minimize round trips with index/delete ops
• Individual response for every request action
• In order
• Failure of one action will not stop subsequent actions.
• localhost:9200/_bulk
• No pretty-printing. Use n
35
{ "delete" : { "_index" : “imdb", "_type" : “movie", "_id" : "2" } }n
{ "index" : { "_index" : “imdb", "_type" : “actor", "_id" : "1" } }n
{ "first_name" : "Tony", "last_name" : "Soprano" }n
...
{ “update" : { "_index" : “imdb", "_type" : “movie", "_id" : "3" } }n
{ doc : {“title" : “Blade Runner" } }n

Percolate API
• Reversing Search
• Store queries and filter (percolate) documents through them.
36
curl -XPUT localhost:9200/_percolator/stocks/alert-on-nokia -d '{
"query" : {
"boolean" : {
"must" : [
{ "term" : { "company" : "NOK" }},
{ "range" : { "value" : { "lt" : "2.5" }}}
]
}
}
}'
curl -X GET localhost:9200/stocks/stock/_percolate -d '{
"doc" : {
"company" : "NOK",
"value" : 2.4
}
}'

Java Integration
• Client list: http://www.elasticsearch.org/guide/clients/
• Java Client, JEST
• Limited
• https://github.com/searchbox-io/Jest
• Spring Data:
• Uses TransportClient
• Implementation of ElasticsearchRepository aligns with generic
Repository interfaces.
• ElasticSearchCrudRepository extends PagingandSortingRepository
• https://github.com/spring-projects/spring-data-elasticsearch
37
@Document(indexName = "book", type = "book", indexStoreType = "memory", shards = 1, replicas = 0, refreshInterval = "-1")
public class Book {
…
}
public interface ElasticSearchBookRepository extends ElasticsearchRepository<Book, String> {
}

Stuff I didn’t cover…
• Analyzers
• Tokenizers
• Token Filters
• Rivers
• RabbitMQ, MySQL, JDBC
38
$ curl -XGET
'localhost:9200/_analyze?tokenizer=whitespace&filters=lowercase
,stop&pretty=1' -d '
The quick Fox Jumped
'
{
"tokens" : [ {
"token" : "quick",
"start_offset" : 5,
"end_offset" : 10,
"type" : "word",
"position" : 2
}, {
"token" : "fox",
"start_offset" : 11,
"end_offset" : 14,
"type" : "word",
"position" : 3
}, {
"token" : "jumped",
"start_offset" : 15,
"end_offset" : 21,
"type" : "word",
"position" : 4
} ]
}

B’what about Mongo?
• Mongo:
• General purpose DB
• ElasticSearch:
• Distributed text search engine
… that’s all I have to say about that.
39

ElasticSearch AJUG 2013

Recommended

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to ElasticSearch AJUG 2013 (20)

Recently uploaded (20)

ElasticSearch AJUG 2013