Использование Elasticsearch для организации поиска по сайту

me
Developer at Twinslash
email: dima.zhlobo@gmail.com
skype: dima.zhlobo
github: proghat
twitter: @proghat
Dmitry Zhlobo

search is hard
● speed vs. relevancy

search is hard
● real time

search is hard
● real time
● different kinds of data

usual approach
● SELECT * FROM posts WHERE `body`
LIKE '%query%'

usual approach
● SELECT * FROM posts WHERE `body`
LIKE '%query%'
● gem 'thinking-sphinx'
…
Article.search(params[:q])

how search works?
● document 1:
flexible and powerful open source, distributed real-
time search and analytics engine for the cloud...
● document 2:
Apache Mahout has implementations of a wide
range of machine learning and data mining...
● document 3:
Our core algorithms for clustering, classification and
batch based collaborative filtering are implemented
on top of Apache Hadoop using the MapReduce...

how search works?
data
mapreduce
learning
classification
recommenders
analysis

how search works?
data
mapreduce
learning
classification
recommenders
analysis
1 2 3
1 2
2 3
2 3
2
3

● full text search
● real time data
● opensource
● restful api
● distributed
● schema free & document oriented
elasticsearch

analysis
Flexible and powerful <strong>search</strong> engine

analysis
Flexible and powerful <strong>search</strong> engine
char
filters
Mapping
HTML Strip
Pattern Replace

analysis
char
filters
Flexible and powerful search engine

analysis
char
filters
tokenizer
Path Hierarchy
Keyword
Letter
Lowercase
NGram
Standard
Whitespace
Pattern
Edge NGram

analysis
char
filters
tokenizer

analysis
char
filters
tokenizer
token
filters
Stop
Lowercase
Snowball
Synonym
TrimUnique
Normalization
Stemmer
Shingle
Truncate
Reverse

analysis
tokenizer
token
filters
char
filters
flexible powerful search engine

russian morphology
“Он отлично рассказал о лучшем поисковом движке”

russian morphology
отлично отличный

russian morphology
отлично отличный рассказать

russian morphology
отлично отличный рассказать хороший

russian morphology
поисковый

russian morphology
движокпоисковый

russian morphology
движок
“хороший поисковой движок”
хороший движок
поисковый
поисковый

russian morphology
движок
“хороший поисковой движок”
хороший
поисковый
движокпоисковый

phonetic analysis
Eyjafjallajökull

phonetic analysis
Eyjafjallajökull
Eiyafyalayokul
iofiolDkul

analysis
char
filters
tokenizer
token
filters

analysis
char
filters
tokenizer
token
filters
analyzer

analysis
char
filters
tokenizer
token
filters
analyzer
● has name

analysis
char
filters
tokenizer
token
filters
analyzer
● has name
● reusable

analysis
char
filters
tokenizer
token
filters
analysis:
analyzer:
rus_morphology:
type: "custom"
char_filter: ["html_strip"]
tokenizer: "standard"
filter: ["lowercase", "russian_morphology", "stopwords"]

getting started
# Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'

curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }'
curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }'
getting started

# Search documents
curl -XGET "localhost:9200/posts/_search?q=data"
curl -XGET "localhost:9200/posts/_search?q=title:elasticsearch"
getting started

# Search documents
curl -XGET "localhost:9200/posts/_search?q=data"
curl -XGET "localhost:9200/posts/_search?q=title:elasticsearch"
# Update and delete documents
curl -XPUT "localhost:9200/posts/post/1" -d '{ title: “Data" }'
curl -XDELETE "localhost:9200/posts/post/whatever"
getting started

mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
repository: {
type: "string"
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
repository: {
type: "string",
boost: 5
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
repository: {
type: "string",
boost: 5,
analyzer: "repo_name"
}
repo_name: {
tokenizer: "letter",
filter: ["lowercase","phonetic"]
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
description: {
type: "string"
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
description: {
type: "string",
analyzer: "english_text"
}
english_text: {
tokenizer: "standard",
filter:
["lowercase", "stopwords", "snowball"]
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
maintainer: {
properties: {
}
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
maintainer: {
properties: {
name: {
type: "string"
}
}
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
maintainer: {
properties: {
name: {
type: "string",
analyzer: "phonetic"
}
}
}
phonetic: {
tokenizer: "standard",
filter: ["lowercase", "stopwords",
"beidermorse"]
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
maintainer: {
properties: {
email: {
type: "string"
}
}
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
maintainer: {
properties: {
email: {
type: "string",
index: "not_analyzed"
}
}
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
language: {
type: "string"
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
language: {
type: "string",
analyzer: "programming_lang"
}
programming_lang: {
tokenizer: "keyword",
filter: ["lowercase"]
}

mapping
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}
created_at: {
type: "date",
format: "YYYY-MM-DD"
}

mapping
curl -XPOST "localhost:9200/repositories" -d '
settings: {
analysis: {
analyzer: { ... },
filter: { ... }
}
},
mappings: {
repository: {
properties: { ... }
}
}'

mapping
curl -XPOST "localhost:9200/repositories" -d '...'
curl -XPOST "localhost:9200/repositories/repository" -d '
{
maintainer: {
name: "Shay Banon",
},
created_at: "2010-02-08"
}'

search
curl -XGET "localhost:9200/repositories/repository/_search?q=engine"

search
curl -XPOST "localhost:9200/repositories/repository/_search" -d '
{
query: {
match: {
description: "search"
}
}
}'

search
"hits" : {
"total" : 3,
"hits" : [
{
"_score" : 0.22295055,
"_source" : { repository: "elasticsearch/elasticsearch" }
}, {
"_score" : 0.22295055,
"_source" : { repository: "ankane/searchkick" }
}, {
"_score" : 0.095891505,
"_source" : { repository: "karmi/tire" }
}
]
}

search
{
query: {
match: {
_all: "elasticsearch"
}
}
}'

search
"hits" : {
"total" : 2,
"hits" : [
{
"_score" : 5.46875,
}, {
"_score" : 0.04746387,
"_source" : { repository: "karmi/tire" }
}
]
}

facets
{
query: {
match: {
_all: "search"
}
},
facets: {
language: {
terms: {
field: "languages"
}
}
}
}'

facets
"hits" : {
"total" : 2,
"hits" : [ ... ]
},
"facets" : {
"language" : {
"terms" : [
{ "term" : "ruby", "count" : 2 },
{ "term" : "shell", "count" : 1 },
{ "term" : "java", "count" : 1 }
]
}
}

filters
{
query: {
filtered: {
query: {
match: {
_all: "search"
}
},
filter: {
term: { "languages": "java" }
}
}
}
}'

filters
"hits" : {
"total" : 1,
"hits" : [
{
"_score" : 5.46875,
}
]
}

performance and scaling
elasticsearch is web scale

random facts
● bulk operations
● real time
● highlights
● geo types and geo distance facets
● attachments
● “did you mean?” and completions
● common terms
● filters and caching
● river

Использование Elasticsearch для организации поиска по сайту

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Использование Elasticsearch для организации поиска по сайту (20)

More from Olga Lavrentieva (20)

Recently uploaded (20)

Использование Elasticsearch для организации поиска по сайту