An Introduction to Elastic Search.

Jurriaan Persyn
@oemebamo – CTO Engagor

SELECT *
FROM myauwesomewebsite
WHERE `text` LIKE
„%shizzle%‟

TEXT SEARCH WITH PHP/MYSQL
• FULLTEXT index
• Only for CHAR, VARCHAR & TEXT columns
• For MyISAM & InnoDB tables
• Configurable Stop Words
• Types:
• Natural Language
• Natural Language with Query Expansion
• Boolean Full Text

MYSQL FULLTEXT BOOLEAN MODE
Operators:
•+ AND
•- NOT
• OR implied
•() Nesting
•* Wildcard
•“ Literal matches

TEXT SEARCH WITH PHP/MYSQL (CONT‟D)
• Typical columns for search table:
• Type
• Id
• Text
• Priority

• Process:
• Blog posts, comments, …
• Save (filtered) duplicate of text in search table.
• When searching …
• Search table and translate to original data via type/id

This is how most php/mysql sites implement their search, right?

SELECT *
FROM mysearchtable
WHERE MATCH(text)
AGAINST („shizzle‟)

SELECT *
FROM mysearchtable
WHERE MATCH(text)
AGAINST („+shizzle –”ma
nizzle”‟ IN BOOLEAN MODE)

SELECT * FROM jobs
WHERE role = „DEVELOPER‟
AND MATCH(job_description)
AGAINST („node.js‟)

SELECT * FROM jobs j
JOIN jobs_benefits jb ON j.id =
jb.job_id
WHERE j.role = „DEVELOPER‟
AND (MATCH(job_description)
AGAINST („node.js -asp‟ IN
BOOLEAN MODE)
AND jb.free_espresso = TRUE

WHAT IS A SEARCH ENGINE?
• Efficient indexing of data
• On all fields / combination of fields
• Analyzing data
• Text Search
• Tokenizing
• Stemming
• Filtering
• Understanding locations
• Date parsing
• Relevance scoring

TOKENIZING
• Finding word boundaries
• Not just explode(„ „, $text);
• Chinese has no spaces. (Not every single character is a
word.)
• Understand patterns:
• URLs
• Emails
• #hashtags
• Twitter @mentions
• Currencies (EUR, €, …)

STEMMING
• “Stemming is the process for reducing inflected (or sometimes
derived) words to their stem, base or root form.”
• Conjugations
• Plurals
• Example:
• Fishing, Fished, Fish, Fisher > Fish
• Better > Good
• Several ways to find the stem:
• Lookup tables
• Suffix-stripping
• Lemmatization
•…
• Different stemmers for every language.

FILTERING
• Remove stop words
• Different for every language
• HTML
• If you‟re indexing web content, not every character is
meaningful.

UNDERSTANDING LOCATIONS
• Reverse geocoding of locations to longitude & latitude
• Search on location:
• Bounding box searches
• Distance searches
• Searching nearby
• Geo Polygons
• Searching a country

(Note: MySQL also has geospatial indeces.)

RELEVANCE SCORING
• From the matched documents, which ones do you show first?
• Several strategies:
• How many matches in document?
• How many matches in document as percentage of length?
• Custom scoring algorithms
• At index time
• At search time
• … A combination

Think of Google PageRank.

“There‟s an app software for
that.”

APACHE LUCENE
• “Information retrieval software library”
• Free/open source
• Supported by Apache Foundation
• Created by Doug Cutting
• Written in 1999

“There‟s software a Java library
for that.”

ELASTICSEARCH
• “You know, for Search”
• Also Free & Open Source
• Built on top of Lucene
• Created by Shay Banon @kimchy
• Versions
• First public release, v0.4 in February 2010
• A rewrite of earlier “Compass” project, now with scalability
built-in from the very core
• Now stable version at 0.20.6
• Beta branch at 0.90 (working towards 1.0 release)
• In Java, so inherently cross-platform

WHAT DOES IT ADD TO LUCENE?
• RESTfull Service
• JSON API over HTTP
• Want to use it from PHP?
• CURL Requests, as if you‟d do requests to the Facebook
Graph API.
• High Availability & Performance
• Clustering
• Long Term Persistency
• Write through to persistent storage system.

$ cd ~/Downloads
$ wget https://github.com/…/elasticsearch-0.20.5.tar.gz
$ tar –xzf elasticsearch-0.20.5.tar.gz
$ cd elasticsearch-0.20.5/
$ ./bin/elasticsearch

$ cd ~/Downloads
$ wget https://github.com/…/elasticsearch-0.20.5.tar.gz
$ tar –xzf elasticsearch-0.20.5.tar.gz
$ git clone https://github.com/elasticsearch/elasticsearch-
servicewrapper.git elasticsearch-servicewrapper
$ sudo mv elasticsearch-0.20.5 /usr/local/share
$ cd elasticsearch-servicewrapper
$ sudo mv service /usr/local/share/elasticsearch-0.20.5/bin
$ cd /usr/local/share
$ sudo ln -s elasticsearch-0.20.5 elasticsearch
$ sudo chown -R root:wheel elasticsearch
$ cd /usr/local/share/elasticsearch
$ sudo bin/service/elasticsearch start

$ sudo bin/service/elasticsearch start
Starting ElasticSearch...
Waiting for ElasticSearch...
.
.
.
running: PID:83071
$

$ curl -XPUT http://localhost:9200/test/stupid-hypes/planking
-d '{"name":"Planking", "stupidity_level":"5"}'

{"ok":true,"_index":"test","_type":"stupid-
hypes","_id":"planking","_version":1}

$ curl -XPUT http://localhost:9200/test/stupid-hypes/gallon-
smashing -d '{"name":"Gallon Smashing",
"stupidity_level":"5"}'

hypes","_id":"gallon-smashing","_version":1}

"stupidity_level":"10"}'


"stupidity_level":"10", "lifetime":30}’


SCHEMALESS, DOCUMENT ORIENTED
• No need to configure schema upfront
• No need for slow ALTER TABLE –like operations
• You can define a mapping (schema) to customize the indexing
process
• Require fields to be of certain type
• If you want text fields that should not be analyzed (stemming,
…)

“Ok, so it‟s a NoSQL store?”

TERMINOLOGY

MySQL Elastic Search
Database Index
Table Type
Row Document
Column Field
Schema Mapping
Index Everything is indexed
SQL Query DSL
SELECT * FROM table … GET http://…
UPDATE table SET … PUT http://…

DISTRIBUTED & HIGHLY AVAILABLE
• Multiple servers (nodes) running in a cluster
• Acting as single service
• Nodes in cluster that store data or nodes that just help in
speeding up search queries.
• Sharding
• Indeces are sharded (# shards is configurable)
• Each shard can have zero or more replicas
• Replicas on different servers (server pools) for failover
• One in the cluster goes down? No problem.
• Master
• Automatic Master detection + failover
• Responsible for distribution/balancing of shards

SCALING ISSUES?
• No need for an external load balancer
• Since cluster does it‟s own routing.
• Ask any server in the cluster, it will delegate to correct node.

• What if …
• More data > More shards.
• More availability > More replicas per shard.

PERFORMANCE TWEAKING
• Bulk Indexing
• Multi-Get
• Avoids network latency (HTTP Api)
• Api with administrative & monitoring interface
• Cluster‟s availability state
• Health
• Nodes‟ memory footprint
• Alternatives voor HTTP Api?
• Java library
• PHP wrappers (Sherlock, Elastica, …)
• But simplicity of HTTP Api is brilliant to work with, latency is
hardly an issue.

Query DSL Example:

(language:nl OR location.country:be OR location.country:aa)
(tag:sentiment.negative) author.followers:[1000 TO *] (-
sub_category:like) ((-status:857.assigned) (-status:857.done))

FACETS
• Instead of returning the matching documents …
• … return data about the distribution of values in the set of
matching documents
• Or a subset of the matching documents
• Possibilities:
• Totals per unique value
• Averages of values
• Distributions of values
•…

TERMINOLOGY (CONT‟D)

MySQL Elastic Search
SELECT field, COUNT(*) Facet
FROM table GROUP BY field

ADVANCED FEATURES
• Nested documents (Child-Parent)
• Like MySQL joins?
• Percolation Index
• Store queries in Elastic
• Send it documents
• Get returned which queries match
• Index Warming
• Register search queries that cause heavy load
• New data added to index will be warmed
• So next time query is executed: pre cached

WHAT ARE MY OTHER OPTIONS?
• RDBMS
• MySQL, …
• NoSQL
• MongoDB, …
• Search Engines
• Solr
• Sphinx
• Xapian
• Lucene itself
• SaaS
• Amazon CloudSearch

… VS. SOLR
•+
• Also built on Lucene
• So similar feature set
• Also exposes Lucene functionality, like Elastic Search, so
easy to extend.
• A part of Apache Lucene project
• Perfect for Single Server search
•-
• Clustering is there. But it‟s definitely not as simple as
ElasticSearch‟
• Fragmented code base. (Lots of branches.)

Engagor used to run on Solr.

… VS. SPHINX
•+
• Great for single server full text searches;
• Has graceful integration with SQL database;
• (Eg. for indexing data)
• Faster than the others for simple searches;
•-
• No out of the box clustering;
• Not built on Lucene; lacks some advanced features;

Netlog & Twoo use Sphinx.

WANT TO USE IT?
• In an existing project:
• As an extra layer next to your data …
• Send to both your database & elasticsearch;
• Consistency problems?;
• Or as replacement for database
• Elastic is as persistent as MySQL;
• If you don‟t need RDBMS features;
• @Engagor: Our social messages are only in Elastic

“Users are incredibly bad at
finding and researching things
on the web.”
Nielsen (March 2013)
http://www.nngroup.com/articles/search-navigation/

“Pathetic and useless are
words that come to mind after
this year‟s user testing.”
Nielsen (March 2013)
http://www.nngroup.com/articles/search-navigation/

“I‟m searching for apples and
pears.”

“It‟s too young. Is it even stable
enough?”
Your boss (Tomorrow Morning)

elasticsearch.org

irc.freenode.net
#elasticsearch

elasticsearch
node.js
socket.io
real time notifications
rabbitmq
backbone.js
gearman

jobs@engagor.com

$ cd /usr/local/share/elasticsearch
$ sudo bin/service/elasticsearch stop

Sources include:
• http://www.elasticsearch.org/videos/2010/02/07/es-introduction.html
• http://www.elasticsearchtutorial.com/
• http://www.slideshare.net/clintongormley/cool-bonsai-cool-an-introduction-to-elasticsearch
• http://www.slideshare.net/medcl/elastic-search-quick-intro
• http://www.slideshare.net/macrochen/elastic-search-apachesolr-10881377
• http://www.slideshare.net/cyber_jso/elastic-search-introduction
• http://www.slideshare.net/infochimps/elasticsearch-v4
• http://engineering.foursquare.com/2012/08/09/foursquare-now-uses-elastic-search-and-on-a-related-note-slashem-also-
works-with-elastic-search/
• http://stackoverflow.com/questions/10213009/solr-vs-elasticsearch
• http://stackoverflow.com/questions/11115523/how-does-amazon-cloudsearch-compares-to-elasticsearch-solr-or-sphinx-
in-terms-o
• http://blog.socialcast.com/realtime-search-solr-vs-elasticsearch/

An Introduction to Elastic Search.

More Related Content

What's hot (20)

Similar to An Introduction to Elastic Search. (20)

More from Jurriaan Persyn (7)

Recently uploaded (20)

An Introduction to Elastic Search.

Editor's Notes