Elastic search and Symfony3 - A practical approach

Nadhiir Rosun / @nads_rosun
E l a s t i c s e a r c h a n d S y m f o n y
31/03/2017
A p r a c t i c a l a p p r o a c h

2
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

3
Introduction
Highly scalable open-source full-text search and
analytics engine
Allows you to store, search, and analyze big volumes of
data quickly and in near real time.
Build on top of Apache Lucene
¬ Popular/powerfull full-text search engine library
Developed in Java

4
Introduction / Projects using ES
Wikipedia
¬ Full-text search
¬ Highlighted search snippets
¬ Search-as-you-type
¬ Did-you-mean suggestions
The Guardian
¬ Real–time analytics of visitor logs
Stack Overflow
¬ Full-text search with geolocation queries
¬ Find related questions and answers
GitHub
¬ Query 130 billion lines of code

5
Introduction / Use case 1
An online store
Customers can search for products
Big volume of products / large amount of transactions
Product catalog to be full-text search, autocomplete
suggestions, did-you-mean suggestions

6
Introduction / Use case 2
Analytics/business intelligence needs
Collect log and transaction data
 Analyze the data to look for trends, statistics &
summarizations
Build custom dashboard to visualize aspects of the data

7
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

8
ES versus RDBMS
Document oriented (NoSQL)
Schema free
Easily distributed & scalable
All fields indexed
Comprehensive and powerfull RESTful API
Advanced search
¬ Full-text search
¬ Aggregations

9
ES versus RDBMS / Teminology
Relational database ES
Database Index
Table Type
Row Document
SELECT GET
INSERT PUT/POST
UPDATE PUT/POST
DELETE DELETE

10
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

11
Talking to ES / Installation
https://www.elastic.co/downloads/elasticsearch
Check if ES is running
¬ http://localhost:9200/
{
"name" : "Cp8oag6",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
"version" : {
"number" : "5.3.0",
"build_hash" : "f27399d",
"build_date" : "2016-03-30T09:51:41.449Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}

12
Talking to ES
RESTful API with JSON over HTTP
¬ Curl
¬ Guzzle
¬ Kibana
PHP Client
¬ Official php client
- https://github.com/elastic/elasticsearch-php
¬ Elastica
- https://github.com/ruflin/Elastica
Framework specific
¬ Symfony : FOSElasticaBundle
- https://github.com/FriendsOfSymfony/FOSElasticaBundle
¬ Drupal 8 : Elasticsearch Connector
- https://www.drupal.org/project/elasticsearch_connector

14
Talking to ES / Index a document
PUT /[index]/[type]/[doc_id]
Request
PUT /providers/accomodation/1
{
"name" : "Beaurivage Appartments",
"about" : "Room or studio accommodation in the centre of the village",
"city" : "Grand Baie",
"star" : 3,
"facility" : ["Internet", "Pool", “Television”, "Parking"]
}

15
Talking to ES / Index a document
Response
{
"_index": "providers",
"_type": "accomodation",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}

16
Talking to ES / Retrieve a document
GET /[index]/[type]/[doc_id]
Request
GET /providers/accomodation/1

17
Talking to ES / Retrieve a document
Response
{
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"name": "Beaurivage Appartments",
"about": "Room or studio accommodation in the centre of the village",
"city": "Grand Baie",
"star": 3,
"facility": ["Internet", "Pool", "Television", "Parking"]
}
}

18
Talking to ES / Update a document
PUT /[index]/[type]/[doc_id]
Request
PUT /providers/accomodation/1
{
"star" : 4,
}

19
Talking to ES / Update a document
Response
{
"_id": "1",
"_version": 2,
"result": “updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}

20
Talking to ES / Delete a document
DELETE /[index]/[type]/[doc_id]
Request
DELETE /providers/accomodation/1

21
Talking to ES / Delete a document
Response
{
"found": true,
"_id": "1",
"_version": 3,
"result": "deleted",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}

22
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

23
Architecture / Cluster
A collection of one or more nodes (servers)
¬ Sharing the same cluster name
¬ Working together to share data/workload
Perfectly fine to have a cluster with only a single node

24
Architecture / Node
A single server that is part of your cluster
¬ A single instance of ES
Master node
¬ One node is elected as MASTER
¬ Manage cluster-wide changes

25
Architecture / Shard
Think of shards as containers for data
¬ Allows to horizontally split/scale your content volume
¬ Allows to distribute and parallelize operations(Performance)
Primary shard :
¬ Document is first indexed on a single primary shard
Replica shard :
¬ Is just a copy of a primary shard
¬ Protect against hardware failure / share server load

26
Architecture / Index
A collection of documents with similar characteristics
¬ An index for :
- Customer data
- Product catalog
- Order data
Create an index
PUT /providers
{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}

27
Architecture / Type
A logical category/partition of your index
Within an index, you can define one or more types
Has a list of fields mapping
A type is defined for documents that have a set of
common fields
¬ Product index with perishable type and non-perishable type √
¬ Blog index with article type and log type ×

28
Architecture / Document
A basic unit of information that can be indexed
A JSON object stored in a single shard
Uniquely identified by its index, type and id
{
"_id": "1",
"_version": 1,
"_source": {
"name": "Beaurivage Appartments",
"about": "Room or studio accommodation in the centre of the village",
"city": "Grand Baie",
"star": 3,
"facility": ["Internet", "Pool", "Television", "Parking"]
}
}

29
Architecture / Mapping
ES automatically creates a mapping/schema
GET /[index]/_mapping/[type]
GET /providers/_mapping/accomodation
{
"providers": {
"mappings": {
"accomodation": {
"properties": {
"about": { …, "type": "text", … },
"city": { …, "type": "text", … },
"facility": { …, "type": "keyword", … },
"name": { …, "type": "text", … },
"stars": { …, "type": "long“, … }
}
…
}

30
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

31
Querying ES / Query DSL
JSON-style domain-specific language to execute queries
GET /providers/accommodation/_search
{
"query": {
"bool": {
"must": [
{ "match": { “name": " Appartments " }},
{ "match": { “about": " Appartments " }}
],
"filter": [
{ "term": { " facility ": “Internet" }},
{ "range": { “stars": { "gte": “3" }}}
]
}
}
}

32
Match all
¬ Simply matches all documents
- SELECT * FROM providers.accommodation
{
"query": {
"match_all": {}
}
}

33
Match
¬ Query a field
- SELECT * FROM providers.accommodation p WHERE p.name =
‘Beaurivage’
{
"query": {
"match": {
“name” : “Beaurivage”
}
}
}

34
Range
¬ Find numbers or dates that fall into a specified range
- SELECT * FROM providers.accommodation p WHERE p.stars > 3
{
"query": {
"range" : {
“stars" : {
"gte" : 3,
}
}
}
}

35
Term
¬ Finds documents that contain the exact term
- SELECT * FROM providers.accommodation p WHERE p.facilities =
‘Internet’
{
"query": {
"term" : { “facility" : “Internet" }
}
}

36
Bool
¬ Combine query clauses
- Must :
› The clause must appear in matching documents
› Contribute to score
- Filter :
› The clause must appear in matching documents
› Scoring is ignored
{
"query": {
"bool": {
"must": [ … ],
"filter": [ … ]
}
}
}

37
Pagination
¬ From
- Similar to offset in sql
¬ Size
- Similar to limit in sql
¬ SELECT * FROM providers.accommodation p LIMIT 10 OFFSET 15
{
"query": {
“match_all” : {}
},
“from” : 15,
“size” : 10
}

38
Querying ES / Full-text search
Structured search
¬ Something either belongs in the set or it does not
¬ Simply includes or excludes documents
¬ Does not worry about document relevance or scoring
¬ Very fast and cachable
Full-text search
¬ Finds how well a documents match the search keywords
¬ Analysis/Inverted index
¬ Each document gets a score
¬ Return documents sorted by relevance

39
Case study : 2 documents
{
"star" : 3,
},
{
"name" : "Hotel Allamanda",
"about" : "Hotel room in front of the best lagoon of the island.",
"city" : "Tamarin",
"star" : 5,
"facility" : ["Internet", “Beach", “Telephone”, "Parking"]
},

40
Case study : Full-text search query
GET /providers/accomodation/_search
{
"query": {
"match": {
"about": "Room accomodation"
}
}
}

41
Response
{ "took": 218, "timed_out": false,
"_shards": { "total": 5, "successful": 5, "failed": 0 },
"hits": {
"total": 2, "max_score": 0.28488502,
"hits": [
{
"_index": "providers", "_type": "accomodation", "_id": "1",
"_score": 0.28488502,
"_source": { … }
},
{
"_index": "providers", "_type": "accomodation", "_id": "2",
"_score": 0.24257512,
"_source": { … }
}
}
}

42
Inverted Index
¬ Designed to allow very fast full-text searches
¬ Consists of :
- a list of all the unique words that appear in any document
- For each word, a list of the documents in which it appears.
1.The quick brown fox jumped over the lazy dog
1.Quick brown foxes leap over lazy dogs in summer

43
Split into separate tokens
List of unique tokens
List in which document each
term appears
Notice:
- Quick and quick appear as
separate terms
- fox and foxes share the same
root word

44
Analyzers
¬ Character Filters
- Tidy up string
- Strip out HTML and ‘&’
¬ Tokenizer
- Split into tokens/terms
- Every whitespace/punctuation
¬ Token Filters
- Can change the tokens
› Lowercase
› Add synonyms
› Remove stopwords (a, and, the…)
Token Filters
Tokenizer
Character
filters

45
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

46
Case study / SF3 resto search engine
Creating a simple restaurant search engine
¬ Full-text search
¬ Facets
¬ Pagination
Techno :
¬ Symfony 3
- FOSElasticaBundle
- PagerBundle
¬ Elasticsearch

47

48
MySQLES
Restaurant
- Name
- About
- City
- facilities
Restaurant
- Name
- About
- City
- photo
- facilities
App 1. New/Update/Delete
Serializer
2. Doctrine object
3. JSON object
Indexing

49
MySQLES
Restaurant
- Name
- About
- City
- facilities
Restaurant
- Name
- About
- City
- photo
- facilities
App
Finder
5.Doctrine object
2. JSON object
1. Search
3,4. Fetch from db
Searching

50
Installing the bundles with composer
your_project/composer.json
"require": {
…
“friendsofsymfony/elastica-bundle": "^3.0",
"ruflin/elastica": "3.1.1 as 2.99.0",
"white-october/pagerfanta-bundle": "dev-master",
}

51
Register the bundles
your_project/app/AppKernel.php
$bundles = [
…
new FOSElasticaBundleFOSElasticaBundle(),
new WhiteOctoberPagerfantaBundleWhiteOctoberPagerfantaBundle(),
];

52
Configuration
¬ Add ES server address in ‘parameters.yml’
your_project/app/config/parameters.yml
parameters:
…
elastica_host: localhost
elastica_port: 9200
¬ Create/import new configuration file ‘fos_elastica.yml’
your_project/app/config/config.yml
imports:
…
- { resource: fos_elastica.yml }

53
your_project/app/config/fos_elastica.yml
fos_elastica:
clients:
default: { host: "%elastica_host%", port: "%elastica_port%" }
indexes:
prov:
client: default
types:
restaurant:
mappings:
name: ~
about: ~
city: ~
facilities:
type: keyword
persistence:
driver: orm
model: AppBundleEntityRestaurant
finder: ~
provider: ~
listener: ~

54
Persistence
¬ Defines how FOSElasticaBundle will index your docs depending
on your symfony entities
Driver
¬ Driver to use (ORM)
Model
¬ Allow to define an ES doc from a symfony entity
Finder
¬ Search interface (deserialize JSON objects into doctrine objects)
Provider
¬ Indexing interface (serialize doctrine objects to JSON objects)
Listener
¬ List of listeners for which the indexing is called
- Default : insert, update & delete

55
File structure

56
Create Entity ‘Restaurant’
your_project/src/AppBundle/Entity/Restaurant.php
class Restaurant
{
protected $id;
protected $name;
protected $about;
protected $city;
protected $photo;
protected $facilities;
//getters and setters
}

57
Index the doctrine objects to ES
¬ Uses the provider defined in fos_elastica.yml
¬ Serialize Doctrine objects to JSON using the Symfony Serializer
component
- Uses the predefined mapping in this process
From the command line
php bin/console fos:elastica:populate

58
Create the form model
¬ Mapped to the FormType
¬ Contain two properties
- title
- facilities
your_project/src/AppBundle/Model/SearchModel.php
class SearchModel
{
protected $title; //string
protected $facilities; //array
//getters and setters
}

59
Create the form type
¬ Contain two fields : title, facilities
your_project/src/AppBundle/Form/Type/SearchType.php
$builder
->add('title', null, array(
'required' => false,
))
->add('facilities', ChoiceType::class, array(
'choices' => array(
'Internet' => 'Internet',
'Parking' => 'Parking',
'Toilet' => 'Toilet',
'Beach' => 'Beach',
'Card Payment' => 'Card Payment',
),
'expanded' => true,
'multiple' => true,
))
->add('search', SubmitType::class);

60
Bundle everything in your search action
your_project/src/AppBundle/Controller/DefaultController.php
$perPage = 8;
// Create form
$searchModel = new SearchModel();
$searchForm = $this->createForm(SearchType::class, $searchModel);
// Get form data
$searchForm->handleRequest($request);
$searchModel = $searchForm->getData();
// Perform query in ES
$elasticaManager = $this->container->get('fos_elastica.manager');
$results = $elasticaManager->getRepository('AppBundle:Restaurant')->search($searchModel);
// Pagination
$adapter = new ArrayAdapter($results);
$pager = new Pagerfanta($adapter);
$pager->setMaxPerPage($perPage);
$pager->setCurrentPage($page);
// return results, pager and form to your template

61
ES search repository
your_project/src/AppBundle/Repository/SearchRepository.php
$boolQuery = new ElasticaQueryBoolQuery();
// Search Field name
$queryName = new ElasticaQueryMatch();
$queryName->setFieldQuery('name', $searchModel->getTitle());
$boolQuery->addShould($queryName);
// Search Field about
$queryAbout = new ElasticaQueryMatch();
$queryAbout->setFieldQuery('about', $searchModel->getTitle());
$queryAbout->setFieldParam('about', 'analyzer', 'english');
$boolQuery->addShould($queryAbout);
…
// Terms
if (count($searchModel->getFacilities()) > 0) {
$tagsQuery = new ElasticaQueryTerms();
$tagsQuery->setTerms('facilities', $searchModel->getFacilities());
$boolQuery->addMust($tagsQuery);
}
$query = ElasticaQuery::create($boolQuery);

62
1. Introduction
2. ES versus RDBMS
3. Talking to ES
4. Architecture
5. Querying ES
6. Case study
7. Conclusions

63
Conclusion
Suggester API
Message broker (RabbitMQ)

extension interactive
14, avenue des Lataniers
72238 Quatre Bornes
Thank you 

Elastic search and Symfony3 - A practical approach

Recommended

More Related Content

What's hot (20)

Similar to Elastic search and Symfony3 - A practical approach (20)

Recently uploaded (20)

Elastic search and Symfony3 - A practical approach