Mastering ElasticSearch with Ruby and Tire
15 likes7,814 views
A tutorial on what is ElasticSearch and how to use it effectively in a real project. The talk discusses how to integrate a search experience in an existing application, showing all the steps from downloading&configuring elastic search, to building the UI and wire the search logic (in a Rails application). The talk was presented at RubyConf 2013.
![Who am I?
I’m Luca Bonmassar (@openmosix)
# 31
# Italian living in San Francisco (and Stockholm)
I work at Gild
I love building products
[for fun, profit and boredom]
Tuesday, 5 November 13](https://image.slidesharecdn.com/rubytireandelasticsearch-131108122003-phpapp01/85/Mastering-ElasticSearch-with-Ruby-and-Tire-2-320.jpg)
![ElasticSearch
II
Open source search+analytics engine
Distributed
[near] Realtime search
Multi tenant
Built on Apache Lucene
REST APIs
JSON documents
Tuesday, 5 November 13](https://image.slidesharecdn.com/rubytireandelasticsearch-131108122003-phpapp01/85/Mastering-ElasticSearch-with-Ruby-and-Tire-14-320.jpg)
![Tire setup
cat “gem ‘tire’” > Gemfile && bundle install
> cat config/initializers/elasticsearch.rb
...
Tire::Configuration.url('http://localhost:9200')
Tire.configure { logger "#{Rails.root}/log/elasticsearchqueries.log" } if ENV['ES_LOG']
Tuesday, 5 November 13](https://image.slidesharecdn.com/rubytireandelasticsearch-131108122003-phpapp01/85/Mastering-ElasticSearch-with-Ruby-and-Tire-22-320.jpg)
Ad
More Related Content
What's hot (19)
Similar to Mastering ElasticSearch with Ruby and Tire (20)
Ad
Recently uploaded (20)
Ad
Mastering ElasticSearch with Ruby and Tire
- 1. Mastering ElasticSearch w/ Ruby (+ Tire) Luca Bonmassar, Gild RubyConf 2013 Tuesday, 5 November 13
- 2. Who am I? I’m Luca Bonmassar (@openmosix) # 31 # Italian living in San Francisco (and Stockholm) I work at Gild I love building products [for fun, profit and boredom] Tuesday, 5 November 13
- 3. Search use case You’re building a product User generated content Let (other) users find or discover this content Tuesday, 5 November 13
- 4. Search is NOT easy It usually starts as but then you want to support AND, OR, NOT, double quotes on multiple fields so and then it goes like Tuesday, 5 November 13
- 5. And so it goes... Tuesday, 5 November 13
- 6. Agenda Let’s define a “pet project” Boilerplate (download, install, scaffold, config, bla bla bla, yadda, yadda, yadda) Build a website w/ simple search Build a more advanced search What next (homework) Tuesday, 5 November 13
- 7. Project! (for fun and no profit) Tuesday, 5 November 13
- 8. Project: build a full textsearch for RubyGems.org Tuesday, 5 November 13
- 9. RubyGems Search Architecture RubyGems.org Web Spider Elastic Search Rails App (Tire) Tuesday, 5 November 13 MongoDB
- 10. RubyGems crawler github.com/openmosix/rubygems-crawler 1. Download all ruby gem names from https://rubygems.org/gems 2. Use gems API to download each gem info: read JSON - write JSON (MongoDB) Tuesday, 5 November 13
- 11. RubyGems Crawler II Tuesday, 5 November 13
- 12. RubyGems Search v1 Tuesday, 5 November 13 Code: github.com/openmosix/rubygems-search Spoiler: http://rubyconf.bonmassar.it
- 14. ElasticSearch II Open source search+analytics engine Distributed [near] Realtime search Multi tenant Built on Apache Lucene REST APIs JSON documents Tuesday, 5 November 13
- 15. ElasticSearch III 1. Download / set up a ES cluster 2. Define settings and data mapping (opt) 3. Index Data 4. Query Index MongoDB Elastic Search { "ok" : true, "status" : 200, "version" : { "number" : "0.90.5", "lucene_version" : "4.4" }, "tagline" : "You Know, for Search" } Tuesday, 5 November 13 curl -‐X GET 'h.p://localhost:9200/ ruby_gems/_search?from=0&size=25&pre.y' > Curl
- 16. ES download + setup > wget http://download.elasticsearch.org/ elasticsearch/elasticsearch/ elasticsearch-0.90.6.tar.gz > tar zxvf elasticsearch-0.90.6.tar.gz > sudo mv elasticsearch-0.90.6 /usr/local/ Hint #1: you need Java Hint #2: you need Oracle Java Tuesday, 5 November 13
- 17. ES config > ls elasticsearch-0.90.6/config/ Logging.yml: where to log, how much to log Elasticsearch.yml: all server config. Defines: Name of the cluster (change it!!!) Node parameters (master/slave, store data/router) Sharing and # replicas Paths Plugins Memory (JVM, heap, memory locking) Network config “Gateway” (cluster backup) Recovery Discovery Slow log + GC log Default options are good enough for dev env Tuesday, 5 November 13
- 18. ES boot + test Start: bin/elasticsearch Test: curl http://localhost:9200/ { "ok" : true, "status" : 200, "name" : "Iron Fist", "version" : { "number" : "0.90.6", "lucene_version" : "4.5.1" }, "tagline" : "You Know, for Search" } Stop: curl -XPOST 'http://localhost:9200/ _shutdown' {"cluster_name":"elasticsearch","nodes": {"jm2Z3J4dSzebjJ7Px2fAJg":{"name":"Iron Fist"}}} Tuesday, 5 November 13
- 19. Profit (you are now an ElasticSearch expert - go and tell the world) Tuesday, 5 November 13
- 20. ElasticSearch operations Create a “RubyGem” Index Defines a “RubyGem” Index data mapping Index data (e.g. upload data from MongoDB to ES index = POST) Query (= GET) Tuesday, 5 November 13
- 21. Tire now Re-Tire ;( A ruby gem wrapping ElasticSearch REST APIs into a powerful ruby DSL ActiveModel integration Rake tasks and utilities to load and query ElasticSearch Tuesday, 5 November 13
- 22. Tire setup cat “gem ‘tire’” > Gemfile && bundle install > cat config/initializers/elasticsearch.rb ... Tire::Configuration.url('http://localhost:9200') Tire.configure { logger "#{Rails.root}/log/elasticsearchqueries.log" } if ENV['ES_LOG'] Tuesday, 5 November 13
- 23. Define an ES index (with Tire DSL) Tuesday, 5 November 13
- 24. Indexing Get a record Convert it to JSON format (to_indexed_json) Push it to Elastic Search (.update_index) ...under the hood... Tuesday, 5 November 13
- 25. Index (all data) Naive (POST on index for each record): Use bulk updates: ...under the hood... Tuesday, 5 November 13
- 26. Search I Tuesday, 5 November 13
- 27. Search II Tuesday, 5 November 13
- 28. Simple Search Tuesday, 5 November 13
- 29. Highlight matches Text ...add some CSS... Tuesday, 5 November 13
- 31. Advanced Search II Tuesday, 5 November 13
- 32. Advanced Search III Tuesday, 5 November 13
- 33. Facets Tuesday, 5 November 13
- 34. ES facets Tuesday, 5 November 13
- 35. ES facets (running) Tuesday, 5 November 13
- 36. Facets - UI Tuesday, 5 November 13
- 37. Bonsai Cool I Tuesday, 5 November 13 Search Suggesters (Did you mean... ?)
- 38. Bonsai Cool II Tuesday, 5 November 13 “Similar to this” (aka “More Like This” API)
- 39. Bonsai Cool III Tuesday, 5 November 13 Percolate API
- 40. Deployment I Run your own cluster Some learnings: at least 3 nodes memory profiling / GC install very good monitoring (github.com/ karmi/elasticsearch-paramedic) more RAM is (always) better Check IOPS (if on AWS) Pros: Total control Cheaper (lot cheaper) Cons: Can be a nightmare / Require dedicated devop Tuesday, 5 November 13
- 41. Deployment II ElasticSearch as a service http://found.no http://searchly.com http://bonsai.io Pros: Get cluster up & running in a minute Focus on dev, not troubleshooting Professional support Cons: Expensive Can be in the wrong region / hosting provider Expensive Did I say expensive? Tuesday, 5 November 13
- 42. Thanks! Code: github.com/openmosix/rubygems-search github.com/openmosix/rubygems-crawler Demo (will be down by the end of rubyconf): http://rubyconf.bonmassar.it Say “hi”: Luca Bonmassar - [email protected] github.com/openmosix twitter.com/openmosix linkedin.com/in/lucabonmassar Tuesday, 5 November 13