How the Lucene More Like This Works
2 likes11,518 views
The More Like This search functionality is a key feature in Apache Lucene that allows to find similar documents to an input one (text or document). Being widely used but rarely explored, this presentation will start introducing how the MLT works internally. The focus of the talk is to improve the general understanding of MLT and the way you could benefit from it. Building on the introduction the focus will be on the BM25 text similarity function and how this has been (tentatively) included in the MLT through a conspicious refactor and testing process, to improve the identification of the most interesting terms from the input that can drive the similarity search. The presentation will include real world usage examples, proposed patches, pending contributions and future developments such as improved query building through positional phrase queries.
![Apache Lucene/Solr London User Group
! Origin from Probabilistic Information Retrieval
! Default Similarity from Lucene 6.0 [1]
! 25th iteration in improving TF-IDF
! TF
! IDF
! Document Length
[1] LUCENE-6789
BM25 Term Scorer](https://image.slidesharecdn.com/lucenesolrmeetup2019-190604153720/85/How-the-Lucene-More-Like-This-Works-14-320.jpg)
![[236] 카카오의데이터파이프라인 윤도영](https://cdn.slidesharecdn.com/ss_thumbnails/236-161025031702-thumbnail.jpg?width=560&fit=bounds)
![[2D1]Elasticsearch 성능 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/2d1elasticsearch-140929192211-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to How the Lucene More Like This Works (20)
Ad
More from Sease (20)
Ad
Recently uploaded (20)
How the Lucene More Like This Works
- 1. Apache Lucene/Solr London User Group How the Lucene More Like This Works Alessandro Benedetti, Software Engineer 16th May 2019
- 2. Apache Lucene/Solr London User GroupWho I am ▪ Search Consultant ▪ R&D Software Engineer ▪ Master in Computer Science ▪ Apache Lucene/Solr Enthusiast ▪ Semantic, NLP, Machine Learning Technologies passionate ▪ Beach Volleyball Player & Snowboarder Alessandro Benedetti
- 3. Apache Lucene/Solr London User GroupSease Search Services ● Open Source Enthusiasts ● Apache Lucene/Solr experts ! Community Contributors ● Active Researchers ● Hot Trends : Learning To Rank, Document Similarity, Search Quality Evaluation, Relevancy Tuning
- 4. Apache Lucene/Solr London User Group ● Document Similarity ● Apache Lucene More Like This ! Term Scorer ! BM25 ● Interesting Terms Retrieval ● Query Building ! DEMO ! Future Work ! JIRA References Agenda
- 5. Apache Lucene/Solr London User Group Document Similarity Problem : find similar documents to a seed one Solution(s) : ● Collaborative approach (users interactions) ● Content Based ● Hybrid Similar ? ● Documents accessed in similar manners by similar people ● Terms distributions ● All of above
- 6. Apache Lucene/Solr London User Group Real World Use Cases - Streaming Services
- 7. Apache Lucene/Solr London User Group Real World Use Cases - Hotels
- 8. Apache Lucene/Solr London User Group Apache Lucene Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. Apache Lucene is an open source project available for free download.
- 9. Apache Lucene/Solr London User Group ● Search Library (java) ● Structured Documents ! Inverted Index ! Similarity Metrics ( TF-IDF, BM25) ! Fast Search ! Support for advanced queries ! Relevancy tuning Apache Lucene
- 10. Apache Lucene/Solr London User Group Inverted Index Indexing
- 11. Apache Lucene/Solr London User Group Input Document More Like This Params Interesting Terms Retriever Term Scorer Query Builder QUERY More Like This - Break Up
- 12. Apache Lucene/Solr London User Group Responsibility : define a set of parameters (and defaults) that affect the various components of the More Like This module ● Regulate MLT behavior ● Groups parameters specific to each component ● Javadoc documentation ● Default values ! Useful container for various parameters to be passed More Like This Params
- 13. Apache Lucene/Solr London User Group ● Field Name ● Field Stats ( Document Count) ● Term Stats ( Document Frequency) ! Term Frequency ! TF-IDF -> tf * (log ( numDocs / docFreq + 1) + 1) ! BM25 Term Scorer Responsibility : assign a score to a term that measure how distinctive is the term for the document in input
- 14. Apache Lucene/Solr London User Group ! Origin from Probabilistic Information Retrieval ! Default Similarity from Lucene 6.0 [1] ! 25th iteration in improving TF-IDF ! TF ! IDF ! Document Length [1] LUCENE-6789 BM25 Term Scorer
- 15. Apache Lucene/Solr London User Group BM25 Term Scorer - Inverse Document Frequency IDF Score has very similar behavior
- 16. Apache Lucene/Solr London User Group BM25 Term Scorer - Term Frequency TF Score approaches asymptotically (k+1) k=1.2 in this example
- 17. Apache Lucene/Solr London User Group BM25 Term Scorer - Document Length Document Length / Avg Document Length affects how fast we saturate TF score
- 18. Apache Lucene/Solr London User Group Responsibility : retrieve from the document a queue of weighted interesting terms Params Used ! Analyzer ! Max Num Token Parsed ! Min Term Frequency ! Min/Max Document Frequency ! Max Query Terms ! Query Time Field Boost Interesting Term Retriever ! Analyze content / Term Vector ! Skip Tokens ! Score Tokens ! Build Queue of Top Scored terms
- 19. Apache Lucene/Solr London User Group Params Used ! Term Boost Enabled More Like This Query Builder Field1 : Term1 Field2 : Term2 Field1 : Term3 Field1 : Term4 Field3 : Term5 3.0 4.0 4.5 4.8 7.5 Q = Field1:Term1^3.0 Field2:Term2^4.0 Field1:Term3^4.5 Field1:Term4^4.8 Field3:Term5^7.5
- 20. Apache Lucene/Solr London User Group Term Boost ! on/off ! Affect each term weight in the MLT query ! It is the term score ( it depends of the Term Scorer implementation chosen) More Like This Boost Field Boost ! field1^5.0 field2^2.0 field3^1.5 ! Affect Term Scorer ! Affect the interesting terms retrieved N.B. a highly boosted field can dominate the interesting terms retrieval
- 21. Apache Lucene/Solr London User Group More Like This Usage - Lucene Classification ! Given a document D to classify ! K Nearest Neighbours Classifier ! Find Top K similar documents to D ( MLT) ! Classes are extracted ! Class Frequency + Class ranking -> Class probability
- 22. Apache Lucene/Solr London User Group More Like This Usage - Apache Solr ! More Like This query parser ( can be concatenated with other queries) ! More Like This search component ( can be assigned to a Request Handler) ! More Like This handler ( handler with specific request parameters)
- 23. Apache Lucene/Solr London User Group More Like This Demo - Movie Data Set This data consists of the following fields: ● id - unique identifier for the movie ● Title - Name of the movie ● Directors - The person(s) who directed the making of the film ● Genres - The genre(s) that the movie belongs to
- 24. Apache Lucene/Solr London User Group More Like This Demo - Tuned ! Enable/Disable Term Boost ! Min Term Frequency ! Min Document Frequency ! Field Boost
- 25. Apache Lucene/Solr London User Group Future Work ! Query Builder just use Terms and Term Score ! Term Positions ? ! Phrase Queries Boost (for terms close in position) ! Sentence boundaries ! Field centric vs Document centric ( should high boosted fields kick out relevant terms from low boosted fields)
- 26. Apache Lucene/Solr London User Group Future Work - More Like These ! Multiple documents in input ! Interesting terms across documents ● Useful for Content Based recommender engines
- 27. Apache Lucene/Solr London User Group Pros ● Apache Lucene Module ! Advanced Params ! Input : - structured document - just text ! Build an advanced query ! Leverage the Inverted Index ( and additional data structures) More Like This Cons ● Massive single class ! Low cohesion ! Low readability ! Minimum test coverage ! Difficult to extend ( and improve)
- 28. Apache Lucene/Solr London User Group ● LUCENE-7498 - Introducing BM25 Term Scorer ● LUCENE-7802 - Architectural Refactor ● LUCENE-8326 - MLT Params Refactor JIRA References
- 29. Apache Lucene/Solr London User Group Questions ?
- 30. Apache Lucene/Solr London User GroupThanks!