Natural Language Search in Solr

Natural language search in Solr
Tommaso Teofili, Sourcesense
t.teofili@sourcesense.com, October 19th 2011

Agenda
 An approach to natural language search in
Solr
 Main points
• Solr-UIMA integration module
• Custom Lucene analyzers for UIMA
• OSS NLP algorithms in Lucene/Solr
• Orchestrating blocks to build a sample
system able to understand natural language
queries
 Results

My Background
 Software engineer at Sourcesense
• Enterprise search consultant
 Member of the Apache Software Foundation
• UIMA
• Clerezza
• Stanbol
• DirectMemory
• ...

The Challenge
 Improved recall/precision
• ‘articles about science’ (concepts)
• ‘movies by K. Spacey’ vs ‘movies with K. Spacey’
 Easier experience for non-expert users
• ‘people working at Google’ - ‘cities near London’
 Horizontal domains (e.g. Google)
 Vertical domains

Hurdles
 understanding documents’ text/user queries
 extract domain-specific/wide entities and
concepts
 index/search performance

Use Case
 search engine for an online movies magazine
 Solr based
 non technical users
 time / cost
• Solr 3.x setup : 2 mins
• NLS setup / tweak : 5 days
 expecting
• improved recall / precision
• more time (clicks) on site ($)

General approach
 Natural language processing
 Processing documents at indexing time
• document text analysis
• write enriched text in (dedicated) fields
• add custom types / payloads to terms
 Processing queries at searching time
• query analysis
• higher boosts to entities/concepts
• in-sentence search
• ...

NLP
 AI discipline
• Computers understanding and managing
information written in human language
 analyze text at various levels
 incrementally enrich / give structure
 extract concepts and named entities

Technical detail
 NLP algorithms plugged via Apache UIMA
 Indexing time
• UpdateProcessor plugin (solr/contrib/uima)
• Custom tokenizers/filters
 Search time
• Custom QParserPlugin

Why Apache UIMA?
 OASIS standard for UIM
 TLP since March 2010
 Deploy pipelines of Analysis Engines
 AEs wrap NLP algorithms
 Scaling capabilities

NLP and OSS
 Sentence Split
• OpenNLP, UIMA Addons, StanfordNLP
 PoS tagging
• OpenNLP, UIMA Addons, StanfordNLP
 Chunking/Parsing
• OpenNLP, StanfordNLP
 NER
• OpenNLP, UIMA Addons, Stanbol, StanfordNLP
 Clustering/Classifying
• Mahout, OpenNLP, StanfordNLP
 ...

Lucene analysis & UIMA
 Type : denote lexical types for tokens
 Payload : a byte array stored at each term
position
 tokenize / filter tokens covered by a certain
annotation type
 store UIMA annotations’ features in types /
payloads

Solr NLS QParser
 analyze user query
 extract (and query on) concepts / entities
 use types/PoS in the query for
• boosting terms
• synonim expansion
 search within sentences
 faceting / clustering using entities
 identify ‘place queries’ and expand Solr spatial
queries (for filtering / boosting)

Performance
 basic (in memory)
• slower with NRT indexing
• search could be significantly impacted
 ReST (SimpleServer)
• faster
• need to explictly digest results
 UIMA-AS
• fast also with NRT indexing
• fast search
• scales nicely with lots of data

Wrap up
 general purpose architecture
 generally improved recall / precision
 NLP algorithms accuracy make the difference
 lots of OSS alternatives
 performances can be kept good

Sources
 Resources
• http://svn.apache.org/repos/asf/lucene/dev/trunk/
solr/contrib/uima/
• https://github.com/tteofili/le11-nls
 Links
• http://wiki.apache.org/solr/SolrUIMA
• http://googleblog.blogspot.com/2010/01/helping-
computers-understand-language.html

Thanks
 http://www.sourcesense.com

 t.teofili@sourcesense.com

 @tteofili

Natural Language Search in Solr

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Natural Language Search in Solr (20)

More from Tommaso Teofili (16)

Recently uploaded (20)

Natural Language Search in Solr