How to cache your searches_ an open source implementation.pptx

Jun 13, 2022Download as PPTX, PDF0 likes127 views

This document discusses caching search results in Solr. It describes two main caches: the QueryResultCache, which caches query results including the document IDs, filter queries, and sort order; and the FilterCache, which caches filter clauses to improve performance of queries using common filters. It recommends using the CacheViewHandler tool to view cached results and provide examples of how to effectively cache queries using filters and sorting.

How to cache your searches:
an open source implementation
Daniele Antuzi, R&D Software Engineer
30th March 2022

‣ R&D Search Software Engineer
‣ Master Degree in Computer Science
‣ Passionate about coding
‣ Food and sport lover
Who We Are
Daniele Antuzi

Agenda
What is a cache
Caches in Solr
CacheViewHandler tool
Caches in practise

A cache is a hardware or software component that
stores previous client’s requests into a fast memory so
that future requests for that data can be served faster.
What caches are
Data provider
Cache
Client

Client
Data provider
Cache
Cache search engine results
Client
Client
Lucene

● QueryResultCache
● FilterCache
Cache query results in Solr

● First cache involved in search
● Caches an ordered list of document IDs identified by
○ q (query)
○ fq (filterQuery)
○ sort
○ minExactCount
QueryResultCache

$/select?q=name:bend&fq=-genre:drama&sort=name asc QueryResultCache - Lucene Query parser { query: "name:bend", filterQuery: "-genre:drama", sort: "name asc", minExactCount: Integer.MAX_VALUE } queryResultCache$

$/select?defType=dismax &fq=genre:drama&q=will bend&qf=author name QueryResultCache - Dismax Query Parser { query: "+((author:will | name:will) (author:bend | name:bend))", filterQuery: "genre:drama", sort: null, minExactCount: Integer.MAX_VALUE } queryResultCache$

● Store filter clauses (&fq= OR filter())
● Unordered set
● Positive logic
● Store &q= clause if:
○ parameter useFilterForSortedQuery = true
○ resultset not sorted by score
FilterCache

CacheViewHandler
https://github.com/elyograg/cacheviewhandler

FilterCache - Positive fq clause
useFilterForSortedQuery
/solr/films/select?q=name:bend&fq=genre:drama

FilterCache - Negative fq clause
useFilterForSortedQuery
/solr/films/select?q=name:bend&fq=-genre:drama
Positive logic

FilterCache - Negative fq clause
useFilterForSortedQuery
/solr/films/select?q=name:bend&fq=-genre:drama
Positive logic
fq=-genre:drama ⇔ fq=*:*&fq=-genre:drama

FilterCache - q result cached
useFilterForSortedQuery
/solr/films/select?q=name:bend&fq=genre:drama&sort=name asc

FilterCache - no sort anymore
useFilterForSortedQuery
/solr/films/select?q=name:bend&fq=genre:drama

FilterCache - q result cached
useFilterForSortedQuery
/solr/films/select?q=name:bend&fq=genre:drama
/solr/films/select?q=name:bend
&fq=genre:drama
&sort=score desc

FilterCache - composed fq clause
/select?q=*:*&fq=genre:sports OR directed_by:russell

FilterCache - composed fq clause
/select?q=*:*&fq=genre:sports OR name:messi

FilterCache - use filter()
/select?q=*:*&fq=filter(genre:sports) OR filter(directed_by:russell)

FilterCache - use filter()
/select?q=*:*&fq=filter(genre:sports) OR filter(name:messi)

A new blog post is coming
Thank You!
Keep an eye on our Blog page, as more is coming!
STAY
TUNED!

- Signal is a text analytics startup that uses Elasticsearch to analyze large volumes of news articles and provide search and analytics services to customers. - Signal faced challenges in providing low latency search across thousands of heterogeneous users querying large and spiky loads of data while continuing to improve their AI models. - Joachim Draeger led experiments with Elasticsearch configurations and monitoring to optimize performance and scaling, finding that fewer, larger shards and reducing the number of search terms improved query latency. Proper monitoring was also essential to identify bottlenecks and expensive searches.

Scale Your Data Tier with Windows Server AppFabricWim Van den Broeck

This document discusses using Windows Server AppFabric to scale the data tier of web applications. It describes the typical challenges of scaling a web application's data tier, such as databases becoming saturated and services slowing down. It then introduces Windows Server AppFabric as a solution, which provides a distributed in-memory cache that can be shared across servers and services. This allows caching data across multiple machines, reducing database load and eliminating duplicate requests. It provides examples of how AppFabric can be used to cache reference data, integrate with sessions, and support optimistic/pessimistic locking for shared resources.

Big Data at Speedmarkgrover

Use Cases for Elastic Search PercolatorMaxim Shelest

The document discusses the use of Elastic Search's percolator feature. The percolator allows storing queries in an index and then indexing documents to retrieve matching queries. This is opposite of traditional search where documents are indexed and queries retrieve them. The percolator works in real-time so queries can immediately be used. Examples are provided of adding percolator queries and using them to match documents. Additional use cases discussed include alerting, notifications, and monitoring systems.

Reflected intelligence evolving self-learning data systemsTrey Grainger

In this presentation, we’ll talk about evolving self-learning search and recommendation systems which are able to accept user queries, deliver relevance-ranked results, and iteratively learn from the users’ subsequent interactions to continually deliver a more relevant experience. Such a self-learning system leverages reflected intelligence to consistently improve its understanding of the content (documents and queries), the context of specific users, and the collective feedback from all prior user interactions with the system. Through iterative feedback loops, such a system can leverage user interactions to learn the meaning of important phrases and topics within a domain, identify alternate spellings and disambiguate multiple meanings of those phrases, learn the conceptual relationships between phrases, and even learn the relative importance of features to automatically optimize its own ranking algorithms on a per-query, per-category, or per-user/group basis.

Journey through high performance django applicationbangaloredjangousergroup

Serverless Clojure and ML prototyping: an experience reportMetosin Oy

This document summarizes a serverless Clojure and machine learning application for prototyping. It describes the team building the application, the serverless AWS infrastructure using Terraform, CI/CD pipelines with GitHub Actions, the Clojure applications including a dashboard SPA and event processor, and natural language processing using Haystack and SageMaker batch transforms. The application allows users to upload documents and questionnaires, select questions, and trigger inferences to find answers from the documents using ML models.

Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...Databricks

It is common for consumer Internet companies to start off with popular third-party tools for analytics needs. Then, when the user base and the company grows, they end up building their own analytics data pipeline and query engine to cope with their data scale, satisfy custom data enrichment and reporting needs and achieve high quality of their data. That’s exactly the path that was taken at Grammarly, the popular online proofreading service. In this session, Grammarly will share how they improved business and marketing analytics, previously done with Mixpanel, by building their own in-house analytics engine and application on top of Apache Spark. Chernetsov wil touch upon several Spark tweaks and gotchas that they experienced along the way: – Outputting data to several storages in a single Spark job – Dealing with Spark memory model, building a custom spillable data-structure for your data traversal – Implementing a custom query language with parser combinators on top of Spark sql parser – Custom query optimizer and analyzer when you want not exactly sql – Flexible-schema storage and query against multi-schema data with schema conflicts – Custom aggregation functions in Spark SQL

The hidden engineering behind machine learning products at HelixaAlluxio, Inc.

Splunk, SIEMs, and Big Data - The Undercroft - November 2019Jonathan Singer

MongoDB Sharding Webinar 2014Dylan Tong

Sharding is a technique for partitioning and distributing data across multiple servers to enable scaling to large data volumes and workloads. It involves defining a shard key to partition data into chunks that are distributed across shards. The document discusses different types of sharding strategies like range, hash, and tag-aware sharding and how they apply to different use cases around scale, geo-distribution, and hardware optimization. It also covers best practices for building a sharded cluster like pre-splitting data, capacity planning, and using tools like MongoDB Management Service for production operations.

Introduction to ElasticsearchRuslan Zavacky

Intelligent MonitoringIntelie

LISA18: Hidden Linux Metrics with Prometheus eBPF ExporterIvan Babrou

Presented at LISA18: https://www.usenix.org/conference/lisa18/presentation/huynh While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include: * Disk latency and io size histograms * Run queue (scheduler) latency * Page cache efficiency * Directory cache efficiency * LLC (aka L3 cache) efficiency * Kernel timer counters * System-wide TCP retransmits Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it. If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics. In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.

Wattpad - Spark StoriesRylan Halteman

Apache Spark Toronto Meetup, July 27, 2016. Wattpad talks about their experiences with Apache Spark. From starting in 2014 with Shark, to building distributed recommendation algorithms using ANN, to improving search results using a sessionized query log. We also talk about some of the issues we faced building our analytics pipeline, including getting spark to work with Luigi, an open source project by Spotify.

Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...DataStax

Dyn delivers exceptional Internet Performance. Enabling high quality services requires data centers around the globe. In order to manage services, customers need timely insight collected from all over the world. Dyn uses DataStax Enterprise (DSE) to deploy complex clusters across multiple datacenters to enable sub 50 ms query responses for hundreds of billions of data points. From granular DNS traffic data, to aggregated counts for a variety of report dimensions, DSE at Dyn has been up since 2013 and has shined through upgrades, data center migrations, DDoS attacks and hardware failures. In this webinar, Principal Engineers Tim Chadwick and Rick Bross cover the requirements which led them to choose DSE as their go-to Big Data solution, the path which led to SPARK, and the lessons that we’ve learned in the process.

Building a data pipeline to ingest data into Hadoop in minutes using Streamse...Guglielmo Iozzia

Netezza fundamentals for developersBiju Nair

This document provides an introduction to the Netezza database appliance, including its architecture and key components. The Netezza uses an Asymmetric Massively Parallel Processing (AMPP) architecture with an array of servers (S-Blades) connected to disks. Each S-Blade contains a Database Accelerator card that offloads processing from the CPU. The document outlines the various hardware components and how they work together to process queries in parallel. It also defines common Netezza objects like users, groups, tables and databases that can be created and managed.

Smart Searching Through Trillion of Research Papers with Apache Spark MLKnoldus Inc.

Smart Searching Through Trillion of Research Papers with Apache Spark ML with...Databricks

Every publication has a rich set of documents that contain information about different domains. Mostly, these documents keeps on sitting in data warehouses. If used wisely, they can prove to be a golden set for companies operating in domains like pharma, medical, or financial institutions. For example, today it takes any pharmaceutical company upto 12 years and $2 billion to bring a single new drug to market. Despite the huge spend, scientists in Pharma don’t have a way to find the data on the work which is already done. They just redo the whole thing, wasting money on duplicate work. The biggest challenge in making those documents searchable is that they need to be tagged with their corresponding topics for which SMEs [Subject Matter Experts] are required. SMEs would read the document and fetch the topics, tag it with the topics. This way of tagging documents is slow and expensive. This talk explains how we can apply Spark ML to tag 100s of thousands of documents. Applying ML will not only make tagging process faster & less expensive but also can explore new fields which are overlooked by SMEs.

Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...Databricks

As Apache Spark applications move to a containerized environment, there are many questions about how to best configure server systems in the container world. In this talk we will demonstrate a set of tools to better monitor performance and identify optimal configuration settings. We will demonstrate how Prometheus, a project that is now part of the Cloud Native Computing Foundation (CNCF: https://www.cncf.io/projects/), can be applied to monitor and archive system performance data in a containerized spark environment. In our examples, we will gather spark metric output through Prometheus and present the data with Grafana dashboards. We will use our examples to demonstrate how performance can be enhanced through different tuned configuration settings. Our demo will show how to configure settings across the cluster as well as within each node.

Cloud Cost Management and Apache Spark with Xuan WangDatabricks

The document discusses managing cloud costs using Apache Spark and Databricks. It describes taking an in-house approach to have flexibility and deeper understanding of costs. Key aspects covered include: - Treating cost attribution as a data problem by extracting and transforming raw data from cloud providers into a data lake for analysis. - Viewing cost control as a process of prioritizing optimization, monitoring for deviations, and automating shutdown of unused resources. - Specific solutions discussed include optimizing for reserved instances, setting alerts on cost predictions, and using Custodian to automate infrastructure management.

Introduction to Forensics and Steganography by Pardhasaradhi Cn|u - The Open Security Community

The document discusses the key aspects of computer forensics including the goals, processes, rules, software, and reporting. The goal of computer forensics is to perform a structured investigation while maintaining evidence to determine what happened on a computer and who was responsible. The typical stages are preparation, search and seizure, acquisition, analysis and reporting. Key rules include never mishandling evidence, never trusting the subject system, documenting everything, and never working on original evidence. Common software used includes FTK Imager, Stegsuite, and Helix. Important files that can be analyzed include SAM, SYSTEM32, index, and NTLDER. Sites listed provide access to forensics tools, software, and resources.

OpenSearch.pdfAbhi Jain

This document provides an overview of OpenSearch, including what it is, its benefits and uses, how to use it, and its key features. OpenSearch is an open source search and analytics engine. It was created as a fork of Elasticsearch 7.10.2 and is powered by the Apache Lucene library. The document discusses how to migrate from Elasticsearch to OpenSearch using various approaches like snapshots, rolling upgrades, or full cluster restarts. It also covers OpenSearch concepts like clusters, nodes, mappings, aggregations, data streams, and plugins.

intro to forensicsPardhasaradhi ch

The document discusses computer forensics, which involves gathering digital evidence from computers to investigate security incidents and cybercrimes. It outlines the goals, stages, rules, and tools used in the digital investigation process. Key aspects include maintaining a documented chain of custody, never modifying original evidence, using forensic software to acquire disk images while calculating hash values for authentication. The summary provides an overview of the key topics covered in the document.

[DSC DACH 24] Increasing user adoption with GenAI offerings - Martin FlechlDataScienceConferenc1

One of the most successful GenAI business models is increasing user productivity by enabling artefact generation via natural language prompts. This allows adoption of product features by every user - even of those which previously have only been accessible by people with the required technical skills. In this talk we will discuss how to design GenAI systems for this purpose, based on examples from the observability domain.

Q con shanghai2013-罗婷-performance methodologyMichael Zhang

This document summarizes a presentation on performance methodology at Salesforce given at QCon Beijing 2014. It discusses: - The importance of performance for user experience, decreasing costs, and serving more customers. - Salesforce's dedicated performance team is organized by areas like UI, mobile, platforms, and infrastructure. - Key performance metrics include response time, throughput, CPU/memory utilization, and database metrics. - Performance is tested proactively via feature and regression tests, and passively via production analysis. Automated testing uses tools like JMeter and internal frameworks. Profilers like Yourkit and HeapAudit help identify causes.

Hybrid Search with Apache Solr Reciprocal Rank FusionSease

Vector-based search gained incredible popularity in the last few years: Large Language Models fine-tuned for sentence similarity proved to be quite effective in encoding text to vectors and representing some of the semantics of sentences in a numerical form. These vectors can be used to run a K-nearest neighbour search and look for documents/paragraphs close to the query in a n-dimensional vector space, effectively mimicking a similarity search in the semantic space (Apache Solr KNN Query Parser). Although exciting, vector-based search nowadays still presents some limitations: – it’s very difficult to explain (e.g. why is document A returned and why at position K?) – it doesn’t care about exact keyword matching (and users still rely on keyword searches a lot) Hybrid search comes to the rescue, combining lexical (traditional keyword-based) search with neural (vector-based) search. So, what does it mean to combine these two worlds? It starts with the retrieval of two sets of candidates: – one set of results coming from lexical matches with the query keywords – a set of results coming from the K-Nearest Neighbours search with the query vector The result sets are merged and a single ranked list of documents is returned to the user. Reciprocal Rank Fusion (RRF) is one of the most popular algorithms for such a task. This talk introduces the foundation algorithms involved with RRF and walks you through the work done to implement them in Apache Solr, with a focus on the difficulties of the process, the distributed support(SolrCloud), the main components affected and the limitations faced. The audience is expected to learn more about this interesting approach, the challenges in it and how the contribution process works for an Open Source search project as complex as Apache Solr.

Blazing-Fast Serverless MapReduce Indexer for Apache SolrSease

Indexing data from databases to Apache Solr has always been an open problem: for a while, the data import handler was used even if it was not recommended for production environments. Traditional indexing processes often encounter scalability challenges, especially with large datasets. In this talk, we explore the architecture and implementation of a serverless MapReduce indexer designed for Apache Solr but extendable to any search engine. By embracing a serverless approach, we can take advantage of the elasticity and scalability offered by cloud services like AWS Lambda, enabling efficient indexing without needing to manage infrastructure. We dig into the principles of MapReduce, a programming model for processing large datasets, and discuss how it can be adapted for indexing documents into Apache Solr. Using AWS Step Functions to orchestrate multiple Lambdas, we demonstrate how to distribute indexing tasks across multiple resources, achieving parallel processing and significantly reducing indexing times. Through practical examples, we address key considerations such as data partitioning, fault tolerance, concurrency, and cost. We also cover integration points with other AWS services such as Amazon S3 for data storage and retrieval, as well as DynamoDB for distributed lock between the lambda instances.

More Related Content

More from Sease (20)

Hybrid Search with Apache Solr Reciprocal Rank FusionSease

Blazing-Fast Serverless MapReduce Indexer for Apache SolrSease

From Natural Language to Structured Solr Queries using LLMsSease

This talk draws on experimentation to enable AI applications with Solr. One important use case is to use AI for better accessibility and discoverability of the data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints. That is where AI – and most importantly, Natural Language Processing and Large Language Model techniques – could make a difference. This natural language, conversational engine could facilitate access and usage of the data leveraging the semantics of any data source. The objective of the presentation is to propose a technical approach and a way forward to achieve this goal. The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries based on the Solr index’s metadata. This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents within Apache Solr. The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook. The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.

Hybrid Search With Apache SolrSease

Multi Valued Vectors LuceneSease

Since the introduction of native vector-based search in Apache Lucene happened, many features have been developed, but the support for multiple vectors in a dedicated KNN vector field remained to explore. Having the possibility of indexing (and searching) multiple values per field unlocks the possibility of working with long textual documents, splitting them into paragraphs, and encoding each paragraph as a separate vector: a scenario that is often encountered by many businesses. This talk explores the challenges, the technical design and the implementation activities that happened during the work for this contribution to the Apache Lucene project. The audience is expected to get an understanding of how multi-valued fields can work in a vector-based search use case and how this feature has been implemented.

When SDMX meets AI-Leveraging Open Source LLMs To Make Official Statistics Mo...Sease

This intervention draws on experimentations ongoing in the context of the OECD-led Statistical Information System Collaboration Community (SIS-CC) to enable AI applications with SDMX. One important use case is to use AI for better accessibility and discoverability of the data: whilst UX techniques, lexical search improvements, and data harmonisation can take statistical organisations to a good level of accessibility, however, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints. That is where AI – and most importantly, NLP and LLM techniques – could potentially make a difference. The “StatsBot” could be this natural language, conversational engine that could facilitate access and usage of the data. The “StatsBot” could leverage the semantics of any SDMX source. The objective of the presentation is to propose a technical approach and a way forward to achieve this goal and create the StatsBot as a universal, open asset usable by all statistical organisations. In a first step, the concept tested is to use Large Language Models with the Apache Solr index of SDMX objects so as to transform natural language queries into SDMX queries. In a second step, results could be framed as a natural language statement complementing the top-k search results. For the purpose of initial PoCs – aimed to demonstrate functional features and feasibility – a commercial LLM (such as OpenAI GPT-4) will be used; in a later stage substitution with an open source LLM will be analysed. The presentation will include the results of the first experimental work, lessons learnt, and scope future work that should lead to defining the path for production-grade, fully open source, and universal StatsBot.

How To Implement Your Online Search Quality Evaluation With KibanaSease

Online testing represents a fundamental method to assess the performance of a ranking model in practical applications, providing the information needed to improve and better understand its behavior. Despite the advantages, the currently available evaluation tools have certain limitations. For this reason, we will present an alternative and customized approach to evaluate ranking models using Kibana. The talk will begin with an overview of online testing, including its benefits and drawbacks. Then, we will provide an in-depth exploration of our Kibana implementation, detailing the reasons behind our approach. Attendees will learn about the various tools provided by Kibana, and with practical examples, we will show how to create visualizations and dashboards, complete with queries and code, to compare different rankers. Attending this presentation will provide participants with valuable knowledge on how to leverage Kibana for the purpose of evaluating ranking models on custom metrics and on specific contexts such as the most popular and “populous” queries.

Introducing Multi Valued Vectors Fields in Apache LuceneSease

Since the introduction of native vector-based search in Apache Lucene happened, many features have been developed, but the support for multiple vectors in a dedicated KNN vector field remained to explore. Having the possibility of indexing (and searching) multiple values per field unlocks the possibility of working with long textual documents, splitting them in paragraphs and encoding each paragraph as a separate vector: scenario that is often encountered by many businesses. This talk explores the challenges, the technical design and the implementation activities happened during the work for this contribution to the Apache Lucene project. The audience is expected to get an understanding of how multi-valued fields can work in a vector-based search use-case and how this feature has been implemented.

Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...Sease

Interleaving is an online evaluation approach for information retrieval systems that compares the effectiveness of ranking functions in interpreting the users’ implicit feedback. Previous work such as Hofmann et al. (2011) has evaluated the most promising interleaved methods at the time, on uniform distributions of queries. In the real world, usually, there is an unbalanced distribution of repeated queries that follows a long-tailed users’ search demand curve. This paper first aims to reproduce the Team Draft Interleaving accuracy evaluation on uniform query distributions and then focuses on assessing how this method generalises to long-tailed real-world scenarios. The replicability work raised interesting considerations on how the winning ranking function for each query should impact the overall winner for the entire evaluation. Based on what was observed, we propose that not all the queries should contribute to the final decision in equal proportion. As a result of these insights, we designed two variations of the ∆AB score winner estimator that assign to each query a credit based on statistical hypothesis testing. To reproduce, replicate and extend the original work, we have developed from scratch a system that simulates a search engine and users’ interactions from datasets from the industry. Our experiments confirm our intuition and show that our methods are promising in terms of accuracy, sensitivity, and robustness to noise.

How does ChatGPT work: an Information Retrieval perspectiveSease

In this talk, we will explore the underlying mechanisms of ChatGPT, a large-scale language model developed by OpenAI, from the perspective of Information Retrieval (IR). We will delve into the process of training the model using massive amounts of data, the techniques used to optimize the model’s performance, and how the IR concepts such as tokenization, vectorization, and ranking are used in generating responses. We will also discuss how ChatGPT handles contextual understanding and how it leverages the power of transfer learning to generate high-quality and relevant responses. Software engineers will gain insights into how a modern conversational AI system like ChatGPT works, providing a better understanding of its strengths and limitations, and how to best integrate it into their software applications. This abstract has been fully written by ChatGPT with the simple prompt in input <Write an abstract for a talk called “How does ChatGPT work? An Information Retrieval perspective”, the audience is software engineers>.

How To Implement Your Online Search Quality Evaluation With KibanaSease

Online testing remains the optimal way to prove how your ranking model performs in your real-world scenario. It can lead to many advantages such as having a direct interpretation of the results and confirming the estimation of offline tests. It gives a better understanding of the ranking model behaviour and builds a solid foundation to learn from to improve it. Nowadays, the available evaluation tools have some limitations and in this talk, we will describe an alternative and customised approach for evaluating ranking models through the use of Kibana. First of all, we give an overview of online testing, highlighting the pros and cons and describing the state-of-the-art. We then dive into Kibana’s implementation and the reasons behind it. We will explore the tools Kibana provides, with their constraints for real-world applications, and show, through practical examples, how to create dashboards (with queries and code) to compare different models.

Neural Search Comes to Apache SolrSease

Learning To Rank has been the first integration of machine learning techniques with Apache Solr allowing you to improve the ranking of your search results using training data. One limitation is that documents have to contain the keywords that the user typed in the search box in order to be retrieved(and then reranked). For example, the query “jaguar” won’t retrieve documents containing only the terms “panthera onca”. This is called the vocabulary mismatch problem. Neural search is an Artificial Intelligence technique that allows a search engine to reach those documents that are semantically similar to the user’s information need without necessarily containing those query terms; it learns the similarity of terms and sentences in your collection through deep neural networks and numerical vector representation(so no manual synonyms are needed!). This talk explores the first Apache Solr official contribution about this topic, available from Apache Solr 9.0. We start with an overview of neural search (Don’t worry - we keep it simple!): we describe vector representations for queries and documents, and how Approximate K-Nearest Neighbor (KNN) vector search works. We show how neural search can be used along with deep learning techniques (e.g, BERT) or directly on vector data, and how we implemented this feature in Apache Solr, giving usage examples! Join us as we explore this new exciting Apache Solr feature and learn how you can leverage it to improve your search experience!

Large Scale IndexingSease

Dense Retrieval with Apache Solr Neural Search.pdfSease

This document provides an overview of dense retrieval with Apache Solr neural search. It discusses semantic search problems that neural search aims to address through vector-based representations of queries and documents. It then describes Apache Solr's implementation of neural search using dense vector fields and HNSW graphs to perform k-nearest neighbor retrieval. Functions are shown for indexing and searching vector data. The document also discusses using vector queries for filtering, re-ranking, and hybrid searches combining dense and sparse criteria.

Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...Sease

The first integrations of machine learning techniques with search allowed to improve the ranking of your search results (Learning To Rank) – but one limitation has always been that documents had to contain the keywords that the user typed in the search box in order to be retrieved. For example, the query “tiger” won’t retrieve documents containing only the terms “panthera tigris”. This is called the vocabulary mismatch problem and over the years it has been mitigated through query and document expansion approaches. Neural search is an Artificial Intelligence technique that allows a search engine to reach those documents that are semantically similar to the user’s query without necessarily containing those terms; it avoids the need for long lists of synonyms by automatically learning the similarity of terms and sentences in your collection through the utilisation of deep neural networks and numerical vector representation.

Word2Vec model to generate synonyms on the fly in Apache Lucene.pdfSease

f you want to expand your query/documents with synonyms in Apache Lucene, you need to have a predefined file containing the list of terms that share the same semantic. It’s not always easy to find a list of basic synonyms for a language and, even if you find it, this doesn’t necessarily match with your contextual domain. The term “daemon” in the domain of operating system articles is not a synonym of “devil” but it’s closer to the term “process”. Word2Vec is a two-layer neural network that takes as input a text and outputs a vector representation for each word in the dictionary. Two words with similar meanings are identified with two vectors close to each other.

Online Testing Learning to Rank with Solr InterleavingSease

Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...Sease

Rated Ranking Evaluator Enterprise is an enterprise version of the open source Rated Ranking Evaluator search quality evaluation tool. It features query discovery to automatically extract queries from a search API, rating generation using both explicit ratings and implicit feedback, and an interactive UI for exploring and comparing evaluation results. The UI provides overview, exploration, and comparison views of evaluation data to meet the needs of business stakeholders and software engineers. Future work aims to improve the tool's capabilities around configuration, multimedia support, insights generation, and click modeling.

Apache Lucene/Solr Document ClassificationSease

This presentation will start by introducing how Apache Lucene can be used to classify documents using data structures that already exist in your index instead of having to generate and supply external training sets. The focus will be on extensions of the Lucene Classification module that come in Lucene 6.0 and the Lucene Classification module's incorporation into Solr 6.1. These extensions will allow you to classify at a document level with individual field weighting, numeric field support, lat/lon fields etc. The Solr ClassificationUpdateProcessor will be explored and how to use it including basic and advanced features like multi class support and classification context filtering. The presentation will include practical examples and real world use cases.

Advanced Document Similarity with Apache LuceneSease

This document provides an overview of using Apache Lucene to perform document similarity searches. It discusses how Lucene indexes documents and calculates term scores using techniques like TF-IDF and BM25. The document demonstrates Lucene's More Like This module, which finds similar documents by building a query from weighted interesting terms in the input document. It also proposes future work, such as leveraging term positions and fields across multiple input documents to improve similarity searches.

Hybrid Search with Apache Solr Reciprocal Rank FusionSease

Blazing-Fast Serverless MapReduce Indexer for Apache SolrSease

From Natural Language to Structured Solr Queries using LLMsSease

Hybrid Search With Apache SolrSease

Multi Valued Vectors LuceneSease

When SDMX meets AI-Leveraging Open Source LLMs To Make Official Statistics Mo...Sease

How To Implement Your Online Search Quality Evaluation With KibanaSease

Introducing Multi Valued Vectors Fields in Apache LuceneSease

Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...Sease

How does ChatGPT work: an Information Retrieval perspectiveSease

How To Implement Your Online Search Quality Evaluation With KibanaSease

Neural Search Comes to Apache SolrSease

Large Scale IndexingSease

Dense Retrieval with Apache Solr Neural Search.pdfSease

Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...Sease

Word2Vec model to generate synonyms on the fly in Apache Lucene.pdfSease

Online Testing Learning to Rank with Solr InterleavingSease

Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...Sease

Apache Lucene/Solr Document ClassificationSease

Advanced Document Similarity with Apache LuceneSease

Recently uploaded (20)

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

Build Your Own Copilot & Agents For DevsBrian McKeiver

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Splunk Security Update | Public Sector Summit Germany 2025Splunk

Cyber Awareness overview for 2025 month of securityriccardosl1

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025 https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/ Is AI just another technology, or does it fundamentally change the way we live and think? Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater. At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts. At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Build Your Own Copilot & Agents For DevsBrian McKeiver

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Splunk Security Update | Public Sector Summit Germany 2025Splunk

Cyber Awareness overview for 2025 month of securityriccardosl1

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

How to cache your searches_ an open source implementation.pptx

1. How to cache your searches: an open source implementation Daniele Antuzi, R&D Software Engineer 30th March 2022

2. ‣ R&D Search Software Engineer ‣ Master Degree in Computer Science ‣ Passionate about coding ‣ Food and sport lover Who We Are Daniele Antuzi

3. Agenda What is a cache Caches in Solr CacheViewHandler tool Caches in practise

4. A cache is a hardware or software component that stores previous client’s requests into a fast memory so that future requests for that data can be served faster. What caches are Data provider Cache Client

5. Client Data provider Cache Cache search engine results Client Client Lucene

6. ● QueryResultCache ● FilterCache Cache query results in Solr

7. ● First cache involved in search ● Caches an ordered list of document IDs identified by ○ q (query) ○ fq (filterQuery) ○ sort ○ minExactCount QueryResultCache

8. /select?q=name:bend&fq=-genre:drama&sort=name asc QueryResultCache - Lucene Query parser { query: "name:bend", filterQuery: "-genre:drama", sort: "name asc", minExactCount: Integer.MAX_VALUE } queryResultCache

9. /select?defType=dismax &fq=genre:drama&q=will bend&qf=author name QueryResultCache - Dismax Query Parser { query: "+((author:will | name:will) (author:bend | name:bend))", filterQuery: "genre:drama", sort: null, minExactCount: Integer.MAX_VALUE } queryResultCache

10. ● Store filter clauses (&fq= OR filter()) ● Unordered set ● Positive logic ● Store &q= clause if: ○ parameter useFilterForSortedQuery = true ○ resultset not sorted by score FilterCache

11. CacheViewHandler https://github.com/elyograg/cacheviewhandler

12. Some practical examples Demo

13. FilterCache - Positive fq clause useFilterForSortedQuery /solr/films/select?q=name:bend&fq=genre:drama

14. FilterCache - Negative fq clause useFilterForSortedQuery /solr/films/select?q=name:bend&fq=-genre:drama Positive logic

15. FilterCache - Negative fq clause useFilterForSortedQuery /solr/films/select?q=name:bend&fq=-genre:drama Positive logic fq=-genre:drama ⇔ fq=*:*&fq=-genre:drama

16. FilterCache - q result cached useFilterForSortedQuery /solr/films/select?q=name:bend&fq=genre:drama&sort=name asc

17. FilterCache - no sort anymore useFilterForSortedQuery /solr/films/select?q=name:bend&fq=genre:drama

18. FilterCache - q result cached useFilterForSortedQuery /solr/films/select?q=name:bend&fq=genre:drama /solr/films/select?q=name:bend &fq=genre:drama &sort=score desc

19. FilterCache - composed fq clause /select?q=*:*&fq=genre:sports OR directed_by:russell

20. FilterCache - composed fq clause /select?q=*:*&fq=genre:sports OR name:messi

21. FilterCache - use filter() /select?q=*:*&fq=filter(genre:sports) OR filter(directed_by:russell)

22. FilterCache - use filter() /select?q=*:*&fq=filter(genre:sports) OR filter(name:messi)

23. A new blog post is coming Thank You! Keep an eye on our Blog page, as more is coming! STAY TUNED!

How to cache your searches_ an open source implementation.pptx

Recommended

More Related Content

Similar to How to cache your searches_ an open source implementation.pptx (20)

More from Sease (20)

Recently uploaded (20)

How to cache your searches_ an open source implementation.pptx