Advanced Retrieval Augmented Generation Techniques

May 23, 20240 likes624 views

While achieving a basic Retrieval Augmented Generation (RAG) is relatively straightforward, attaining superior results requires tuning and optimizing various factors, such as a careful selection of embedding models. Additionally, applying advanced techniques, such as multi-stage retrieval with rerankers, is essential. A methodology for quality evaluation is also critical to success in crafting the best strategy for your specific use case. This talk will introduce the landscape of available optimization techniques and provide advice on best practices.

1 | © Copyright 2024 Zilliz
1
1 | © Copyright 9/25/23 Zilliz
1 | © Copyright 9/25/23 Zilliz
Speaker
Jiang Chen
Ecosystem & AI Platform
jiang.chen@zilliz.com
@jiangc1010

2 | © Copyright 2024 Zilliz
2
Fantastic RAG Techniques
And Where to Find Them
Jiang Chen @ Zilliz

3 | © Copyright 2024 Zilliz
3
LLMs are great, but …
You still need to battle hallucination
with retriever, just like the Niffler

4 | © Copyright 2024 Zilliz
4
The evolution of AI made the semantic search of
unstructured data possible
Search by Probability
Statistical analyses of common
datasets established the foundation for
processing unstructured data, e.g. NLP,
and image classification
AI Model Breakthrough
The advancements in BERT, ViT, CBT
etc. have revolutionized semantic
analysis across unstructured data
Vectorization
Word2Vec, CNNs, Deep Speech pioneered
unstructured data embeddings, mapping the
words, images, videos into high-dimensional
vectors

5 | © Copyright 2024 Zilliz
5
01 Review of RAG basics
CONTENTS
02 Advanced RAG techniques
RAG in action with Milvus Lite
03

6 | © Copyright 2024 Zilliz
6
01 Review of RAG basics

7 | © Copyright 2024 Zilliz
7
Why RAG?
RAG vs. LLM
- Knowledge of LLM is out-of-date
- LLM can not get your private knowledge
- Hallucinations
- Transparency and interpretability
RAG vs. Fine-tune
- Fine-tune is expensive
- Fine-tune spent much time
- RAG is pluggable

9 | © Copyright 2024 Zilliz
9
02 Advanced RAG techniques

10 | © Copyright 2024 Zilliz
10
First thing first
Measure it before you attempts to improve it!

11 | © Copyright 2024 Zilliz
11
Indexing
Query Retrieval Prompt&
Generation

12 | © Copyright 2024 Zilliz
12
Types of RAG Enhancement Techniques
● Divide & Conquer
○ Query Enhancement: better express or process the query intent.
○ Indexing Enhancement: data cleanup, better parser and chunking
○ Retriever Enhancement: more retrievers and hybrid search strategy
○ Generator Enhancement: prompt engineering and more powerful LLM
● Thinking outside the box
○ Agents? Other tools than retriever?

13 | © Copyright 2024 Zilliz
13
Query Enhancement

16 | © Copyright 2024 Zilliz
16
What are the differences in features
between Milvus and Zilliz Cloud?
Sub query1: What are the features of Milvus?
Sub query2: What are the features of Zilliz Cloud?

18 | © Copyright 2024 Zilliz
18
Indexing Enhancement

19 | © Copyright 2024 Zilliz
19
Good dishes come from good ingredients
• Data collection
• Data cleaning
• Parsing & Chunking
• DNN-native data?

21 | © Copyright 2024 Zilliz
21
Retriever Enhancement

25 | © Copyright 2024 Zilliz
25
Generator Enhancement

32 | © Copyright 2024 Zilliz
32
03 RAG in action with Milvus Lite

34 | © Copyright 2024 Zilliz
34
Seamless integration with all popular AI toolkits

35 | © Copyright 2024 Zilliz
35
35 | © Copyright 9/25/23 Zilliz
35 | © Copyright 9/25/23 Zilliz
Simplify and streamline
the conversion of
unstructured data into
state-of-the-art vector
embeddings, using
intuitive UI and Restful
APIs.
Pipelines
Easy. High-quality. Scalable.
Simplify the workflow
for developers, from
converting
unstructured data into
searchable vectors to
retrieving them from
vector databases
Deliver excellence in
every phase of vector
search pipeline
development and
deployment,
regardless of their
expertise
Ensure scalability for
managing large
datasets and
high-throughput
queries, maintaining
high performance with
min. customization or
infra changes
Zilliz Cloud Pipelines

36 | © Copyright 2024 Zilliz
36
T H A N K Y O U

Embark on a comprehensive exploration of Retrieval Augmented Generation (RAG) in this illuminating session. Delve into the architecture seamlessly merging retrieval and generation models and uncover its versatile applications. From refining search processes to enhancing content generation, RAG is reshaping the landscape of natural language processing. Join us for a brief yet comprehensive Introduction to RAG and its transformative potential, along with insights into its applications.

Introduction to Open Source RAG and RAG EvaluationZilliz

You’ve heard good data matters in Machine Learning, but does it matter for Generative AI applications? Corporate data often differs significantly from the general Internet data used to train most foundation models. Join me for a demo on building an open source RAG (Retrieval Augmented Generation) stack using Milvus vector database for Retrieval, LangChain, Llama 3 with Ollama, Ragas RAG Eval, and optional Zilliz cloud, OpenAI.

Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz

AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379

How to fine-tune and develop your own large language model.pptxKnoldus Inc.

generative-ai-fundamentals and Large language modelsAdventureWorld5

Thank you for the detailed review of the protein bars. I'm glad to hear you and your family are enjoying them as a healthy snack and meal replacement option. A couple suggestions based on your feedback: - For future orders, you may want to check the expiration dates to help avoid any dried out bars towards the end of the box. Freshness is key to maintaining the moist texture. - When introducing someone new to the bars, selecting one in-person if possible allows checking the flexibility as an indicator it's moist inside. This could help avoid a disappointing first impression from a dry sample. - Storing opened boxes in an airtight container in the fridge may help extend the freshness even further when you can't

LLaMA 2.pptxRkRahul16

- LLaMA 2 is a family of large language models developed by Meta in partnership with Microsoft and others. It has been pretrained on 2 trillion tokens and has three model sizes up to 70 billion parameters. - LLaMA 2 was trained using an auto-regressive transformer and reinforcement learning from human feedback to improve safety and alignment. It can generate text, translate languages, and answer questions. - The models were pretrained on Meta's research supercomputers then fine-tuned for dialog using supervised learning and reinforcement learning from human feedback to further optimize safety and usefulness.

Introduction to LLMsLoic Merckel

The Twelve-Factor Appで考えるAWSのサービス開発Amazon Web Services Japan

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfPo-Chuan Chen

The document describes the RAG (Retrieval-Augmented Generation) model for knowledge-intensive NLP tasks. RAG combines a pre-trained language generator (BART) with a dense passage retriever (DPR) to retrieve and incorporate relevant knowledge from Wikipedia. RAG achieves state-of-the-art results on open-domain question answering, abstractive question answering, and fact verification by leveraging both parametric knowledge from the generator and non-parametric knowledge retrieved from Wikipedia. The retrieved knowledge can also be updated without retraining the model.

Generative modelsBirger Moell

The document discusses generative models and their applications in artificial intelligence. Generative adversarial networks (GANs) use two neural networks, a generator and discriminator, that compete against each other. The generator learns to generate new data that looks real by fooling the discriminator, while the discriminator learns to better identify real from fake data. GANs have been used for tasks like image generation and neural style transfer. They show potential to generate art, music and other creative forms through machine learning.

Gen AI Cognizant & AWS event presentation_12 Oct.pdfPhilipBasford

This document provides an overview of generative AI capabilities and architectures on AWS. It discusses the evolution of generative AI and some of its potential uses including generative search, smart data analytics assistance, text summarization, personalization, simulation, and automating routine tasks. It outlines several generative AI architectures available on AWS including Stable Diffusion, Claude, Jurassic-2z, Titan, Command & Embed, and models available through Hugging Face. The document discusses Amazon SageMaker and Amazon Bedrock as flagship services for foundational models on AWS. It also presents the Enterprise Knowledge Navigator solution for advanced question answering, retrieval-augmented generation, security, and interacting with data lakes. The document concludes with two case studies

2024.05.01 RAG 세미나: 자연어처리의 정보검색 기법과 최신 RAG 모델Cheoneum Park

Generative AI - Responsible Path Forward.pdfSaeed Al Dhaheri

Generative AI: Responsible Path Forward Dr. Saeed Aldhaheri discusses the potential and risks of generative AI and proposes a responsible path forward. He outlines that (1) while generative AI shows great economic potential and can augment human capabilities, it also poses new ethical risks if not developed responsibly. (2) Current approaches by the tech industry are not sufficient, and a human-centered perspective is needed. (3) Building responsible generative AI requires moving beyond technical solutions to address sociotechnical issues through principles of ethics by design, governance, risk frameworks, and responsible data practices.

Using MLOps to Bring ML to Production/The Promise of MLOpsWeaveworks

In this final Weave Online User Group of 2019, David Aronchick asks: have you ever struggled with having different environments to build, train and serve ML models, and how to orchestrate between them? While DevOps and GitOps have made huge traction in recent years, many customers struggle to apply these practices to ML workloads. This talk will focus on the ways MLOps has helped to effectively infuse AI into production-grade applications through establishing practices around model reproducibility, validation, versioning/tracking, and safe/compliant deployment. We will also talk about the direction for MLOps as an industry, and how we can use it to move faster, with more stability, than ever before. The recording of this session is on our YouTube Channel here: https://youtu.be/twsxcwgB0ZQ Speaker: David Aronchick, Head of Open Source ML Strategy, Microsoft Bio: David leads Open Source Machine Learning Strategy at Azure. This means he spends most of his time helping humans to convince machines to be smarter. He is only moderately successful at this. Previously, David led product management for Kubernetes at Google, launched GKE, and co-founded the Kubeflow project. David has also worked at Microsoft, Amazon and Chef and co-founded three startups. Sign up for a free Machine Learning Ops Workshop: http://bit.ly/MLOps_Workshop_List Weaveworks will cover concepts such as GitOps (operations by pull request), Progressive Delivery (canary, A/B, blue-green), and how to apply those approaches to your machine learning operations to mitigate risk.

Natural Language Processing NLP (Transformers)Hichem Felouat

Responsible AIData Con LA

Data Con LA 2020 Description More and more organizations are embracing AI technology by infusing it in their products and services to to differentiate themselves against their competitors. AI is being utilized in some sensitive areas of human life. In this session let's look at some of principles governing adoption of AI in a responsible manner. Why companies are accelerating adoption of AI? Increasingly organization are accelerating adoption of AI to differentiate their product and services in the market. Outcomes of this digital transformation that we have seen in the areas of optimizing operations, engaging customers, empowering employees and transforming their products and services. *List some of the sensitive use cases where AI is being applied *Why governing AI is important and what are those principles? *How Microsoft is approaching it? Speaker Suresh Paulraj, Microsoft, Principal Cloud Solution Architect Data & AI

Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.

What is MLOpsHenrik Skogström

UNLEASHING INNOVATION Exploring Generative AI in the Enterprise.pdfHermes Romero

The document provides an overview of generative AI, including its key concepts and applications. It discusses transformer models versus neural networks, explaining that transformer models use self-attention to capture long-range dependencies in sequential data like text. Large language models (LLMs) based on the transformer architecture have shown strong performance in natural language generation tasks. The document outlines the evolution of generative AI techniques from early machine learning to modern large pretrained models. It also surveys some commercial generative AI applications in industries like healthcare, finance, and gaming.

MLOps - The Assembly Line of MLJordan Birdsell

The catalyst for the success of automobiles came not through the invention of the car but rather through the establishment of an innovative assembly line. History shows us that the ability to mass produce and distribute a product is the key to driving adoption of any innovation, and machine learning is no different. MLOps is the assembly line of Machine Learning and in this presentation we will discuss the core capabilities your organization should be focused on to implement a successful MLOps system.

分散トレーシングAWS:X-Rayとの上手い付き合い方Recruit Lifestyle Co., Ltd.

Chicago AWS Solutions Architect Mehdy Haghy recaps the new AI/ML releases and...AWS Chicago

This document summarizes Amazon Web Services' generative AI and machine learning services, including Amazon Q, Amazon Bedrock, Amazon CodeWhisperer, and Amazon SageMaker. It outlines new capabilities and updates to each service, such as customizing AI models with domain-specific data in Amazon Bedrock, improved coding suggestions from Amazon CodeWhisperer, and new tools for distributed training and inference in Amazon SageMaker. The document is intended to recap announcements from re:Invent 2023 about AWS's generative AI and AI/ML services.

Building NLP applications with TransformersJulien SIMON

The document discusses how transformer models and transfer learning (Deep Learning 2.0) have improved natural language processing by allowing researchers to easily apply pre-trained models to new tasks with limited data. It presents examples of how HuggingFace has used transformer models for tasks like translation and part-of-speech tagging. The document also discusses tools from HuggingFace that make it easier to train models on hardware accelerators and deploy them to production.

Real World End to End machine Learning PipelineSrivatsan Srinivasan

Purpose of this presentation is to highlight how end to end machine learning looks like in real world enterprise. This is to provide insight to aspiring data scientist who have been through courses or education in ML that mostly focus on ML algorithms and not end to end pipeline. Architecture and components mentioned in Slide 11 will be discussed in detailed in series of post on LinkedIn over the course of next few month To get updates on this follow me on LinkedIn or search/follow hashtag #end2endDS. Post will be active in August 2019 and will be posted till September 2019

Vector Similarity Search & Indexing MethodsKate Shao

The document discusses various indexing methods for efficient vector similarity search on large datasets. It introduces graph-based indexes that use approximate nearest neighbor algorithms based on navigable small-world graphs. Space partition indexes are also discussed, including inverted multi-indexes and optimizations for billion-scale approximate nearest neighbors. Product quantization encoding is covered as another indexing approach. The document concludes by proposing a layered framework that decomposes vector search into space partitioning, candidate filtering, and result validation layers to balance accuracy, speed and system requirements.

230309_LoRaYongSang Yoo

This document proposes a Low-Rank Adaptation (LoRA) approach to efficiently adapt large language models. LoRA hypothesizes that weight updates during model adaptation have low intrinsic rank. It parametrizes update matrices with low-rank matrices to reduce storage and improve efficiency. Empirical experiments show LoRA can adapt models using 3 times fewer parameters and computations than full fine-tuning, with no inference latency. LoRA is compatible with other adaptation methods and can efficiently tune large models for different tasks.

A comprehensive guide to Agentic AI SystemsDebmalya Biswas

A deep-dive into Agentic AI covering the following topics: Linking to the corresponding articles on Medium. The article on Long-term #Memory Management is now 3rd among the 10 most read stories on #AgenticAI on Medium -:) - Agentic AI Platform Reference #Architecture https://medium.com/datadriveninvestor/ai-agent-platform-reference-architecture-0be5b19d0eba - AI Agents #Marketplace & Discovery for #MultiAgentSystems https://medium.com/ai-advances/ai-agents-marketplace-discovery-for-multi-agent-systems-27a31b6b1ca6 - Personalizing #UX for Agentic AI https://medium.com/ai-advances/personalized-ux-for-agentic-ai-ab132f2eeb03 - Agent #Observability https://medium.com/ai-advances/stateful-and-responsible-ai-agents-7af386268554 - Long-term #Memory Management https://medium.com/ai-advances/long-term-memory-for-agentic-ai-systems-4ae9b37c6c0f - Agentic AI Scenarios: Agentic #RAGs: extending RAGs to #SQL Databases https://medium.com/ai-advances/agentic-rags-extending-rags-to-sql-databases-1509b25ca3e7 #ReinforcementLearning Agents for #ControlSystems https://medium.com/ai-advances/reinforcement-learning-agents-for-industrial-control-systems-b917b513f0c4 - Responsible AI Agents #Privacy Risks of LLMs https://medium.com/ai-advances/privacy-risks-of-large-language-models-llms-5c0f96dccc56 Responsible #LLMOps in Towards Data Science https://towardsdatascience.com/responsible-llmops-985cd1af3639

2024-10-28 All Things Open - Advanced Retrieval Augmented Generation (RAG) Te...Timothy Spann

2024-10-28 All Things Open - Advanced Retrieval Augmented Generation (RAG) Techniques Timothy Spann https://2024.allthingsopen.org/sessions/advanced-retrieval-augmented-generation-rag-techniques In 2023, we saw many simple retrieval augmented generation (RAG) examples being built. However, most of these examples and frameworks built around them simplified the process too much. Businesses were unable to derive value from their implementations. That’s because there are many other techniques involved in tuning a basic RAG app to work for you. In this talk we will cover three of the techniques you need to understand and leverage to build better RAG: chunking, embedding model choice, and metadata structuring.

Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI...Zilliz

More Related Content

What's hot (20)

The Twelve-Factor Appで考えるAWSのサービス開発Amazon Web Services Japan

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfPo-Chuan Chen

Generative modelsBirger Moell

Gen AI Cognizant & AWS event presentation_12 Oct.pdfPhilipBasford

2024.05.01 RAG 세미나: 자연어처리의 정보검색 기법과 최신 RAG 모델Cheoneum Park

Generative AI - Responsible Path Forward.pdfSaeed Al Dhaheri

Using MLOps to Bring ML to Production/The Promise of MLOpsWeaveworks

Natural Language Processing NLP (Transformers)Hichem Felouat

Responsible AIData Con LA

Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.

What is MLOpsHenrik Skogström

UNLEASHING INNOVATION Exploring Generative AI in the Enterprise.pdfHermes Romero

MLOps - The Assembly Line of MLJordan Birdsell

分散トレーシングAWS:X-Rayとの上手い付き合い方Recruit Lifestyle Co., Ltd.

Chicago AWS Solutions Architect Mehdy Haghy recaps the new AI/ML releases and...AWS Chicago

Building NLP applications with TransformersJulien SIMON

Real World End to End machine Learning PipelineSrivatsan Srinivasan

Vector Similarity Search & Indexing MethodsKate Shao

230309_LoRaYongSang Yoo

A comprehensive guide to Agentic AI SystemsDebmalya Biswas

The Twelve-Factor Appで考えるAWSのサービス開発Amazon Web Services Japan

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfPo-Chuan Chen

Generative modelsBirger Moell

Gen AI Cognizant & AWS event presentation_12 Oct.pdfPhilipBasford

2024.05.01 RAG 세미나: 자연어처리의 정보검색 기법과 최신 RAG 모델Cheoneum Park

Generative AI - Responsible Path Forward.pdfSaeed Al Dhaheri

Using MLOps to Bring ML to Production/The Promise of MLOpsWeaveworks

Natural Language Processing NLP (Transformers)Hichem Felouat

Responsible AIData Con LA

Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.

What is MLOpsHenrik Skogström

UNLEASHING INNOVATION Exploring Generative AI in the Enterprise.pdfHermes Romero

MLOps - The Assembly Line of MLJordan Birdsell

分散トレーシングAWS:X-Rayとの上手い付き合い方Recruit Lifestyle Co., Ltd.

Chicago AWS Solutions Architect Mehdy Haghy recaps the new AI/ML releases and...AWS Chicago

Building NLP applications with TransformersJulien SIMON

Real World End to End machine Learning PipelineSrivatsan Srinivasan

Vector Similarity Search & Indexing MethodsKate Shao

230309_LoRaYongSang Yoo

A comprehensive guide to Agentic AI SystemsDebmalya Biswas

Similar to Advanced Retrieval Augmented Generation Techniques (20)

2024-10-28 All Things Open - Advanced Retrieval Augmented Generation (RAG) Te...Timothy Spann

Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI...Zilliz

10-25-2024_BITS_NYC_Unstructured Data and LLM_ What, Why and HowTimothy Spann

10-25-2024_BITS_NYC_Unstructured Data and LLM_ What, Why and How https://www.bletchley.org/bits-2024 Tim Spann Milvus Zilliz https://github.com/tspannhw/SpeakerProfile https://www.bletchley.org/bits-2024 Data Science & Machine Learning Unstructured Data and LLM: What, Why and How Timothy Spann  Tim Spann is a Principal Developer Advocate at Zilliz, where he focuses on technologies such as Milvus, Towhee, GPTCache, Generative AI, Python, Java, and various Apache tools like NiFi, Kafka, and Pulsar. With over a decade of experience in IoT, big data, and distributed computing, Tim has held key roles at Cloudera, StreamNative, and HPE. He also runs a popular Big Data meetup in Princeton & NYC, frequently speaking at conferences like ApacheCon, Pulsar Summit, and DeveloperWeek. In addition to his work, Tim is an active contributor to DZone as the Big Data Zone leader. He holds a BS and MS in computer science.

What Makes "Deep Research"? A Dive into AI AgentsZilliz

About this webinar: Unless you live under a rock, you will have heard about OpenAI’s release of Deep Research on Feb 2, 2025. This new product promises to revolutionize how we answer questions requiring the synthesis of large amounts of diverse information. But how does this technology work, and why is Deep Research a noticeable improvement over previous attempts? In this webinar, we will examine the concepts underpinning modern agents using our basic clone, Deep Searcher, as an example. Topics covered: Tool use Structured output Reflection Reasoning models Planning Types of agentic memory

Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz

2025-04-05 - Block71 Event - The Landscape of GenAI and Ecosystem.pdfIvan Tang

17-October-2024 NYC AI Camp - Step-by-Step RAG 101Timothy Spann

17-October-2024 NYC AI Camp - Step-by-Step RAG 101 https://github.com/tspannhw/AIM-BecomingAnAIEngineer https://github.com/tspannhw/AIM-Ghosts AIM - Becoming An AI Engineer Step 1 - Start off local Download Python (or use your local install) https://www.python.org/downloads/ python3.11 -m venv yourenv source yourenv/bin/activate Create an environment https://docs.python.org/3/library/venv.html Use Pip https://pip.pypa.io/en/stable/installation/ Setup a .env file for environment variables Download Jupyter Lab https://jupyter.org/ Run your notebook jupyter lab --ip="0.0.0.0" --port=8881 --allow-root Running on a Mac or Linux machine is optimal. Setup environment variables source .env Alternatives Download Conda https://docs.conda.io/projects/conda/en/latest/index.html https://colab.research.google.com/ Other languages: Java, .Net, Go, NodeJS Other notebooks to try https://zilliz.com/learn/milvus-notebooks https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/build_RAG_with_milvus.ipynb References Guides https://zilliz.com/learn HuggingFace Friend https://zilliz.com/learn/effortless-ai-workflows-a-beginners-guide-to-hugging-face-and-pymilvus Milvus https://zilliz.com/milvus-downloads https://milvus.io/docs/quickstart.md LangChain https://zilliz.com/learn/LangChain Notebook display https://ipywidgets.readthedocs.io/en/stable/user_install.html References https://medium.com/@zilliz_learn/function-calling-with-ollama-llama-3-2-and-milvus-ac2bc2122538 https://github.com/milvus-io/bootcamp/tree/master/bootcamp/RAG/advanced_rag https://zilliz.com/learn/Retrieval-Augmented-Generation https://zilliz.com/blog/scale-search-with-milvus-handle-massive-datasets-with-ease https://zilliz.com/learn/generative-ai https://zilliz.com/learn/what-are-binary-vector-embedding https://zilliz.com/learn/choosing-right-vector-index-for-your-project

08-13-2024 NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)Timothy Spann

08-13-2024 NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus) https://www.meetup.com/unstructured-data-meetup-new-york/events/302512791/ This is an in-person event! Registration and photo identification is required to get in. Topic: Connecting your unstructured data with Generative LLMs What we’ll do: Have some food and refreshments. Hear three exciting talks about unstructured data and generative AI. 5:30 - 6:00 - Welcome/Networking/Registration 6:00 - 6:25 - Tim Spann, Principal DevRel, Zilliz 6:25 - 6:50 - Bill Qarbine, CTO, Quarbine 7:00 - 7:30 - Corey Nolet, Principal Engineer, NVIDIA :30 - 8:00 - Jacob Marks, Senior Machine Learning Engineer and Researcher, Voxel51 8:00 - 8:30 - Networking Tech talk 1: Quick intro to unstructured data, edge ai and Milvus Speaker: Timothy Spann, Principal Developer Advocate, Zilliz Tech talk 2: Modern Analytics & Reporting with Milvus Vector DB and GenAI Speaker: Bill Qarbine, CTO, Quarbine See how Milvus Cloud features can be applied up in the stack by developers to deliver interactive analytics using Milvus vector search and GenAI across the application life-cycle. Qarbine also includes a unique SQL-like option for a more approachable query interface than what the Milvus API provides. ![img](https://images.lumacdn.com/editor-images/5o/334942df-f888-4cca-aae6-4ef9d17007c4.png) Tech talk 3: cuVS+Milvus ![img](https://images.lumacdn.com/editor-images/uy/b4ee8afe-6bd5-48a1-a949-be97c4ef9407.png) Tech talk 4: Combining Hugging Face Transformer Models and Visual Data with FiftyOne Datasets and Models are the two pillars of modern machine learning, but connecting the two can be cumbersome and time-consuming. In this lightning talk, you will learn how the seamless integration between Hugging Face and FiftyOne simplifies this complexity, enabling more effective data-model co-development. By the end of the talk, you will be able to download and visualize datasets from the Hugging Face hub with FiftyOne, apply state-of-the-art transformer models directly to your data, and effortlessly share your datasets with others. Jacob Marks is a Senior Machine Learning Engineer and Researcher at Voxel51, where he conducts research in representation learning, synthetic data evaluation, and data-centric AI. He also leads open-source efforts in search and generative AI for the FiftyOne data-centric AI toolkit, including building VoxelGPT and integrations with Hugging Face, vector databases, and more. Prior to joining Voxel51, Jacob worked at Google X, Samsung Research, and Wolfram Research. In a past life, he was a theoretical physicist: in 2022, he completed his Ph.D. at Stanford, where he investigated quantum phases of matter. ![img](https://images.lumacdn.com/editor-images/em/d759369b-cbdb-4ce3-aa7b-756e728cfddc.png) Who Should attend: Anyone interested in talking and learning about Unstructured Data and Generative AI Apps. When: August 13 2024 5:30PM Where: This is an in-person event! Registration

NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)Zilliz

09-03-2024_UnstructuredDataAndAIDiscussion.pdfTimothy Spann

https://www.startupgrind.com/events/details/startup-grind-princeton-presents-building-the-future-ai-amp-startups-in-action/cohost-princeton Building the Future: AI & Startups in Action Sep 3, 6:00 – 8:00 PM Princeton 23 Orchard Rd, 23 Orchard Road, Montgomery, 08558 Step into the future! and see AI & Startups in Action. SeeExplore how AI drives innovation with live demos, no-code tools, and insights from startup founders. Join us to see how AI transforms ideas into impactful, real-world solutions! RAG

NYCMeetup07-25-2024-Unstructured Data Processing From Cloud to EdgeTimothy Spann

NYCMeetup07-25-2024-Unstructured Data Processing From Cloud to Edge https://www.meetup.com/unstructured-data-meetup-new-york/ https://www.meetup.com/unstructured-data-meetup-new-york/events/301720478/ Details This is an in-person event! Registration is required to get in. Topic: Connecting your unstructured data with Generative LLMs What we’ll do: Have some food and refreshments. Hear three exciting talks about unstructured data and generative AI. 5:30 - 6:00 - Welcome/Networking/Registration 6:05 - 6:30 - Tim Spann, Principal DevRel, Zilliz 6:35 - 7:00 - Chris Joynt, Senior PMM, Cloudera 7:05 - 7:30 - Lisa N Cao, Product Manager, Datastrato 7:30 - 8:30 - Networking Tech talk 1: Unstructured Data Processing From Cloud to Edge Speaker: Tim Spann, Principal Dev Advocate, Zilliz In this talk I will do a presentation on why you should add a Cloud Native vector database to your Data and AI platform. He will also cover a quick introduction to Milvus, Vector Databases and unstructured data processing. By adding Milvus to your architecture you can scale out and improve your AI use cases through RAG, Real-Time Search, Multimodal Search, Recommendations Engines, fraud detection and many more emerging use cases. As I will show, Edge devices even as small and inexpensive as a Raspberry Pi 5 can work in machine learning, deep learning and AI use cases and be enhanced with a vector database. Tech talk 2: RAG Pipelines with Apache NiFi Speaker: Chris Joynt, Senior PMM, Cloudera Executing on RAG Architecture is not a set-it-and-forget-it endeavor. Unstructured or multimodal data must be cleansed, parsed, processed, chunked and vectorized before being loaded into knowledge stores and vector DB's. That needs to happen efficiently to keep our GenAI up to date always with fresh contextual data. But not only that, changes will have to be made on an ongoing basis. For example, new data sources must be added. Experimentation will be necessary to find the ideal chunking strategy. Apache NiFi is the perfect tool to build RAG pipelines to stream proprietary and external data into your RAG architectures. Come learn how to use this scalable and incredible versatile tool to quickly build pipelines to activate your GenAI use case. Tech Talk 3: Metadata Lakes for Next-Gen AI/ML Speaker: Lisa N Cao, Datastrato Abstract: As data catalogs evolve to meet the growing and new demands of high-velocity, unstructured data, we see them taking a new shape as an emergent and flexible way to activate metadata for multiple uses. This talk discusses modern uses of metadata at the infrastructure level for AI-enablement in RAG pipelines in response to the new demands of the ecosystem. We will also discuss Apache (incubating) Gravitino and its open source-first approach to data cataloging across multi-cloud and geo-distributed architectures. Who Should attend: Anyone interested in talking and learning about Unstructured Data and Generative AI Apps. When: July 25, 2024 5:30PM

Building Production Ready Search Pipelines with Spark and MilvusZilliz

Read more: https://zilliz.com/blog/building-production-ready-search-pipelines-spark-milvus Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.

Multimodal Embeddings (continued) - South Bay Meetup SlidesZilliz

2025-02-24 - AWS meetup - Zilliz presentation.pdfIvan Tang

Zilliz's presentation in AWS x Apache Doris meetup on 24th Feb 2025 in Singapore. In this presentation, I shared a live demo on how you can outsource thinking and reasoning to Amazon Nova for generating a research report on any topic. Code repo for live demo can be found here: https://github.com/zilliztech/deep-searcher I've also shared about the hard tradeoffs you need to make when choosing vector indexes and lastly my top 5 favourite features of Zilliz Cloud.

Multimodal Retrieval-Augmented Generation (RAG) with Vector DatabaseZilliz

Multi-modality elevates the capabilities of neural network models to a whole new level. By leveraging contrastive learning and specialized model architectures, we can create a unified vector space for images and text, enhancing multimodal representations. This talk will share insights into building image-text search and Composite Image Retrieval (CIR) using multimodal embeddings and the Milvus open source vector database, demonstrating how multi-modality unlocks new use cases in Retrieval-Augmented Generation (RAG).

Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & ...Zilliz

About this webinar Discover how to integrate Vision Language Models with Browser Use and Milvus to create an agentic system capable of real-time visual and textual analysis. Ideal for developers who want to learn how to use Agents that can see, take action, and remember what they saw. This Session Will: - Demonstrate a workflow where Browser Use extracts dynamic web data, while Milvus stores and retrieves the data, that way you can always come back to what the agent saw. - Showcase practical use cases, such as querying live web content with AI agents that reason over historical and visual data. - Explore balancing autonomy and control in agentic systems, including challenges like hallucination mitigation and performance optimization.

Supercharge Spark: Unleashing Big Data Potential with Milvus for RAG systemsZilliz

Apache Spark dominates the big data processing world, but efficient vector similarity search on massive datasets remains a bottleneck. This talk will show how you can seamlessly integrate Milvus with Spark to unlock the true power of vector similarity search. We'll explore how Milvus integrates with Spark, enabling efficient vector search within Spark workflows. Real-world applications showcasing the combined power of Spark and Milvus in tackling complex similarity search challenges will be presented. Finally, we'll shed light on the significant performance gains achieved through this integration. Whether you're dealing with recommendation systems, image retrieval, or any other application requiring vector similarity search, this talk will equip you with the knowledge to leverage Spark and Milvus to their maximum potential. Join us on this exploration of how Spark and Milvus can enhance your big data processing capabilities with fast similarity search even at scale!

09-12-2024 - Milvus, Vector database used for Sensor Data RAGTimothy Spann

09-12-2024 - Milvus Sensor Data RAG https://voxel51.com/blog/the-computer-vision-interface-for-vector-search/ https://www.linkedin.com/feed/update/urn:li:activity:7233322212370300929/ It’s in the Air Tonight. Sensor Data in RAG– Tim Spann 🥑 at Milvus #computervision#ai#artificialintelligence#machinevision#machinelearning#datascience https://voxel51.com/computer-vision-events/ai-machine-learning-computer-vision-meetup-sept-12-2024/ will do a quick overview of the basics of Vector Databases and Milvus and then dive into a practical example of how to use one as part of an application. I will demonstrate how to consume air quality data and ingest it into Milvus as vectors and scalars. We will then use our vector database of Air Quality readings to feed our LLM and get proper answers to Air Quality questions. I will show you how to all the steps to build a RAG application with Milvus, LangChain, Ollama, Python and Air Quality Reports. Preview the demo on Medium. About the Speaker Tim Spann is a Principal Developer Advocate for Zilliz and Milvus. He works with Milvus, Generative AI, HuggingFace, Python, Big Data, IoT, and Edge AI. Tim has over twelve years of experience with the IoT, big data, distributed computing, messaging, machine learning and streaming technologies.

Introduction to Large Language Model Customization.pdfZilliz

09-19-2024 AI Camp Hybrid Seach - Milvus for Vector DatabaseTimothy Spann

2024-10-28 All Things Open - Advanced Retrieval Augmented Generation (RAG) Te...Timothy Spann

Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI...Zilliz

10-25-2024_BITS_NYC_Unstructured Data and LLM_ What, Why and HowTimothy Spann

What Makes "Deep Research"? A Dive into AI AgentsZilliz

Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz

2025-04-05 - Block71 Event - The Landscape of GenAI and Ecosystem.pdfIvan Tang

17-October-2024 NYC AI Camp - Step-by-Step RAG 101Timothy Spann

08-13-2024 NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)Timothy Spann

NYC Meetup Unstructured Data Processing From Cloud to Edge (Milvus)Zilliz

09-03-2024_UnstructuredDataAndAIDiscussion.pdfTimothy Spann

NYCMeetup07-25-2024-Unstructured Data Processing From Cloud to EdgeTimothy Spann

Building Production Ready Search Pipelines with Spark and MilvusZilliz

Multimodal Embeddings (continued) - South Bay Meetup SlidesZilliz

2025-02-24 - AWS meetup - Zilliz presentation.pdfIvan Tang

Multimodal Retrieval-Augmented Generation (RAG) with Vector DatabaseZilliz

Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & ...Zilliz

Supercharge Spark: Unleashing Big Data Potential with Milvus for RAG systemsZilliz

09-12-2024 - Milvus, Vector database used for Sensor Data RAGTimothy Spann

Introduction to Large Language Model Customization.pdfZilliz

09-19-2024 AI Camp Hybrid Seach - Milvus for Vector DatabaseTimothy Spann

More from Zilliz (20)

Smarter RAG Pipelines: Scaling Search with Milvus and FeastZilliz

About this webinar Learn how Milvus and Feast can be used together to scale vector search and easily declare views for retrieval using open source. We’ll demonstrate how to integrate Milvus with Feast to build a customized RAG pipeline. Topics Covered - Leverage Feast for dynamic metadata and document storage and retrieval, ensuring that the correct data is always available at inference time - Learn how to integrate Feast with Milvus to support vector-based retrieval in RAG systems - Use Milvus for fast, high-dimensional similarity search, enhancing the retrieval phase of your RAG model

Webinar - Zilliz Cloud Monthly Demo - March 2025Zilliz

Join our monthly demo for a technical overview of Zilliz Cloud, a highly scalable and performant vector database service for AI applications Topics covered - Zilliz Cloud's scalable architecture - Key features of the developer-friendly UI - Security best practices and data privacy - Highlights from recent product releases - This webinar is an excellent opportunity for developers to learn about Zilliz Cloud's capabilities and how it can support their AI projects. Register now to join our community and stay up-to-date with the latest vector database technology.

Combining Lexical and Semantic Search with Milvus 2.5Zilliz

In short, lexical search is a way to search your documents based on the keywords they contain, in contrast to semantic search, which compares the similarity of embeddings. We’ll be covering: Why, when, and how should you use lexical search What is the BM25 distance metric How exactly does Milvus 2.5 implement lexical search How to build an improved hybrid lexical + semantic search with Milvus 2.5

Bedrock Data Automation (Preview): Simplifying Unstructured Data ProcessingZilliz

Bedrock Data Automation (BDA) is a cloud-based service that simplifies the process of extracting valuable insights from unstructured content—such as documents, images, video, and audio. Come learn how BDA leverages generative AI to automate the transformation of multi-modal data into structured formats, enabling developers to build applications and automate complex workflows with greater speed and accuracy.

Deploying a Multimodal RAG System Using Open Source Milvus, LlamaIndex, and vLLMZilliz

About this webinar: While text-based RAG systems have been everywhere in the last year and a half, there is so much more than text data. Images, audio, and documents often need to be processed together to provide meaningful insights, yet most RAG implementations focus solely on text. Think about automated visual inspection systems understanding manufacturing logs and production line images, or robotics systems correlating sensor data with visual feedback. These multimodal scenarios demand RAG systems that go beyond text-only processing. In this talk, we'll walk through how to build a Multimodal RAG system that helps solve this problem. We'll explore the architecture that makes it possible to run such a system and demonstrate how to build one using Milvus, LlamaIndex, and vLLM for deploying open-source LLMs on your infrastructure. Through a live demo, we'll showcase a real-world application processing both images and text queries. Whether you're looking to reduce API costs, maintain data privacy, or gain more control over your AI infrastructure, this session will provide you with actionable insights to implement Multimodal RAG in your organization. Topics covered: - vLLM and self hosting LLMs - Multimodal RAG Demo: a real-world application processing both images and text queries

February Product Demo: Discover the Power of Zilliz CloudZilliz

Join our monthly demo for a technical overview of Zilliz Cloud, a highly scalable and performant vector database service for AI applications Topics covered - Zilliz Cloud's scalable architecture - Key features of the developer-friendly UI - Security best practices and data privacy - Highlights from recent product releases This webinar is an excellent opportunity for developers to learn about Zilliz Cloud's capabilities and how it can support their AI projects. Register now to join our community and stay up-to-date with the latest vector database technology.

Full Text Search with Milvus 2.5 - UD Meetup Berlin Jan 23Zilliz

"Milvus 2.5 introduces text search by introducing native full text search capabilities, seamlessly combining term-based matching with vector similarity in a single system. This feature automatically handles text-to-vector conversion and real-time BM25 scoring, eliminating the complexity of manual embedding generation and external processing pipelines. Through a live demo, we'll showcase how easy we make it to use Full Text search now :D"

Building the Next-Gen Apps with Multimodal Retrieval using Twelve Labs & MilvusZilliz

"This session dives deep into the power of Multimodal Retrieval, a revolutionzing approach that enhances personalization by seamlessly integrating diverse data sources for more intuitive product interactions. Explore the foundational concepts of Multimodal Embedding and Any-to-Any Search, and learn how to leverage these technologies to build next generation products. Discover how to seamlessly integrate the Twelve Labs Embed API and Milvus into your projects. Through live demos, you’ll see how Fashion Product Search is redefined with deeper insights into the architecture, and discover how this approach is revolutionizing user interactions, especially with bots. We’ll also explore real world case studies that demonstrate the ease and power of building multimodal apps."

Voice-to-Value- LLM-Powered Customer Interaction Analysis.pdfZilliz

"Explore the transformative potential of Voice AI in customer interaction analysis powered by LLMs. Learn how Gemini 2.0 enables transcription, summarization, and actionable insight extraction to streamline ticket resolution and enhance customer experiences. This session delves into the architecture and practical applications of LLM-powered systems, showcasing how they revolutionize customer support workflows through real-world examples and insights"

Accelerate AI Agents with Multimodal RAG powered by Friendli Endpoints and Mi...Zilliz

AI agents are transforming industries, especially with recent vision-language models like Llama 3.2 Vision that enable AI agents to go beyond text-based understanding by integrating multimodal capabilities. Building such advanced AI agents can feel complex, but FriendliAI simplifies the process by offering end-to-end solutions, from creating your own custom models to deploying them in production. In this webinar, we’ll learn about the AI developer workflow from model fine-tuning to inference serving. We’ll also work through building a simple AI agent with advanced multimodal RAG capabilities using Friendli Serverless Endpoints and Milvus DB. This session is ideal for those looking to learn more about large-language model inference serving, start building AI agents with RAG capabilities, or explore multimodal RAG queries in greater depth

1 Table = 1000 Words? Foundation Models for Tabular DataZilliz

Tables form the backbone of modern data storage, powering everything from relational databases to enterprise systems. Yet despite their ubiquity, we've barely scratched the surface of their potential. While Deep Learning has revolutionized our ability to process text and images, its impact on tabular data has been surprisingly limited. This gap is now being bridged through groundbreaking research in multimodal modeling, particularly with innovations like the TableGPT2 model. In this talk, we'll explore how these new multimodal foundation models are trained to understand tabular data, and demonstrate practical ways to unlock hidden value in your organization's data assets.

How Milvus allows you to run Full Text SearchZilliz

How to Optimize Your Embedding Model Selection and Development through TDA Cl...Zilliz

About this webinar: Embedding models are a crucial layer in vector database applications, yet figuring out which embedding model is best for your dataset has been a notoriously difficult task. However, an efficient and intuitive approach for many use cases can be produced through Topological Data Analysis (TDA) on your evaluation dataset. Identifying patterns of weak performing behavior in your model is made easy and scalable through a table that reveals the performance of different semantic categories of queries being made to your vector database. Topics covered: - Risks and limitations of current evaluation approaches for embeddings - Compare embedding models on your own dataset using Navigable TDA clusters - ML lifecycle case studies in ecommerce: model selection, fine-tuning, and post-deployment

Milvus: Scaling Vector Data Solutions for Gen AIZilliz

Milvus, an LF AI project, is an open-source vector database built to power Gen AI solutions. 80% of the data in the world is unstructured data, and vector databases are the databases that help you get valuable insights from unstructured data. With this in mind, we built Milvus as a distributed system on top of other open-source solutions, including MinIO and Kafka, to support vector collections that exceed billion-scale. This session will deeply dive into the architecture decisions that make this cloud-native vector database seamlessly scale horizontally, provide users with tunable consistency, orchestrate in-memory and on-disk indexing, and scalable search strategies.

Keeping Data Fresh: Mastering Updates in Vector DatabasesZilliz

Managing and extracting value from unstructured data has become a critical challenge as the volume of data continues to grow. This virtual event brings together industry experts to explore the latest techniques in Retrieval Augmented Generation (RAG) and vector databases. Discover how RAG systems are revolutionizing natural language processing by seamlessly integrating information retrieval techniques, enabling more accurate and contextual language generation. Gain practical insights into building and optimizing these applications. This session will also cover how vector databases like Milvus, play a key role in RAG and working with unstructured data. Learn proven strategies for maintaining data freshness, accuracy, and efficiency, ensuring your organization stays ahead of the curve.

GraphRAG Agents with Neo4j, Milvus and GPT4Zilliz

Using LLM Agents with Llama 3.2, LangGraph and MilvusZilliz

Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!Zilliz

Vector Databases for Enhanced ClassificationZilliz

What will you learn? In this webinar, we dive into the use of Milvus as a high-performance vector database tailored for handling large-scale document collections, focusing on European Commission and Parliament acts. Our approach shifts from traditional RAG-based classification to a hybrid search method, leveraging K-Nearest Neighbor (KNN) for pinpointing top documents relevant to classification tasks. This session is ideal for those aiming to refine classification accuracy by leveraging vector-based indexing and hybrid retrieval in vast datasets. Topics covered: KNN and Sparse Search Integration: How KNN retrieval combined with sparse search helps extract top documents aligned with classification needs. Versatile Embeddings for Multilingual and Multi-Domain Applications: The BGE M3-Embedding model is designed to provide robust, high-quality embeddings across multiple languages and domains, making it adaptable for diverse tasks in multilingual and cross-functional environments. Real-World Application: Step-by-step demonstration using European legislative acts to showcase KNN-driven retrieval and classification workflows.

Building an Accuracy Flywheel for your LLM RAG AppsZilliz