28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Member-only story

Optimizing Dialog LLM Chatbot Retrieval

Augmented Generation with a Swarm
Architecture
Anthony Alcaraz · Follow
5 min read · Aug 25

Listen Share More

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 1/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Art

Retrieval augmented generation (RAG) has become a dominant paradigm for

creating conversational AI agents like LLM chatbots.

By retrieving relevant information and context, RAG allows dialog models to go

beyond their training data and have more natural, knowledgeable conversations.

However, as RAG scales to real-world production use, several challenges emerge.

In this article, I discuss how a swarm architecture can help optimize and solve some
of these RAG challenges for dialog chatbots.

What is Retrieval Augmented Generation (RAG)?

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 2/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

RAG combines a powerful neural dialog generator model like GPT-3 with the ability
to retrieve and incorporate external knowledge and context.

At its core, RAG consists of two main components:

Retriever: Responsible for finding and retrieving relevant context for the current
conversation from various sources like:

Vector databases: Stores embeddings of documents and uses semantic

similarity search to find related context.

Knowledge graphs: Directly queries a knowledge graph to find relevant entities

and relationships.

Generator: A large language model that incorporates the retrieved context and
generates a response.

By providing relevant external information to the generator, RAG reduces

hallucination and repetition while improving specificity and factual grounding
compared to conversation without retrieval.

Challenges of Scaling RAG

As RAG moves from prototypes to production conversational systems at scale,
several key challenges emerge:

Slow or inadequate retrieval: Errors and latency from the retriever harm the
user experience.

Repetitive or irrelevant retrieval: Bringing the same context repetitively

degrades responses.

Scaling compute: RAG is computationally heavy due to retrieval per turn and
generator model size.

Prompt engineering: Hard to manually craft optimal prompts with new topics,
users, and contexts.

Brittle pipelines: Complex RAG systems with many components can fail in
unexpected ways.

Data silos: Information retrieval limited to only certain corpora or sources.

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 3/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Catastrophic forgetting: Conversation history and context is lost across turns.

How Swarm Architecture Can Revolutionize RAG for Chatbots

The swarm architecture: an ensemble-based approach that offers solutions by
distributing tasks across multiple agents.

1. Understanding Swarm Architecture: A swarm architecture operates like a hive

mind. Instead of relying on a single agent, it employs an ensemble of diverse,
loosely coupled agents. These agents work together, coordinating and sharing
information to solve problems. Imagine a team where each member has a unique
skill and they communicate effectively to produce a holistic solution; that’s the
essence of a swarm system.

2. Incorporating Diverse Prompting Techniques: One of the major advantages of the

swarm approach is its ability to incorporate multiple prompting techniques. This is
how it’s achieved:

Manager Agent: An overseer that dictates prompt strategy and delegates tasks
based on agent capabilities.

Dedicated Prompt Agents: Each focuses on a specific prompting technique — be

it input-output prompting, chain of thought, or skeleton prompts.

Collaborative Efforts: Agents can offer partial prompts that others can then
build upon, allowing for a mosaic of ideas.

Automated Optimization: Prompt agents propose prompt variations, with the

manager picking the most fitting one.

3. Combining Diverse Models: Diversity isn’t just limited to prompting. Here’s how
the swarm architecture ensures model diversity:

Specialization: Agents can house models fine-tuned for specific skills, topics, or
modalities.

Niche Performance: Introducing new agents into the swarm enables better
performance on specialized tasks.

Flexibility: Different agents, different model sizes. The right agent can be
chosen for the right task.

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 4/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Dynamic Growth: As new models are developed, they can be effortlessly

integrated into the swarm.

4. Optimizing RAG for Chatbots: Swarm architecture doesn’t merely introduce

diversity; it actively optimizes RAG. Here’s how:

Parallel Retrieval: Specialized agents allow for simultaneous querying, speeding

up processes.

Redundancy: Multiple retrievers provide varied contexts, enhancing relevance

while minimizing repetition.

Learning in Unison: The swarm, in its diversity, can refine prompt construction
via mutual reinforcement learning.

Automated Prompting: Tailored prompts based on dialog history and user

profiles enhance personalization.

Resilience and Redundancy: A breakdown in one agent doesn’t cripple the

system. The swarm ensures business as usual.

Incremental Growth: As the technology evolves, new agents can be added

without disrupting existing operations.

Shared Memory: The swarm can remember and leverage past interactions,
making conversations more context-aware.

Scalability: Distribute agents across available computing resources for efficient

scaling.

The swarm architecture, with its properties of distribution, diversity, redundancy,

and flexible coordination, is poised to bring about a revolution in how we perceive
and deploy RAG in dialog systems.

Detailed Swarm Architecture for Dialog RAG

Let’s now look at a more concrete implementation sketch with sample Python
pseudo-code.

Swarm Architecture
At an abstract level, our swarm consists of the following components:

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 5/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Shared memory — Central storage for conversation context, facts, and retrieved
information. All agents can access this.

Task queue — Holds incoming user queries and resulting dialog tasks that
agents can work on.

Manager — Handles task assignment and oversees swarm coordination.

Retrieval agents — Specialized agents that focus on efficient context retrieval

from diverse sources.

Prompt agents — Agents that suggest prompt variations and formats tailored to
the dialog.

Generator agent — Single agent that incorporates retrieval results into prompts
for the generator model.

Orchestrator — Central component that interfaces with the outside world.

The key idea is that instead of a monolithic pipeline, responsibilities are distributed
across decentralized agents that share information and coordinate as needed to
have an ongoing conversation with the user.

The loose coupling provided by the swarm architecture makes the system robust,
flexible, and scalable.

Sample Implementation

# Shared memory
memory = VectorDatabase()

# Task queue
task_queue = TaskQueue()

# Manager agent
manager = ManagerAgent(memory, task_queue)
# Specialized retriever agents
vector_retriever = VectorRetrieverAgent(memory)
graph_retriever = GraphRetrieverAgent(memory)
web_retriever = WebRetrieverAgent(memory)

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 6/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

# Prompt engineering agents

template_agent = TemplatePromptAgent(memory)
profile_agent = UserPromptAgent(memory)
# Single generator agent
generator = GeneratorAgent(memory)
# Orchestrator
orchestrator = Orchestrator(memory, task_queue)

# Start all agents

start_agents([manager, vector_retriever, graph_retriever,
web_retriever,
template_agent, profile_agent, generator])
# Main dialog loop
while True:
# Get next user query
user_query = orchestrator.get_input()

# Create dialog task

task = DialogTask(user_query)
# Add task to queue
task_queue.put(task)

# Process swarm until task is complete

while not task.complete():
# Each agent does work
for agent in [vector_retriever, graph_retriever, web_retriever,
template_agent, profile_agent, generator]:
agent.process()
# Manager handles coordination
manager.optimize_swarm()

# Output final response

orchestrator.output(task.get_response())

This provides a rough sketch of how a swarm architecture enables distributing key
Open agents
RAG components across loosely coupled in app that share context and coordinate as
needed to have an ongoing dialog with the user. 1
Search Medium

The agents leverage parallelism while the swarm provides resilience and flexibility
to improve and scale dialog RAG capabilities in an incremental manner.

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 7/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

AI Machine Learning Deep Learning Llmops Software Development

Follow

Written by Anthony Alcaraz

3.4K Followers

Chief AI Officer, ML&LLMOps expert, passionate about decision making.

https://www.linkedin.com/in/anthony-alcaraz-b80763155/ https://aldecis.com/

More from Anthony Alcaraz

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 8/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Anthony Alcaraz in GoPenAI

Knowledge Graph Prompting: A New Approach for Multi-Document

Question Answering
Multi-document question answering (MD-QA) involves answering questions that require
synthesizing information across multiple documents.

· 8 min read · Aug 28

424 8

Anthony Alcaraz in Artificial Intelligence in Plain English

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-architect… 9/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Vector Search Is Not All You Need

Introduction

· 6 min read · Sep 18

356 10

Anthony Alcaraz in GoPenAI

The Complete Overview to Retrieval Augmented Generation (RAG)

Retrieval augmented generation (RAG) is an exciting technique that is transforming how
natural language processing systems work. In this…

· 6 min read · Sep 1

172 2

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 10/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Anthony Alcaraz in Artificial Intelligence in Plain English

The Rise of AI Agents : A Focus On Multi-Agents Systems

The concept of an “agent” has roots tracing back millennia to ancient philosophers like
Aristotle.

· 7 min read · Sep 16

132

See all from Anthony Alcaraz

Recommended from Medium

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 11/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Jayita Bhattacharyya in GoPenAI

Primer on Vector Databases and Retrieval-Augmented Generation (RAG)

using Langchain, Pinecone &…
Vector Databases Generation (RAG) Langchain Pinecone HuggingFace Large Language model
generative ai

9 min read · Aug 16

212 1

Dominik Polzer in Towards Data Science

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 12/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

All You Need to Know about Vector Databases and How to Use Them to
Augment Your LLM Apps
A Step-by-Step Guide to Discover and Harness the Power of Vector Databases

· 24 min read · Sep 17

960 8

Lists

Predictive Modeling w/ Python

20 stories · 427 saves

Practical Guides to Machine Learning

10 stories · 489 saves

The New Chatbots: ChatGPT, Bard, and Beyond

13 stories · 127 saves

Natural Language Processing

652 stories · 253 saves

Cking in newmathdata

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 13/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Harnessing the Power of Large Language Models for Knowledge Graph

Creation
· The role of large language models in creating knowledge graphs from unstructured data. ·
Comparison of Top Models · Storage Platforms for…

11 min read · Sep 13

25

Waveline

Extract Data from Documents with ChatGPT

Guide on how to extract data from documents like PDFs using Large Language Models (LLMs)

4 min read · Jul 19

250

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 14/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

ai geek (wishesh)

Best Practices for Deploying Large Language Models (LLMs) in

Production
Large Language Models (LLMs) have revolutionized the field of natural language processing
and understanding, enabling a wide range of AI…

10 min read · Jun 26

66 1

Sachin Kulkarni

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 15/16
28/9/23, 8:52 Optimizing Dialog LLM Chatbot Retrieval Augmented Generation with a Swarm Architecture | by Anthony Alcara…

Generative AI with Enterprise Data

Create business value add Enterprise knowledge to Large Language Models

6 min read · Jul 25

269 5

See more recommendations

https://medium.com/@alcarazanthony1/optimizing-dialog-llm-chatbot-retrieval-augmented-generation-with-a-swarm-archite… 16/16