LLM +

Automation

Framework

Guide for
Table of Contents

I. Introduction ........................................................................................................1
I.1 Overview & Challenge ................................................................................................................................................ 1

I.2 Benefits ........................................................................................................................................................................ 2

II. Prerequisite ......................................................................................................2

III.1 Runtime Dependencies ........................................................................................................................................... 3

III.2 Host Python Backend using Docker ....................................................................................................................... 3

III. Framework & Solution .....................................................................................4

II.1 Solution Overview ..................................................................................................................................................... 4

II.2 Solution Architecture .............................................................................................................................................. 10

II.3 Framework Components ........................................................................................................................................ 11

II.4 Reusable Components ............................................................................................................................................ 12

IV. Quick Start Use Cases .................................................................................... 13

IV.1 Conversational (Default) ........................................................................................................................................ 13

IV.2 Transactional .......................................................................................................................................................... 15

IV.3 Transactional with human in the loop ................................................................................................................. 17

V. Appendix ......................................................................................................... 20
V.1 Future Enhancements............................................................................................................................................. 20

V.2 Known Issues and Limitations ............................................................................................................................... 20

V.3 Additional sources of documentation .................................................................................................................. 20

[Type here]
I. Introduction

I.1 Overview & Challenge

Leverage this LLM Framework template along with its Generative AI powered

custom activities to make LLM App & Automation Development much easier,

more flexible, more reliable and built for enterprise level usage with

automation.

We've seen how powerful large language models can be, whether it's for generating

professional looking essays, responding to emails, extracting entities from

unstructured sources, generating works of art, music, etc - that's why we fondly call

them generative AI.

Its power relies on the huge amount and quality of data it's trained with, that

ironically is also where the limiting factor is. But through other integrations it can

make these models more powerful and reliable.

The challenge with large language models by itself is it's built to be general

purpose. It has been trained on millions of pieces of text so it can "understand

semantically" the human language.

It might be able to answer specific questions based on the information it was

trained on, but in reality, large language models don’t know of the underlying

context that language describes, as the technology aims to generate text that

sounds nice, grammatically, and semantically, but they don’t really have some kind

of objectivity or fact-based context aside from just satisfying statistical consistency

with the prompt.

So essentially, without the proper integrations or framework, LLMs can perpetuate

and amplify harmful biases through hallucinations and might, in turn, reinforce

today's pandemic issue of misinformation and disinformation.

I.2 Benefits

By marrying UiPath with Generative AI capabilities (specifically LLMs) through this

solution, it can help you achieve the following:

1. Extend the generative capabilities of large language models by providing

context from any types of sources.

2. Achieve full automation mileage whether its attended or unattended

automations, by supplementing the output derived from LLM with custom

actions.

3. Address general concerns around information accuracy, by leveraging

automation to help speed up adoption at an enterprise level.

4. Quickly build your own Generative AI powered automation using Large

Language Models, using this framework that is both flexible and reusable.

5. This template leverages custom connectors/API built on top of LangChain

framework which is an industry standard in LLM app development.

II. Prerequisite

The custom activity package used throughout the framework is the main pre-

requisite to utilize the template for your LLM powered projects. This section depicts

the steps to set up the custom activities included in the package.

See Appendix ‘V.3. Additional sources of documentation’ for full video tutorials
III.1 Runtime Dependencies

o Docker - Download Docker Desktop | Docker

o OpenAI API Key (https://platform.openai.com/) or Claude API Key (Anthropic

Console)
o Pinecone Account - https://app.pinecone.io/

A. Register and create a project.

B. Create/Name your index.

C. Index Dimensions should be 1536 (OpenAI output embeddings dimension).

III.2 Host Python Backend using Docker

The following step portrays docker container running on a local machine. In a

production scenario it’s best to host the API on a proper remote app server.

a. Download & Install Docker - https://www.docker.com/products/docker-desktop/

b. Make sure Docker is running by starting up the application. Verify docker is running

by checking docker icon in the right icons panel

 c. Run below command in command line

docker run -p 5000:5000 zell12/qblangchainapi:latest

Where first 5000 -> external port accessed by client; and second 5000 -> internal port from code e.g. waitress-server --

listen=0.0.0.0:5000 app:app

d. Verify if http://localhost:5000 is running in the browser, which means the API is

running.

If hosted on a remote webserver, localhost will be replaced with the remote host name or IP

III. Framework & Solution

II.1 Solution Overview

The solution comprises of 2 main components:

A. UiPath Studio Template (LLM Framework Studio Template)

B. Custom Activity Package (LLM Framework Activities)

A. LLM Framework Studio Template

This template orchestrates the reusability of the solution across a wide array

of use cases powered up large language models, which can either be

Conversational (chat based/context aware conversations), Transactional (queue

based) and Transactional with human in the loop.

Along with this reusability, there is an important consideration for flexibility in the

framework so that it can cater to different types of downstream process which may

or may not involve UI/API automation.

It is created to fit all of the best practices regarding logging, exception handling,

application initialization, and others, being ready to tackle a complex conversational

and transactional business scenario.

The template contains several pre-made State containers for initializing

applications, retrieving input data, processing it, and ending the transaction. All of

these States are connected through multiple Transitions which cover almost every

need in a standard automation scenario. There are also multipleinvoked workflows,

each handling particular aspects of the project.

 B. Custom Activities

LLM Powered Custom Activities built using LangChain Framework. These

activities build up the core elements to enable context-aware and memory

backed interaction with your favorite large language models.

a. Ingest Knowledge Source – Create and serialize vector database out of

provided sources such as csv, pdf, YouTube video links, word documents,

etc.

Properties

- Authentication
API Endpoint (String)
This is the endpoint where the backend API is hosted (either locally or on a remote app

server) + the api route = <API_Endpoint_BaseUrl>/api/vectordb

For instance, if docker container is ran in local machine where the bot is, then the API

endpoint will be the, where <API_Endpoint_BaseUrl> = http://localhost:5000

Resulting endpoint will be http://localhost:5000/api/vectordb

Embeddings API Key (String)

This is using OpenAI Embeddings so the key will come from https://platform.openai.com >

Account > View API Keys (API keys - OpenAI API)

 Pinecone API Key (String)
API Key for the vector database. Must have a Pinecone Index created either at

https://app.pinecone.io/ or from Azure Resources. Fetch API Key from Pinecone Project >

API Keys. The following video guide includes tutorial on how to setup and get required

properties from Pinecone. https://youtu.be/xCqTXhPVakY

- Input
Source (String)
File path or URL sources. Base Path (e.g. c:\temp\file.csv -> file.csv or

https://youtube.com/watch?v=dafsf790 -> watch?v=dafsf790) will be used as a

namespace value in the vector store

- Options
Pinecone Environment (String)
The environment name of the Pinecone vector database e.g. us-east1-gcp

Pinecone Index Name (String)

The index name of the Pinecone vector database. Pinecone project > Indexes

- Output
Output Context (String)
Reference (i.e. namespace) to the vector embeddings or documents that will be used to

generate context using semantic search.

a. Converse With Context – Context aware LLM chain with conversational

buffer memory. Multi-input (llm model, context docs, memory/history,

prompt template) LLM chain.

Properties

- Authentication
API Endpoint (String)
 This is the endpoint where the backend API is hosted (either locally or on a remote app

server) + the api route = <API_Endpoint_BaseUrl>/api/llmchain

For instance, if docker container is ran in local machine where the bot is, then the API

endpoint will be the, where <API_Endpoint_BaseUrl> = http://localhost:5000

Resulting endpoint will be http://localhost:5000/api/llmchain

Embeddings API Key (String)

Pinecone API Key (String)
LLM API Key
API Key for the chosen LLM provider. Get API key from the LLM provider’s portal.

OpenAI -> https://platform.openai.com > Account > View API Keys (API keys - OpenAI API)

Anthropic -> https://console.anthropic.com > Account > API Keys

- Input
Source or Context (String)

Message
[Any] Question or instruction [to the provided context]

Memory
Serialized memory file injected into the LLM chain.

- Options
Pinecone Environment (String)

Pinecone Index Name (String)

LLM Provider
LLM provider that will be used for the completions. Defaults to OpenAIGPT.

LLM Model Name

Specific model name for chose LLM provider.

Defaults to gpt-4-0613.

OpenAI GPT Models -> https://platform.openai.com/account/rate-limits

Anthropic Claude Models -> https://www-

files.anthropic.com/production/images/model_pricing_july2023.pdf

- Output

AI Message
Context-aware completions Output

Updated Memory
Same reference to input memory, with updated context after the LLM chain run
 a. Fetch Knowledge Source – Fetch document/s from vector database for

inspection purposes

Properties

- Authentication
API Endpoint (String)

Embeddings API Key (String)

Pinecone API Key (String)

- Input
Source (String)

- Options
Pinecone Environment (String)

Pinecone Index Name (String)

- Output
Args (String)
Embeddings generation JSON string arguments. Includes keys - env, index_name,

namespace

Doc Content Array (String[])

Content of documents fetched from vector store

Doc Count (String)

Count of documents fetched from the vector store

C. Config Mapping

Global variables that is being used throughout the framework are being
mapped from the config.xlsx values in the 00_InitializeSource.xaml
II.2 Solution Architecture

Below is a high-level solution/technical architecture of the different gears or moving

parts powering up the solution. (Zoom in to 200% to view in more clarity)

II.3 Framework Components

Similar to REFramework, this template leverages state machines to flexibly handle

the flow for various types of use cases. The studio template includes some

components of REFramework stored in Framework\REFRamework folder, while the

other core framework components are stored in the root, Framework, Forms and

Data folder which are enumerated below:

a. LLMFramework.xaml – This framework revolves around standard usages of

LLM for various real-world use cases, which can generally be categorized as

Transactional (Input – Process – Output) and Conversational (Message –

Response)

b. Framework Folder – Core set of files that builds up the framework

b.1. 00_InitializeSource
Connect to various sources, convert into embeddings and store into a vector database, to be

used as context for the prompt template. Currently supported sources: CSV, DOCX, PDF,

YOUTUBE VIDEOS

b.2. 10_InteractLLM.xaml
Initialize Prompt Template, Fetch or Restore stored memory, (Re)Initialize LLM chain or

agent, LLM Interrogation, Interaction Flow Rules

b.3. 15_ActionCenter.xaml
Customizable action center form for your specific use case where the LLM output would

require human validation.

b.4. 20_PerformActions.xaml
 Add process flows to interact with ANY downstream systems either through UI or API using

the response from the LLM.

b.5. 25_PostProcessResponse.xaml
Add process flows to perform any post processing step before completing the current

transaction or conversation.

c. Data Folder – Contains configuration files, temporary data, sample files

d. Forms Folder – Scenario specific. Conversational. Sample Forms

II.4 Reusable Components

Reusable components are stored in the Framework\Reusable components folder

which currently consists of the following:

e. AfterResumeWorkflow.xaml

Customizable Post Resume Workflow e.g.- Verifying task action/button

executed i.e. submit, reject, etc.- Post processing of validated data based on

the task action

f. InteractionFlowRulesValidation.xaml

Use this workflow to determine the next step after the LLM interaction.
IV. Quick Start Use Cases

IV.1 Conversational (Default)

Step1: In Project Settings , make sure ‘Supports Persistence’ is disabled.

Step 2: Publish Project to Orchestrator. Make sure the customizable fields are

filled up or just remove other samples xamls that a not relevant for this

deployment.

Step 3: Create process in Orchestrator. Select Sample-Chatmode.xaml as entry

point.

Step 4: Run from Assistant

Sample Scenario: Chat With Your Own Data

Sample-ChatMode.xaml

THIS CAN BE DEPLOYED AS IS, OR CUSTOMIZED. MARK THIS AS MAIN OR CHOOSE

THIS AS ENTRY POINT WHEN DEPLOYING (ATTENDED PROCESS). THIS REQUIRES

ROBOT VERSION >= 23.4

Leverage context aware LLM Framework with memory to ingest data source and

answer any questions (in a conversational manner) related or unrelated to the

ingested source.

Related Artifacts (Forms, Workflows, Triggers)

MAIN Framework -> Framework\LLMFramework.xaml

Framework Folder -> Framework\*

Forms & Triggers -> Forms\Forms-Chat\*

End Sample Scenario

IV.2 Transactional

Step 1: In Project Settings , make sure ‘Supports Persistence’ is enabled.

Step 2: Create process in Orchestrator. Select LLMFramework.xaml (Default Main)

as entry point.

Step 3: [IMPORTANT]: Create necessary Queues, Storage Buckets, Queue Trigger,

in Orchestrator as defined in Config.xlsx

Step 4: Run the process as unattended robot in Orchestrator and start consuming

queues.

Sample Scenario: AR PO Extraction & Analytics

Sample-TransactionMode1.xaml

-> Invoice Dispatcher -> Queue -> DU

-> Data Service -> LLM Framework [LLM Interaction (Analyze) > ERP (Act) > LLM

Powered AP Analytics Dashboard (Respond)]

Related Artifacts

MAIN Framework -> Framework\LLMFramework.xaml

Framework Folder -> Framework\*

End Sample Scenario

IV.3 Transactional with human in the loop

Step 1: In Project Settings , same as with Transactional, ‘Supports Persistence’ is

enabled.

Step 2: Also, ‘Wait for Form Task and Resume’ MUST BE ADDED to 1.5 wait for

human action state in the LLMFramework.Xaml (See Appendix: Known

Issues and Limitations for more details on the reason)

Step 2: Publish Project to Orchestrator

Step 3: Create process in Orchestrator. Select LLMFramework.xaml (Default Main)

as entry point.

Step 4 [IMPORTANT]: Create necessary Queues, Storage Buckets, Queue Trigger,

in Orchestrator as defined in Config.xlsx

Step 5: Run the process as unattended robot in Orchestrator and start consuming

queues.

Sample Scenario: Contracts Indemnification Review

Sample-TransactionMode2.xaml

THIS IS A SIMULATION OF A QUEUE BASED TRANSACTIONAL PROCESS.

LLMFRAMEWORK WORKS LIKE REFRAMEWORK AND DEPLOYED AS THE MAIN XAML.

AFTER RUNNING THIS DISPATCHER, RUN OR DEPLOY LLMFRAMEWORK.XAML

THIS DISPATCHER XAML SIMULATES below steps:

> Upload to Storage Bucket

> Add queue item

AFTER SIMULATING DISPATCHER, RUN LLMFRAMEWORK.xaml to consume the

transactions, using the following parameters

> in_TransactionMode = TRUE

> in_UseAgent = FALSE

> in_Query (Prompt Template) = “Identify if the document is a one-sided

indemnification or a mutual indemnification agreement? Provide examples from

the text to justify your answer. If both parties agree to indemnify each other it is

mutual indemnification. If only one party is required to indemnify the other (

example supplier indemnifies company, even if company has to notify the supplier)

it is one-sided indemnification. Provide examples of clauses to justify your answer.”

> in_Source = BLANK. Comes from GetTransaction Data - TransactionItem

Reference
Indemnification in a contract is a contractual clause that shifts liability, costs, and

expenses from one party to another. After the LLM agent is used to identify if an

MSA document has one-sided or two-sided indemnification (with extracted excerpts

from the contract) it will push to human for further review via the custom action

center form

Related Artifacts

MAIN Framework -> Framework\LLMFramework.xaml

Framework Folder -> Framework\*

End Sample Scenario

V. Appendix

V.1 Future Enhancements

➢ UiPath Integration Services Custom Connection for the LLM Framework

APIs/Activities

➢ Enable web page knowledge ingestion and cascading URL web scraping from

base URL.

➢ Prompt Customization/Pre-loaded prompts/prompt store

➢ Include other LLM options LLaMA2

➢ Include support for output parsers: unstructured input -> structured output

V.2 Known Issues and Limitations

As the framework is still in its nascent stage, there are still some known bugs and

limitations that are acknowledged, such as below.

➢ Persistence support needs to be enabled in Studio and the Wait activity needs to

be added when using the HITL flow, otherwise, adding the Wait activity without

Persistence enabled would keep the project from building or enabling

Persistence while using Chat Mode would throw Persistence Bookmarks error

during runtime.

V.3 Additional sources of documentation

Supporting guides and video tutorials

 Additional Process Documentation

Full How to Power up your Automations with https://youtu.be/X0FmJoT8Mms

Guide GenAI using LLM Framework Template in

UiPath

Custom How to Setup Python API Backend for https://youtu.be/xCqTXhPVakY

LLM Framework Custom LangChain
Activity
Activities in UiPath
Setup