EIS-Webinar-Agent-Approaches-2024-08-21.pdf
0 likes•506 views
The webinar discusses agent-based applications of large language models (LLMs) led by industry experts, focusing on practical implementations rather than hype. Key topics include the differentiation between AI agents, assistants, and chatbots, along with governance, orchestration, and workflow optimization. The session also emphasizes the importance of context, transparency, and security in enhancing the efficiency and reliability of AI-driven processes.
EIS-Webinar-Agent-Approaches-2024-08-21.pdf
- 1. www.earley.com WEBINAR WEBINAR Agent Based LLMApplications:Separating the Hype from PracticalApplications SETH EARLEY CEO & FOUNDER EARLEY INFORMATION SCIENCE Media Sponsor
- 2. www.earley.com Today’s Speakers [email protected] https://www.linkedin.com/in/sethearley/ Seth Earley Founder & CEO Earley Information Science Sanjay Mehta Principal Solution Architect Earley Information Science [email protected] https://www.linkedin.com/in/sanjaymehta/ Dominique Legault Founder ReliableGenius Alexander Kline Founding Partner Arcana Concept [email protected] https://www.linkedin.com/in/alexander-kline- futurist-/ [email protected] https://www.linkedin.com/in/dominique- legault-84824199//
- 3. www.earley.com BeforeWe Get Started WE ARE RECORDING SESSION WILL BE 50 MINUTES PLUS 10 MINUTES FOR Q&A YOUR INPUT IS VALUED Link to recording & slides will be sent by email after the webinar Use the Q&A box to submit questions Participate in the polls during the webinar Feedback survey afterward (~1.5 minutes) Thank you to our media partners :VKTR 3
- 4. www.earley.com About Earley Information Science 4 Proven methodologies to organize information and data. SELL MORE PRODUCT SERVICE CUSTOMERS EFFICIENTLY INNOVATE FASTER 1994 YEAR FOUNDED. Boston HEADQUARTERED. 50+ SPECIALISTS & GROWING.
- 5. www.earley.com Agenda 5 Agents,Assistants and Chatbots LLMs and Process Automation The role of Retrieval Augmented Generation (RAG) Task agent –approach to iterative processing of data records LLM Agent Engineering Orchestration agent– “conductor” of agents Contextualization of agents – data retrieval, system, process, function, organization context, user preferences Guidelines and guardrails – traceability, privacy, governance, IP protection (respecting communication and role context) Getting started
- 6. www.earley.com Poll 6 1. Not on the radar 2. Planning stages for Agent based AI 3. Controlled experiments using Agent based AI 4. Agent based AI usage is currently banned 5. Implemented PoC’s (internal or externally facing) 6. Agent based AI applications deployed 7. None of the above Where are you on your Agentic AI journey?
- 7. www.earley.com Agents, Assistants and Chatbots 7 AI agents act autonomously towards solving broad challenges.They can take actions without or with limited human intervention. AI assistants serve a supporting role for specific human needs. Can handle a narrow set of objectives without autonomy (decisions require human approval). Chatbots handle simpler tasks – often information retrieval – in a conversational format.
- 8. www.earley.com LLMs and ProcessAutomation 8 Customer service, content creation, data analysis, personal assistant functions Requirements: • Do you understand the process in detail? (you can’t automate what you don’t understand) • Are the correct data sources available? • Is data of sufficient quality? • Is it structured for retrieval?
- 9. www.earley.com Foundational Elements of an Agentic Solution Capability Tasks & Applications, Definition & Goals (service assistant, sales assistant, operational task assistant...) Governance Rules, Guardrails, Evaluation (security, restrictions, performance...) Platform Infrastructure, AI/LLM Model, Sources & Systems (hosting, components, data, applications, information architecture) Configuration Interface, Prompting, Execution, Memory (human-agent, agent-agent, ux, search, chat, navigation, data, analytics...)
- 10. www.earley.com Agent Template Example parameters metadata System Prompt 01 Tools/Functions 03 Profiles Identity / Security Memory Personalization User 04 Data 02 Information Architecture guardrails Parsing Indexing Retrievers Embeddings Inputs -> Outputs <-
- 11. Copyright © 2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com Complex Analysis/Action Product Support Product Configuration Complex Workflows Domain Complexity Transaction Support Knowledge Retrieval & Action Information/ status inquiries/ order processing Action Complexity 11 Action Complexity versus Domain Complexity Multi AgentWorkflows Knowledge Bots or Configuration Agents Transaction Bots or Agents
- 12. www.earley.com Why LLMAgents? 12 • Improving the functions and capabilities of LLMs • Overcome intrinsic limitations of LLMs around currency of information, hallucinations, weaknesses in non language tasks (mathematics for example) • Enable LLM to call a function that executes an action • Increasing focus on enabling LLMs to learn to call API’s https://arxiv.org/pdf/2302.04761
- 13. Copyright © 2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com 13 Orchestration “conductor” of agents
- 14. Arcana Concept • 45 people from all different backgrounds, skillsets and geographic locations • ideate, discuss world changing inventions, businesses cases, prototype & develop • hardware + software + resources + growth lab : studio
- 15. Acceleration Ideation, Research, Communication, Coordination, Presentation, Preservation
- 16. The Recipe • Expert on Material Science, Lean startup • Personality is a combination of the Merlin Archetype and absent minded professor • Speaks like you would expect a wizard to speak • Moves conversations forward • Tracks concurrent projects • Coordinates with his owl to fetch external information Purpose, Personality, Ability The many iterations of Fizban
- 17. Who is Fizban? • Tell us about his personality • What does he do for you? • What are the challenges and how are they addressed? Open AI GPT 4-Omni Slack API integration Custom Memory System Search
- 18. Orchestration Agent • Hands off tasks to specialized agents • Manages and coordinates responses and outputs • Tracks multiple threads and conversations • Manages short- and long-term memory while respecting privacy and privileged conversations • Provides references and audit trails through Retrieval Augmented Generation Note: AI lies
- 19. www.earley.com 19 Agent LLM Engineering Lessons Learned DevelopingAIT echnology Dominique Legault Aug 21st, 2024
- 20. Finding the balance between context size, cost and efficiency Context Relevance 1. Small Context Size & Low Accuracy (Bottom Left Quadrant) • Insufficient Context: The LLM lacks necessary details, leading to vague or generic responses. The cost is low, but so is accuracy. • Example: A chatbot with minimal context providing incomplete or inaccurate answers. 2 .Small Context Size & High Accuracy (Top Left Quadrant) • Optimized Small Context: With highly relevant but minimal context, the LLM provides accurate responses efficiently. The cost remains low, and accuracy is high. • Example: A well-tuned model with just enough context to deliver precise answers. 3. Large Context Size & Low Accuracy (Bottom Right Quadrant) • Information Overload: The LLM is given too much context, including irrelevant information. Accuracy drops as the model struggles to focus on relevant details. Cost is high, but accuracy is low. • Example: An AI overwhelmed by extensive data, leading to slower and less accurate responses. 4. Large Context Size & High Accuracy (Top Right Quadrant) • Optimally Managed Large Context: The LLM efficiently processes a large amount of relevant information, providing highly accurate results. Cost is high, but so is accuracy. • Example: A system with advanced context management techniques like pruning irrelevant data to maintain accuracy.
- 21. Choosing the right workflow for your problem Workflow Design Single Shot Workflows: • Definition: A single agent handles the entire task from start to finish, suitable for simple tasks. • Advantages: ◦ Simplicity: Easier to design and implement with minimal complexity. ◦ Speed: Quick task completion with no need for agent coordination. ◦ Consistency: More reliable outcomes since one model processes all information. • Use Cases: Ideal for simple queries like answering specific questions or generating short text, and straightforward processes like report generation. Multi Agentic Workflows: • Definition: Multiple agents collaborate, each specializing in different parts of a task. • Advantages: ◦ Specialization: Agents optimize for specific subtasks, increasing efficiency. ◦ Scalability: Handles complex, multi-step tasks more effectively. ◦ Flexibility: Adapts to dynamic tasks, activating different agents as needed. • Use Cases: Best for complex problem solving, like multi-stage research or content creation, and dynamic environments where tasks evolve, such as real-time data analysis. Choosing the Right Workflow: • Task Complexity: Single Shot Workflows are sufficient for simple tasks, while Multi Agentic Workflows excel in complex scenarios. • Resource Allocation: Single Shot Workflows use fewer resources; Multi Agentic Workflows offer greater flexibility and scalability, ideal for complex environments. • Outcome Reliability: Single Shot Workflows are consistent, while Multi Agentic Workflows provide tailored solutions for more dynamic needs. Impact on Business: • Efficiency: Workflow choice affects business efficiency, particularly in automation and customer service. • Scalability: Multi Agentic Workflows are better suited for scaling complex operations.
- 22. Decision Making Autonomy, Objective Alignment, and Impact on Workflow Efficiency Agent Perspective Autonomous Agents: • Definition: Operate independently, making decisions based on pre-defined goals without human intervention. • Characteristics: ◦ Self-Sufficient: Function without continuous human oversight. ◦ Goal-Oriented: Focus on achieving specific outcomes like process optimization. ◦ Adaptable: Learn and adapt to changing conditions. • Use Cases: Process automation, exploration, and research. • Challenges: Trust and control, ethical considerations. Personal Assistants: • Definition: AI agents that assist a specific human, enhancing productivity and task management. • Characteristics: ◦ User-Centric: Tailor actions to individual preferences. ◦ Context-Aware: Use personal data to provide relevant assistance. ◦ Proactive/Reactive: Suggest actions or respond to commands. • Use Cases: Executive support, personalized customer service. • Challenges: Privacy, security, accuracy, and relevance. Comparative Analysis: • Decision-Making Autonomy: ◦ Autonomous Agents: High autonomy, goal-driven. ◦ Personal Assistants: Human-dependent, supports decision-making. • Impact on Workflow: ◦ Autonomous Agents: Automate processes, reduce human intervention. ◦ Personal Assistants: Enhance human tasks, improve productivity.
- 23. Balancing Accessibility with Data Protection Securing AI Agents: Public vs Private Publicly Accessible Agents • Definition: AI agents open to the public via websites, chatbots, or APIs. • Key Traits: ◦ Wide Accessibility: Designed for public interaction. ◦ Limited Data Access: To mitigate risks, they handle non-sensitive information. • Security Concerns: ◦ Data Exposure: Potential leakage of sensitive information. ◦ External Threats: Vulnerable to attacks like phishing. ◦ Compliance Risks: Must adhere to privacy regulations like GDPR, CCPA. • Best Practices: ◦ Information Filtering: Ensure only public data is accessible. ◦ Access Controls: Restrict access to sensitive data. ◦ Continuous Monitoring: Regular checks for security breaches. Internally Accessible Agents • Definition: AI agents used exclusively within an organization. • Key Traits: ◦ Restricted Access: Available only to authorized personnel. ◦ Deep Context Integration: Can access sensitive internal data for precise tasks. • Security Concerns: ◦ Internal Data Handling: Breaches could lead to severe consequences. ◦ Insider Threats: Risks of misuse by internal staff. ◦ Data Retention: Prevent unauthorized data retention. • Best Practices: ◦ Encryption & Logs: Secure data and monitor access. ◦ RBAC: Limit data access based on roles. ◦ Regular Audits: Ensure compliance with security policies. Comparative Analysis: • Risk Exposure: Public agents face external threats, while internal agents are prone to insider risks. • Information Sensitivity: Public agents should limit data access; internal agents need strong protection for sensitive data. • Deployment Strategy: Public agents require external safeguards, while internal agents need a robust internal security framework. Organizational Impact: • Data Governance: Clear policies for handling sensitive data. • Incident Response: Quick action plans for breaches. • Training & Awareness: Regular employee education on AI security.
- 24. Triggering Mechanisms and Action Execution Optimizing LLM Agent Workflows Triggering LLM Workflows: • Manual: User commands initiate specific workflows (e.g., generating reports). • Automated: Event-based triggers like incoming emails or data changes. • Scheduled: Time-based tasks (e.g., daily reports). • Conditional: Rule-based actions triggered by predefined conditions. Executing Actions: • Text Generation: Creating content like emails, reports, or posts. • Data Processing: Analyzing data to provide insights and recommendations. • Task Automation: Executing commands, updating systems, or sending emails. • Workflow Orchestration: Performing complex, multi-step processes. Business Implications: • Efficiency: Reduces manual tasks and boosts productivity. • Scalability: Handles increased workloads effortlessly. • Customization: Tailors workflows to business needs. • Integration: Aligns with existing systems for seamless operations.
- 25. The role of Transparency and Debugging in AI-Driven Processes Enhancing Trust and Efficiency in LLM Workflows` 1. Transparency in Workflows • Building Trust: Transparency fosters user trust by clarifying LLM decision-making. • Accountability: Transparent processes ensure errors and biases are addressed. • Credibility: Referencing sources builds credibility and informed decision-making. • Explainability: Clear reasoning enhances understanding in complex scenarios. 2. Debugging Multi-Agent Systems • Challenges: Multi-agent systems can be complex, leading to issues like miscommunication. • Tool Frameworks: Tools like LangSmith offer workflow visualization and step-by- step debugging. • Example: Tracing incomplete responses in customer inquiries. 3. Gaining Insight into LLM Behavior • Performance Metrics: Monitor response time, accuracy, and relevance for fine- tuning. • Behavioral Analysis: Debugging tools reveal how LLMs interpret inputs and outputs. • Continuous Improvement: Feedback loops and regular analysis enhance LLM alignment with user needs. 4. Business Impact • Reliability & Efficiency: Transparent, well-debugged workflows reduce errors and improve system performance. • User Confidence: Regular monitoring and debugging increase user trust and adoption of AI solutions.
- 27. Copyright ©2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com www.earley.com Generation vs Retrieval 27 User query Process query using LLM to understand user intent Generate Retrieve Process response using LLM to provide conversational format LLMs used to process query and present results Response Proprietary information - No exposure of IP - Corporate knowledge - Auditability Publicly available information - Prone to hallucination - Exposure of IP - Missing audit trail
- 28. Copyright ©2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com www.earley.com 28 Source: https://arxiv.org/abs/2312.10997 Retrieval Augmented Generation has Evolved
- 29. © 2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com Source Data to be Enriched Original Product Name Original Description Attribute Names Original Catalog Context SCHEMA/ MDM/PIM/ERP Related Artifacts Web Pages/Content Digital Assets (images, diagrams) Existing Search Index Product Sheets, Manuals Knowledge Base Target Audiences / Segments Feed, API, Crawl, Direct Merge by key – series, category or attribute code Unified Document Repository Rules & Governance Standards Bodies (ISO, TC, SC) Restrictions Style Guide Branding Examples (Positive & Negative) Existing Search Rules 3rd Party Intelligence Google Knowledge API & Graph Competitor Websites & Search Results Large Language Models EIS MRO Knowledge Graph User context Signals / Telemetry Search Analytics Clickstream Analytics Transactional Analytics Performance Metrics KPI's Audience / Segment / Profile Prepare/process: normalize, classify, tag Generate Doc Embeddings Create Graph Index Generate Contextual Embeddings (Application, Audience, Behavioral) Generate Query / Phrase / Q&A Embeddings Generate Industry & Organization Specific Ontological Embeddings Create Vector Index Generate LLM Context and Metadata
- 30. Copyright ©2024 Earley Information Science, Inc. All Rights Reserved. Sample Output Data 30 Category Code: "E3107000000“ Category Name: "Jumper Bars“ Brand Code: “ACM“ Brand Name: “ACME“ Series Code: 110400166560 Series Name Original: "Dedicated Short Bar“ New Name: “ACME Shorting Bar for Electrical Terminal Blocks“ New Description: "A dedicated shorting bar designed for use with electrical terminal blocks to create a secure electrical connection and prevent short circuits. Suitable for automotive, manufacturing, and automation applications.“ Keywords: "shorting bar", "electrical terminal blocks", "short circuit prevention", "automotive", "manufacturing", "automation“ Relevant Categories: "Electrical Components", "Terminal Blocks“ Reasoning: "The new name and description provide a clear understanding of the product's purpose and applications, while the keywords and categories ensure it can be easily found by the target audience.” Original New • New names, descriptions and normalized attribute values will draw from both internal (ACME) sources and publicly available data • Output results will conform to ACME style guidelines and industry standards • Enriched SEO information & metadata (e.g. reasoning) will be provided all output records
- 31. www.earley.com 31 Guidelines and Guardrails
- 32. © 2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com Guidelines and Guardrails 32 How do you define “reasoning ability”? What is control flow and how is it managed? Which agent reports to other agents? Is there a hierarchy? How are agents making “decisions” and how are they being enabled? What are the restrictions on what agents in the wild can do? What are the governance controls and considerations? How do enterprises need to think about agent automations and restrictions?
- 33. www.earley.com Poll 33 1. They are not aware of the benefit/risk profile 2. They are aware but do not have the appetite for agent- based approaches 3. They are aware and have appetite for agent based approaches 4. The organization is already pursuing these approaches 5. Something else (let us know in comments) What is the stance of your leadership in terms of agent based LLM approaches?
- 34. www.earley.com Getting Started 34 Consider high value processes that typically require human intervention Map end to end data flows Establish baselines for processes to measure impact of remediation Consider guardrails for agent functionality
- 35. Copyright ©2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com www.earley.com Requirements Discovery Session for Agent Based Approaches The AI Agent Requirements sprint will quickly identify how the organization can deploy LLM Agents to address issues and challenges that arise from LLM deployments that seek to improve efficiencies and effectiveness of organizational processes. The sprint consists of the following: • Stakeholder interviews • Education and alignment • Current state maturity • Review of systems and tools • LLM Agent Proof of Value (PoV) plan
- 36. Copyright ©2023 Earley Information Science, Inc. All Rights Reserved. AI Agent Working Session Topics Topic Overview Goals Questions to Address Key Concepts and Success Criteria AI and LLM deployment considerations including current state processes and technology ecosystem • Level set on key terms and foundational understanding. • How can the organization make use of AI Agents? • What is the role of LLMs and automation? Target Processes Overview of business objectives and use cases for LLM Agents • Orient client team to multiple agentic approaches. • Get initial response on which are most important. • Which business priorities must the AI serve first? • Which teams will be most involved? Current Landscape Definition Map processes, knowledge, data and content by people and systems to identify how business processes are currently supported. • Develop more complete view of scope and scale of systems. • Map target process • Develop domain model • Where are key automation leverage points/processes? • Where are key areas of changes re: people and process? PoV Planning Plan and proposal for LLM Agent Proof of Value. • Plan to implement AI Agent powered capability • What is the level of effort needed for infrastructure and execution?
- 37. Copyright ©2024 Earley Information Science, Inc. All Rights Reserved. www.earley.com www.earley.com www.earley.com The EIS/Arcana AI Agent Working Session Can Help Get You There Through a combination of interviews, questionnaires, surveys and working sessions, the EIS/Arcana AI Agent Assessment: 1. Educates executives and stakeholders about AI technologies – capabilities and limitations 2. Evaluates business value and target use cases for Agentic Approaches 3. Outlines success factors and metrics 4. Examines critical areas of the enterprise for AI Agent readiness: • Business alignment and process clarity • Knowledge and Data readiness and technology infrastructure • Ongoing governance, decision making and success measures 5. Summarizes the current state in an executive working session designed to identify gaps, set realistic goals and prioritize actions for an AI Agent Proof of Value (PoV) 37 The output of the AI Agent Assessment is a roadmap for deployment of an AI Agent PoV based on a target process for the organization https://www.earley.com/agent-workflow-requirements-working-session
- 39. www.earley.com Contact [email protected] https://www.linkedin.com/in/sethearley/ Seth Earley Founder & CEO Earley Information Science Sanjay Mehta Principal Solution Architect Earley Information Science [email protected] https://www.linkedin.com/in/sanjaymehta/ Dominique Legault Founder ReliableGenius Alexander Kline Founding Partner Arcana Concept [email protected] https://www.linkedin.com/in/alexander-kline- futurist-/ [email protected] https://www.linkedin.com/in/dominique- legault-84824199//
- 40. www.earley.com 40 We Make Information More Useable, Findable, And Valuable Earley Information Science is a professional services firm headquartered in Boston and founded in 1994. With over 50+ specialists and growing, Earley focuses on architecting and organizing data – making it more findable, usable, and valuable. Our proven methodologies are designed to address product data, content assets, customer data, and corporate knowledge bases. We deliver scalable solutions to the world’s leading brands, driving measurable business results.