apidays Paris 2024 - AI-Enhanced API Documentation Bridging Knowledge Gaps and Elevating Developer Experience - Saheed Abiola Lasisi, JPMorgan Chase & Co

Saheed LASISI , JPMorgan Chase & Co
API Days Paris | December 2024
Bridging Knowledge gaps and
Elevating Developer
Experience

Agenda
1. Introduction
2. Challenges in API Documentation
3. Generative AI: A Game Changer
4. Capabilities of AI in API Documentation
5. Real world AI challenges
6. Addressing Challenges and Governance
7. Future Vision and Key Takeaways
8. Q &A

Introduction
❑ Over 50% of buy decisions are
influenced by developers
❑ Developers are decision makers in
companies of all sizes
- DevRelx1
Technical Documentation is key in getting in front of Decision makers
https://www.devrelx.com/post/how-education-helps-developers-reach-purchasing-decisions-and-product-adoption
of-b2b-sales-interactions-between-su

The opportunity
❑ Nearly 40% of developers report
educational resources as top
expectation from vendors1
❑ “80% of B2B sales interactions will
occur in Digital channels by 2025” –
Gartner2
https://www.devrelx.com/post/how-education-helps-developers-reach-purchasing-decisions-and-product-adoption
2 – Source: https://www.gartner.com/en/newsroom/press-releases/2020-09-15-gartner-says-80--of-b2b-sales-interactions-
between-su
❑ When it comes to choosing a public
API, good API documentation
outranked performance or
security3

API Documentation Challenges
❑ Inconsistencies
❑ Inaccuracies
❑ Complexity
❑ Outdated Information
4 – Source https://www.postman.com/state-of-api/2023/executing-on-apis/#executing-on-apis/
❑ 80% of developers find poor documentation to be
their top pain point consuming APIs.4

API
Documentation
Expertise
Navigating documentation challenges

Trends in API Documentation
Interactive
Documentatio
n
Real-time
Collaboration
Focus on
User
Experience
Localization and
Globalization
Doc As Code
Gen AI and
Automation

Generative AI: the Game Changer
Intelligence
Machine
Learning
Deep
Learning
Generative AI
(GenAI)
e.g., LLM chatbot to
generate creative
content, create
images, or summarize
text
Artificial
Intelligence
(AI)
Broad term for the field that enables machines
to mimic human intelligence, solving problems
and performing tasks.
Machine
Learning (ML)
Subset of AI, this involves computers learning
from data to make decisions or predictions
without being explicitly programmed.
Deep Learning
Branch of machine learning that uses multi-
layered (“deep”) artificial neural networks to
handle large data sets and perform complex
tasks.
Generative AI
(GenAI)
Type of deep learning that focuses on creating
new, realistic outputs from unstructured inputs
like text, images, or audio.
Large Language
Models (LLM)
Model trained on vast amounts of text data to
understand, generate, and predict text with
human-like proficiency across various contexts
and tasks.
Retrieval
Augmented
Generation
(RAG)
A technique that enhances the accuracy and
relevance of the model's responses by
integrating external knowledge through
information retrieval systems .

Localization and Globalization: GenAI can
automatically translate API documentation
into multiple languages, making APIs
accessible to a wider audience of
developers.
Unlocking new possibilities:
❑ API spec, test and code generation
❑ Interactive API Mocks
❑ Enhanced AI search for API
discoverability
Enhanced Adoption:
❑ Faster onboarding
❑ Simplified explanations and adoption for
bridging stakeholders
❑ Multilingual support
Enhancing Consistency and Accuracy:
❑ GenAI leverages templates, style-
guides, blueprints, data dictionaries,
glossaries and playbooks to enforce
governance and generate quality
content for cross-platform adaptation.
Improving Accessibility and Engagement:
❑ Enabling Natural language search
❑ Intelligent examples and contextual code
snippets
Automating the grunt work: dynamic
content generation from API artifacts:
specs, code snippets, requirements,
collections, tests etc
Enhancing Developer experience:
❑ Personalized documentation experience
based on profile and usage context.
❑ Interactive tutorials and examples
Interactive guidance and chat assistance
❑ Troubleshooting and debugging
Extending Scale: GenAI can help organize
thoughts and provide basic drafts or
comprehensive outlines . It can automate
content creation, editing, rewriting and
content organization necessary of new
projects.
GenAI is
revolutionizing API
Documentation
1
5
2
4
8
6
7 3
IMPORTANT! Content produced by GenAI must be reviewed, validated and fact checked
against credible sources by a human in the loop before publication or dissemination.
Capabilities of GenAI in API Documentation

Real world Use-cases
❏ Plaid embeds AI Robot into its API documentation: “Ask
Bill”1.
❏ Stripe, known for its excellent developer experience:
❏ Brings AI enabled Stripe docs to VSCode IDE
through Github copilot extension2
❏ Stripe leverages GPT4 and Stripe docs for
enhanced developer support 3.
Result: Reduction in support effort.
1: https://plaid.com/docs/support/?showChat=true
2: https://insiders.stripe.dev/t/join-the-vs-code-extension-github-copilot-preview/107
3: https://openai.com/index/stripe/

Challenges in AI enabled API
documentation
● Maintaining Accuracy, Consistency, and Style.
● Risks of Hallucinations and bias in AI-Generated Content.
● Risk of Prompt injections and brute-force attacks
● Compliance risks
“These challenges are not unique to API
Documentation.”

Real world GenAI
Challenges
Generative AI is no longer a futuristic
fantasy. Everyone is racing to integrate,
but are they overlooking critical roadblocks
?
(Source: Writer.com, State of AI Report 2024)
➔ Security, Data Protection and compliance are top pain
points in AI integrations. - (Postman, State of API Report
2024 and Writer.com State of AI Report 2024)

AI
RISKS
Compliance
Bias
Data
Privacy
Accuracy Copyright
Ethics
Building Trust in the
age of GenAI
❏ To truly unlock the potential of AI, we need to
build systems that are trustworthy, reliable,
and ethical.
❏ "88% of companies estimate it will take at
least 6 months to have a reliable in-house AI
solution." (Source: State of AI Report 2004,
Writer)
This highlights the need for careful
planning, investment, and a long-
term perspective on AI adoption.

○ "Data controls and access leads security
controls needed for generative AI "
○ "61% of companies have already experienced
accuracy issues with generative AI." - (Source:
Writer.com, State of AI Report 2024)
Start with Data Risks and
governance
This underscores the critical need for
robust data strategies to manage
data access, control, and
governance.
(Source: Writer.com, State of AI Report 2024)

❏ Unreliable outputs
❏ AI hallucinations and confabulations
❏ Weak Operational integrity of AI driven
processes
Impact:
❏ Misinformed or flawed decisions
❏ Misleading, false and harmful information
❏ Erosion of Trust and acceptance
Mitigation Strategies:
❏ Regularly update training data to improve accuracy.
❏ Implement feedback mechanisms for ongoing model
improvement.
❏ Implement validation mechanisms (e.g., fact-checking)
to detect and prevent AI hallucinations and factual
grounding.
❏ Incorporate human oversight and intervention in
critical decision-making processes.
❏ Continuously train AI models on real-world data to
reduce hallucination tendencies (aka self-evolving
models).
❏ Use techniques like retrieval-augmented generation
(RAG) (contextual and knowledge graphs) to improve
generated content accuracy and relevance.
❏ Continuously score trusted, untrusted and new
retrieval sources to establish varying degree of trust.
❏ Continuously certify training data sets and knowledge
sources for augmented retrieval.
❏ Implement explainable AI (XAI) methods such as
LIME(Local Interpretable Model-Agnostic
Explanations), SHAP(SHapley Additive exPlanations), or
counterfactual explanations to provide insights into
model reasoning.
❏ Use a combination of explainability strategies taking
into account context, domain specific knowledge and
invariants.
Tackle Accuracy and Explainability Risks

Transform your lifecycle with DDD

Address Adoption Pitfalls - 1
Stay up-to-date with RAG
Retrieval-Augmented Generation (RAG)1 is an advanced
technique in natural language processing that enhances
the capabilities of LLMs by integrating external
information retrieval mechanisms to augment the
generation process.
❏ This approach enables LLMs to access up-to-date
and domain-specific data.
❏ Contextual RAGs, Semantic chunking, high fidelity
agentic retrieval and Knowledge graphs aims to
solve semantics and context problems even
further.
Research shows higher relevance, accuracy
and lower hallucinations - (RobustQA Benchmark, ACL 2023,
https://aclanthology.org/2023.findings-acl.263/)
1: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401

Confidential Copyright ©
(Source: Weaviate - https://weaviate.io/blog/what-is-agentic-rag)
❏ RAG is a pattern that can
improve the efficacy of LLM
applications by using your
own custom data.
❏ It can facilitate the
development of high-quality
articles, documentation,
reports, chatbots, search
engines and summaries using
relevant context.
Contd ..

01
02
Mitigate Data Privacy Risk
Potential Risks:Unauthorized access
Impact: violation of privacy regulations
Mitigation Strategies: Explicit user consent for data usage,
access controls, encryption, anonymization, audits and
compliance with privacy regulations, input sanitization,
zero-trust architecture
Mitigate Bias and Fairness
Potential Risks: Biased training datasets
Impact: discriminatory outcomes
Mitigation Strategies: Use diverse dataset for training,
data pre-processing, conduct fairness assessments, data
annotations and labelling, establish a feedback loop,
update models regularly, incorporate automatic fairness
verification and fairness awareness learning in model
training.

20
03
04
Mitigate Prompt Injection and Security attacks
Potential Risks: Unauthorized access
Impact: misinformation, privacy breaches and system
compromises.
Mitigation Strategies: input sanitization, rate limiting, RBAC, code
review and static analysis, encryption, secure storage and zero
trust architecture.
Maintanability
Potential Risks: Difficulties in maintaining, updating, and
refactoring designs
Impact: Technical debt and misinformation arising from poor and
outdated designs
❏ Adopt standardized diagramming frameworks ("diagram as
code") such as Mermaid, PlantUML, and Architecture as
Code tools.
❏ Enforce version control for Documentation diagrams just
like code.

21
05
Compliance
Potential Risks: Non-compliance with legal and industry
standards
Impact: Legal repercussions and penalties
❏ Shift-left design approach to address compliance risks
with a compliance styleguide.
❏ Implement automated compliance guardrails based on a
compliance styleguide as part of every genAI app stack.
❏ Establish clear compliance monitoring and reporting
procedures.
❏ Stay informed about the latest laws, regulations, and
industry standards.
❏ Establish patterns and mechanisms to mitigate compliance
risks.
❏ Collaborate early with compliance, legal and regulatory
experts to ensure compliance is incorporated early in the
development lifecycle.

Envisioning the future
❏ Reduced Developer Friction: fewer support
tickets .
❏ Faster Onboarding: Developers onboard nX
faster.
❏ Improved Satisfaction: Higher adoption rates
and enhanced feedback.
❏ APIs that are Self-Documenting and
Developer-Centric.
❏ Context-aware Dynamic Content Adaptation
Based on Developer Interactions.

Wrapping up
❏ The Age of AI Enablement: powering API documentation to deliver
dynamic and intelligent experiences that adapt to developers'
needs.
❏ Elevate the developer experience: Combine AI with strong
governance, patterns and standards to drive API adoption and
client satisfaction.
❏ Shift-left Governance: Shift-left to design-time, risk, compliance,
and conformance to establish a robust governance framework to
unlock GenAI's full potential.
❏ Knowledge is power: Continuously or retrospectively build
certified knowledge bases to fuel AI accuracy and explainability.
❏ Don't wait, innovate: Start small, iterate, learn, refine and evolve.

Questions
Connect: https://www.linkedin.com/in/saheed-abiola-lasisi-4b62891b/

APPENDIX

GenAI Risk Mitigation Strategies -1
Risk Category Potential Risks IMPACT Mitigation Strategies
Data Privacy Unauthorized data access Violation of privacy regulations
Implement robust access controls, data encryption, and
anonymization techniques.
Regularly audit and monitor data access.
Acquire explicit user consent for data usage.
Adhere to relevant data privacy regulations (e.g., GDPR,
CCPA).
Educate employees on data privacy best practices.
Bias and
Fairness
Biased training datasets Discriminatory outcomes Utilize diverse and representative data for AI model
training.
Regularly audit, annotate dataset and mitigate
potential biases in AI models
Conduct fairness assessments to identify and address
biases.
Include diverse stakeholders in the AI model
development lifecycle and a feedback loop.
Maintain transparency in the AI decision-making
process.
Continuously monitor and update AI models to address
emerging biases.

GenAI Risk Mitigation Strategies - 2
Compliance Non-compliance with legal and
industry standards
Legal repercussions and
penalties
Stay informed about the latest
laws, regulations, and industry
standards.
Establish clear compliance
monitoring and reporting
procedures.
Conduct regular compliance
audits to identify gaps.
Provide comprehensive training
to employees on compliance
requirements.
Collaborate with legal and
regulatory experts to ensure
compliance.
Shift-left design approach for
compliance mitigation.
Develop mechanisms for
addressing and reporting non-
compliance.

Ethical
Considerations
Unforeseen harmful outcomes Negative societal effects
Develop and implement clear ethical guidelines for
AI development and deployment.
Conduct ethical impact assessments to evaluate
potential societal implications.
Actively engage with relevant stakeholders to
discuss ethical concerns.
Promote transparency and openness in AI
development and usage.
Regularly review and update ethical guidelines to
adapt to evolving AI capabilities.
Cultivate a culture of ethical AI use within the
organization.
Establish channels for reporting ethical concerns
related to AI.
Regularly educate employees on ethical
considerations in AI development and use.
Encourage interdisciplinary collaboration to
address ethical challenges.

Accuracy Inaccurate or unreliable outputs Misinformed or flawed decisions Regularly update training data to improve
accuracy.
Conduct rigorous testing and validation of AI
models.
Implement feedback mechanisms for ongoing
model improvement.
Monitor and address biases in training and
evaluation data.
Provide clear explanations and maintain
transparency in AI predictions.
AI hallucinations or confabulations Misleading, false or harmful information
Implement validation mechanisms (e.g., fact-
checking) to detect and prevent AI hallucinations.
Regularly review and update language models to
minimize hallucination risks.
Incorporate human oversight and intervention in
critical decision-making processes.
Continuously train AI models on real-world data to
reduce hallucination tendencies.
Use techniques like retrieval-augmented
generation (RAG) to improve generated content
accuracy and relevance.
Continuously certify training data sets and
knowledge sources for augmented retrieval.

Copyright Infringement of copyrighted materials Legal repercussions and penalties Obtain necessary licenses and
permissions for using copyrighted
materials.
Implement content filtering and
copyright protection measures.
Maintainability
and Scalability
Difficulties in maintaining, updating, and
refactoring designs (workflows,
flowcharts, UML, sequence diagrams,
architecture diagrams, etc.)
Technical debt arising from poor
and outdated designs
Adopt standardized diagramming
frameworks ("diagram as code")
such as Mermaid, PlantUML, and
Architecture as Code tools.
Enforce version control for
diagrams, just like code.

Explainability Lack of understanding of how AI models
arrive at decisions
Erosion of Trust and acceptance: If users
and stakeholders don't understand the
reasoning behind AI-driven decisions, they
may be hesitant to adopt and utilize these
systems, especially in high-stakes
applications
Implement explainable AI (XAI) methods such
as LIME(Local Interpretable Model-Agnostic
Explanations), SHAP(SHapley Additive
exPlanations), or counterfactual
explanations to provide insights into model
reasoning.
Develop techniques to visualize and
interpret model behavior, such as attention
maps or decision trees.
Document model architecture, training data,
and decision-making processes to enhance
transparency.
Use a combination of explainability
strategies taking into account context,
domain specific knowledge and constraints.
Verification Difficulty in verifying the correctness and
reliability of AI models
Misinformation and safety risks Establish robust testing and validation
procedures for AI models, including unit
tests, integration tests, and A/B testing.
Erosion of Trust Use diverse datasets and scenarios for
comprehensive testing, including edge cases
and adversarial examples.
Reputational damage Incorporate human review and feedback in the
verification process to identify potential biases or
errors.

Jurisdiction and
Regulatory Risks
Varying legal and regulatory requirements
across different jurisdictions
Operational Complexities in
managing compliance
Conduct thorough research and
analysis of the legal and regulatory
landscape in each relevant
jurisdiction.
Uncertainty regarding the applicability of
existing laws and regulations to AI Market entry barriers
Engage with legal experts and
regulatory bodies to ensure
compliance with current and
evolving requirements.
Potential conflicts between different legal
frameworks
Increased legal and operational
cost
Develop a comprehensive
regulatory strategy that addresses
the specific requirements of each
jurisdiction.

apidays Paris 2024 - AI-Enhanced API Documentation Bridging Knowledge Gaps and Elevating Developer Experience - Saheed Abiola Lasisi, JPMorgan Chase & Co

Recommended

More Related Content

Similar to apidays Paris 2024 - AI-Enhanced API Documentation Bridging Knowledge Gaps and Elevating Developer Experience - Saheed Abiola Lasisi, JPMorgan Chase & Co (20)

More from apidays (20)

Recently uploaded (20)

apidays Paris 2024 - AI-Enhanced API Documentation Bridging Knowledge Gaps and Elevating Developer Experience - Saheed Abiola Lasisi, JPMorgan Chase & Co