SlideShare a Scribd company logo
DevFest Croatia, October 2024
Function Calling
with the Vertex AI Gemini API
Márton Kodok
Software Architect at REEA.net
1. Intro
2. Gemini disambiguation
3. What is Vertex AI?
4. Function calling with Gemini
5. Demo
Agenda
Function Calling with the Vertex AI Gemini API @martonkodok
● Google Developer Expert on Cloud technologies (2016→)
● Champion of Google Cloud Innovators program (2021→)
● Among the Top 3 romanians on Stackoverflow 207k reputation
● Crafting Cloud Architecture+ML backends at REEA.net
Articles: martonkodok.medium.com
Twitter: @martonkodok
Slideshare:martonkodok
StackOverflow: pentium10
GitHub: pentium10
Function Calling with the Vertex AI Gemini API @martonkodok
About me
@martonkodok
Gemini
Part #1
@martonkodok
Gemini comes in different sizes
Gemini 1.5 Flash
fastest, most cost-efficient
model yet for high volume tasks
GA
Gemini 1.5 Pro
Multimodal reasoning for longer
prompts, 1 million context
window
GA
1M
high volume 2000 (RPM),
lower latency and cost
2M
Large prompt,
1000+ pages of PDF
* October 2024 @martonkodok
A bit about pricing
For 1M tokens Gemini 1.5 Flash Gemini 1.5
Pro
C3.5 Haiku GPT-4o GPT-4o-mini
Input Token $0.075 - $0.15 $1.25 - $2.50 $0.25 $2.50 $0.15
Output Token $0.30 - $0.60 $5.00 - $10.00 $1.25 $10.00 $0.60
1 million tokens (for prompts up to 128K tokens — for prompts longer than 128K)
Function Calling with the Vertex AI Gemini API @martonkodok
Gemini everywhere
1. Gemini consumer app
2. Gemini in the AI Studio (API key)
3. Gemini in Vertex AI Studio (Google Cloud - service account key file)
4. Gemini Code Assist (your IDE, VsCode, JetBrains …)
5. Gemini for Google Workspace (for teams by your company)
@martonkodok
Where to use
Build with Gemini
Google AI Studio
aistudio.google.com
- API key
- 2-15 QPM
- Libraries in Python/Go/Node/Dart/Android
console.cloud.google.com
- service account key
- Vertex AI SDK
- LangChain integration + libraries
- access to 120 models via Model Garden
- available for individual/edu/business
Vertex AI Studio
Vertex AI
+
Model Garden
Part #2
“VertexAI is the Google Cloud product group for ML
End-to-end ML Platform
@martonkodok
Function Calling with the Vertex AI Gemini API @martonkodok
Model Garden
Task Specific
AutoML and APIs
Open Source
Models
Foundation
Models
Model Garden
Function Calling with the Vertex AI Gemini API @martonkodok
Models
Gemini 1.5 Flash
fastest, most cost-efficient
model yet for high volume tasks
Gemini 1.5 Pro
Multimodal reasoning for longer
prompts, 1 million context
window
Imagen 3
Generate images from
Text prompts
Multimodal
Embeddings
Extract semantic information
Chirp for
Speech to Text
Build voice enabled applications
Function Calling with the Vertex AI Gemini API @martonkodok
Vertex AI Summer 2024 Launches
End-to-end model building
platform with choice at every
level
Develop and deploy faster,
grounded in your domain
Build on a foundation of scale &
enterprise readiness
Context Caching
Gemini 1.5
Grounding on Google
Search
Function Calling
“ demo…
@martonkodok
Part #3
Gemini
Function Calling
Function Calling with the Vertex AI Gemini API @martonkodok
Function calling in Gemini
A framework to connect LLMs
to real-time data
delegate certain data processing tasks
to functions
Function Calling with the Vertex AI Gemini API @martonkodok
API parameters External API
Processing
Model Output
Function Calls Function Responses
Data
User
Prompt
Generating
Function calling in Gemini
Database
Function Calling with the Vertex AI Gemini API @martonkodok
Elements of a Function call
1. Function Declaration (name, description, parameters, required)
2. Tools (group functions into tools)
3. Attach (tools to Gemini initialization)
4. Response Parts (parts that are returned to Gemini for response generation)
Function Calling with the Vertex AI Gemini API @martonkodok
Function Declaration in Gemini tools
list_datasets_func = FunctionDeclaration(
name="list_datasets",
description="Get a list of datasets that will help answer the user's question"
)
list_tables_func = FunctionDeclaration(
name="list_tables",
description="List tables in a dataset that will help answer the user's question",
parameters={
type": "object",
"properties": {
"dataset_id": { type": "string",
"description": "Dataset ID to fetch tables from.",
} },
"required": [
"dataset_id",
],
},
)
Function Calling with the Vertex AI Gemini API @martonkodok
Gemini Tools
sql_query_tool = Tool(
function_declarations=[
list_datasets_func,
list_tables_func,
…
],
)
model = GenerativeModel(
"gemini-1.5-pro",
generation_config={"temperature": 0},
safety_settings={
HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
…
},
tools=[sql_query_tool],
)
Function Calling with the Vertex AI Gemini API @martonkodok
Gemini Fetch Functions
def extract_function_calls(response: GenerationResponse) -> list[dict]:
function_calls: list[dict] = []
if response.candidates[0].function_calls:
for function_call in response.candidates[0].function_calls:
function_call_dict: dict[str, dict[str, Any]] = {function_call.name: {}}
for key, value in function_call.args.items():
function_call_dict[function_call.name][key] = value
function_calls.append(function_call_dict)
return function_calls
Prompt: What datasets exists in our database?
Response: [{'list_datasets': {}}]
function_call {
name: "list_datasets"
args {
}
}
Function Calling
Demo
Part #4
Function Calling with the Vertex AI Gemini API @martonkodok
API parameters External API
Processing
Model Output
Function Calls Function Responses
Data
User
Prompt
Generating
Function calling in Gemini
Database
Function Calling with the Vertex AI Gemini API @martonkodok
Gemini Function calling also supports
1. Parallelfunction processing
2. Nestedparameters and complex schemas
3. Function calling modes:
a. AUTO, The model decides whether to predict a function call or a natural language response
b. ANYmode forces the model to predict a function call
c. NONEmode instructs the model to not predict function calls
Function Calling with the Vertex AI Gemini API @martonkodok
Gemini Function calling benefits
1. Ability to feed in real time results from own app/database
2. Adheres output to a specific schema - receiving consistently formatted responses.
3. Ability to work on multimodal inputs
4. Agent and workflow processing
5. Reasoning engine, and orchestrator jobs STEPS to simulate compile/run of output
Part #5
Takeaways
Gen AI … repository
Function Calling with the Vertex AI Gemini API @martonkodok
@martonkodok
Code repository GenAI for Google Cloud
goo.gle/gen-ai-github
@martonkodok
Sample notebooks and apps
Description Sample Name
Intro to Function Calling in Gemini intro_function_calling.ipynb
Working with Parallel Function Calls and Multiple Function Responses in
Gemini
parallel_function_calling.ipynb
Working with Data Structures and Schemas in Gemini Function Calling function_calling_data_structures.ipynb
Forced Function Calling with Tool Configurations in Gemini forced_function_calling.ipynb
SQL Talk: Natural Language to BigQuery with Gemini's Function Calling sql-talk-app
Using Gemini Function Calling to get Real-Time Company News and Insights use_case_company_news_and_insights.ipy
nb
Intro to ReAct Agents with Gemini & Function Calling intro_diy_react_agent.ipynb
Conclusion
Function Calling with the Vertex AI Gemini API @martonkodok
1. Build with the groundbreaking ML tools that power Gemini
2. Accelerate ML with tooling for fetching business data via function calls
3. End-to-end integration for data and AI with agents that outperform and solve complex ML tasks
4. Ability to combine multimodal inputs and complex schemas to maintain output format
Gemini Function calling: Enhanced ML developer experience
Function Calling with the Vertex AI Gemini API @martonkodok
“ But there is more…
Function Calling with the Vertex AI Gemini API @martonkodok
“
@martonkodok
Article about Imagen 3
Vector Search and
multimodal embeddings
Remote Functions in BQ
size color
living
Linkedin: @martonkodok
Thank you. Q&A.
Reea.net - Integrated web solutions driven by creativity
to deliver projects.
Follow for articles:
martonkodok.medium.com
Slides available on:
slideshare.net/martonkodok

More Related Content

Similar to Function Calling with the Vertex AI Gemini API (20)

PPTX
Build with AI Event master deck final final
DiyaVerma14
 
PDF
[SHARED ONLINE] - GDG Cloud Iasi - Build with AI - Sponsored by Qodea.pdf
athlonica
 
PDF
AI Agents with Gemini 2.0 - Beyond the Chatbot
Márton Kodok
 
PDF
[SHARED ONLINE] - GDG Cloud Iasi - Make it happen with Google AI - Sponsored ...
athlonica
 
PDF
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
PPTX
Intro To Gemini API - Build with AI.pptx
NitinChauhan439626
 
PDF
[BuildWithAI] Introduction to Gemini.pdf
Sandro Moreira
 
PPTX
Certification Study Group - NLP & Recommendation Systems on GCP Session 5
gdgsurrey
 
PPTX
Building .NET AI Applications with Google AI: Leveraging Vertex AI and Gemini
Udaiappa Ramachandran
 
PDF
Build with AI on Google Cloud Session #3
Margaret Maynard-Reid
 
PDF
Build with AI on Google Cloud Session #4
Margaret Maynard-Reid
 
PPTX
GDG MIT Generative AI : Getting started with the study jam
mitgdsc
 
PDF
DevFest SG 2024 - What’s new in On-device Generative AI
Hassan Abid
 
PDF
AI Intelligence: Exploring the Future of Artificial Intelligence
sayalikerimova20
 
PDF
Building the Next Generation of Android Apps with AI.pdf
Veronica Anggraini
 
PPTX
geminipro (google developer student clubs Haldia Institute of Technology 2023...
XAnLiFE
 
PPTX
DeepSeek vs ChatGPT vs Gemini Which AI Best Fits Your Needs.
Davis Smith
 
PDF
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Márton Kodok
 
PDF
AI Deep Dive_ A Journey through Heroku_OpenAI Integration.pdf
mattmeyers15
 
PDF
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
James Anderson
 
Build with AI Event master deck final final
DiyaVerma14
 
[SHARED ONLINE] - GDG Cloud Iasi - Build with AI - Sponsored by Qodea.pdf
athlonica
 
AI Agents with Gemini 2.0 - Beyond the Chatbot
Márton Kodok
 
[SHARED ONLINE] - GDG Cloud Iasi - Make it happen with Google AI - Sponsored ...
athlonica
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Intro To Gemini API - Build with AI.pptx
NitinChauhan439626
 
[BuildWithAI] Introduction to Gemini.pdf
Sandro Moreira
 
Certification Study Group - NLP & Recommendation Systems on GCP Session 5
gdgsurrey
 
Building .NET AI Applications with Google AI: Leveraging Vertex AI and Gemini
Udaiappa Ramachandran
 
Build with AI on Google Cloud Session #3
Margaret Maynard-Reid
 
Build with AI on Google Cloud Session #4
Margaret Maynard-Reid
 
GDG MIT Generative AI : Getting started with the study jam
mitgdsc
 
DevFest SG 2024 - What’s new in On-device Generative AI
Hassan Abid
 
AI Intelligence: Exploring the Future of Artificial Intelligence
sayalikerimova20
 
Building the Next Generation of Android Apps with AI.pdf
Veronica Anggraini
 
geminipro (google developer student clubs Haldia Institute of Technology 2023...
XAnLiFE
 
DeepSeek vs ChatGPT vs Gemini Which AI Best Fits Your Needs.
Davis Smith
 
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Márton Kodok
 
AI Deep Dive_ A Journey through Heroku_OpenAI Integration.pdf
mattmeyers15
 
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
James Anderson
 

More from Márton Kodok (20)

PDF
Vector search and multimodal embeddings in BigQuery
Márton Kodok
 
PDF
BigQuery Remote Functions for Dynamic Mapping of E-mobility Charging Networks
Márton Kodok
 
PDF
Gen Apps on Google Cloud PaLM2 and Codey APIs in Action
Márton Kodok
 
PDF
DevBCN Vertex AI - Pipelines for your MLOps workflows
Márton Kodok
 
PDF
Discover BigQuery ML, build your own CREATE MODEL statement
Márton Kodok
 
PDF
Cloud Run - the rise of serverless and containerization
Márton Kodok
 
PDF
BigQuery best practices and recommendations to reduce costs with BI Engine, S...
Márton Kodok
 
PDF
Vertex AI: Pipelines for your MLOps workflows
Márton Kodok
 
PDF
Cloud Workflows What's new in serverless orchestration and automation
Márton Kodok
 
PDF
Serverless orchestration and automation with Cloud Workflows
Márton Kodok
 
PDF
Serverless orchestration and automation with Cloud Workflows
Márton Kodok
 
PDF
Serverless orchestration and automation with Cloud Workflows
Márton Kodok
 
PDF
BigdataConference Europe - BigQuery ML
Márton Kodok
 
PDF
DevFest Romania 2020 Keynote: Bringing the Cloud to you.
Márton Kodok
 
PDF
BigQuery ML - Machine learning at scale using SQL
Márton Kodok
 
PDF
Applying BigQuery ML on e-commerce data analytics
Márton Kodok
 
PDF
Supercharge your data analytics with BigQuery
Márton Kodok
 
PDF
Vibe Koli 2019 - Utazás az egyetem padjaitól a Google Developer Expertig
Márton Kodok
 
PDF
BigQuery ML - Machine learning at scale using SQL
Márton Kodok
 
PDF
Google Cloud Platform Solutions for DevOps Engineers
Márton Kodok
 
Vector search and multimodal embeddings in BigQuery
Márton Kodok
 
BigQuery Remote Functions for Dynamic Mapping of E-mobility Charging Networks
Márton Kodok
 
Gen Apps on Google Cloud PaLM2 and Codey APIs in Action
Márton Kodok
 
DevBCN Vertex AI - Pipelines for your MLOps workflows
Márton Kodok
 
Discover BigQuery ML, build your own CREATE MODEL statement
Márton Kodok
 
Cloud Run - the rise of serverless and containerization
Márton Kodok
 
BigQuery best practices and recommendations to reduce costs with BI Engine, S...
Márton Kodok
 
Vertex AI: Pipelines for your MLOps workflows
Márton Kodok
 
Cloud Workflows What's new in serverless orchestration and automation
Márton Kodok
 
Serverless orchestration and automation with Cloud Workflows
Márton Kodok
 
Serverless orchestration and automation with Cloud Workflows
Márton Kodok
 
Serverless orchestration and automation with Cloud Workflows
Márton Kodok
 
BigdataConference Europe - BigQuery ML
Márton Kodok
 
DevFest Romania 2020 Keynote: Bringing the Cloud to you.
Márton Kodok
 
BigQuery ML - Machine learning at scale using SQL
Márton Kodok
 
Applying BigQuery ML on e-commerce data analytics
Márton Kodok
 
Supercharge your data analytics with BigQuery
Márton Kodok
 
Vibe Koli 2019 - Utazás az egyetem padjaitól a Google Developer Expertig
Márton Kodok
 
BigQuery ML - Machine learning at scale using SQL
Márton Kodok
 
Google Cloud Platform Solutions for DevOps Engineers
Márton Kodok
 
Ad

Recently uploaded (20)

PPTX
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pptx
Varsha Nayak
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PDF
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
PDF
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
PDF
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
PPTX
Transforming Mining & Engineering Operations with Odoo ERP | Streamline Proje...
SatishKumar2651
 
PDF
SciPy 2025 - Packaging a Scientific Python Project
Henry Schreiner
 
PPTX
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
PPTX
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PPTX
In From the Cold: Open Source as Part of Mainstream Software Asset Management
Shane Coughlan
 
PPTX
Homogeneity of Variance Test Options IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
Revenue streams of the Wazirx clone script.pdf
aaronjeffray
 
PPTX
Hardware(Central Processing Unit ) CU and ALU
RizwanaKalsoom2
 
PDF
Online Queue Management System for Public Service Offices in Nepal [Focused i...
Rishab Acharya
 
PPTX
Human Resources Information System (HRIS)
Amity University, Patna
 
PDF
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PDF
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
PDF
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
PDF
4K Video Downloader Plus Pro Crack for MacOS New Download 2025
bashirkhan333g
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pptx
Varsha Nayak
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
Transforming Mining & Engineering Operations with Odoo ERP | Streamline Proje...
SatishKumar2651
 
SciPy 2025 - Packaging a Scientific Python Project
Henry Schreiner
 
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
In From the Cold: Open Source as Part of Mainstream Software Asset Management
Shane Coughlan
 
Homogeneity of Variance Test Options IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
Revenue streams of the Wazirx clone script.pdf
aaronjeffray
 
Hardware(Central Processing Unit ) CU and ALU
RizwanaKalsoom2
 
Online Queue Management System for Public Service Offices in Nepal [Focused i...
Rishab Acharya
 
Human Resources Information System (HRIS)
Amity University, Patna
 
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
4K Video Downloader Plus Pro Crack for MacOS New Download 2025
bashirkhan333g
 
Ad

Function Calling with the Vertex AI Gemini API

  • 1. DevFest Croatia, October 2024 Function Calling with the Vertex AI Gemini API Márton Kodok Software Architect at REEA.net
  • 2. 1. Intro 2. Gemini disambiguation 3. What is Vertex AI? 4. Function calling with Gemini 5. Demo Agenda Function Calling with the Vertex AI Gemini API @martonkodok
  • 3. ● Google Developer Expert on Cloud technologies (2016→) ● Champion of Google Cloud Innovators program (2021→) ● Among the Top 3 romanians on Stackoverflow 207k reputation ● Crafting Cloud Architecture+ML backends at REEA.net Articles: martonkodok.medium.com Twitter: @martonkodok Slideshare:martonkodok StackOverflow: pentium10 GitHub: pentium10 Function Calling with the Vertex AI Gemini API @martonkodok About me
  • 5. @martonkodok Gemini comes in different sizes Gemini 1.5 Flash fastest, most cost-efficient model yet for high volume tasks GA Gemini 1.5 Pro Multimodal reasoning for longer prompts, 1 million context window GA 1M high volume 2000 (RPM), lower latency and cost 2M Large prompt, 1000+ pages of PDF
  • 6. * October 2024 @martonkodok A bit about pricing For 1M tokens Gemini 1.5 Flash Gemini 1.5 Pro C3.5 Haiku GPT-4o GPT-4o-mini Input Token $0.075 - $0.15 $1.25 - $2.50 $0.25 $2.50 $0.15 Output Token $0.30 - $0.60 $5.00 - $10.00 $1.25 $10.00 $0.60 1 million tokens (for prompts up to 128K tokens — for prompts longer than 128K)
  • 7. Function Calling with the Vertex AI Gemini API @martonkodok Gemini everywhere 1. Gemini consumer app 2. Gemini in the AI Studio (API key) 3. Gemini in Vertex AI Studio (Google Cloud - service account key file) 4. Gemini Code Assist (your IDE, VsCode, JetBrains …) 5. Gemini for Google Workspace (for teams by your company)
  • 8. @martonkodok Where to use Build with Gemini Google AI Studio aistudio.google.com - API key - 2-15 QPM - Libraries in Python/Go/Node/Dart/Android console.cloud.google.com - service account key - Vertex AI SDK - LangChain integration + libraries - access to 120 models via Model Garden - available for individual/edu/business Vertex AI Studio
  • 10. “VertexAI is the Google Cloud product group for ML End-to-end ML Platform @martonkodok
  • 11. Function Calling with the Vertex AI Gemini API @martonkodok Model Garden Task Specific AutoML and APIs Open Source Models Foundation Models Model Garden
  • 12. Function Calling with the Vertex AI Gemini API @martonkodok Models Gemini 1.5 Flash fastest, most cost-efficient model yet for high volume tasks Gemini 1.5 Pro Multimodal reasoning for longer prompts, 1 million context window Imagen 3 Generate images from Text prompts Multimodal Embeddings Extract semantic information Chirp for Speech to Text Build voice enabled applications
  • 13. Function Calling with the Vertex AI Gemini API @martonkodok Vertex AI Summer 2024 Launches End-to-end model building platform with choice at every level Develop and deploy faster, grounded in your domain Build on a foundation of scale & enterprise readiness Context Caching Gemini 1.5 Grounding on Google Search Function Calling
  • 16. Function Calling with the Vertex AI Gemini API @martonkodok Function calling in Gemini A framework to connect LLMs to real-time data delegate certain data processing tasks to functions
  • 17. Function Calling with the Vertex AI Gemini API @martonkodok API parameters External API Processing Model Output Function Calls Function Responses Data User Prompt Generating Function calling in Gemini Database
  • 18. Function Calling with the Vertex AI Gemini API @martonkodok Elements of a Function call 1. Function Declaration (name, description, parameters, required) 2. Tools (group functions into tools) 3. Attach (tools to Gemini initialization) 4. Response Parts (parts that are returned to Gemini for response generation)
  • 19. Function Calling with the Vertex AI Gemini API @martonkodok Function Declaration in Gemini tools list_datasets_func = FunctionDeclaration( name="list_datasets", description="Get a list of datasets that will help answer the user's question" ) list_tables_func = FunctionDeclaration( name="list_tables", description="List tables in a dataset that will help answer the user's question", parameters={ type": "object", "properties": { "dataset_id": { type": "string", "description": "Dataset ID to fetch tables from.", } }, "required": [ "dataset_id", ], }, )
  • 20. Function Calling with the Vertex AI Gemini API @martonkodok Gemini Tools sql_query_tool = Tool( function_declarations=[ list_datasets_func, list_tables_func, … ], ) model = GenerativeModel( "gemini-1.5-pro", generation_config={"temperature": 0}, safety_settings={ HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE, … }, tools=[sql_query_tool], )
  • 21. Function Calling with the Vertex AI Gemini API @martonkodok Gemini Fetch Functions def extract_function_calls(response: GenerationResponse) -> list[dict]: function_calls: list[dict] = [] if response.candidates[0].function_calls: for function_call in response.candidates[0].function_calls: function_call_dict: dict[str, dict[str, Any]] = {function_call.name: {}} for key, value in function_call.args.items(): function_call_dict[function_call.name][key] = value function_calls.append(function_call_dict) return function_calls Prompt: What datasets exists in our database? Response: [{'list_datasets': {}}] function_call { name: "list_datasets" args { } }
  • 23. Function Calling with the Vertex AI Gemini API @martonkodok API parameters External API Processing Model Output Function Calls Function Responses Data User Prompt Generating Function calling in Gemini Database
  • 24. Function Calling with the Vertex AI Gemini API @martonkodok Gemini Function calling also supports 1. Parallelfunction processing 2. Nestedparameters and complex schemas 3. Function calling modes: a. AUTO, The model decides whether to predict a function call or a natural language response b. ANYmode forces the model to predict a function call c. NONEmode instructs the model to not predict function calls
  • 25. Function Calling with the Vertex AI Gemini API @martonkodok Gemini Function calling benefits 1. Ability to feed in real time results from own app/database 2. Adheres output to a specific schema - receiving consistently formatted responses. 3. Ability to work on multimodal inputs 4. Agent and workflow processing 5. Reasoning engine, and orchestrator jobs STEPS to simulate compile/run of output
  • 27. Gen AI … repository Function Calling with the Vertex AI Gemini API @martonkodok
  • 28. @martonkodok Code repository GenAI for Google Cloud goo.gle/gen-ai-github
  • 29. @martonkodok Sample notebooks and apps Description Sample Name Intro to Function Calling in Gemini intro_function_calling.ipynb Working with Parallel Function Calls and Multiple Function Responses in Gemini parallel_function_calling.ipynb Working with Data Structures and Schemas in Gemini Function Calling function_calling_data_structures.ipynb Forced Function Calling with Tool Configurations in Gemini forced_function_calling.ipynb SQL Talk: Natural Language to BigQuery with Gemini's Function Calling sql-talk-app Using Gemini Function Calling to get Real-Time Company News and Insights use_case_company_news_and_insights.ipy nb Intro to ReAct Agents with Gemini & Function Calling intro_diy_react_agent.ipynb
  • 30. Conclusion Function Calling with the Vertex AI Gemini API @martonkodok
  • 31. 1. Build with the groundbreaking ML tools that power Gemini 2. Accelerate ML with tooling for fetching business data via function calls 3. End-to-end integration for data and AI with agents that outperform and solve complex ML tasks 4. Ability to combine multimodal inputs and complex schemas to maintain output format Gemini Function calling: Enhanced ML developer experience Function Calling with the Vertex AI Gemini API @martonkodok
  • 32. “ But there is more… Function Calling with the Vertex AI Gemini API @martonkodok
  • 33. “ @martonkodok Article about Imagen 3 Vector Search and multimodal embeddings Remote Functions in BQ size color living
  • 34. Linkedin: @martonkodok Thank you. Q&A. Reea.net - Integrated web solutions driven by creativity to deliver projects. Follow for articles: martonkodok.medium.com Slides available on: slideshare.net/martonkodok