Code Review Automation

with Gemini
Mansun Kuo
2024-11-30
Outline

● Introduction
● Code Review with Gemini
● AI Code Review in CI/CD Pipeline
● Key Takeaways
Introduction
A MEME of Code Review
Code reviewers can sometimes be nitpicky, inconsistent, or overly
complicated

https://glif.app/glifs/clxtc53mi0000ghv10g6irjqj
What is Code Review
The process of examining source code among team members

● Design
● Functionality Merge with
Approval
● Complexity
● Tests
● Naming Trunk (main)
● Comments
● Style
Code
● Consistency Short-lived Review
● Documentation Feature Branch

Pull Request

https://google.github.io/eng-practices/review/reviewer/looking-for.html
https://trunkbaseddevelopment.com/
Efficiency in Code Review Drives Team Success
Team with faster code reviews have 50% higher software delivery
performance
● Code
improvements
● Knowledge
transfer
● Shared code
ownership
● Team ownership
● Transparency

https://dora.dev/research/2023/dora-report/2023-dora-accelerate-state-of-devops-report.pdf#21
Speed of Code Review
Slower review, lower productivity

● One business day for initial

response
● Avoid large pull request
○ Breakdown into smaller tasks
● What is small
○ One self-contained change
○ Include test
○ Complete context
○ Few hundreds lines with reasonable
number of files

https://google.github.io/eng-practices/review/reviewer/speed.html
https://google.github.io/eng-practices/review/developer/small-cls.html
https://www.reddit.com/r/ProgrammerHumor/comments/1apz2tp/scrollingspeedoncodereviewgraph/?rdt=61474
Task Reliance of AI
Only about 50% people relied on AI for code review
● A 25% increase in AI
adoption is associated
with 3.1% increase in
code review speed
● AI adoption may have
negative impact for
software delivery
● code reviews and
testing are safeguards
that build trust about
using GenAI
https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report
https://dora.dev/research/2024/trust-in-ai/#five-strategies-for-fostering-developers-trust-in-gen-ai
https://dora.dev/research/2024/dora-report/
Continuous Merge
Enables automatic approval and efficient routing of complex pull
requests
● Automating merge
approvals for
small, low-risk pull
requests
○ Safe changes
○ Small fixes
○ PRs from
trusted teams
● Ensure the reviews
go to the most
appropriate people

https://linearb.io/blog/what-is-continuous-merge
2024 Software Engineering Benchmarks Repot
Shorter pickup/review time and smaller PR size for elite group

● 2000+ teams, 3.6 M

PRs in 64 countries
● Average on 4 groups
○ Elite – Top 10%
○ Good – 11 - 30%
○ Fair – 31 - 60%
○ Needs Focus – Bottom
40%

https://linearb.io/resources/engineering-benchmarks
https://linearb.io/blog/engineering-metrics-benchmarks-what-makes-elite-teams
https://linearb.io/blog/pull-request-pickup-time
How to Write Comments

● Be courtesy and respectful Bad:

● Explain Why “Why did you use threads here when there’s
obviously no benefit to be gained from
● Giving Guidance
concurrency?”
● Label common severity
Good:
“The concurrency model here is adding
complexity to the system without any actual
performance benefit that I can see. Because
there’s no performance benefit, it’s best for this
code to be single-threaded instead of using
multiple threads.”
https://google.github.io/eng-practices/review/reviewer/comments.html
Code Review with Gemini
Gemini is Good at Coding

https://cloud.google.com/products/gemini/code-assist
Strength of Gemini with Code
Long input token with affordable price

● Longest input tokens

○ Gemini 1.5 Pro: 2 million
○ Gemini 1.5 Flash: 1 million
● 1 million tokens would look
like:
○ 50,000 lines of code (with the
standard 80 characters per line)
○ All the text messages you have
sent in the last 5 years
○ 8 average-length English novels
○ Transcripts of over 200 average
length podcast episodes
https://cloud.google.com/vertex-ai/generative-ai/docs/long-context
https://artificialanalysis.ai/leaderboards/models
Gemini Code Assist - Review Selected Code
Uses your selected code as reference and responds to your prompt

https://cloud.google.com/gemini/docs/codeassist/write-code-gemini#prompt-gemini-selected-code
Gemini Code Assist - Review Whole Project
Local codebase awareness

https://cloud.google.com/gemini/docs/codeassist/write-code-gemini#prompt-gemini-selected-code
GenAI for Developers
Showcases from GCP
● Prompts for code review
● CLI for different review
scenarios
● Integrate Gemini with
GitHub, GitLab, Bitbucket,
CircleCI, Jira, and more.
● ReAct agents with LangChain
to create Jira/GitLab issues
● LangSmith Integration
● RAG
https://github.com/GoogleCloudPlatform/genai-for-developers
https://codelabs.developers.google.com/genai-for-dev-code-review
https://pretalx.com/devconf-us-2024/speaker/US877R/
Prompt for Code Review - Instruction
Provide context and goals
You are a senior software engineer and architect with over 20 years of experience, specializing in the language of the provided
code snippet and adhering to clean code principles. You are meticulous, detail-oriented, and possess a deep understanding of
software design and best practices.

Your task is to perform a comprehensive code review of the provided code snippet. Evaluate the code with a focus on the
following key areas:

● Correctness: Ensure the code functions as intended, is free of errors, and handles edge cases gracefully.
● Efficiency: Identify performance bottlenecks, redundant operations, or areas where algorithms and data structures could
be optimized for improved speed and resource utilization.
● Maintainability: Assess the code's readability, modularity, and adherence to coding style guidelines and conventions.
Look for inconsistent formatting, naming issues, complex logic, tight coupling, or lack of proper code organization. Suggest
improvements to enhance clarity and maintainability.
● Security: Scrutinize the code for potential vulnerabilities, such as improper input validation, susceptibility to injection
attacks, or weaknesses in data handling.
● Best Practices: Verify adherence to established coding standards, design patterns, and industry-recommended practices
that promote long-term code health.
https://github.com/GoogleCloudPlatform/genai-for-developers/blob/ae484700f402ff1e4a18d24f745ec9cb31308d6c/devai-cli/src/devai/commands/review.py#L179
-L188
Prompt for Code Review - Output Format
Apply best practice of code review
Structure
Organize your findings by class and method names This provides clear context for the issues and aids in refactoring.

Tone
Frame your findings as constructive suggestions or open-ended questions. This encourages collaboration and avoids a purely
critical tone. Examples:
● "Could we explore an alternative algorithm here to potentially improve performance?"
● "Would refactoring this logic into smaller functions enhance readability and maintainability?"

Specificity
Provide detailed explanations for each issue in your own words. Don't repeat terms from the prompt without any explanations.
This helps the original developer understand the reasoning and implement effective solutions.
Prioritization
If possible, indicate the severity or potential impact of each issue (e.g., critical, high, medium, low). This helps prioritize fixes. If
no major issues are found, state: "No major issues found. The code appears well-structured and adheres to good practices."
Frame your feedback as constructive suggestions or open-ended questions to foster collaboration and avoid a purely critical
tone. Example: "Could we explore an alternative algorithm here to potentially improve performance?"
https://github.com/GoogleCloudPlatform/genai-for-developers/blob/ae484700f402ff1e4a18d24f745ec9cb31308d6c/devai-cli/src/devai/commands/review.py#L143
-L156
Prompt for Code Review - Few Shot Examples
A List with query and response
First questions are to detect violations of coding style guidelines and conventions. Identify inconsistent formatting, naming
conventions, indentation, comment placement, and other style-related issues. Provide suggestions or automatically fix the
detected violations to maintain a consistent and readable codebase if this is a problem.

package main
package main
import "fmt"
import "fmt"
Query
func main() {{
func main()
name :=
name := "Alice"
"Alice"
greeting
greeting:=
:=fmt.Sprintf("Hello,
fmt.Sprintf("Hello,%s!", name)
%s!", name)
fmt.Println(greeting)
fmt.Println(greeting)
}
}

● Indentation: yes - Code is consistently indented with spaces (as recommended by Effective Go)
● Variable Naming: yes - Variables ("name", "greeting") use camelCase as recommended
● Line Length: yes - Lines are within reasonable limits
Respons
● Package Comments: n/a - This code snippet is too small for a package-level comment e
https://github.com/GoogleCloudPlatform/genai-for-developers/blob/ae484700f402ff1e4a18d24f745ec9cb31308d6c/devai-cli/src/devai/commands/review.py#L193
-L291
Code Review with AI
Mimic what we used to do in a code review
● Treat AI as a supplemental reviewer
○ AI can make mistakes
○ AI tends to find some issues for you.
○ Developers should take responsible to the final answer
● AI only focus on the diff to have a fixed scope of code review
○ LLM understand Git diff
○ Human can address more concern when there is any
● Post the result of AI code review in the comment of a pull request
Build an Interactive Chat for Code Review
Disable response validation if your code is not perfect 😀
from vertexai.generative_models import GenerativeModel
from google.cloud.aiplatform import telemetry

MODEL_NAME = "gemini-1.5-flash"
USER_AGENT = "cloud-solutions/genai-for-developers-v1.0"
qry = "THE_PROMPT_IN_PREVIOUS_SLIDES"
source = "THE_SOURCE_CODE_YOU_WOULD_LIKE_TO_REVIEW"
code_chat_model = GenerativeModel(MODEL_NAME)
with telemetry.tool_context_manager(USER_AGENT):
code_chat = code_chat_model.start_chat(response_validation=False)
code_chat.send_message(qry)
response = code_chat.send_message(source)
https://github.com/GoogleCloudPlatform/genai-for-developers/blob/ae484700f402ff1e4a18d24f745ec9cb31308d6c/devai-cli/src/devai/commands/review.py#L433
-L440
https://ai.google.dev/gemini-api/docs/text-generation?lang=python#chat
Enable LangChain for Flexibility

system_as_human = False
if self.llm.get_name() == "ChatGoogleGenerativeAI":
system_as_human = True
review_prompt = self.prompt.get_review_prompt(system_as_human=system_as_human)
chain = review_prompt | self.llm
if path:
code = self._get_code_from_local(path, commit1, commit2)
elif remote_url:
code = self._get_code_from_remote_git(remote_url, remote_branch, commit1,
commit2)
else:
raise ValueError("Either 'path' or 'remote_url' must be provided.")
result = chain.invoke(code)
https://python.langchain.com/v0.1/docs/expression_language/get_started/
LangChain - Few Shot Examples

def get_few_shot_prompt(self) -> FewShotChatMessagePromptTemplate:

example_prompt = ChatPromptTemplate.from_messages([
("human", "{query}"),
("ai", "{response}"), [

]) {

examples = self.review_examples 'query': "...",

return FewShotChatMessagePromptTemplate( 'response':

examples=examples, "...",

example_prompt=example_prompt, },

) {
'query': "...",

def get_review_prompt(self, system_as_human: bool = False) -> ChatPromptTemplate: 'response':

review_prompt = ChatPromptTemplate([ "...",

("system", self.review_instructions), },

("system", self.review_output_format), ]

self.get_few_shot_prompt(),
("human", "{code}"),
])

https://python.langchain.com/v0.1/docs/modules/model_io/prompts/few_shot_examples_chat/
AI Code Review in CI/CD Pipeline
GenAI Code Review in CI/CD
Enable instant reviews and enforce AI code reviews for compliance
● LLM privider
● CLI Base Merge with
commit Approval
○ git diff base_commit head_commit
○ Review code with LLM
○ Post comment to a pull request
Trunk (main) Code
● CI/CD pipeline Review
○ Webhook event of pull request
■ Pull request number
■ Base commit SHA
■ Head commit SHA AI Code
○ Secrets Review
■ Git platform access token for posting
review as a comment Head
Pull Request
■ SSH key for private repo commit
■ LLM access key for trigger LLM
https://docs.github.com/en/webhooks/webhook-events-and-payloads#pull_request
https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html#merge-request-events
https://support.atlassian.com/bitbucket-cloud/docs/event-payloads/#Pull-request
Cloud Build
Native serverless CI/CD platform in GCP

● Pros
○ Serverless
○ Container-based CI/CD pipeline
○ Nice integration with GCP
services
○ Integrated with major Git
platforms
● Cons
○ Don’t have straightforward way
to reuse useful steps
○ Don’t have advanced pipeline
control like retry, conditional
step and finally
Behaviors of Cloud Build

● Cloud Build trigger

○ Not able to distinguish different
action types (opened, synchronize,
…)
○ Comment won’t trigger a build
● Cloud Build
○ Default substitutions for a pull
request
○ Use git clone –-depth 1 <repo>
■ Need to fetch commits manually
○ Pull Head branch as master
■ git rev-parse $_BASE_BRANCH (x)
■ git rev-parse origin/$BASE_BRANCH
(o)
https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values#using_default_substitutions
Code Stingray
A lightweight CLI for GenAI code review with CI/CD

● Git platform
○ GitHub
● LLM
○ Google AI
○ Google Cloud Vertex AI
● CI/CD
○ Cloud Build
○ (Container-based CI/CD tools)

https://github.com/mansunkuo/code-stingray
A Step to Have Gemini Code Review in Cloud Build
Expose GITHUB_TOKEN as env var from Secret Manager and execute
CLI
steps:
- name: ghcr.io/mansunkuo/code-stingray:0.1.2
secretEnv: ['GITHUB_TOKEN']
entrypoint: bash
args:
- '-c'
- |
git fetch origin $_BASE_BRANCH $_HEAD_BRANCH
python -m code_stingray.cli --path . -c1 $(git rev-parse origin/$_BASE_BRANCH) -c2 $(git rev-parse
origin/$_HEAD_BRANCH) \
google_cloud --google_cloud_project $PROJECT_ID \
github --github_repo_owner $(echo $REPO_FULL_NAME | cut -d '/' -f 1) --github_repo_name $REPO_NAME --github_pr_number
$_PR_NUMBER
logsBucket: ${PROJECT_ID}_cloudbuild # Change it as the Bucket you want to use.
serviceAccount: projects/${PROJECT_ID}/serviceAccounts/${PROJECT_ID}@${PROJECT_ID}.iam.gserviceaccount.com
options:
logging: GCS_ONLY
availableSecrets:
secretManager:
- versionName: projects/${PROJECT_ID}/secrets/GITHUB_TOKEN/versions/latest
https://github.com/mansunkuo/code-stingray/blob/main/config/cicd/cloudbuild.yaml
env: GITHUB_TOKEN
Sample Result
Overall impression with suggestions for each file

https://github.com/mansunkuo/code-stingray/pull/1#issuecomment-2466840668
GitHub Personal Access Token
Read and write access of pull requests

https://github.com/settings/personal-access-tokens/new
Protect your Pull Request in Publiuc Repositorues

Cloudbuild: GitHub:
Key Takeaways
Key Takeaways
AI Code review reshape the way of code review
● AI increases the speed of code review
● A human gatekeeper in the team is the key to ensure the quality
● Gemini code assist is your best friend in development stage
● Code review automation requires LLM, CLI and CI/CD with
appropriate prompt
● Enable AI code review within CI/CD to boost your code review
process
● Extended topics
○ Agentic AI code review
○ RAG-based code review
References

● GenAI for developers

○ GoogleCloudPlatform/genai-for-developers
○ Code Review Automation with GenAI | Google Codelabs
○ GitHub - Code Review Automation with GenAI | Google Codelabs
○ Bitbucket - Code Review Automation with GenAI | Google Codelabs
● Code Review
○ Code review - Wikipedia
○ About pull request reviews - GitHub Docs
○ Google Engineering Practices Documentation | eng-practices
○ Announcing the 2024 DORA report | Google Cloud Blog
○ https://dora.dev/research/2023/dora-report/2023-dora-accelerate-state-of-devops-rep
ort.pdf
○ What is Continuous Merge? | LinearB Blog
● https://github.com/mansunkuo/code-stingray
Thanks

https://tinyurl.com/yy2uxpt9