17/07/2024, 08:07 Email - Richard Holt - Outlook

AI Tooling for Software Engineers in 2024: Reality Check (Part 1)

The Pragmatic Engineer <[email protected]>
Tue 16/07/2024 17:33
To:Richard Holt <[email protected]>
View in browser

👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic

Engineer Newsletter. In every issue, I cover challenges at Big Tech and
startups through the lens of engineering managers and senior engineers.

AI Tooling for Software Engineers

in 2024: Reality Check (Part 1)
How do software engineers utilize GenAI tools in their software
development workflow? We sidestep the hype, and look to the reality
of tech professionals using LLMs for coding and other tasks.
GERGELY OROSZ AND ELIN NILSSON
JUL 16 ∙ PAID

READ IN APP

In April last year we published The Productivity Impact of AI Coding Tools

based on a survey of subscribers to this newsletter, about how new AI tools
were helping developers with coding. Back then, ChatGPT and GitHub
Copilot were the dominant tools and more were on the way during that time
of experimentation.
Based on readers’ feedback at the time, the article advised playing around
with AI coding tools to find out what worked or not, and predicted: “AI
coding tools will move the industry forward.”

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 1/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Fast forward to 2024, and AI coding tools are more widespread than ever.
GitHub Copilot has passed 1 million paying customers (safe to assume
mostly developers,) and there’s been a surge in startups building AI
software engineering tools, along with no shortage of hype.
A recent peak of the AI hype cycle saw some startups raise funding to
“replace developers with AI engineers.” This publication’s take on that
“mission” and similar ones is that they’re overheated marketing slogans in
response to the popularity of GitHub Copilot, and aren’t the reality.
But how are engineers really using these tools in 2024?
In order to sidestep the hype and tackle that question, we recently launched
a new survey asking software engineers and engineering managers about
your hands-on experience with AI tooling; which tools are being used this
year, what parts of the development workflow are AI-augmented, what
works well, and what doesn’t?
As far as we know, this is the biggest survey yet on how tech professionals
are using AI tools, and this publication’s business model means we’re free
from bias on the subject. The volume of responses was such that it’s taken
a couple of months to compile the data, but today we present it!
We analyze input from subscribers to this newsletter and seek to offer a
balanced, pragmatic, and detailed view of where LLM-powered
development tooling is today.
This article covers:
1. Survey overview. Most of the data in this survey is from software
engineers, with a roughly even split between people working with AI
tools for less than 6 months, between 6-12 months, or for more than a
year.
2. Popular software engineering AI tools. ChatGPT and GitHub Copilot
remain the standouts in popularity. Google’s Gemini, Antrophic’s
Claude, Jetbrains AI, Codeium, and others follow.
3. AI-assisted software engineering workflows. Some of the most
common workflows use Copilot in the IDE, chatting with AI bots instead
of googling, AI-assisted debugging, and picking up unfamiliar
languages and frameworks. There’s a long tail of helpful, innovative
use cases.
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 2/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

4. The good. When AI tools work well, they’re a massive help in

completing projects, increasing test coverage, and making
experimentation easier.
5. The bad. Poor output, hallucinations, and devs over-trusting these
tools, top the list of complaints.
6. What’s changed since last year? Surprisingly, not too much!
Interactive rubber-ducking is more common, and teams are
experimenting more with AI agents.
The bottom of this article could be cut off in some email clients. Read the
full article uninterrupted, online.
Read the full article online

1. Overview
A total of 211 tech professionals took part in the survey, an increase on the
175 responses to last year’s AI tooling questionnaire.
Positions: most respondents are individual contributors (circa 62%.) The
remainder occupy various levels of engineering management:

Respondents’ job titles

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 3/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

We asked engineering leaders additional questions because these people

are in a great position to see the impact of AI tooling on teams. We’ll cover
the engineering leaderships’ views in a follow-up issue.
Experience: The lengths of respondents’ careers is segmented into five-
year periods, up to 20+ years of experience (YOE) in the tech industry:

Respondents’ time spent in tech

Fun fact: one respondent had 60 years of professional development
experience (!!) This developer is now semi-retired. His outlook on these
tools – which he plans to use in the future – is positive, writing:
“I have seen many other technologies over the past 60 years that were
touted as software developer eliminators. All of them failed to live up to
the hype. This is a new age and AI looks like it may be a promising
opportunity to push the boundaries in software design and
development. Exciting time to be alive and see the future blossom
before our eyes.”
Time spent using AI tools: again, a pretty even split between professionals
who are new to using AI tools (6 months or less,) people who have used
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 4/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

them for 6-12 months, and those who have done so for over a year.

How long spent using AI tools for software development

Responses from people who don’t use AI tooling may seem a bit out of
place in a survey about these tools, but we wanted to reach such
developers and ask specific questions, in order to find out why some folks
do not, will not, or cannot employ them. We’ll dive into this in a future
article.
Size of company: a roughly even split in size from tiny businesses, all the
way to mega-corporations. Companies with between 1-50 people are the
majority.

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 5/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Size of company worked at

2. Popular software engineering AI tools
ChatGPT and GitHub Copliot need little introduction as the market leaders
in the software engineering sector. But exactly how popular are they, and
which other tools do developers use?
The Tools
As just mentioned, ChatGPT and GitHub Copilot are heavily represented in
the survey. The surprise was the degree of their popularity:

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 6/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

AI tools software developers use

The responses reveal that as many professionals are using both ChatGPT
and GitHub Copilot as all other tools combined! Below is a breakdown of
this “other” category; note, the next most popular tool, Google’s Gemini,
has only 14% of the mentions garnered by GitHub Copilot:

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 7/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Long tail of AI tools tried or used for software development

The difference in mindshare across respondents is significant, as becomes
clear by visualizing all mentions together:

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 8/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

ChatGPT and GitHub Copilot dominate the AI tooling race

This chart lists all tools with at least two mentions in the survey. Those
mentioned only once include Microsoft Teams AI, Amazon Q, Meta AI,
Vercel’s v0.dev (UI generation from prompts,) Databricks AI assistant, Replit
Ghostwriter, Ellipsis.dev (AI code reviews & bugfixes,) Mutable.ai (creating
living documentation,) CodeRabbit AI (AI-powered code reviews,)
StartCoder (code completion,) and Aider (AI pair programming in a
terminal.) It’s great that so many tools are being tried out!
Favorite AI coding tools
We asked developers which tools are their favorites.
GitHub Copilot and ChatGPT. As the charts show, these got the most
mentions. Here’s a selection:
“Github Copilot chat is the real deal.” – engineer at a dev tools startup
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 9/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

“I use GitHub Copilot because it also has built-in chat” – Ruby on Rails
developer at a quality control software vendor
“I use GitHub Coliplot for daily coding and ChatGPT 4 for complex,
open-ended design discussions” – a data engineer
“GitHub Copilot autocomplete is nice [but] I’ve found its chat function
chat useless. It’s the same with the “generate code from description”
functionality. When I need a leetcode function such as “partition array X
based on Y”, I like using chatGPT, because it works fine. But ChatGPT
gets stuck on harder problems. For example, building complex
typescript generics is too much to handle.” – engineer at a scaleup
“I only use Copilot currently. I did use the free ChatGPT, and every so
often I come back to it for something specific.” – software engineer at a
cybersecurity startup
GitHub Copilot Chat is mentioned quite a lot, mostly positively. A big plus is
that it offers an alternative to opening a browser to use ChatGTP. Still, not
everyone is blown away by what the market-leading AI can do, including a
senior software engineer who’s saltiness is hard to miss:
“My favorite is GitHub Copilot. It’s the least bad of all the AI tools I have
access to.”
Another respondent shares that their company evaluated 8 AI coding tools
and settled on GitHub Copilot, which seems to have an advantage with
companies buying department-wide licenses, and developers moving over
to it from other tools:
“I started off with Tab9. After that, my company supplied me with a free
GitHub Copilot option, so I started to use Copilot!” – Ruby on Rails
developer at a quality control software vendor
Several respondents say ChatGPT is the only tool they use, and that they
like it. A software engineer at an eSports company shares:
“ChatGPT is where I go if I want to reason about something and I don’t
have colleagues around me available.”
Other tools earned honorable mentions as some devs’ favorite tools:
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 10/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Claude – the Opus model – was mentioned several times as the best
coding model. This was before the Claude 3.5 Sonnet model was
released, which is much more proficient with coding tasks, according
to the Anthropic team, who uses this model to develop their own
product; meaning we can expect Claude’s popularity to increase.
Gemini. The most mentioned after ChatGPT and Copilot. Several
respondents reckon Gemini is better for coding tasks than ChatGPT,
and prefer to use it over OpenAI’s chatbot. One respondent says they
alternate between Gemini and Claude to gauge which one works better
for each use case.
Codieum has several mentions as a favorite IDE, thanks to its code-
aware autocomplete.
Cursor also gets several mentions, with one respondent calling it a
“game-changer.”
Perplexity and Phind are cited a few times as preferred search tools
for software engineering-related topics.
Other tools: Aider (pair programming in the terminal), JetBrains AI,
AWS CodeWhisperer and Rewatch (meeting notes) each had one
mention
3. AI-assisted software engineering
workflows
We asked respondents who have used AI tools for more than six months
what an AI-assisted workflow looks like for them and/or their team. Some
trends can be observed:
Most common use cases
Integrated AI-assisted IDE workflow. The single most common response.
The most common integration is GitHub Copilot in Visual Studio Code, or in
a JetBrains IDE. A few quotes:
“GitHub Copilot integrated into Jetbrains IDE. Copilot suggests tests
when doing test driven development (TDD.) This speeds up the TDD
cadence quite a bit, which makes testing even more joyful!” – developer
with 29 years of experience (YOE)
“I use GitHub Copilot in Visual Studio Code IDE. My workflow isn't much
different, just faster, and my test coverage is better. SQL has always
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 11/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

been my achilles heel, but I've been writing much more complex
queries.” – director of engineering in security, 12 YOE
“GitHub Copilot is part of the IDE. It’s like ‘autocomplete on steroids!’
Copilot Chat/Bing/Gemini can make devops iterations faster, as it helps
with syntax rules and acts as a behavior bank.” – software engineer, 2
YOE
“Coding: For GitHub Copilot, it's enabled in the IDE via a plugin (VS
Code or JetBrains.) I recently needed to make changes on our Google
Cloud environments, and found that Copilot was helpful in scripting
Google Cloud commands. On our client side, much of the code is C++ or
C#, and it's proven useful there as well. I did an iOS project in Objective-
C, and since Xcode didn't support Copilot natively, I opened the project
in VS Code in order to get some suggestions for calling certain Apple
Frameworks. It was less helpful there, but still useful.” - Director of
engineering, 30 YOE
A better Google, Stack Overflow, or documentation resource. Another
very popular use case is using AI tools as interactive search,
documentation, and for explainers:
“Previously, I'd go to Google or the official docs for help with an
exception or something unfamiliar; now chat GPT is my first point of call.
What's great is that I can follow up with further questions and it takes
into account the context.” – DevOps engineer, 9 YOE
“Before the advent of ChatGPT/Gemini I would have to go through
unfocused tutorials on the technologies I am unfamiliar with, and search
Google/Stack Overflow for particular questions. Now, I have a
conversation with the AI, driving the discussion in the direction of my
exploration." – CTO, 17 YOE
“I ask it to explain blocks of code, or prompt the chat for specific queries
about how to use a specific framework or language. I've started
defaulting to Copilot Chat more and more instead of Google, especially
since it's integrated right into the IDE." - Senior engineer, 10 YOE.
Debugging. This use case is for AI chatbots like ChatGPT, GitHub Copilot
Chat, Gemini, or Claude. Survey responses mention:
Using when debugging tricky production issues
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 12/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Helping generate a set of possible issues to explore for debugging

Debugging things like config issues in a YAML file
Creating a set of possible issues to be explored during debugging
Debugging root causes of minor implementation errors
Identifying errors with SQL queries
Debugging issues with tooling and third-party software
New ideas and theories about what to debug
A software architect of 20 years has a word of warning about over relying
on AI:
“As a (slightly depressing) anecdote, I interviewed a candidate at
roughly a senior level. They had done well in earlier coding interviews
and were joining me for a debugging session. Not only did they get
stuck immediately and struggle to progress on their own, when I asked
them leading questions to unblock them they would copy the
corresponding code into ChatGPT and ask it instead. When it failed to
give them the result they were looking for, they gave up on trying to
respond and went back to aimlessly staring at the code 😞 .”
Working with unfamiliar languages or frameworks is another very
popular use case which the AI tools help with:
Writing scripts and tests in languages the dev has not worked with for
some time
Writing code in languages the dev has not yet used before, such as
AppleScript, Bash, the jq programming language, etc.
Explaining code, or how to do something in the language
Asking questions about implementation practices for unfamiliar
frameworks and languages
Creative use cases
These are mentioned only rarely by respondents, but are neat and
innovative.
Filling in for absent infra/DevOps expertise::
“This tooling has been invaluable for helping us with infrastructure tasks.
We no longer have a dedicated DevOps person, so it's been on us
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 13/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

backend devs to maintain a fairly complex Kubernetes setup. Being able

to go into GPT4 and describe our setup, tell it what we want to do, and
then getting an answer that we dig into has been a lifesaver more than
once.” – senior staff engineer, 25 YOE, using AI tools for 2 years.
Treating AI agents as junior developers. Another fascinating use case is
AI coding agents, which was the topic of a recent deep dive. A staff
engineer with 20 YOE explains:
“An experimental workflow we are trying out involves giving work to
agents as we would to a junior engineer. We then work on our own code
until it's time to review the agent's work. We've experimented with this
transactionally; it's akin to giving someone a ticket to work on in a sprint.
This approach has never worked for any interesting amount of work,
though.“
AI-augmented coding experience. Some devs take collaboration with AI
tools to a new level. A software architect with 14 YOE has been using these
tools for nearly 2 years, and says:
“I break down the task to the smallest level which can be unit tested.
Then I prompt and code. In case I need to learn something new, or
adjacent to what I know, I go to ChatGPT or Perplexity.ai to help get
started and write the unit test cases. All of these have significantly
reduced the time to develop, shortened the feedback loop and have
increased my confidence.“
Improving automation. From a senior engineer with 8 YOE:
“The latest big thing AI has helped me with is writing command line tools
to automate small tasks I do often. I was pretty good about this in
general, as I would write shell scripts and aliases that compose
commands quite frequently. But now ChatGPT helps me write these
much faster.
For slightly more complex shell scripts, I often get lazy to write them
because I didn't want to take a couple hours of my day to write and test
the script. Why do this for a task that only takes me a few minutes to
complete? However, with ChatGPT I can cut this down to minutes, which
makes it more likely for me to automate these tasks!”

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 14/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Interactive rubber ducking. As in describing a problem to an AI chat tool,

then seeing how it responds. Here’s how one data engineer does it:
“When I encounter a novel problem, I start typing into the chat, trying to
describe the problem well enough to get a usable answer. Often, this
already ‘rubber ducks’ the chatbot and I have my answer. Otherwise, it’s
back and forth until I have something I can try out. It’s still mostly at the
“consult stack overflow and bother colleagues less stage” for me. Which
is still good: more focus time for all involved!"
Using these tools to get unblocked by describing what needs to be
achieved is helpful, as is using AI as a study or “thought” partner.”
More use cases
Auto-generating tests. Another popular use case where Copilot is
mentioned the most. The tool is not only good at generating tests for the
code you wrote, but also for creating test data. From a senior software
engineer (8 YOE):
“Copilot has been quite useful while testing. I always start with
describing a scenario and Copilot does a really good job of
autogenerating the test I want.
I tend to work with Spark and Flink, and the most annoying part of
testing is generating seed data for the test, especially when there are
lots of tables. After creating one example myself, Copilot does a really
good job of using that example as reference to help me autogenerate
many more scenarios. This has saved me hours of time. It has also
prevented me from being lazy with tests because of how time
consuming some are to write.”
Refactoring. A very common use case of asking a chatbot or tool to
refactor existing code, such as:
Reactoring simple enough but tedious changes
Refactoring small components and microservices
Limiting refactoring to around 100 lines of code is what one
respondent finds helpful with ChatGPT; i.e.: giving the prompt "convert
these Ruby Sorbet classes to comparable Typescript interfaces."
Code generation and scripting, such as:
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 15/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Generating boilerplate structures when starting a new project or

component
Generating individual functions
Using for simple-enough code generation, as an alternative to
StackOverflow
Creating scripts for data exports, bash scripts, or for cleaning and
manipulating data
Review code. Asking it to review and critique code, and suggest how to
improve it.
Other use cases mentioned:
Reply to PR comments. A full stack engineer with 5 YOE uses this tool
to address PR comments they receive and to speed up their pace of
work
Help with SQL queries and regular expressions. A fairly popular use
case, given these tools do well in both areas
Autocomplete “on steroids.” GitHub Copilot is often described this
way for its in-line coding suggestions
Getting started/unstuck. Taking the first step in naming things,
coming up with an API structure, generating example code, or just to
generate ideas on how to attack a problem at hand
Write meeting notes. Summarize the meetings for engineering teams.
This use case is not unique to software engineering, but it sure is
useful!
Write requirements and user stories. Engineering managers and
product managers use these tools to gather and record requirements,
and create user stories.
Refactor tickets for JIRA tasks. These tools are great at writing, so
making tickets more readable is something they help with.
Better writing. Writing is a strength of LLMs, so devs use it to help
them write in correct English, and refine ideas and wording.
Some teams and devs use AI everywhere in their workflows
Several devs mention they use AI augmentation for pretty much everything:

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 16/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Plan: brainstorm ideas, consult on various topics, task and ticket

generation
Code: code autocomplete, scaffolding, code analysis, code
generation, test generation, explain unfamiliar languages and
frameworks
Deploy: optimize deployment processes
Other: write documents, emails, status updates
One senior software engineer applies AI augmentation everywhere:
Debugging and brainstorming issues: consult the chat assistant
when stuck, to get unstuck.
Avoid context switching: an AI tool can write out smaller functions,
thereby reducing distractions when in deep-focus mode.
Get familiar with new architecture: ask questions to get familiar with
the technologies for a new project, or alternative architectural designs
Coaching it to not hallucinate: Training the tool in areas it’s weak in,
like SQL generation
Sharing approaches with other devs: this engineer works with two
other developers, and they share useful knowledge for using the tool
“Newbie” and engineering manager use cases
For those relatively new to AI tools, who have used them for 6 months or
less, the most commonly-mentioned use cases are:
Getting started with coding faster / generate ideas
Bugfixing support
A more powerful research tool than Google
Superior autocomplete that works better than existing IDE
autocompletes
Improved coding
Improved writing
For engineering managers and founders, use cases are understandably
more varied:
Write/review emails, improve written communication
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 17/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Brainstorm on product and business strategies. Use it to set pricing,

create marketing strategies, advise on technology stack
Discuss sensitive topics which are hard to raise with peers at work
Create reports from bullet-point lists
Learn faster by using it as an “interactive tutor”
4. The Good
We asked readers who have been using AI tools for 6 months or more, what
their best experiences have been. These were enjoyable to read because
they show these tools can work fantastically well today, rather than just
promising to in the future.
Massive help in completing specific projects. The most-cited positive
experience is that AI tooling is a huge help for completing projects, such as:
Rewriting a firmware from C++ to Rust. The tooling helps with training,
offering library alternatives, and with initial training on Rust.
Converting existing bash scripts that only worked on Linux, to cross-
platform Python3 scripts that also work on Windows. This would
otherwise take hours or days.
Remove jQuery from a codebase in one afternoon, which could’ve
taken days.
A tech lead with very little Kubernetes experience used it to deploy a
Kubernetes cluster on their own
A software architect with 25 years of experience wrote a new service
with 5,000 lines of code in what felt like record time
Better testing leading to better code quality. Several respondents
highlight that AI tooling makes testing much easier, which leads to improved
code quality:
“The tooling keeps our test driven development vibe going. This has
made our codebase easier to maintain.” – software engineer, 29 YOE
“We are seeing better use of test cases, better code quality, and better
documentation. This is true both at the individual, and team-level.” –
web developer, 7 YOE

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 18/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

“We now have an increased number of test cases. This positively

impacts our belief in our code.” – senior software engineer, 6 YOE
Educational. Readers share many cases of this, and LLMs do indeed seem
well suited to helping tech workers learn new technologies and frameworks
faster, in a more interactive way. A few examples:
A principal engineer with 20 YOE says it flattens the learning curve
with new technologies by removing the need to wade through
documentation
An engineering lead with 10 YOE also says this, and that it makes
learning and experimenting with new technologies less arduous
A senior software engineer with 10 YOE uses it to integrate SDKs and
third-party APIs because they no longer have to deal so much with
integration documents
A software engineer with 4 YOE says learning Flutter for a personal
project was faster than using the official Google documentation
A software engineer with 4 YOE says it’s faster to adjust to using a new
programming language, and it helps helps them pick up syntax faster
Easier to prototype and experiment. Building a prototype version of a
product is much easier, share several respondents. Here’s a CTO and
cofounder with 18 YOE:
“This tooling was key to being able to ship my startup prototype. To
build this prototype, I had to become a ‘full stack engineer’: doing
everything from cloud to frontend, including Python, Typescript,
Javascript, and advanced Java.”
A software engineer with 25 years of experience writes that it makes them
more likely to experiment with unfamiliar technologies:
“I've built projects I wouldn't have considered building without LLM
assistance; in AppleScript and Go, for example. As an experiment, I built
an entire tool just using Claude 3 Opus to write the code, and it worked
really well.”
Better documentation = better communication. Many mentions of this in
responses:
Boosting the quality of documentation
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 19/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Reviewing documents with AI tools

Finding relevant information with tools like Notion AI
Making messy documents more professional-looking. One software
architect says they transformed a chaotic document from a vendor into
a neat integration document that their team now uses
Reducing the language barrier for non-native English speakers
Productivity and quality improvements. A distinguished engineer with 25
YOE shares:
“Personally, I feel like it has supercharged me. I am able to process more
information, come to more conclusions, write more, make more notes -
it's a force multiplier.”
A director of engineering with 30 YOE reckons the efficiency of their teams
has jumped significantly:
“For engineers who use it, they get a boost as they're writing less
"boilerplate" code, and can spend more time in other areas. I have heard
"10-20%" increase from my teams, although I can't back that up with
data.”
“Rusty” folks rediscover coding. A staff engineer with 30 YOE says:
“I am seeing architects and ‘rusty seniors’ who weren't coding as much
in their daily jobs get back into shipping code regularly. We even have a
VP (who used to be a coder 10+ years ago) sharing stories about
shipping code again, thanks to GitHub Copilot.”
Non-engineers can contribute code. An interesting use case is shared by
an engineering manager:
“It has empowered some of our non-engineers who are mildly
comfortable with Python to write more code and scale their non-code
written content.”
Consultancies can build more ambitious solutions for customers on a
budget. A software engineer working at a consultancy observes:
“The most valuable thing I noticed is that AI tooling unlocks the
possibility of developing features in our software solutions that were
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 20/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

simply unachievable before, due to cost inefficiency. There are many

features and products for which the cost to the client was unacceptable.
With these AI tools, we can now get it done within their budget.”
More enjoyable side projects. A developer with 4 YOE finds more joy in
side projects:
“On my side projects: dramatic decrease in the tedious parts of
programming, making them even more enjoyable to work on. Faster time
to a minimum viable (side) project!”
5. The Bad
Of course, there’s not just upsides to AI tooling. Overall, there’s almost as
many frustrations with these tools as there is praise for them in people’s
responses. Here are the most common complaints:
Hallucination. This is an unavoidable feature of LLMs because of how they
work; they essentially guess the next most likely token (aka word.) This
occasionally leads to phrases (including coding phrases) that do not exist,
and the tools can and do invent functions, classes, APIs, etc.
Examples of hallucinations shared by respondents include nonexistent SQL
functions, methods, libraries, and APIs. When not caught in time, these add
time for debugging, finding and removing hallucinations.
Bad output. Generating bad, insecure, or duplicated code, which is
“garbage.” A specific weakness of some AI tools seems to be a failure to
understand implicit dependencies, by making code suggestions that break
them.
Some developers trust AI too much. Several respondents noted how they
see developers becoming overly dependent on AI-generated code
suggestions, and failing to critically evaluate correctness. Sometimes, code
looks good at first glance, but contains bugs which become evident upon
closer inspection.
If the output of AI tools is blindly trusted, the results can be a waste of time.
A staff engineer with 20 YOE shares:
“One developer added an entirely AI-generated function and generated
tests for it using AI. Then, they didn't review the code well. It had bugs
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 21/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

which took far longer to find and fix than it would have taken to write the
code correctly – even without AI!”
Meanwhile, the benefit of AI tools empowering inexperienced or rusty
engineers to contribute code caused a stir at one company. A software
architect with 9 YOE shares:
“Ooof, we had an incident about a year ago. Our not-very-technical CEO
pushed a change entirely written by ChatGPT. Their code turned out to
work completely incorrectly.”
This incident is a good reminder to always review your AI generated code,
especially if you're working with people who are unfamiliar with either
coding or the language/framework itself!
Code reviews take more effort now. Due to hallucinations, developers
need to spend more time on code reviews. A tech lead with 8 YOE
elaborates:
“I have to be more focused on code reviews since there are often
duplicated statements in the AI-generated code. I also catch issues in
the test code, including missing tests for new code added.”
Wasted time. Hallucinations are the biggest time waster mentioned in
responses, along with the way AI chat tools can get stuck. An engineer with
4 YOE explains:
“There have been times when I [was] pretty clear what I'm looking for,
but the tool continually misunderstands, and the code just gets worse
and worse. I've also had code languages just change in the middle of a
set of questions when I've established what language I'm using.”
It’s not just less experienced engineers who can get stuck, thanks to
hallucinations. Here’s a principal engineer with 20 years of experience
saying the same:
“Occasionally I'll lose time because it hallucinates a capability that a
library doesn't have. This usually becomes obvious very quickly but
sometimes I go down a bit of a rabbit hole figuring out if it was a minor
mistake or if the capability isn't actually there at all."

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 22/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

It can make communication (emails, summaries) worse. Half a dozen

people complain that these tools are over-used in written communication,
which degrades its quality. As an example, here’s a principal software
engineer with 20 YOE:
“People in my organization are trying to use it for tech designs and to
write emails. The results are terrible. The AI writes in a very specific way:
the generated text is full of empty phrases, and uses ‘consultant slang.’
It’s not efficient communication.”
One CTO is disappointed with the tools’ inability to correctly summarize
industry jargon:
“I tried using a video call summarization assistant to summarize my calls.
This was a complete bummer: the voice-to-text transcription gets totally
thrown off by any sort of industry-specific terminology. Even when it
gets particular words correct, the automatic summary of the text
produces outcomes that have zero overlap with how I would summarize
the meeting.”
Overuse of AI writing and summarization tools. An engineering manager
is concerned about this, and summarizes the problem:
“I am concerned that if everyone relies too much on AI for writing and
summarizing, we'll wind up with incredibly long-winded content that's
about 60% the quality it could have been if a human wrote it. At the
same time, it’s harder to read because it’s long-winded. Also, people are
using AI to summarize text, meaning we have an AI tool summarizing
writing that is AI-generated, just to uncover what the human intended to
originally say!”
Other complaints:
At one company, the “old guard” is pushing back against AI coding
tools, fearing less experienced developers will copy-paste the outputs
uncritically
There’s general annoyance with AI hype, and how it’s hard to separate
fact from fiction
A few people say the code generated is outdated, likely due to the tool
being trained on older versions of frameworks
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 23/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

More experienced devs complain that code is being generated but not
understood
6. What’s changed since 2023?
In last year’s survey, we asked people how they used AI tools for coding,
and what problem areas they saw. Let’s compare the responses back then
with today’s.
How developers use AI tools
The same coding workflows which devs said AI tools helped them with last
year, are also mentioned in this year’s survey. A recap of workflows which
developers used GitHub Copilot for in 2023:
Scaffolding
Writing boilerplate code
Autocomplete
Generating tests and documentation
Refactoring
Writing tests
Getting started faster
Benefits of ChatGPT mentioned in 2023’s responses:
Faster learning
Completing routine, boring tasks quicker
Kickstarting things more rapidly
Improving code quality
Easier prototyping
Faster debugging
An interactive rubber duck
In the responses to our new, latest survey, the use cases mentioned are:
Interactive rubber ducking is mentioned by far more people than a
year ago, when it was mentioned just once. This use case seems to be
getting popular!
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 24/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

AI agents as junior engineers is an interesting use case. In this year's

survey, a participant shares that they are experimenting using
autonomous AI agents for simple tasks. We’ll likely see far more of this
approach.
Embraced by leadership. In the responses, managers share that they
rely on tools like ChatGPT and Claude for matters like business
strategy, communication, analyzing large data sets, etc.
Non-coding work tasks. Devs use tools like ChatGPT heavily for
writing-related tasks, producing meeting summaries, planning, drafting
emails, etc.
Chief complaints about AI tools: last year and now
Last year, the most common complaints about LLM-based coding
assistants were:
Generated code with errors
Didn’t work so well for specialized frameworks
Out-of-date knowledge
Struggled with complex tasks that need more context as input
Generated dated code
This year, these complaints are present again. The only one that’s less
mentioned is out-of-date knowledge. New issues are mentioned in this
year’s survey:
AI tools are over-trusted. Several respondents note that they or their
colleagues invest too much trust in these tools, and copy-and-paste
code that they have not taken time to thoroughly review.
Incidents and bugs. As a consequence of being over-trusted and
under-scrutinized, AI-generated code becomes hard to debug and
causes production incidents.
Hype and inflated expectations. Last year, this was not mentioned in
the survey at all, even though there was no shortage of hype at the
time. In contrast, the latest survey responses contain skepticism about
the true nature of AI’s much-vaunted productivity gains, and some
readers say it’s hard to tell if the tools really make everyone more
productive.

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 25/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

Quality of written documents. A new complaint is that tools struggle

with industry-specific text, and are not very useful for architecture
documents. Then again, last year we focused on coding use cases,
and these go beyond just coding!
It seems many new complaints derive from AI-assisted workflows being
much more widely used this year, than in 2023. Over time, issues have
emerged like AI-introduced bugs being shipped to production simply
because these tools are constantly used. Of course, we know many bugs
make it into production without AI’s help, too!
We’re learning more about how AI tools are helpful, and where they remain
difficult.
Takeaways
AI is a fast-moving sector when it comes to raising funding and launching
new products. And yet, the AI developer tooling market seems to be moving
slower.
In some ways, little has changed since last year: ChatGPT and GitHub
Copilot are still the top tools of choice for software engineers, developer
use cases remain pretty similar, and the downsides of the tools haven’t
disappeared.
Simultaneously, we’re seeing wide adoption with more people and teams
using AI tools for longer. There are also innovative, new approaches, like
experimenting with AI agents on simple problems, and using these tools for
rubber ducking.
We’re in the midst of a significant tooling change, with AI-augmented
software engineering becoming widespread across tech. Basically,
these tools have too many upsides for developers to ignore them: it’s easier
and faster to switch between stacks, easier to get started on projects, and
simpler to become productive in unfamiliar codebases. Of course there are
also downsides, but being aware of them means they can be mitigated.
I hope you’ve enjoyed this analysis of what might be the largest survey to
date of software engineers using AI tools. Your responses are very
interesting, so we’re going to follow up with a second part in the near
future; a “reality check” on AI coding tools. Expect plenty of information
about what you can reliably use these tools for, today, and also a look into
https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 26/27
17/07/2024, 08:07 Email - Richard Holt - Outlook

how much time AI tools really save. And of course, more observations and
opinions from tech professionals using AI.
Last but not least, thank you to everyone who took part in the survey!

A guest post by
Elin Nilsson
Tech industry researcher, engineer, enthusiast of things 🧑‍💻 Doing
research for The Pragmatic Engineer.
Subscribe to Elin

This post is only for paying subscribers of The Pragmatic Engineer. This
email is intended for a single recipient, but occasional forwarding is totally
fine.
Like & Comment

As a subscriber, you also have access to 🔒 subscriber-only resources for

engineering managers and engineers.
Know someone who would benefit from the newsletter? Consider gifting a
subscription.
Give a gift subscription

LIKE COMMENT RESTACK

© 2024 Gergely Orosz

548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe

https://outlook.office.com/mail/inbox/id/AAQkADRiMDcwY2MyLTc0ODEtNGRkOC05NjU5LTEyNGZjMWQ0ZDA3NwAQAO1nzgbMF%2FxCrDeM17s… 27/27