On the Robustness of Code Generation Techniques:

An Empirical Study on GitHub Copilot

Antonio Mastropaolo∗ , Luca Pascarella∗ , Emanuela Guglielmi† , Matteo Ciniselli∗
Simone Scalabrino† , Rocco Oliveto† , Gabriele Bavota∗
∗ SEART @ Software Institute, Università della Svizzera italiana (USI), Switzerland
† University of Molise, Italy

Abstract—Software engineering research has always being Intuitively, the ability of the developer to provide “proper” in-
concerned with the improvement of code completion approaches, puts to the model will become central to boost the effectiveness
which suggest the next tokens a developer will likely type while of its recommendations. In the concrete example of GitHub
arXiv:2302.00438v1 [cs.SE] 1 Feb 2023

coding. The release of GitHub Copilot constitutes a big step

forward, also because of its unprecedented ability to automati- Copilot, the natural language description provided to the model
cally generate even entire functions from their natural language to automatically generate a code function could substantially
description. While the usefulness of Copilot is evident, it is influence the model output. This means that two developers
still unclear to what extent it is robust. Specifically, we do providing different natural language descriptions for the same
not know the extent to which semantic-preserving changes in function they would like to automatically generate could receive
the natural language description provided to the model have
an effect on the generated code function. In this paper we two different recommendations. While this would be fine in
present an empirical study in which we aim at understanding case the two descriptions are actually different in the semantics
whether different but semantically equivalent natural language of what they describe, receiving different recommendations for
descriptions result in the same recommended function. A negative semantically equivalent natural language descriptions would
answer would pose questions on the robustness of deep learning pose questions on the robustness and usability of DL-based
(DL)-based code generators since it would imply that developers
using different wordings to describe the same code would obtain code recommenders.
different recommendations. We asked Copilot to automatically This is the main research question we investigate in this
generate 892 Java methods starting from their original Javadoc
description. Then, we generated different semantically equivalent paper: We study the extent to which different semantically
descriptions for each method both manually and automatically, equivalent natural language descriptions of a function result in
and we analyzed the extent to which predictions generated by different recommendations (i.e., different synthesized functions)
Copilot changed. Our results show that modifying the description by GitHub Copilot. The latter is selected as representative of
results in different code recommendations in ∼46% of cases. DL-based code recommenders since it is the de facto state-of-
Also, differences in the semantically equivalent descriptions might
impact the correctness of the generated code (±28%). the-art tool when it comes to code generation.
Index Terms—Empirical Study, Recommender Systems We collected from an initial set of 1,401 open source projects
a set of 892 Java methods that are (i) accompanied by a Doc
I. I NTRODUCTION Comment for the Javadoc tool, and (ii) exercised by a test
One of the long lasting dreams in software engineering re- suite written by the project’s contributors. Then, as done in
search is the automated generation of source code. Towards this the literature [23], [32], we considered the first sentence of
goal, several approaches have been proposed. The first attempts the Doc Comments as a “natural language description” of the
targeted the relatively simpler problem of code completion, method. We refer to this sentence as the “original” description.
that has been tackled exploiting historical information [50],
We preliminarily checked whether existing automated para-
coding patterns mined from software repositories [21], [42],
phrasing techniques are suitable for robustness testing, i.e., if
[56], [10], [41], [45], [19] and, more recently, Deep Learning
they can be used to create semantically equivalent descriptions
(DL) models [62], [27], [29], [8], [53], [15].
of the methods to generate. We validated two state-of-the-
The release of GitHub Copilot [14] pushed the capabilities
art approaches in this scenario: PEGASUS [66], a DL-based
of these tools to whole new levels. The large-scale training
paraphrasing tool, and Translation Pivoting (TP), a heuristic-
performed on the OpenAI’s Codex model allows Copilot to
based approach. We used both techniques to generate a
not limit its recommendations to few code tokens/statements
paraphrase for each original description in our dataset. Then,
the developer is likely to write: Copilot is able to automatically
we manually inspected the obtained paraphrases and classified
synthesize entire functions just starting from their signature
them as semantically equivalent or not. We obtained positive
and natural language descriptions.
results for both the approaches, with TP being the best
This new generation of code recommender systems has the
performing one with 77% of valid paraphrases.
potential to change the way in which developers write code
[18] and comes with a number of questions concerning how to Then, to answer our main research question, we generated
effectively exploit them to maximize developers’ productivity. different paraphrases for each original description.
 We used the two previously described automated approaches, We also excluded forked projects to avoid duplicates. The
i.e., PEGASUS and TP, and we also manually generated decision to focus on a single programming language aimed
paraphrases by distributing the original descriptions among four instead at simplifying the non-trivial toolchain needed to run
of the authors, each of which was in charge of paraphrasing a our study. The whole repositories selection process has been
subset of them. performed using the GitHub search tool by Dabic et al. [17].
Therefore, for each original description, we obtained a set ofAt this stage, we obtained 1,401 repositories.
semantically equivalent paraphrased descriptions. We provided In our experimental design, we use the passing/failing tests as
both the original and the paraphrased descriptions as input to a proxy to assess the correctness of the predictions generated by
Copilot, asking it to generate the corresponding method body. Copilot. Thus, we need the projects to use a testing framework
We analyze the percentage of cases in which the paraphrased and to be compilable. We selected all projects that used Maven
descriptions result in a different code prediction as compared as build automation tool and for which the build of their latest
to the original one, with a particular focus on the impact release succeeded. We obtained 214 repository. By parsing
on the prediction quality, e.g., cases in which the original the POM (Project Object Model) file1 we only considered
description resulted in the recommendation of a method passing projects having as dependencies both jUnit [4] — a well-
its associated test cases while switching to a paraphrased known unit testing framework — and Jacoco [2] — a code
description made Copilot recommending a method failing its coverage library. We analyzed the Jacoco reports and selected
related tests. as methods subject of our experiment those having at least
Our results show that paraphrasing a description results 75% of statement coverage. This gives us confidence that the
in a change in the code recommendation in ∼46% of cases. related test cases exercise an acceptable number of behaviors
The resulting changes also cause substantial variations in and, therefore, could allow to spot cases in which different
the percentage of correct predictions. Such findings indicate generated functions for semantically-equivalent descriptions
the central role played by the model’s input in the code actually behave differently. We are aware that passing tests does
recommendation and the need for testing and improving the not imply correctness. We discuss this aspect in Section IV.
robustness of DL-based code generators. Given our goal to use the method’s description as input for
Data and code used in our study are publicly available [6]. Copilot, we also exclude methods not having any associated
Doc Comment for the Javadoc tool. Then, we process the
II. S TUDY D ESIGN Doc Comment of each method in our dataset to extract from
The goal of our study is to understand how robust is a state- it the first sentence (i.e., from the beginning to the first “.”).
of-the-art DL-based code completion approach (i.e., GitHub This is the same approach used in the literature when building
Copilot). We aim at answering the following research questions: datasets aimed at training DL-based techniques for Java code
summarization (see e.g., [23], [32]), with the training set
RQ0 : To what extent can automated paraphrasing composed by pairs <method, code_description>, with
techniques be used to test the robustness of DL-based the latter being the first sentence of the Doc Comment. To
code generators? Not always natural language processing ensure that the extracted sentence contains enough wording for
techniques can be used out of the box on software-related the code description, we exclude all methods having less than
text [35]. Therefore, with this preliminary RQ, we want 10 tokens in the extracted first sentence, since their description
to understand whether existing automated techniques for may not be sufficient for synthesizing the method.
generating natural language paraphrases are suitable for SE
task at hand (i.e., paraphrasing a function description). TABLE I
O UR DATASET OF 892 METHODS FROM 33 REPOSITORIES
RQ1 : To what extent is the output of GitHub Copilot
influenced by the code description provided as input by the Avg Median St. Dev.
developer? This RQ aims at understanding whether Copilot,
as a representative of DL-based code generators, is likely to # Tokens 154.3 92.0 218.2
generate different recommendations for different semantically # Parameters 1.6 1.0 1.2
equivalent natural language descriptions provided as input. # Cyclomatic Complexity 5.3 3.0 7.6
In the following we detail the context for our study (Sec- % Coverage 96.1 100.0 6.7
tion II-A) and how we collected (Section II-B) and analyzed
(Section II-C) the data needed to answer our RQs. The above-described process resulted in the collection of
892 Java methods. Table I shows descriptive statistics about
A. Context Selection their characteristics in terms of number of tokens, parameters
The context of our study is represented by 892 Java methods and cyclomatic complexity. These three together provide an
collected through the following process. We selected all GitHub idea about the complexity of the task Copilot was asked to
Java repositories having at least 300 commits, 50 contributors, perform (i.e., the complexity of the methods it had to generate).
and 25 stars. These filters have been used in an attempt to
exclude personal/toy projects. 1 POM files are used in Maven to declare dependencies towards libraries.
 Full Context Such a technique relies on natural language translation
public class Hook implements Resultsable {
services to translates the original description o from English
// Start: attributes from JSON file report
into a foreign language (i.e., French), obtaining oE → F . Then,
private final Result result = null;
private final Match match = null;
oE → F is translated back in the original language (oE→F →E )
@JsonDeserialize(using = OutputsDeserializer.class)
obtaining a paraphrase.
@JsonProperty("output")
private final Output[] outputs = new Output[0];
We provide each technique with the original description
as input. TP failed to generate a valid paraphrase (i.e., a
// foe Ruby reports
private final Embedding[] embeddings = new Embedding[0]; sentence different from the original one) in 100 cases (out
// End: attributes from JSON file report
of 892), while this only happened once with PEGASUS. We
@Override
public Result getResult() { manually analyzed whether the valid paraphrases we obtained
return result;
} were actually semantically equivalent to the original description.
/** Return the embedding vector */ For such a process, each of the 1,683 paraphrases (892 for
public Embedding[] getEmbeddings() {
| each of the two tools minus the 101 invalid ones) has been
} Method to be predicted
independently inspected by two authors who classified it as
/** Checks if the hook has content meaning as it has at least
* attachment or result with error semantically equivalent or not. Conflicts, that arisen in 11.9%
* message.
*/
(PEGASUS) and 16.54% (TP) of cases, have been solved by
public boolean hasContent() {
if (embeddings.length > 0) {
a third author not involved in the first place.
return true;
}
Concerning RQ1 , we start from the original description
if (StringUtils.isNotBlank(result.getErrorMessage())) { and we generate semantically equivalent descriptions by (i)
return true;
}
// TODO: hook with 'output' should be treated
using the two automated tools, i.e., PEGASUS [5] and TP,
/ as empty or not?
return false;
and (ii) manually generating paraphrases. For the manual
} paraphrasing, we split the 892 methods together with their
}
original description into four sets and assigned each of
them to one author. Each author was in charge of writing a
semantically equivalent but different description of the method
Non Full Context
by looking at its code and original description. This resulted
public class Hook implements Resultsable {
in a dataset (available in [6]) in which, for each subject
// Start: attributes from JSON file report
private final Result result = null; method, we have its original and paraphrased description. In
private final Match match = null;
the end, for each original sentence, we had between one and
@JsonDeserialize(using = OutputsDeserializer.class)
@JsonProperty("output") three paraphrases: paraphrasedPEGASUS , paraphrasedTP , and
private final Output[] outputs = new Output[0];
paraphrasedmanual . While paraphrasedmanual is available for
// foe Ruby reports
private final Embedding[] embeddings = new Embedding[0];
all the methods, paraphrasedPEGASUS and paraphrasedTP are
// End: attributes from JSON file report not. Indeed, we exclude the cases in which each of such tools
@Override
public Result getResult() {
failed to generate paraphrases (1 and 100, respectively) and the
}
return result; ones that were not considered as semantically equivalent in our
manual check (based on the results of RQ0 ). The maximum
/** Return the embedding vector */
public Embedding[] getEmbeddings() { number of semantically equivalent paraphrases is 2,575 (up to
|
} Method to be predicted 891 with PEGASUS, up to 792 with TP, and 892 manually).
} The paraphrases, as well as the original description, have
been used as input to Copilot, simulating developers asking it
to synthesize the same Java method by using different natural
language descriptions. At the time of our study, Copilot does not
Fig. 1. GitHub Copilot’s input for both code context representations
provide open APIs to access its services. The only way to use
it is through a plugin for one of the supported IDEs. Manually
Statistics about the coverage show, instead, the by-design high invoking Copilot for the thousands of times needed (up to 6,934,
statement coverage we ensure for the included methods. as we will explain later) was clearly not an option. For this
reason, we developed a toolchain able to automatically invoke
B. Data Collection Copilot on the subject instances: We exploit the AppleScript
language to automate this task on a MacBook Pro, simulating
To address RQ0 , we experiment with two state-of-the-art the developer’s interaction with Visual Studio Code (vscode).
paraphrasing techniques. The first is named PEGASUS [66], For each method mi in our dataset, we created up to four
and it is a sequence-to-sequence DL model pre-trained using different versions of the Java file containing it (one for each
self-supervised objectives specifically tailored for abstractive of the experimented descriptions). In all such versions, we
text summarization and fine-tuned for the task of paraphrasing (i) emptied mi ’s body, just leaving the opening and closing
[5]. As for the second technique, we opted for Translation curly bracket delimiting it; and (ii) removed the Doc Comment,
Pivoting (TP). replacing it with one of the four code descriptions we prepared.
 Starting from these files, the automation script we imple- While the original Levenshtein distance works at character-
mented (available in our replication package [6]) performs the level, it can be easily generalized at token-level (each unique
following steps on each file Fi . token is represented as a specific character). In this case, a token
First, it opens Fi in vscode and moves the cursor within the is a word in the text. The normalized token-level Levenshtein
curly brackets of the method mi of interest. Then, it presses distance provides an indication of the percentage of words
“return” to invoke Copilot, waiting up to 20 seconds for its that must be changed in the original description to obtain a
recommendation. Finally, it stores the received recommendation, paraphrased one.
that could possibly be empty (i.e., no recommendation received). Then, we analyze the percentage of methods for which the
To better understand this process, the top part of Fig. 1 depicts paraphrased descriptions result in a different method prediction
how the invocation of Copilot is performed. The gray box as compared to the original one. When they are different, we
represents the whole Java file (i.e., the context used by Copilot also assess how far the methods obtained by using a given
for the prediction). The emptied method (i.e.,getEmbeddings) paraphrased description is from the method recommended
is framed with a black border, with the cursor indicating the when providing the original description as input. Also in this
position in which Copilot is invoked. The green comment on case we use the token-level Levenshtein distance as metric. The
top of the method represents one of the descriptions we created. latter is computed with the same formula previously reported
As it can be seen, Fig. 1 includes for the same Java file two for the natural text descriptions; in this case, however, the
different scenarios, named Full context and Non-full context. In tokens are not the words but the Java syntactic tokens. Thus,
the Full context scenario (top part of Fig. 1) we provide Copilot NTLev indicates in this case the percentage of code tokens
with the code preceding and following the emptied method, that must be changed to convert the method obtained through
simulating a developer adding a new method in an already the original description into the one recommended with one
existing Java file. In the Non-full context scenario, instead, we of the paraphrases.
only provide as context the code preceding the emptied method Finally, we study the “quality” of the recommendations
(bottom part of Fig. 1), simulating a developer writing a Java obtained using the different descriptions both in the Full
file sequentially and implementing a new method. context and Non-full context scenarios. Given the sets of
The basic idea behind these two scenarios is that the methods generated from the original description and each of the
contextual information provided to Copilot can play a role paraphrasing approach considered, we present the percentages
in its ability to predict the emptied method. Overall, the of methods for which Copilot: (i) synthesized a method passing
maximum number of Copilot invocations needed for our study all the related test cases (PASS); (ii) synthesized a method that
is 6,934 (892 original descriptions plus up to 2,575 paraphrases, does not pass at least one of the test cases (FAIL); (iii) generated
each of which for 2 context scenarios). After having collected an invalid method (i.e., with syntactic errors) (ERROR); (iv)
Copilot’s recommendations, we found out that sometimes they did not generate any method (EMPTY). Syntactic errors have
did not only include the method we asked to generate, but been identified as recommendations for which Java Parser [3]
also additional code (e.g., other methods). To simplify the data did not manage to identify a valid recommended method (i.e.,
analysis and to make sure we only consider one recommended cases in which Java Parser fails to identify a method node in
method, we wrote a simple parsing tool to only extract from the AST generated for the obtained recommendation). On top
the generated recommendation the first valid method (if any). of the passing/failing methods, we also compute the token-level

C. Data Analysis Levenshtein distance and the CodeBLEU [49] between each
synthesized method and the target one (i.e., the one originally
Concerning RQ0 , we report the number and the percentage implemented by the developers). CodeBLEU measures how
of 892 methods for which automatically generated paraphrases similar two methods are. Differently from the BLEU score
(i.e., those generated by PEGASUS and by TP) have been [46], CodeBLEU evaluates the predicted code considering not
classified as semantically equivalent to the original description. only the overlapping n-grams but also syntactic and semantic
This provides an idea of how reliable these tools are when match of the two pieces of code (predicted and reference) [49].
used for testing the robustness of DL-based code generators.
Also, this analysis allows to exclude from RQ1 automatically
generated paraphrases that are not semantically equivalent. D. Replication Package
To answer RQ1 , we preliminarily assess how far the
paraphrased descriptions are from the original ones (i.e., the The code and data used in our study are publicly available
percentage of changed words) by computing the normalized [6]. In particular, we provide (i) the dataset of manually defined
token-level Levenshtein distance [31] (NTLev) between the and automatically generated paraphrases; (ii) the AppleScript
original (do ) and any paraphrased description (dp ): code used to automate the Copilot triggering; (iii) the code used
to compute the CodeBLEU and the Levenshtein distance; (iv)
TLev (do , dp )
N T Lev(do , dp ) = the dataset of 892 methods and related tests used in our study;
max({|do |, |dp |}) (v) the scripts used to automatically generate the paraphrased
with TLev representing the token-level Levenshtein distance descriptions using PEGASUS and TP; and (vi) all raw data
between the two descriptions. output of our experiments.
 Results Achieved With the Original and the Manually Paraphrased Descriptions
Original
Paraphrased Min: 0 0 1 1 0 0

Max: 1,779 2,721 1,779 2,721 271 249

652 644

112 122
96 99

32 27

FAIL PASS ERROR EMPTY ALL FAIL PASS ALL FAIL PASS

Unit Test Results CodeBLEU Levenshtein Distance on Code

Fig. 2. Results achieved by Copilot when considering the Full context code representation on paraphrasesmanual .

III. R ESULTS D ISCUSSION These findings suggest that the two paraphrasing techniques
can be adopted as testing tools to assess the robustness of
As previously explained, in RQ1 we conducted our exper- DL-based code recommenders. In particular, once established
iments both in the Full context and in the Non-full context a reference description (e.g., the original description in our
scenario. Since the obtained findings are similar, due to space study), these tools can be applied to paraphrase it and verify
limitations we only discuss in the paper the results achieved whether, using the reference and the paraphrased descriptions,
in the Full context scenario (i.e., the case in which we provide the code recommenders generate different predictions.
Copilot with all code preceding and following the method
object of the prediction). The results achieved in the Non-full
Answer to RQ0 . State-of-the-art paraphrasing techniques
context scenario are available in our replication package [6].
can be used as starting point to test the robustness of DL-
based code recommenders, since they are able to generate
A. RQ0 : Evaluation of Automated Praphrase Generators semantically equivalent descriptions of a reference text in
up to 77% of cases.
TABLE II
N UMBER OF SEMANTICALLY EQUIVALENT OR NONEQUIVALENT B. RQ1 : Robustness of GitHub Copilot
PARAPHRASED DESCRIPTIONS OBTAINED USING PEGASUS AND TP.
Performance of Copilot when using the original and the
Equivalent Nonequivalent Invalid paraphrased description as input. Fig. 2 summarizes the
performance achieved by Copilot when using the original
PEGASUS 666 (74.7%) 225 (25.2%) 1 (0.1%)
description (light blue) and the manually generated paraphrased
TP 688 (77.1%) 104 (11.7%) 100 (11.2%)
description (dark blue) as input. Similarly, we report in Fig. 3
the performance obtained when considering the paraphrases
Table II reports the number of semantically equivalent and generated with the two automated techniques, i.e., PEGASUS
nonequivalent descriptions obtained using the two state-of-the- and TP (top and bottom of Fig. 3, respectively). It is worth
art paraphrasing techniques, namely PEGASUS and Translation noticing that, in the latter, we only considered in the analysis
Pivoting (TP), together with the number of invalid paraphrases the paraphrases manually considered as equivalent in RQ0 , i.e.,
generated. Out of the 892 original descriptions on which they 666 for PEGASUS and 688 for TP.
have been run, PEGASUS generated 666 (75%) semantically A first interesting result is that, as it can be noticed
equivalent descriptions, while TP went up to 688 (77%). If from Fig. 2 and Fig. 3, the results obtained with the three
we do not consider the invalid paraphrases, i.e., the cases for methodologies are very similar. For this reason, to avoid
which the techniques do not actually provide any paraphrase, repetitions, in the following, we will mainly focus on the
the latter obtains ∼87% of correctly generated paraphrases. results obtained with the manually generated paraphrases.
 Results Achieved With the Original and the Automatically Generated Paraphrased Descriptions
Original
Pegasus Min: 0 0 1 1 0 0

Max: 1,441 2,747 1,441 2,747 271 283

479
463

88 92 87
73

26 24

FAIL PASS ERROR EMPTY ALL FAIL PASS ALL FAIL PASS

Unit Test Results CodeBLEU Levenshtein Distance on Code

Original
Min: 0 0 1 1 0 0
Translation-Pivoting
Max: 1,779 2,625 1,779 2,625 165 175

509
495

87 83
77 77

25 23

FAIL PASS ERROR EMPTY ALL FAIL PASS ALL FAIL PASS

Unit Test Results CodeBLEU Levenshtein Distance on Code

Fig. 3. Results achieved by Copilot when considering the Full context code representation on paraphrasesPEGASUS and paraphrasesTP .

Also, as we will discuss, the quality of Copilot’s recom- Only ∼13% of instances (112 and 122 depending on the
mendations is very similar when using the original and the used description) resulted in test-passing methods. While such
paraphrased descriptions. a result seems to indicate limited performance of Copilot,
In Fig. 2, the bar chart in the left side reports the number of it must be considered the difficulty of the code generation
methods recommended by Copilot (out of 892) that resulted in tasks involved in our study. Indeed, we did not ask Copilot to
failing tests, passing tests, syntactic errors, and no (i.e., empty) generate simple methods possibly implementing quite popular
recommendation. Looking at such a chart, the first thing that routines (e.g., a method to generate an MD5 hash from a string)
leaps to the eyes is the high percentage of Java methods (∼73% but rather randomly selected methods that, as shown in Table I,
for the original and ∼72% for the paraphrased description) are composed, on average, by more than 150 tokens (median =
for which Copilot was not able to synthesize a method passing 92) and have an average cyclomatic complexity of 5.3 (median
the related unit tests. = 3.0).
 Target Method Target Method

public void removeListener(IChemObjectListener col){ public static void translateAllPositive(IAtomContainer atomCon){

if (chemObjectListeners == null) { double minX = Double.MAX_VALUE;
return; double minY = Double.MAX_VALUE;
} Iterator<IAtom> atoms = atomCon.atoms().iterator();
List<IChemObjectListener> listeners = lazyChemObjectListeners(); while (atoms.hasNext()) {
if (listeners.contains(col)) { IAtom atom = (IAtom) atoms.next();
listeners.remove(col); if (atom.getPoint2d() != null) {
} if (atom.getPoint2d().x < minX) {
} minX = atom.getPoint2d().x;
}
if (atom.getPoint2d().y < minY) {
Recommended method starting from the original description minY = atom.getPoint2d().y;
}
public void removeListener(IChemObjectListener col){ PASS
if (chemObjectListeners == null) { }
return; }
} logger.debug("Translating: minx=" + minX + ", minY=" + minY);
lazyChemObjectListeners().remove(col); translate2D(atomCon, minX * -1, minY * -1);
} }

CodeBLEU: 0.45 Recommended method starting from the original description

public static void translateAllPositive PASS
(IAtomContainer atomCon){
double minX = Double.POSITIVE_INFINITY;
Fig. 4. Example of recommended method that passes the unit tests but reports double minY = Double.POSITIVE_INFINITY;
a low CodeBLEU score compared to the oracle (i.e., target method). double minZ = Double.POSITIVE_INFINITY;
for (IAtom atom : atomCon.atoms()) {
if (atom.getPoint2d() != null) {
minX = Math.min(minX, atom.getPoint2d().x);
minY = Math.min(minY, atom.getPoint2d().y);
}
Thus, we consider the successful generation of more than if (atom.getPoint3d() != null) {
minX = Math.min(minX, atom.getPoint3d().x);
110 of these methods a quite impressive result for a code minY = Math.min(minY, atom.getPoint3d().y);
minZ = Math.min(minZ, atom.getPoint3d().z);
recommender. The remaining ∼15% of instances resulted }
}
either in a parsing error (∼100 methods) or in an empty for (IAtom atom : atomCon.atoms()) {
if (atom.getPoint2d() != null) {
recommendation (∼30 methods). atom.setPoint2d(new Point2d(
atom.getPoint2d().x - minX,
The box plot in the middle part of Fig. 2 depicts the results }
atom.getPoint2d().y - minY));

achieved in terms of CodeBLEU [49] computed between if (atom.getPoint3d() != null) {

atom.setPoint3d(new Point3d(
the recommended methods and the target one (i.e., the one atom.getPoint3d().x - minX,
atom.getPoint3d().y - minY,
implemented by the original developers). Higher values indicate atom.getPoint3d().z - minZ));
}
higher similarity between the compared methods. Instead, in the }
}
right box plot, we show the normalized Levenshtein distance,
Levenshtein distance: 165
for which lower values indicate higher similarity.
For both metrics, we depict the distributions when con-
sidering all generated predictions, the ones failing tests, and Fig. 5. Example of recommended methods that pass the unit tests but would
require 165 edit actions to match the target method.
the ones passing tests. As expected, higher (lower) values of
CodeBLEU (Levenshtein distance) are associated with test-
passing methods. Indeed, for the latter, the median CodeBLEU
is ∼0.80 (Levenshtein = ∼0.10) as compared to the ∼0.40 Similarly, Fig. 5 shows an example of prediction passing the
(Levenshtein = ∼0.58) of test-failing methods. Despite such tests but that, accordingly to the Levenshtein distance, would
an expected finding, it is interesting to notice that 25% of require 165 token-level edits to match the target prediction
test-passing methods have a rather low CodeBLEU <0.50. (NTLev=63%). Differently from the previous example, it is
Fig. 4 shows an example of recommended method having clear that, in this case, the two methods do not have the same
a CodeBLEU with the target method of 0.45 and passing the behavior since the recommended one also treats 3D points,
related tests. The recommended method, while substantially while the original one only 2D points. In other words, the tests
different from the target, captures the basic logic imple- fail to capture the difference in the behavior.
mented in it. The target method first checks if the object These examples provide two interesting observations. The
chemObjectListeners is null and, if not, it proceeds re- first is that, metrics such as CodeBLEU and Levenshtein
moving from the listeners list the element matching the one distance may result in substantially wrong assessments of
provided as parameter (i.e.,col). The method synthesized by the quality of a prediction. Indeed, while the discussed
Copilot avoids the second if statement by directly performing predictions have low CodeBLEU/high Levenshtein values and,
the remove operation after the null check. thus, would be considered as unsuccessful predictions in most
Note that there the two implementations are equivalent: The of the empirical evaluations, it is clear that they are valuable
remove method of java.util.List preliminarily checks recommendations for a developer, even when not 100% correct
whether the passed element is contained in the list before (see Fig. 5). This poses questions on the usage of these metrics
removing it. While the check in the original method has in the evaluation of code recommenders. Second, also the
no functional role, together with the introduction of the testing-based evaluation shows, as expected, some limitations
listeners variable, it might have been introduced to make as in the second example, in which the two methods do not
the method more readable and self-explanatory. implement the same behavior but both pass the tests.
 As a final note, it is also interesting to observe as 25% Answer to RQ1 . Different (but semantically equivalent)
of test-failing predictions exhibit high values (>∼0.60) of natural language descriptions of the same method are likely
CodeBLEU, indicating a high code similarity that, however, to result in different code recommendations generated by
does not reflect in test-passing recommendations. DL-based code generation models. Such differences can
Impact of paraphrasing the input descriptions. Out of result in a loss of correct recommendations (∼28% of
the 892 manually paraphrased descriptions, 408 (46%) result test-passing methods can only be obtained either with the
in different code recommendations as compared to the original original or the paraphrased descriptions). These findings
description. This means that Copilot synthesizes different suggest that testing the robustness of DL-based code
methods when it is provided as input with the original recommenders may play an important role in ensuring
description and with the manually paraphrased description, their usability and in defining possible guidelines for the
which are supposed to summarize the same piece of code. developers using them.
Note that at this stage we are not focusing on the “quality”
of the obtained predictions in any way. We are just observing
that different input descriptions have indeed an impact on IV. T HREATS TO VALIDITY
the recommended code. This implies that developers using
different wordings to describe a needed method may end Threats to construct validity concern the relationship between
up with different recommendations. Such differences also the theory and what we observe. Concerning the performed
result in the potential loss of correct recommendations. Indeed, measurements, we exploit the passing tests as a proxy for the
out of the 112 test-passing predictions obtained with the correctness of the recommendations generated by Copilot. We
original description and the 122 obtained with the manually acknowledge that passing tests does not imply code correctness.
paraphrased description, only 98 are in overlap, indicating that However, this it can provide hints about the code behavior.
there are 38 correct recommendations only generated either by To partially address this threat we focused our study on
the original (14) or the paraphrased (24) description. methods having high statement coverage (median = 100%).
To have a deeper look into the 408 different predictions Also, we complemented this analysis with the CodeBLEU and
generated by Copilot with the original and the paraphrased the normalized token-level Levenshtein distance. As for the
description, the left part of Fig. 6 (light blue) shows the execution of our study, we automatically invoked Copilot rather
normalized token-level Levenshtein distance between (i) the than using it as actual developers would do: We automatically
original description and the paraphrased description (see accepted the whole recommendations and did not simulate a
the boxplot labeled with “Description”), and (ii) the method scenario in which a developer selects only parts of the provided
obtained using the original description and that recommended recommendations. In other words, while our automated script
using the paraphrased description (“Code”). The “Description” simulates a developer invoking Copilot for help, it cannot
boxplot depicts the percentage of words that must be changed simulate the different usages a developer can make of the
to convert the paraphrased description into the original one. received code recommendation.
As it can be seen, while describing the same method, the Threats to internal validity concern factors, internal to our
paraphrased descriptions can be substantially different as study, that could affect our results. While in RQ2 we had
compared to the original ones, with 50% of them requiring multiple authors inspecting the semantic equivalence of the
changes to more than 70% of their words. Similarly, the paraphrasing generated by the automated tools, in RQ1 we
different methods recommended in the 408 cases under analysis, relied on a single author to paraphrase the original description.
can be substantially different, with a median of ∼30% of code This introduces some form of subjectivity bias. However, the
tokes that must be changed to convert the recommendation whole point of our paper is that, indeed, subjectivity plays
obtained with the original description into the one obtained a role in the natural language description of a function to
using the paraphrased description (see the “Code” boxplot). generate and we are confident that the written descriptions
These findings are confirmed for the automatically para- were indeed semantically equivalent to the original one. Indeed,
phrased descriptions (see the middle and the right part of Fig. 6 the authors involved in the manual paraphrasing have an
for the results achieved with the PEGASUS and TP paraphrases, average of seven years of experience in Java. Also related
respectively). As it can be seen, the main difference as to internal validity is our choice of using the first sentence of
compared to the results of the manually paraphrased description the Doc Comments as the original natural language description.
(left part of Fig. 6) is that TP changes a substantially lower These sentences may be of low quality and not representative
number of words in the original description as compared to of how a developer would describe a method they want to
PEGASUS and to the manual paraphrasing. Such a finding automatically generate. This could substantially influence our
is expected considering that TP just translates the original findings, especially in terms of the effectiveness of Copilot (i.e.,
description back and forth from English to French, thus rarely its ability to generate test-passing methods). However, such
adding new words to the sentence, something that is likely a threat is at least mitigated by the fact that Copilot has also
to happen using PEGASUS or by paraphrasing the sentence been invoked using the manually written descriptions, showing
manually. a similar effectiveness. A final threat regards the projects used
for our study.
 408 out of 892 Manually Paraphrased Descriptions Resulted in 327 out of 666 Pegasus Paraphrased Descriptions Resulted in 328 out of 688 TP Paraphrased Descriptions Resulted in
Changes of the Recommended Code Changes of the Recommended Code Changes of the Recommended Code
L L L
e e e
v v v
e e e
n n n
s s s
h h h
t t t
e e e
i i i
n n n

D D D
i i i
s s s
t t t
a a a
n n n
c c c
e e e
Description Code Description Code Description Code

Fig. 6. Levenshtein distance between the original description and (i) the manually paraphrased descriptions (left part) and (ii) the descriptions automatically
paraphrased by PEGASUS (middle part) and Translate Pivoting (right). Similarly, we report the Levenshtein distance between the method recommended using
the original description and the three paraphrases. The latter is only computed for recommendations in which the obtained output differs.

Those are open-source projects from GitHub, and it is likely A. Empirical Studies on Code Recommenders
that at least some of them have been used for training Copilot
itself. In other words, the absolute actual effectiveness reported Proksch et al. [48] conducted an empirical study aimed
might not be reliable. However, the objective of our study is to at evaluating the performance of code recommenders when
understand the differences when different paraphrases are used suggesting method calls. Their study has been run on a real-
rather than the absolute performance of Copilot, like previous world dataset composed of developers’ interactions captured
studies did (e.g., [43]). in the IDE. Results showed that commonly used evaluation
techniques based on synthetic datasets extracted by mining
Threats to external validity are related to the possibility to
released code underperform due to a context miss.
generalize our results. Our study has been run on 892 methods
we carefully selected as explained in Section II-A. Rather than On a related research thread, Hellendoorn et al. [20]
going large-scale, we preferred to focus on methods having compared code completion models on both real-world and
a high test coverage and a verbose first sentence in the Doc synthetic datasets. Confirming what observed by Proksch et al.,
Comment. Larger investigations are needed to corroborate they found that the evaluated tools are less accurate on the
or contradict our findings. Similarly, we only focused on real-world dataset, thus concluding that synthetic benchmarks
Java methods, given the effort required to implement the are not representative enough. Moreover, they found that
toolchain needed for our study, and in particular the script the accuracy of code completion tools substantially drops in
to automatically invoke Copilot and parse its output. Running challenging completion scenarios, in which developers would
the same experiment with other languages is part of our future need them the most.
agenda. Mărăs, oiu et al. [37] analyzed how practitioners rely on
code completion during software development. The results
showed that the users actually ignore many synthesized
V. R ELATED W ORK suggestions. Such a finding has been corroborated by Arrebola
and Junior [9], who stressed the need for augmenting code
Recommender systems for software developers are tools recommender systems with the development’s context.
supporting practitioners in daily activities [38], [51], such Jin and Servant [26] and Li et al. [33] investigated the hidden
as documentation writing and retrieval [64], [39], [40], [24], costs of code recommendations. Jin and Servant found that
refactoring [11], [55], bug triaging [54], [63], bug fixing [30], IntelliSense, a code completion tool, sometimes underperforms
[58], [34], etc. Among those, code recommenders, such as code by providing the suitable recommendation far from the top of
completion tools, have became a crucial feature of modern the recommended list of solutions. Consequently, developers
Integrated Development Environments (IDEs) and support in are discouraged from picking the right suggestion. Li et al.,
speeding up code development by suggesting the developers aware of this potential issue, conducted a coding experiment in
code they are likely to write [12], [29], [16]. Given the empirical which they try to predict whether correct results are generated
nature of our work, that focuses on investigating a specific by code completion models, showing that their approach can
aspect of code recommenders, in this section we do not discuss reduce the percentage of false positives up to 70%.
all pervious works proposing novel or improving existing code Previous studies also assessed the actual usefulness of these
recommenders (see e.g., [64], [39], [40], [30], [58], [34], [61], tools. Xu et al. [65] ran a controlled experiment with 31
[44], [36], [28], [7], [29], [27], [60], [57]). Instead, we focus on developers who were asked to complete implementation tasks
empirical studies looking at code recommenders from different with and without the support of two code recommenders. They
perspectives (Section V-A) and on studies specifically focused found a marginal gain in developers’ productivity when using
on GitHub Copilot (Section V-B). the code recommenders.
 Ciniselli et al. [15] empirically evaluated the performance of Two previous studies aimed at evaluating the security of
two state-of-the-art Transformer-based models in challenging the solutions recommended by Copilot. Hammond et al. [47]
coding scenarios, for example, when the code recommender investigated the likelihood of receiving from Copilot recom-
is required to generate an entire code block (e.g., the body mendations including code affected by security vulnerabilities.
of a for loop). The two experimented models, RoBERTa They observed that vulnerable code is recommended in 40%
and Text-To-Text Transfer Transformer (T5), achieved good of cases out of the completion scenarios they experimented
performance (∼69% of accuracy) in the more classic code with. On a similar note, Sobania et al. [52] evaluated GitHub
completion scenario (i.e., predicting few tokens needed to Copilot on standard program synthesis benchmark problems
finalize a statement), while reported a substantial drop of and compared the achieved results with those from the genetic
accuracy (∼29%) when dealing with the previously described programming literature. The authors found that the performance
more complex block-level completions. of the two approaches are comparable. However, approaches
Our study is complementary to the ones discussed above. based on genetic programming are not mature enough to be
Indeed, we investigate the robustness of DL-based code deployed in practice, especially due to the time they require to
recommenders supporting what it is know in the literature as synthesize solutions. In our study, we do not focus on security,
“natural language to source code translation”. We show that but only on the correctness of the suggested solutions.
semantically equivalent code descriptions can result in different Albert Ziegler, in a blog post about GitHub Copilot2
recommendations, thus posing questions on the usability of investigated the extent to which the tool suggestions are copied
these tools. from the training set they used. Ziegler reports that Copilot
rarely recommends verbatim copies of code taken from the
B. Empirical Studies on GitHub Copilot training set.

GitHub Copilot has been recently introduced as the state- VI. C ONCLUSIONS AND F UTURE W ORK
of-the-art code recommender, and advertised as an “AI pair
programmer” [1], [22]. Since its release, researchers started We investigated the extent to which DL-based code recom-
investigating its capabilities. menders tend to synthesize different code components when
Most of the previous research aimed at evaluating the starting from different but semantically equivalent natural
impact of GitHub Copilot on developers’ productivity and its language descriptions. We selected GitHub Copilot as the tool
effectiveness (in terms of correctness of the provided solutions). representative of the state-of-the-art and asked it to generate 892
Imai [25] investigated to what extent Copilot is actually a non-trivial Java methods starting from their natural language
valid alternative to a human pair programmer. They observed description. For each method in our dataset we asked Copilot
that Copilot results in increased productivity (i.e., number of to synthesize it using: (i) the original description, extracted
added lines of code), but decreased quality in the produced as the first sentence in the Javadoc; and (ii) paraphrased
code. Ziegler et al. [67] conducted a case study in which they descriptions. We did this both by manually modifying the
investigated whether usage measurements about Copilot can original description and by using automated paraphrasing tools,
predict developers’ productivity. They found that the acceptance after having assessed their reliability in this context.
rate of the suggested solutions is the best predictor for perceived We found that in ∼46% of cases semantically equivalent but
productivity. Vaithilingam et al. [59] ran an experiment with different method descriptions result in different code recom-
24 developers to understand how Copilot can help developers mendations. We observed that some correct recommendations
complete programming tasks. Their results show that Copilot can only be obtained using one of the semantically equivalent
does not improve the task completion time and success rate. descriptions as input.
However, developers report that they prefer to use Copilot Our results highlight the importance of providing a proper
because it recommends code that can be used as a starting code description when asking DL-based recommenders to
point and saves the effort of searching online. synthesize code. In the new era of AI-supported programming,
Nguyen and Nadi [43] used LeetCode questions as input developers must learn how to properly describe the code
to Copilot to evaluate the solutions provided for several components they are looking for to maximize the effectiveness
programming languages in terms of correctness — by running of the AI support.
the test cases available in LeetCode — and understandability Our future work will focus on answering our first research
— by computing their Cyclomatic Complexity and Cognitive question in vivo rather than in silico. In other words, we aim
Complexity [13]. They found notable differences among the at running a controlled experiment with developers to assess
programming languages in terms of correctness (between the impact of the different code descriptions they write on
57%, for Java, and 27%, for JavaScript). On the other the received recommendations. Also, we will investigate how
hand, Copilot generates solutions with low complexity for to customize the automatic paraphrasing techniques to further
all the programming languages. While we also measure the improve their performance on software-related text (such as
effectiveness of the solutions suggested by Copilot, our main methods’ descriptions).
focus is on understanding its robustness when different inputs
are provided. 2 https://docs.github.com/en/github/copilot/research-recitation
 ACKNOWLEDGMENTS [23] X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,”
in Proceedings of the 26th Conference on Program Comprehension, ICPC
This project has received funding from the European 2018, Gothenburg, Sweden, May 27-28, 2018, F. Khomh, C. K. Roy, and
Research Council (ERC) under the European Union’s Horizon J. Siegmund, Eds. ACM, 2018, pp. 200–210.
[24] ——, “Deep code comment generation,” ser. ICPC ’18, 2018.
2020 research and innovation programme (grant agreement No. [25] S. Imai, “Is github copilot a substitute for human pair-programming?
851720). an empirical study,” in 2022 IEEE/ACM 44th International Conference
on Software Engineering: Companion Proceedings (ICSE-Companion).
R EFERENCES IEEE, 2022, pp. 319–321.
[26] X. Jin and F. Servant, “The hidden cost of code completion: Understand-
[1] “Github copilot https://copilot.github.com.” ing the impact of the recommendation-list length on its efficiency,” in
[2] Jacoco, https://www.eclemma.org/jacoco/. Proceedings of the 15th International Conference on Mining Software
[3] Java Parser, https://github.com/javaparser/javaparser. Repositories, 2018, pp. 70–73.
[4] jUnit, https://junit.org/junit5/. [27] R. Karampatsis and C. A. Sutton, “Maybe deep neural networks are
[5] PEGASUS fine-tuned for paraphrasing, https://huggingface.co/tuner007/ the best choice for modeling source code,” CoRR, vol. abs/1903.05734,
pegasus_paraphrase. 2019. [Online]. Available: http://arxiv.org/abs/1903.05734
[6] Replication package, https://github.com/antonio-mastropaolo/ [28] J. Kim, S. Lee, S. Hwang, and S. Kim, “Adding examples into java
robustness-copilot. documents,” in 2009 IEEE/ACM International Conference on Automated
[7] M. Allamanis, E. T. Barr, C. Bird, and C. Sutton, “Learning natural coding Software Engineering, 2009, pp. 540–544.
conventions,” in Proceedings of the 22nd ACM SIGSOFT International [29] S. Kim, J. Zhao, Y. Tian, and S. Chandra, “Code prediction by feeding
Symposium on Foundations of Software Engineering, ser. FSE 2014, trees to transformers,” arXiv preprint arXiv:2003.13848, 2020.
2014, pp. 281–293. [30] C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, “A systematic
[8] U. Alon, R. Sadaka, O. Levy, and E. Yahav, “Structural language models study of automated program repair: Fixing 55 out of 105 bugs for $8
of code,” arXiv, pp. arXiv–1910, 2019. each,” in 2012 34th International Conference on Software Engineering
[9] F. V. Arrebola and P. T. A. Junior, “On source code completion assistants (ICSE), 2012, pp. 3–13.
and the need of a context-aware approach,” in International Conference [31] V. I. Levenshtein et al., “Binary codes capable of correcting deletions,
on Human Interface and the Management of Information. Springer, insertions, and reversals,” in Soviet physics doklady, vol. 10, no. 8. Soviet
2017, pp. 191–201. Union, 1966, pp. 707–710.
[10] M. Asaduzzaman, C. K. Roy, K. A. Schneider, and D. Hou, “Context- [32] B. Li, M. Yan, X. Xia, X. Hu, G. Li, and D. Lo, “Deepcommenter: a
sensitive code completion tool for better api usability,” in 2014 IEEE deep code comment generation tool with hybrid lexical and syntactical
International Conference on Software Maintenance and Evolution, 2014, information,” in ESEC/FSE ’20: 28th ACM Joint European Software
pp. 621–624. Engineering Conference and Symposium on the Foundations of Software
[11] G. Bavota, A. D. Lucia, A. Marcus, and R. Oliveto, “Automating extract Engineering, Virtual Event, USA, November 8-13, 2020, P. Devanbu,
class refactoring: an improved method and its evaluation,” Empir. Softw. M. B. Cohen, and T. Zimmermann, Eds. ACM, 2020, pp. 1571–1575.
Eng., vol. 19, no. 6, pp. 1617–1664, 2014. [33] J. Li, R. Huang, W. Li, K. Yao, and W. Tan, “Toward less hidden
[12] M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples cost of code completion with acceptance and ranking models,” in 2021
to improve code completion systems,” in Proceedings of the 7th Joint IEEE International Conference on Software Maintenance and Evolution
Meeting of the European Software Engineering Conference and the ACM (ICSME). IEEE, 2021, pp. 195–205.
SIGSOFT Symposium on The Foundations of Software Engineering, ser. [34] Y. Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code
ESEC/FSE 2009, 2009, pp. 213–222. transformation learning for automated program repair,” in Proceedings of
[13] G. A. Campbell, “Cognitive complexity: An overview and evaluation,” the ACM/IEEE 42nd International Conference on Software Engineering,
in Proceedings of the 2018 international conference on technical debt, ser. ICSE ’20, 2020, p. 602?614.
2018, pp. 57–58. [35] B. Lin, F. Zampetti, G. Bavota, M. D. Penta, M. Lanza, and R. Oliveto,
[14] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, “Sentiment analysis for software engineering: how far can we go?”
H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large in Proceedings of the 40th International Conference on Software
language models trained on code,” arXiv preprint arXiv:2107.03374, Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018,
2021. pp. 94–104.
[15] M. Ciniselli, N. Cooper, L. Pascarella, A. Mastropaolo, E. Aghajani, [36] F. Liu, G. Li, Y. Zhao, and Z. Jin, “Multi-task learning based pre-
D. Poshyvanyk, M. D. Penta, and G. Bavota, “An empirical study on the trained language model for code completion,” in Proceedings of the 35th
usage of transformer models for code completion,” IEEE Transactions IEEE/ACM International Conference on Automated Software Engineering,
on Software Engineering, no. 01, pp. 1–1, 5555. ser. ASE 2020. Association for Computing Machinery, 2020.
[16] M. Ciniselli, N. Cooper, L. Pascarella, D. Poshyvanyk, M. Di Penta, and [37] M. Mărăs, oiu, L. Church, and A. Blackwell, “An empirical investigation
G. Bavota, “An empirical study on the usage of bert models for code of code completion usage by professional software developers,” in Pro-
completion,” in Proceedings of the 18th Working Conference on Mining ceedings of the 26th Annual Workshop of the Psychology of Programming
Software Repositories, ser. MSR ’21, 2021, p. To Appear. Interest Group, 2015.
[17] O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github [38] C. McMillan, D. Poshyvanyk, M. Grechanik, Q. Xie, and C. Fu,
for msr studies,” in 2021 IEEE/ACM 18th International Conference on “Portfolio: Searching for relevant functions and their usages in millions
Mining Software Repositories (MSR). IEEE, 2021, pp. 560–564. of lines of code,” ACM Trans. Softw. Eng. Methodol., vol. 22, no. 4, pp.
[18] N. A. Ernst and G. Bavota, “Ai-driven development is here: Should you 37:1–37:30, 2013.
worry?” IEEE Softw., vol. 39, no. 2, pp. 106–110, 2022. [39] L. Moreno, G. Bavota, M. Di Penta, R. Oliveto, and A. Marcus, “How can
[19] V. J. Hellendoorn and P. Devanbu, “Are deep neural networks the best i use this method?” in Proceedings of the 37th International Conference
choice for modeling source code?” in Proceedings of the 2017 11th Joint on Software Engineering - Volume 1, ser. ICSE ’15, 2015, p. 880?890.
Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017, [40] L. Moreno, G. Bavota, M. D. Penta, R. Oliveto, A. Marcus, and
2017, p. 763?773. G. Canfora, “Arena: An approach for the automated generation of release
[20] V. J. Hellendoorn, S. Proksch, H. C. Gall, and A. Bacchelli, “When notes,” IEEE Transactions on Software Engineering, vol. 43, no. 2, pp.
code completion fails: A case study on real-world completions,” in 106–127, 2017.
2019 IEEE/ACM 41st International Conference on Software Engineering [41] A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “A large-scale study on
(ICSE). IEEE, 2019, pp. 960–970. repetitiveness, containment, and composability of routines in open-source
[21] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the projects,” in Proceedings of the IEEE/ACM 13th Working Conference on
naturalness of software,” in Proceedings of the 34th International Mining Software Repositories (MSR 2016), 2016, pp. 362–373.
Conference on Software Engineering, ser. ICSE 2012. IEEE Press, [42] A. T. Nguyen, T. T. Nguyen, H. A. Nguyen, A. Tamrawi, H. V. Nguyen,
2012, pp. 837–847. J. Al-Kofahi, and T. N. Nguyen, “Graph-based pattern-oriented, context-
[22] G. D. Howard, “Github copilot: Copyright, fair use, creativity, transfor- sensitive source code completion,” in 2012 34th International Conference
mativity, and algorithms,” 2021. on Software Engineering (ICSE), 2012, pp. 69–79.
[43] N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s [62] M. White, C. Vendome, M. Linares-Vásquez, and D. Poshyvanyk,
code suggestions,” in 2022 IEEE/ACM 19th International Conference on “Toward deep learning software repositories,” in Proceedings of the
Mining Software Repositories (MSR). IEEE, 2022, pp. 1–5. 12th Working Conference on Mining Software Repositories, ser. MSR
[44] T. Nguyen, P. C. Rigby, A. T. Nguyen, M. Karanfil, and T. N. Nguyen, ’15. Piscataway, NJ, USA: IEEE Press, 2015, pp. 334–345. [Online].
“T2api: Synthesizing api code usage templates from english texts with Available: http://dl.acm.org/citation.cfm?id=2820518.2820559
statistical translation,” in Proceedings of the 2016 24th ACM SIGSOFT [63] X. Xia, D. Lo, Y. Ding, J. M. Al-Kofahi, T. N. Nguyen, and X. Wang,
International Symposium on Foundations of Software Engineering, ser. “Improving automated bug triaging with specialized topic model,” IEEE
FSE 2016, 2016, p. 1013?1017. Transactions on Software Engineering, vol. 43, no. 3, pp. 272–297, 2017.
[45] H. Niu, I. Keivanloo, and Y. Zou, “Api usage pattern recommendation [64] T. Xie and J. Pei, “Mapo: Mining api usages from open source
for software development,” Journal of Systems and Software, vol. 129, repositories,” ser. MSR ’06, 2006.
pp. 127–139, 2017. [65] F. F. Xu, B. Vasilescu, and G. Neubig, “In-ide code generation from
[46] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for natural language: Promise and challenges,” 2021.
automatic evaluation of machine translation,” in Proceedings of the 40th [66] J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu, “Pegasus: Pre-training with
annual meeting of the Association for Computational Linguistics, 2002, extracted gap-sentences for abstractive summarization,” 2019.
pp. 311–318. [67] A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister,
[47] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “An G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural
empirical cybersecurity evaluation of github copilot’s code contributions,” code completion,” in Proceedings of the 6th ACM SIGPLAN International
arXiv preprint arXiv:2108.09293, 2021. Symposium on Machine Programming, 2022, pp. 21–29.
[48] S. Proksch, S. Amann, S. Nadi, and M. Mezini, “Evaluating the
evaluations of code recommender systems: a reality check,” in 2016 31st
IEEE/ACM International Conference on Automated Software Engineering
(ASE). IEEE, 2016, pp. 111–121.
[49] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan,
M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic
evaluation of code synthesis,” CoRR, vol. abs/2009.10297, 2020.
[Online]. Available: https://arxiv.org/abs/2009.10297
[50] R. Robbes and M. Lanza, “Improving code completion with program
history,” Automated Software Engineering, vol. 17, no. 2, pp. 181–212,
2010.
[51] M. P. Robillard, W. Maalej, R. J. Walker, and T. Zimmermann,
Recommendation Systems in Software Engineering. Springer Publishing
Company, Incorporated, 2014.
[52] D. Sobania, M. Briesch, and F. Rothlauf, “Choose your programming
copilot: A comparison of the program synthesis performance of github
copilot and genetic programming,” arXiv preprint arXiv:2111.07875,
2021.
[53] A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intelli-
code compose: Code generation using transformer,” arXiv preprint
arXiv:2005.08025, 2020.
[54] A. Tamrawi, T. T. Nguyen, J. M. Al-Kofahi, and T. N. Nguyen, “Fuzzy
set and cache-based approach for bug triaging,” in Proceedings of the
19th ACM SIGSOFT Symposium and the 13th European Conference
on Foundations of Software Engineering, ser. ESEC/FSE ’11, 2011, p.
365?375.
[55] N. Tsantalis, T. Chaikalis, and A. Chatzigeorgiou, “Ten years of jdeodor-
ant: Lessons learned from the hunt for smells,” in 25th International
Conference on Software Analysis, Evolution and Reengineering, SANER
2018, R. Oliveto, M. D. Penta, and D. C. Shepherd, Eds. IEEE Computer
Society, 2018, pp. 4–14.
[56] Z. Tu, Z. Su, and P. Devanbu, “On the localness of software,” in
Proceedings of the 22nd ACM SIGSOFT International Symposium on
Foundations of Software Engineering, ser. FSE 2014. New York,
NY, USA: Association for Computing Machinery, 2014, p. 269–280.
[Online]. Available: https://doi.org/10.1145/2635868.2635875
[57] M. Tufano, D. Drain, A. Svyatkovskiy, and N. Sundaresan, “Generating
accurate assert statements for unit test cases using pretrained transformers,”
CoRR, vol. abs/2009.05634, 2020.
[58] M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and
D. Poshyvanyk, “An empirical study on learning bug-fixing patches
in the wild via neural machine translation,” ACM Trans. Softw. Eng.
Methodol., vol. 28, no. 4, pp. 19:1–19:29, 2019.
[59] P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs.
experience: Evaluating the usability of code generation tools powered
by large language models,” in CHI Conference on Human Factors in
Computing Systems Extended Abstracts, 2022, pp. 1–7.
[60] C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On
learning meaningful assert statements for unit test cases,” in Proceedings
of the 42nd International Conference on Software Engineering, ICSE
2020, 2020, p. To Appear.
[61] F. Wen, E. Aghajani, C. Nagy, M. Lanza, and G. Bavota, “Siri, write the
next method,” in 43rd IEEE/ACM International Conference on Software
Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 2021,
pp. 138–149.