IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO.

7, JULY 2022 2339

CTOS: Compiler Testing for Optimization

Sequences of LLVM
He Jiang , Member, IEEE, Zhide Zhou , Zhilei Ren, Jingxuan Zhang , and Xiaochen Li

Abstract—Optimization sequences are often employed in compilers to improve the performance of programs, but may trigger critical
compiler bugs, e.g., compiler crashes. Although many methods have been developed to automatically test compilers, no systematic
work has been conducted to detect compiler bugs when applying arbitrary optimization sequences. To resolve this problem, two main
challenges need to be addressed, namely the acquisition of representative optimization sequences and the selection of representative
testing programs, due to the enormous number of optimization sequences and testing programs. In this study, we propose CTOS, a
novel compiler testing method based on differential testing, for detecting compiler bugs caused by optimization sequences of LLVM.
CTOS first leverages the technique Doc2Vec to transform optimization sequences into vectors to capture the information of
optimizations and their orders simultaneously. Second, a method based on the region graph and call relationships is developed in
CTOS to construct the vector representations of the testing program, such that the semantics and the structure information of programs
can be captured simultaneously. Then, with the vector representations of optimization sequences and testing programs, a “centroid”
based selection scheme is proposed to address the above two challenges. Finally, CTOS takes in the representative optimization
sequences and testing programs as inputs, and tests each testing program with all the representative optimization sequences. If there
is an output that is different from the majority of others of a given testing program, then the corresponding optimization sequence is
deemed to trigger a compiler bug. Our evaluation demonstrates that CTOS significantly outperforms the baselines by up to 24:76%

50:57% in terms of the bug-finding capability on average. Within seven month evaluations on LLVM, we have reported 104 valid bugs
within 5 types, of which 21 have been confirmed or fixed. Most of those bugs are crash bugs (57) and wrong code bugs (24). 47 unique
optimizations are identified to be faulty and 15 of them are loop related optimizations.

Index Terms—Compiler testing, optimization sequences, LLVM, program representation, software testing

1 INTRODUCTION optimizing programs with some optimization sequences,

which may lead to unintended application behavior and
an important infrastructure for software develop-
A S
ment, compilers (e.g., GCC, LLVM) usually provide
many optimizations to improve the performance of pro-
disasters, especially for safety-critical domains [9]. In the
literature, Purini et al. [4] state that “there are optimization
sequences which crash the compiler”. Fursin et al. [1] and
grams, e.g., running time, code size, and throughput. With
Ansel et al. [2] have also reported some bugs that lead com-
these optimizations, many studies [1], [2], [3], [4], [5], [6]
pilers to crash or produce incorrect program execution
have been conducted on the compiler phase-ordering prob-
when applying some optimization sequences. We present
lem [7], [8], namely how to select good optimization
two examples as well in Section 2.1 that illustrate both a
sequences (i.e., a set of compiler optimizations in a certain
crash bug and a wrong code bug of LLVM caused by cer-
order) to achieve satisfactory performance for programs.
tain optimization sequences.
However, potential compiler bugs may be triggered when
Although some methods [9], [10], [11], [12], [13], [14],
[15], [16], [17], [18] have been proposed to automatically test
compilers in recent years, no systematic investigation has
 He Jiang is with the School of Software, and Key Laboratory for Ubiquitous
Network and Service Software of Liaoning Province, Dalian University of
been conducted on compiler bugs caused by optimization
Technology, Dalian 116024, China, and also with the DUT Artificial Intel- sequences. To date, these existing methods to test compilers
ligence Institute, Dalian 116024, China. E-mail: [email protected]. can be roughly divided into three types, namely, Random-
 Zhide Zhou and Zhilei Ren are with the School of Software, and Key Labo- ized Differential Testing (RDT) [10], [11], [12], [13], [14], Dif-
ratory for Ubiquitous Network and Service Software of Liaoning Province,
Dalian University of Technology, Dalian 116024, China. ferent Optimization Levels (DOL, a variant of RDT), and
E-mail: [email protected], [email protected]. Equivalence Modulo Inputs (EMI) [9], [15], [16], [17], [18],
 Jingxuan Zhang is with the College of Computer Science and Technology, [19], [20]. Given a program, RDT detects compiler bugs by
Nanjing University of Aeronautics and Astronautics, Nanjing 210016,
comparing the outputs of distinct compilers implemented
China. E-mail: [email protected].
 Xiaochen Li is with the SnT Centre for Security, Reliability and Trust, based on the same specification. In contrast, DOL and EMI
University of Luxembourg, 4365 Esch-sur-Alzette, Luxembourg. only require one compiler. DOL determines whether a com-
E-mail: [email protected]. piler contains bugs by comparing the outputs of a program
Manuscript received 5 Mar. 2020; revised 6 Feb. 2021; accepted 6 Feb. 2021. optimized by predefined compiler optimization sequences
Date of publication 11 Feb. 2021; date of current version 18 July 2022. (e.g., sequences represented by compiler flags O1, O2 and
(Corresponding author: He Jiang.)
Recommended for acceptance by G. Fraser. O3), while EMI compares the outputs of some equivalent
Digital Object Identifier no. 10.1109/TSE.2021.3058671 variants of a seed program [19]. However, these existing
0098-5589 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See ht_tps://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2340 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

methods only focus on detecting bugs related to predefined

optimization sequences in certain orders, rather than arbi-
trary optimization sequences.
In this paper, we investigate how to detect compiler bugs
caused by optimization sequences for LLVM [21]. LLVM is a
mature and widely used compiler infrastructure. Hundreds
of analysis and transformation optimizations have been
implemented in LLVM [22]. Moreover, many compilers
(e.g., Clang [23], Rust [24], Swift [25], and WebAssembly
[26]) of different programming languages and tools (e.g.,
Klee [27], Phasar [28]) have been implemented based on
LLVM. In contrast to DOL and EMI, two challenges need to
be addressed. On the one hand, the number of potential opti-
mization sequences is extremely huge, such that it is intracta- Fig. 1. Programs in crash bug 41294 and wrong code bug 41720 of
LLVM (trunk 355281). (https://bugs.llvm.org/show_bug.cgi?id=41294,
ble to test each optimization sequence. For example, more https://bugs.llvm.org/show_bug.cgi?id=41720.).
than 150 optimizations exist in LLVM, and there are 15030
optimization sequences1 to be tested, when the sequence Section 4), of which 21 have been confirmed or fixed. Most of
length is set to be 30. Additionally, similar optimization those bugs are crash bugs (57) and wrong code bugs (24).
sequences may trigger duplicate bugs for a given testing pro- 47 unique optimizations have been identified to be faulty
gram. Hence, we need to address the challenge of the acquisi- and 15 of them are loop related optimizations.
tion of representative optimization sequences, namely, how to In summary, the main contributions of this paper are as
acquire a set of representative optimization sequences to follows:
accelerate the testing of LLVM. On the other hand, there is
an almost infinite number of testing programs which may 1) In this paper, we first investigate how to detect com-
incur compiler bugs, and it is impossible to test them all. piler bugs caused by optimization sequences. As to
Moreover, distinct testing programs may incur duplicate the best of our knowledge, this is the first systematic
compiler bugs. Thus, we need to address the second chal- work for this problem.
lenge, namely the selection of representative testing programs. 2) We present a novel method CTOS to find compiler
In this paper, we propose a novel compiler testing method bugs by optimization sequences. In CTOS, efficient
based on differential testing, called Compiler Testing for vector representation methods and a selection
Optimization Sequences (CTOS), for detecting bugs caused scheme are designed to address two challenges,
by optimization sequences for LLVM. CTOS first leverages namely the acquisition of representative optimiza-
the representation technique Doc2Vec [29] in natural lan- tion sequences and the selection of representative
guage processing to transform optimization sequences into testing programs.
vectors on the basis of its optimizations and their orders. 3) Extensive testing efforts have been conducted on
Then, a method based on the region graph and call relation- LLVM. We have reported 104 valid bugs within 5
ships of programs is presented to construct the vector repre- types, of which 21 have been confirmed or fixed.
sentations of testing programs. Next, with the vector These bugs cover 47 unique optimizations, where 15
representations of optimization sequences and testing pro- of them are loop related optimizations.
grams, we present a centroid based selection scheme to select The remainder of this paper is organized as follows.
representative optimization sequences and testing programs, Section 2 shows the background of our study. Section 3
thus addressing the above two challenges. Finally, CTOS uti- presents the testing process and the proposed methods. Next,
lizes differential testing to validate all representative optimi- we describe the evaluation results in Section 4. Section 5
zation sequences and testing programs. Specifically, CTOS presents some discussions about this paper. The threats to
takes in representative optimization sequences and testing validity and related work are described in Sections 6 and 7.
programs as inputs and tests each testing program with all Section 8 concludes our study.
the representative optimization sequences. If there is an out-
put that is different from the majority of others of a given test-
2 BACKGROUND
ing program, then the corresponding optimization sequence
is deemed to trigger a compiler bug. 2.1 Illustrative Examples
To evaluate the effectiveness of CTOS, we conduct a com- In this subsection, we present two concrete examples to
parative experiment that compares CTOS with 14 baselines. illustrate the compiler bugs in LLVM caused by optimiza-
Our evaluation demonstrates that CTOS significantly out- tion sequences. Note that these examples in this section
performs the baselines by detecting 24:76%
50:57% more only aim to show the phenomenon of the compiler bugs
bugs on average. In addition, we conduct an experiment for introduced by optimization sequences.
seven months to show the bug-finding capability of CTOS in Fig. 1a shows a program that triggers a crash bug by the
practice. After running CTOS on LLVM for seven months, loop-vectorizer2 optimization when the program is optimized
we have reported 104 valid bugs within 5 types (see by the optimization sequence “-functionattrs -loop-rotate -licm
-sroa -loop-vectorize.” An assertion fails when loop-vectorizer
1. It is valid for an optimization to appear multiple times in an opti-
mization sequence. 2. https://llvm.org/doxygen/LoopVectorize_8cpp_source.html.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2341

works on the Intermediate Representation (IR) file optimized

by “-functionattrs -loop-rotate -licm -sroa.” This bug occurs
because the loop invariant operands are scalarized, but they
are used outside the loop and should be ignored when com-
puting the scalarization overhead.3 However, when we
delete any optimization from {functionattrs, loop-rotate, licm,
sroa}, or change the order, or only use loop-vectorizer to opti-
mize the program, this bug disappears. In addition, there
does not exist any bug when the program is optimized by the
default standard optimization levels, e.g., O1, O2, and O3
provided by LLVM.
The second program in Fig. 1b incurs a wrong code bug
of LLVM when applying the optimization sequence “-gvn Fig. 2. The framework of DMPV model for Doc2Vec.
-licm -loop-rotate -loop-vectorize”. Variable a is a global vari-
able and is initialized to 0. If the program executes correctly, the default model in the tool that we use to implement our
the output should be 9. However, after optimization, the methods.
output is 15, which is caused by incorrectly optimizing the Fig. 2 shows the framework of DMPV, which is similar to
calculation of variable a on line 5 with loop-vectorizer. Similar the framework of CBOW. In Fig. 2, the top half of the figure
to the first example, this bug cannot be reproduced when is the framework of CBOW. The only change for the DMPV
we remove any optimization in the optimization sequence, model is the additional document token [29]. DMPV con-
or change the order of them, or use the default standard sists of an input layer, an output layer, and a hidden layer.
optimization levels. The hidden layer h is a 1  V vector to represent words in a
From the above two examples, it is evident that optimiza- low dimensional space. Each column in the matrix WVocV is
tion sequences may heavily affect the behavior of LLVM a unique vector representation of a word; and the matrix
and potential bugs of LLVM may be exposed by employing WV0 Voc is the parameter matrix of h; each column in the
certain optimization sequences to optimize programs. This matrix DT V is the unique vector representation of a docu-
is important for ensuring the correctness of the behavior of ment. V is pre-defined by users. T is the documents and Voc
a program when developers tune the compiler optimiza- is the vocabulary of the training set.
tions for specific programs to achieve better performance Initially, the matrix WVocV , WV0 Voc , and DT V are ran-
(e.g., size or speed), especially for safety-critical programs. domly initialized. Each word wx in Voc is represented as a
Compiler testing, currently, is the dominant technique one-hot vector with the size of jVocj, which is a zero vector
for detecting compiler bugs due to its simplicity and easy with the exception of a single 1 to uniquely identify the
application. However, there does not exist any work that word. The document di in T is represented in a similar way
focuses on finding compiler bugs caused by optimization to the word.
sequences. Therefore, this motivates us to detect bugs of Given a document di and its word sequences, DMPV
LLVM by testing it with arbitrary optimization sequences. tries to predict the center word with its surrounding context
To achieve this goal, we need to address two challenges, in a fixed window size k by using the vector representation
namely the acquisition of representative optimization of a document as the context information [29]. Specifically,
sequences and the selection of representative testing pro- DMPV takes in the vectors of the surrounding words wkx ¼
grams, due to the enormous number of optimization fwxk ; . . . ; wx1 ; wxþ1 ; . . . ; wxþk g in a 2k sized window and
sequences and testing programs. the vector of the corresponding document di as inputs.
Then the vector of the center word wx is the expected out-
put. For example, if we want to obtain the vector of word
2.2 Doc2Vec ’sat’ and k ¼ 2, the surrounding words are ’the’, ’cat’, ’on’,
Doc2Vec is a fundamental component to represent the opti- and ’the’. Based on WVocV and DT V , the inputs are propa-
mization sequences and testing programs in our study. It gated to the hidden layer
was originally designed to transform documents (sentences
1

 
or paragraphs) into low-dimensional vectors [29]. Doc2Vec h¼ sum wkx  WVocV þ di  DT V :
is an extension to Word2Vec [30] to extend the learning of 2k þ 1
embeddings from words to word sequences. Similar to Then, the prediction of wx is typically done via softmax
Word2Vec that has two models, i.e., Continuous Bag-Of- function
Words model (CBOW) and continuous Skip-gram model
(Skip-gram), there are also two approaches within Doc2Vec, wx ¼ Softmaxðh  WV0 Voc Þ:
namely, Distributed Bag-Of-Words version of Paragraph
Vector (DBOW) and Distributed Memory model of Para- Finally, the objective of the DMPV is to maximize the aver-
graph Vectors (DMPV) [29]. DBOW and DMPV work in a age log probability
similar way as Skip-gram and CBOW, respectively. We take
1 X
jVocj
the DMPV model as an example to explain Doc2Vec as it is
 
log p wx wkx :
jVocj x¼1
3. https://reviews.llvm.org/D59995.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2342 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

corresponding IR of a program. For the back-end, many

tools are utilized to generate the final executables from IRs,
such as assembler and linker. For simplicity, we also use
Clang as a driver to schedule and execute the tools (e.g.,
linker) of back-end, and it can take in LLVM IRs to generate
executables. The optimizer Opt of LLVM is in charge of
scheduling and executing optimizations.
CTOS is composed of 5 steps. (1) The first step is to select
representative optimization sequences and testing pro-
grams. (2) Then, the front-end of a compiler is used to emit
the IR file of a given testing program without optimizations.
(3) The third step is to optimize the IR produced in the pre-
vious step using the optimizer Opt of LLVM with the
selected optimization sequences. For the IR of a testing pro-
gram, if there are n selected optimization sequences, n opti-
mized IRs (i.e., IRopt1 , IRopt2 , . . . , IRoptn ) will be produced
by the optimizer Opt. (4) In the fourth step, the n optimized
Fig. 3. Framework of CTOS for catching compiler bugs caused by optimi- IRs are loaded by the back-end of a compiler to generate n
zation sequences of LLVM. executables (i.e., exe1 , exe2 , . . . , exen ). (5) The final step is to
obtain the outputs (i.e., O1 , O2 , . . . , On ) of n executables and
After training, the column vectors in WVocV and DT V are compare them to determine whether there are bugs. The
the final vector representation of the words and documents, outputs may be different, but the majority of them should
respectively. be identical. Thus, if there is an output Oi that is different
Unlike Word2Vec that only learns the vector representa- from the majority of {O1 , O2 , . . . , On }, 1  i  n, then the ith
tions of words, Doc2Vec takes word order into consider- optimization sequence is deemed to trigger a compiler bug
ation and can learn the vector representations of word for the given testing program.
sequences. Thus, we adopt Doc2Vec to represent the optimi- From Fig. 3, the first step clearly is the foundation of
zation sequences and the instruction sequences in our CTOS. Generally, we can randomly generate optimization
study. sequences and testing programs. However, this random
In the next section, we present CTOS, a method based on strategy may not be efficient due to the huge space of opti-
differential testing [31] for catching compiler bugs caused mization sequences and testing programs. Meanwhile,
by optimization sequences of LLVM. Based on Doc2Vec, we many duplicate bugs may be triggered by similar optimiza-
present efficient methods to resolve the challenges of the tion sequences and testing programs. Thus, we need to
acquisition of representative optimization sequences and select representative optimization sequences and testing
the selection of representative testing programs. programs.
In Sections 3.2 and 3.3, we first introduce the vector rep-
3 APPROACH resentations of optimization sequences and testing pro-
grams, respectively. Some optimizations in a certain order
In this section, we first introduce the framework of CTOS. constitute an optimization sequence. Thus, if optimization
Then, we present algorithms to address two challenges, sequences have similar optimizations in a similar order,
namely the acquisition of representative optimization they may trigger duplicate compiler bugs. In addition, the
sequences and the selection of representative testing semantics and structure information of a testing program is
programs. the key to distinguish different testing programs. Thus, if
testing programs have similar semantics and structure
3.1 Overview of CTOS information, they may also incur duplicate compiler bugs.
Fig. 3 shows the framework of CTOS. Generally, CTOS is Therefore, a Doc2Vec based method is introduced to trans-
based on differential testing [31]. However, unlike other form an optimization sequence into a vector, which cap-
studies (e.g., [10], [11], [13], [14]) that compare the outputs tures optimizations and their orders of the corresponding
of testing programs compiled by different compilers or opti- optimization sequence simultaneously; a method based on
mized by the different default optimization levels, CTOS the region graph and call relationships of a program is pro-
determines whether there are compiler bugs by comparing posed to represent testing programs as vectors, such that
the outputs of testing programs optimized by distinct opti- the semantics and structure information of a program can
mization sequences. In Fig. 3, the front-end and the back- be captured by vectors. With the vector representations of
end indicate the compiler front-end and the compiler back- optimization sequences and testing programs, we assume
end that are used to transform the source code of a testing that similar optimization sequences and testing programs
program into its LLVM IR and generate executables, respec- are close to each other in their corresponding vector spaces,
tively. We adopt Clang [23] as the front-end, a widely used respectively. Hence, we present a centroid based selection
language front-end and compiler driver based on LLVM for scheme to select representative optimization sequences and
C language family (e.g., C and C++), since C programs are testing programs in Section 3.4, such that the distances
used as the testing programs in our study. As the front-end, among the selected representative optimization sequences
Clang takes in the source code and outputs the and testing programs are maximized, respectively.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2343

3.2 Representation of Optimization Sequences completely different from that in the sequence (b). By using
An optimization sequence is constituted of some optimiza- Doc2Vec, we can resolve the difficulty, which captures opti-
tions in a certain order. Thus, the representation of an opti- mizations and their orders of optimization sequences
mization sequence should reflect the specific optimizations simultaneously.
and their orders contained in the sequence. Intuitively, an
optimization sequence is similar to a sentence in natural lan- 3.3 Representation of Testing Programs
guage, which consists of some words in a certain order. Testing programs are another critical factor to trigger com-
Hence, in this study, we treat optimization sequences as piler bugs caused by optimization sequences. Different test-
sentences, such that efficient representation methods of sen- ing programs may trigger different bugs. Thus, we need to
tences can be adopted to transform optimization sequences select representative testing programs to improve the test
into vectors. However, many state-of-the-art representation efficiency for finding more distinct bugs. Our study focuses
methods of sentences, such as the bag-of-words [32], cannot on finding compiler bugs caused by optimization sequences
reflect the word order. They fail to distinguish different sen- of LLVM, which makes us decide to construct vector repre-
tences with the same words. For capturing optimizations sentations of testing programs using LLVM IR. LLVM IR is
and their orders of an optimization sequence simulta- a light-weight and low-level while expressive, typed, and
neously, we employ Doc2Vec [29], a popular and widely extensible representation of programs [21]. In this subsec-
used sentence vector representation technique to represent tion, we present a vector representation method based on
optimization sequences as vectors. Doc2Vec is an unsuper- the region graph and call relationships generated from the
vised method for learning continuous distributed vector unoptimized IR to transform a testing program into a vec-
representations of sentences or documents. Doc2Vec takes tor. With this approach, we can capture the semantics and
word orders into consideration such that the sequences structure information of programs, which are useful for
with different orders of the same words have different vec- selecting representative testing programs. We divide the
tor representations. In addition, Doc2Vec can be applied to vector representation of a testing program into two parts,
variable-length word sequences, so variable-length optimi- namely, the representation of a function and the representa-
zation sequences can be easily transformed into vector tion of the whole program. First, we transform the instruc-
representations. tion sequences of each edge in the region graph of a
In this study, Doc2Vec is applied in a relatively straight- function into vectors using the Doc2Vec technique; then a
forward way. That is, optimizations and optimization deep region-first algorithm is employed to aggregate vec-
sequences are viewed as words and sentences, respectively. tors of each edge under two constraints to construct the vec-
We leverage the DMPV model (see Section 2.2) of Doc2Vec tor representation of a function. Second, after obtaining the
as the representation method of optimization sequences. vector representations of all functions, we aggregate them
Then we input optimizations and optimization sequences according to their call relationships to form the vector repre-
into the DMPV model of Doc2Vec to obtain the vector repre- sentation of the whole program.
sentations of optimization sequences.
For example, if we only take five optimizations in LLVM
into consideration, i.e., {-functionattrs, -gvn, -loop-rotate, 3.3.1 Representation of a Function
-loop-vectorize, -sroa}, and set the max length of optimization A function consists of basic blocks, branches, and loops.
sequences to 5, we can obtain 51 þ 52 þ 53 þ 54 þ 55 ¼ 3905 Basic blocks contain the basic semantics of a function, while
optimization sequences. Take the following three optimiza- branches and loops control the structure of a function [33].
tion sequences as an example: (a) “-functionattrs -loop-rotate Thus, we use the region graph [34], [35] of a function to con-
-sroa -gvn -loop-vectorize”; (b) “-functionattrs -sroa -loop-vector- struct its vector representation, since the region graph could
ize -loop-rotate -gvn”; (c) “-loop-rotate -sroa -gvn -loop- simultaneously capture the semantics and structure infor-
vectorize”. If we can only test two sequences among them mation of a function [34], [35]. Definition 1 shows the gen-
due to the limitation of resources (e.g., time), testing sequen- eral definition of a region graph.
ces (a) and (b) may uncover more bugs, since sequence (c) is
Definition 1. A region graph is a special control flow graph, in
a subsequence of sequence (a) only without the optimiza-
which each node (i.e., basic block) exactly belongs to a region.
tion “-functionattrs”. However, in this case, the order of opti-
Specifically, a region is a connected subgraph of the control
mizations is hard to be captured by some bag-of-words
flow graph that has exactly two connections to the residual
methods. For instance, we can calculate the similarity
graph [34], [35], [36].
between optimization sequences utilizing the Jaccard simi-
larity coefficient,4 which is defined as the size of the inter- In a region graph, each node (i.e., basic block) exactly
section divided by the size of the union of two sample sets belongs to a region. Fig. 4 shows an example of the region
A and B, i.e., JðA; BÞ ¼ jA \ Bj=jA [ Bj. For the sequences graph. This graph is derived from a simple bubble sort algo-
(a) and (b), they have the same optimizations, i.e., Jða; bÞ ¼ rithm using the tool Opt in LLVM. We remove the contents
1; while Jða; cÞ ¼ 4=5 for the sequences (a) and (c). It indi- (i.e., statement sequences) of some basic blocks for simplic-
cates that sequences (a) and (b) are identical, and sequences ity. Clearly, there are 10 basic blocks, 12 edges, and 4
(a) and (c) should be tested. This contrasts with the observa- regions colored by four colors in Fig. 4. The number next to
tion, since the order of optimizations in the sequence (a) is each edge is its index. In these basic blocks, blocks “%11”
and “%15” are entry nodes of the outer loop and the inner
loop respectively, and block “%19” is the entry node of a
4. https://en.wikipedia.org/wiki/Jaccard_index. branch. From Fig. 4, the structure information (e.g., the
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2344 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

outer loop, the inner loop, and the branch) of the program
are clearly captured by each region.

Fig. 4. Example of a region graph.

region-first algorithm to aggregate the information on each

edge of the region graph.
Specifically, the information of an edge is the concatena-
tion of the instruction sequences from its start node (i.e.,
basic block) to the end node. The reason for this is that if the
program executes from the start node to the end node
through this edge, the code in these two nodes will be exe-
cuted. For instance, the information of the first edge in
Fig. 4 is “alloca alloca alloca alloca alloca store store bitcast bit-
cast bitcast store br load icmp br”. After obtaining the informa-
tion of each edge, we also leverage Doc2Vec to transform
the information into vectors. The reason for this is that the
instructions and instruction sequences can also be treated as
words and sentences in natural language, respectively. In
addition, the instruction sequences are also order sensitive
like the optimization sequences.
Then, the deep region-first algorithm is utilized to aggre-
gate the information of each edge to form the final vector
representation of a function. Algorithm 1 presents the
details of the proposed aggregation algorithm. Given a
region graph of a function with vector representations of
To obtain the semantics of a function, we use the edges, we first recognize the entry node (entry node), the
instruction sequences of each basic block [21] to represent exit node (exit node), and the loop entry nodes (loop entry
the behavior of a function. This is because the proposed nodes) of the region graph in lines 2 to 4. Then all outgoing
representation method of a program is based on LLVM IR edges of entry node are marked as visited in line 7, since the
and each instruction has precise and fine-grained seman- entry node does not have incoming edges. The work list
tics. In addition, the order of instructions significantly (work list) initialized by entry node in line 9 is used to store
impacts the semantics of a function. For example, we can the candidate nodes. Next, from line 10 to 35, the vectors of
easily know that the behavior of the basic block “%11” is incoming edges of a node are aggregated to its outgoing
to load a variable and compare it with 0 through the edges until work list is empty.
instructions “load” and “cmp”. However, we do not simply In the aggregation process, two constraints need to be
translate all the instruction sequences of each basic block satisfied in the proposed aggregation algorithm. The first
to vectors and aggregate them (e.g., sum or average), since constraint (line 11) is called a deep region-first constraint. It
this will lose the structure information of a function. For means the node in the innermost region will be first selected
instance, the block “%58” has no influence on the block to aggregate the vectors of incoming edges to the outgoing
“%19”. Simply aggregating these vectors diminishes the edges. The reason is that the semantics of the outer region is
importance of the blocks introduced by the structure of a based on the inner region. In Fig. 4, after propagating infor-
function. Therefore, in this study, we present a deep mation to edges 2 and 12, nodes “%14” and “%59” are two
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2345

3.3.2 Representation of the Whole Program

Generally, a program may contain many functions with call-
ing relationships. In our study, we construct the vector repre-
sentation of a program based on its call graph. Algorithm 2
presents the procedure to generate the vector representation of
a program. Given a call graph with the vector representation of
each function of a program, the key idea of Algorithm 2 is to
propagate the vectors of callees to their callers. This is because
the callee should have different weights due to the call relation-
ships. For example, in Fig. 5, the function “func6” is only called
by functions “func1” and “func2”. It has no any direct impact
Fig. 5. Example of call graph.
on other functions. Thus, if we only simply aggregate (e.g., sum
or average) all the vectors of functions, we lose the impact of
candidate nodes. However, the region of node “%14” is con- the weight of each function. However, one constraint must be
tained in the region of node “%59”, i.e., the previous region satisfied in the propagation process. That is, all vectors of call-
is “deeper” than the later one, thus node “%14” is selected. ees of a caller must be the final results, i.e., there is not any vec-
When the current node (cur node) within the deepest region tor that needs to be propagated to these callees. Specifically, the
in work list is selected, we delete it from work list and add vectors of end nodes are their final results that have been
it into the list visited nodes that stores the visited nodes marked in line 9.
(line 12 and 13).
Next, the second constraint is employed to aggregate the
vectors of incoming edges and outgoing edges of each suc-
cessor node of cur node in line 15 to 35. The second con-
straint indicates that, for a node with predecessors and
successors in a region graph, such as node “%54” in Fig. 4,
if the information on its incoming edges can be propagated
to the outgoing edges, the information before its predeces-
sors must have been propagated to the incoming edges.
This constraint is valid by line 19, i.e., all incoming edges of
a node must have been visited. For example, if we want to
propagate the information on the incoming edges 6 and 7 to
the outgoing edge 8, the information on edges 5 and 4 must
have been propagated to edges 6 and 7, respectively. In our
study, to aggregate the information of incoming edges and
outgoing edges, we average their sum and assign it to the
outgoing edges (line 22 to 25). Then the successor node of
cur node is added into work list if it does not exist in
work list and visited nodes (line 25 and 26).
However, for the loop entry node, the second constraint
must be relaxed due to the back edge of a loop. For instance,
edge 9 is a back edge and it is also an incoming edge of the
loop entry node “%15”. We cannot propagate any informa-
tion to the outgoing edges of node “%15”, since the informa-
tion before node “%55” has not been propagated to edge 9
under the second constraint. Thus, we propagate the infor-
mation on the incoming edges of the loop entry node sepa-
rately. That is, if the information has been propagated to
edge 3, we will propagate it to edges 4 and 10. Similarly, the
information on edge 9 will be propagated to edges 4 and 10.
Thus, in line 17, we verify that whether a node belongs to
loop entry nodes. Lines 29 to 35 in Algorithm 1 are the corre-
sponding pseudocode for processing the loop entry node. In the algorithm, we first remove edges of recursive calls
Therefore, starting from the entry node (e.g., “%2”) of a in line 2, such that the call graph is a directed acyclic graph.
region graph, when all nodes have been processed, the vec- On the one hand, the recursive calls make it impossible for
tor representation of the edge related to the exit node (e.g., the algorithm to decide which nodes have the final result.
“%59”) is treated as the vector representation of a function. For example, there is an indirect recursive call between
In particular, if there is only one basic block in the region “func1” and “func4” in Fig. 5. According to the constraint, if
graph, the vector representation of the instruction sequence we want to aggregate the vector of “func4” to “func1”, the
in this basic block is treated as the vector representation of a vector representation of “func4” should be the final results.
function. However, at this moment “func1” is the successor of
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2346 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

“func4”, it does not have the final vector representation. On least one instance in each group to be selected. This is
the other hand, the recursive calls have almost no effect on because the distribution of instances may be unbalanced,
improving the representation of functions. For instance, which causes that some instances with special features that
“func7” recursively calls itself. Its vector representation can trigger bugs may be lost. In this paper, we use the
does not change after the aggregation. X-means algorithm [37] to cluster the target instances, since
Then, the start nodes (start nodes) and the end nodes it can automatically determine the best number of groups.
(end nodes) in the call graph are recognized in lines 3 and 4. Next, from line 7 to 18, we select the best candidate instance
start nodes are the nodes without predecessors, while in each iteration until k required instances are selected.
end nodes are the nodes without successors in the call graph.
We add a node “start” (i.e., a fake function with the name
“start”) to the call graph with a zero vector as its representa-
tion. Next, we add edges from the node “start” to
start nodes. The reason is that there may exist some functions
(besides the “main” function) which are not called by any
other functions. Notably, the function without any caller can
also be split into an independent program, but in our study,
we treat the functions in a program generated by Csmith in a
uniform way for simplifying the processing of the program.
The work list (work list) is initialized with end nodes. Thus,
from line 12 to 25, we propagate the vectors of callees to call-
ers until work list is empty. In line 13, the current node
(cur node) is randomly picked up from work list. In line 14,
the predecessor nodes (pre nodes) of (cur node) are selected.
Thus, for each node in pre nodes, we propagate the vectors of
its successors to it from line 15 to 24. The constraint is verified
in line 18. If all nodes in suc nodes of a node have been vis-
ited, we average the sum of its vector and those of its succes-
sors (line 19 and 20). Then this average value is assigned to
the node as its final vector representation in line 22.
In Fig. 5, when the recursive calls are deleted, “func4”,
“func5”, “func6”, and “func7” are the nodes with final vector
representations. Thus, we can propagate these vectors back
to “func1”, “func2”, and “func3” according to their calling
relationships. After obtaining the final vector representa- Specifically, the selection procedure is a “centroid” based
tions of “func1”, “func2”, and “func3”, the vector representa- scheme. Suppose the solar system is a set of instances, the
tions of “func2” and “func3” can be propagated back to task is to select some instances around the center “Sun”. For
function “main”. However, we cannot propagate the vector a candidate instance, we first calculate the distance
representations of “func1” and “main” back to “start”, since dist2center from this instance to the center, such that the
the vector representation of “main” is not the final result. instances with different orbits can be distinguished like the
Lastly, the vector representation of “start” is treated as the “Earth” and “Mars”. In addition, the minimum distance
vector representation of the whole program. min dist2sel from the candidate instance to the selected
instances is calculated for avoiding similar instances in the
same or similar orbits. For example, if the instances “Earth”
3.4 Selection Scheme and “Mars” are selected, the instance “Moon” cannot be
With the vector representations of optimization sequences selected since it is very close to “Earth”. Therefore, to balance
and testing programs, we present a selection scheme in this the effect of these two distances, we leverage the product of
subsection to select representative optimization sequences these two distances as the score of a candidate instance. The
and testing programs. larger the score a candidate instance has, the better it will be.
Given a set of instances (i.e., optimization sequences and In this study, we utilize the euclidean distance function
testing programs), we aim to select a small set of instances distðu; vÞ to calculate the distance between two instances.
with better diversity, since the space of instances is huge Suppose ~ u ¼ ðu1 ; u2 ; . . . ; un Þ and ~ v ¼ ðv1 ; v2 ; . . . ; vn Þ are two
and duplicate bugs may be triggered by the instances with candidate instances represented by n dimensional vectors,
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
high similarities. Algorithm 3 presents the proposed selec- distð~ vÞ ¼ ðu1  v1 Þ2 þ    þ ðun  vn Þ2 . In Algorithm 3,
u; ~
tion scheme. The central idea is to select instances one by
one such that the total distances among the selected instan- for a candidate instance that has not been selected, we first cal-
ces are maximized. First, M instances will be generated by a culate the distance from it to the centroid in line 11; then the
random generator (for the generation of initial optimization minimum distance from it to the selected instances is calcu-
sequences and testing programs, see Section 4.2). We then lated in line 12, the score of the current instance is calculated
cluster these M instances into groups. After that, the central via the product of these two distances in line 13. In lines 10 to
instances of each group are selected as the initialization of 16, the current best instance with the maximum score is
the set of selected instances selected insts, which leads at selected. This process repeats until k instances are selected.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2347

4 EVALUATION training, we obtain the vector representations of the optimi-

zation sequences and testing programs (For vector represen-
In this study, we conduct experiments on LLVM to evaluate
tations of testing programs, we still need the aggregations
the effectiveness of CTOS. Specifically, our evaluation aims
shown in Algorithms 1 and 2). These vectors are then taken
at answering the following two Research Questions (RQs).
as the inputs of the selection algorithm to select representa-
 RQ1. Can the proposed selection scheme help to detect
tive optimization sequences and testing programs. By this
bugs?
setting, CTOS can efficiently learn the representation of
This RQ investigates whether the proposed selection
optimization sequences and testing programs and be less
scheme for selecting representative optimization sequences
influenced by the out-of-vocabulary problem.
and testing programs is helpful for detecting compiler bugs
For the selection scheme presented in Section 3.4, the
of LLVM. Besides the proposed selection scheme, we also
used clustering algorithm X-means is implemented in
consider two alternative methods for acquiring optimiza-
PyClustering [41], which is a data mining library that pro-
tion sequences and four alternative methods for selecting
vides a wide range of clustering algorithms. The parameters
testing programs (see details in Section 4.3). Thus, we con-
of X-means are set to the default values. In Algorithm 3, the
duct an experiment for comparing the results of CTOS
total numbers of initial optimization sequences and testing
under three kinds of optimization sequences and five kinds
programs (i.e., M) are 300,000 and 100,000, respectively,
of testing programs.
they reach the limitation of memory of our system on proc-
 RQ2. How is the bug-finding capability of CTOS in
essing optimization sequences and testing programs. The
practice?
numbers of optimization sequences and testing programs
In this RQ, we investigate the capability of CTOS for
(i.e., k) that should be selected by Algorithm 3 are 3,000 and
detecting LLVM bugs caused by optimization sequences in
1,000, respectively. This is because given the testing period
practice. Specifically, we evaluate CTOS from three aspects,
(i.e., two weeks for one testing process on LLVM in prac-
namely, the number of reported bugs, the type of bugs, and
tice), we can test about 3,000 optimization sequences and
the number of buggy optimizations.
1,000 testing programs. Besides, we set a single testing
period to two weeks, because we hope to timely update and
4.1 Implementations test the latest development version of LLVM.
We implemented CTOS with approximately 2,000 lines of
Python code.5 In our study, the used Doc2Vec [29] model is 4.2 Testing Setup
implemented in Gensim [38], which is a widely used Python Hardware. Our evaluation is conducted on an x86_64 com-
Ò
library for the representation of natural language. For the puter running Ubuntu 18.04 Linux with an Intel CoreTM i7-
vector representation of optimization sequences, we set 7700 CPU @ 3.60GHZ x 8 processor and 16 GB of memory.
vector size ¼ 100, window ¼ 3, alpha ¼ 0:05, and epochs ¼ Compilers. In our study, we only conduct our experiments
20 to train the Doc2Vec model. All the parameters of Doc2- on LLVM. The reason is that, as to our knowledge, only
Vec are set according to the documents of Gensim and the LLVM currently can allow developers to adjust the orders
suggestions by [39]. Algorithms 1 and 2 for the vector repre- of optimizations. For GCC, another mature and widely
sentation of a given program are implemented based on used compiler in both industry and academia, the orders of
NetworkX [40]. First, a C program is transformed into its IR optimizations are fixed.6 Although any order of optimiza-
using Clang without any optimization. Then the optimizer tions can be passed as command-line arguments to GCC,
Opt of LLVM takes in this IR to generate the corresponding the orders of these optimizations cannot affect the behavior
region graph and the call graph. Similarly, we employ Doc2- of GCC. The same case also occurs for CompCert that is a
Vec to obtain the vector representation of instruction verified and high-assurance compiler for the C language.7
sequences on each edge of a region graph, and the parame- The fixed order of optimizations is beneficial for the rapid
ters of Doc2Vec are the same as those in the representation implementation and safety of compilers. However, there is
of optimization sequences except that window ¼ 5. no doubt that the fixed order of optimizations limits the
Note that in this study, we train two Doc2Vec models in capabilities of compilers to optimize programs for different
each iteration to obtain the vector representations of optimi- requirements. In contrast, as a compiler providing support
zation sequences and testing programs, respectively. A typi- for arbitrary orders of optimizations, LLVM has been
cal Doc2Vec model usually requires a large number of widely used to implement many compilers and tools. Our
training data to solve the out-of-vocabulary problem which study aims to improve the reliability of optimizations with
may be time-consuming. Since the space of optimization arbitrary orders for LLVM, which helps to guarantee the
sequences is extremely huge, the out-of-vocabulary problem correctness of different LLVM-based compilers and tools.
may also negatively affect the representation of optimiza- Optimizations. In our study, C programs are used as the
tion sequences (or testing programs). To alleviate this prob- inputs of LLVM. Thus we mainly focus on testing the
lem, in each iteration of testing, we train local Doc2Vec machine independent optimizations of LLVM that are use-
models specialized to learn the knowledge of current opti- ful for the C programming language. We currently do not
mization sequences and testing programs. That is, we train consider the optimizations for object-oriented programming
two new Doc2Vec models using the new randomly gener- languages (e.g., C++ and Objective-C), profile guided
ated optimization sequences and testing programs. After
6. https://stackoverflow.com/questions/33117294/order-of-gcc-
5. All the source code of CTOS will be publicly available after the optimization-flags
publication. 7. https://github.com/AbsInt/CompCert/issues/287
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2348 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

optimizations, and results visualization of optimizations. optimization sequences, we remove each optimization in
Finally, 114 optimizations are selected.8 In LLVM, each opti- the optimization sequence one by one. If the bug still occurs,
mization may depend on certain other optimizations as the which indicates that this optimization has no impact on the
preconditions. LLVM provides a mechanism to manage the bug, we delete it from the optimization sequence; otherwise,
dependencies, i.e., PassManager.9 Thus, we do not need to the removed optimization will be put back into the original
manually manage the dependencies of optimizations. position. This process continues until no optimization can
For generating the initial optimization sequences, we be deleted. Additionally, similar to the related work [9],
assign an index to each optimization. The length of an opti- [13], [17], we also use Creduce [45], a widely used tool for
mization sequence is randomly selected in the range from reducing C, C++, or OpenCL programs, to reduce the test-
50 to 200. This is because there are 84 unique transformation ing programs that have triggered bugs. LLVM warnings
optimizations in the -O3 optimization level of the current and Frama-C are used to detect undefined behaviors during
released LLVM 7.0.1 (the optimizations in -O3 optimization the reduction process to ensure that the resultant program
level may be different in different versions of LLVM). The is valid.
range from 50 to 200 could guarantee that the generated Note that, we first reduce the optimization sequences.
optimization sequences have different lengths. Then we This is because the reduction of testing programs always
leverage a uniform random number generator to randomly takes more time than the reduction of optimization sequen-
generate the indexes of optimizations until the length of the ces. The developers of LLVM think that “the passes are mostly
current optimization sequence is reached. Next, the index designed to operate independently, so if we see an assert/crash,
sequence is translated into the corresponding optimization then we can always blame the last pass in the sequence. And if the
sequence. In addition, the parameters of optimizations are test ends with the same assertion and backtrace in the last pass in
set to the default values. the sequence, then we can assume that it is a duplicate.” (see
Testing Programs. We use Csmith [13], a widely used pro- LLVM Bug#40927 [46]) Thus, if the last optimizations of
gram generator that supports most features of the C pro- the reduced sequences are identical (for the crash bug, the
gramming language to generate initial testing programs. To failed assertion or backtrace also should be identical), the
detect compiler bugs caused by optimization sequences, the corresponding bugs are treated as a duplicate. Therefore,
testing program needs to be valid, free to undefined behav- reducing optimization sequences first can save the total
iors, diverse, and executable. However, other program gen- time to reduce test cases.
erators (e.g., CCG [42], Yarpgen [43], Orion [9]) can not Duplicate Bug Identification. In our study, we also adopt
meet our requirements. For example, CCG can not generate the above strategy to filter out duplicate bugs. We treat the
runnable testing programs, and the maintenance of CCG last optimization in a reduced optimization sequence as a
has stopped for a long time. Yarpgen is a generator to pro- buggy optimization. Thus, if the last optimizations in the
duce correct runnable C/C++ programs, but it only sup- two reduced optimization sequences of two bugs are the
ports a few C/C++ features. Orion is a mutation-based tool same, these two bugs are treated as a duplicate. Besides, for
to generate new programs for seed programs by deleting crash bugs, to improve the accuracy of duplicate bugs iden-
the dead code in the seed programs. For the mutation-based tification, we further use the failed assertion or backtrace to
tools (e.g., Orion), the diversity of the programs generated determine duplicate crash bugs. That is, when two crash
by these tools is limited by the seed programs. In addition, bugs have the same failed assertion or backtrace, we treat
some grammar-based program generators (e.g., Grammari- them as duplicate crash bugs. The reason for adopting this
nator [44]) can also be utilized to generate testing programs, strategy rather than just distinct optimization sequences to
but the generated programs always are invalid and contain identify duplicate bugs is because many duplicate bugs can
undefined behaviors. Thus, we employ Csmith to generate be triggered even though the reduced optimization sequen-
testing programs in this paper. The minimum size of the ces are distinct. This is also the strategy applied by the
generated programs is set to 80KB as suggested in [13]. LLVM developers, as can be seen in LLVM Bug#40926 [47],
Other parameters of Csmith are set to the default values. In #40927 [46], #40928 [48], #40929 [49], and #40933 [50], which
addition, we leverage LLVM warnings and Frama-C10 to are marked as duplicates of LLVM Bug#40925 [51].
detect undefined behaviors of the generated programs, Although the optimization sequences of these bugs are dis-
since the undefined behavior may cause invalid compiler tinct, they are marked as duplicates because the root causes
bugs. of these bugs are introduced in the same last optimization
Test Case Reduction. Similar to other related studies (e.g., in the sequences. Through this strategy, we may avoid
[9], [13], [17]), all test cases that trigger compiler bugs reporting too many duplicate bugs to developers.
should be reduced before we report them to the developers, Bug Types. In our study, we mainly find the following
such that the developers can quickly locate the real reasons five types of compiler bugs caused by optimization sequen-
of the bugs and fix them. Test case reduction includes two ces of LLVM.
parts in our study, namely, optimization sequence reduc-
tion and testing program reduction. For the reduction of (1) Crash. The optimizer Opt of LLVM crashes when
optimizing the IR of a program.
(2) Invalid IR. In LLVM, each optimization takes in the
8. The full list of 114 optimizations can be found on the website, valid IR of a program as input, and its outputs also
https://github.com/CTOS-results/LLVM-Bugs-by-Optimization- should be a valid IR. However, invalid IR may be
sequences
9. https://llvm.org/docs/WritingAnLLVMPass.html
generated by some optimizations due to the interac-
10. http://frama-c.com/ tion among optimizations. In our evaluation, we
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2349

tune on the option “-verify-each11” of the optimizer TABLE 1

Opt to verify whether the output IR is valid after Optimization Sequences and Testing Programs for RQ1
every optimization. If the IR is invalid after an opti-
RS Optimization sequences generated by the random
mization, the optimizer Opt will be stopped and will strategy
output some error messages. SS Optimization sequences selected by the proposed
(3) Wrong code. The optimized IR produced by the opti- selection scheme
mizer Opt may contain different semantics to the 2W Optimization sequences generated by the
original program, which makes the corresponding combinatorial testing technique
executable produce wrong outputs, or occur seg- RP Testing programs randomly generated by Csmith
RPS Testing programs randomly generated by the swarm
mentation faults or floating point exceptions. testing
(4) Performance. When the out-of-memory of the opti- RPSv Testing programs randomly generated by the variant
mizer Opt occurs, we treat it as performance bugs. It of swarm testing
could slow the compilation of programs. In the worst SP Testing programs selected by the proposed selection
case, the computer system may be jammed due to scheme
the performance bug. Generally, we set the maxi- SPS Testing programs selected by the proposed selection
scheme with the static features of programs
mum size of the memory of an optimizer process to
4 GB, since it is sufficient to optimize a testing pro-
gram using 4 GB memory in most cases. Thus if the
maximum memory of the optimizer Opt is greater implemented in ACTS [53] to generate optimization sequen-
than 4 GB, the optimizer will be stopped like a crash ces. ACTS is a widely used test generation tool for con-
bug. We also try not to limit the size of the memory, structing t-way combinatorial test sets to detect failures
but it has the same results as the 4 GB limitation. triggered by interactions of parameters in the software [52].
(5) Code generator bug. The code generator in the back- However, we can only generate 2-way combinatorial opti-
end is used to generate the assembly code of a pro- mization sequences 2W since there are too many optimiza-
gram from the corresponding IR. However, the IR of tions. In our study, we set the length of the optimization
a program optimized by some optimizations may sequences generated by ACTS to 200 for simplicity, and
trigger some bugs in the code generator. Currently, then ACTS generates 34,473 2-way combinatorial optimiza-
we only find one bug for this type, it makes the code tion sequences.
generator not emit a machine instruction and stops In addition, besides the testing programs selected by the
the code generator. proposed selection scheme (SP) in CTOS, four alternative
methods are used to select testing programs. Similar to
other studies (e.g., [13]), we also use Csmith with the default
4.3 Answer to RQ1 configuration to randomly generate testing programs (RP).
To evaluate the effectiveness of the proposed selection Additionally, the swarm testing technique [54] is utilized to
scheme, we design an experiment for comparing the results guide Csmith to generate diverse testing programs by ran-
of CTOS with three kinds of optimization sequences and domizing test configurations. As in the study [55], we also
five kinds of testing programs. The experiment is conducted consider two versions of swarm testing: an original version
on the recently released version of LLVM 7.0.1, such that we of swarm testing [54] that randomly sets the value of each
can verify the detected bugs using the latest development configuration option to be 0 or 100, and a variant of swarm
version of LLVM. For a detected bug, if it does not exist in testing that randomly sets the value of each configuration
the latest development version of LLVM, we then think it option to be a floating-point number ranging from 0 to 100
has been fixed; otherwise, we report it to the developers of [55]. Thus, we obtain two kinds of testing programs by
LLVM to further identify whether it is a valid bug. swarm testing, namely the testing programs generated by
Specifically, there are three kinds of optimization sequen- the original swarm testing (RPS) and the testing programs
ces. Besides the optimization sequences selected by the pro- generated by the variant of swarm testing (RPSv). More-
posed selection scheme (SS) in CTOS, we adopt a random over, apart from the proposed vector representation of test-
strategy and a combinatorial testing technique [52] to gener- ing programs, the static features of programs presented in
ate optimization sequences. For randomly generating opti- [20] are used to select testing programs with the proposed
mization sequences (RS), an index is assigned to each selection scheme (SPS). The static features of programs
optimization, and then a uniform random number genera- include language features, operation features, and structure
tor is utilized to randomly generate the indexes of optimiza- features (see [20] for details). The optimization sequences
tions until the length (that is randomly selected in the range and testing programs used in RQ1 are summarized in
from 50 to 200) of the current optimization sequence is Table 1. Therefore, besides CTOS that takes in SP and SS as
reached. Next, we translate the index sequence into the cor- inputs in this study, we obtain 14 variants of CTOS, namely,
responding optimization sequence. This random strategy is (1) CTOS(RP+RS), (2) CTOS(RP+SS), (3) CTOS(RP+2W), (4)
identical to the strategy described in Section 4.2. The mini- CTOS(RPS+RS), (5) CTOS(RPS+SS), (6) CTOS(RPS+2W), (7)
mum and maximum lengths of RS are the same as those of CTOS(RPSv+RS), (8) CTOS(RPSv+SS), (9) CTOS(RPSv+2W),
SS, and the number of SS and RS is identical (i.e., 3,000). In (10) CTOS(SP+RS), (11) CTOS(SP+2W), (12) CTOS(SPS+
addition, we utilize the combinatorial testing techniques RS), (13) CTOS(SPS+SS), (14) CTOS(SPS+2W).
For CTOS and its 14 variants, we run each 10 times and
11. http://llvm.org/docs/CommandGuide/opt.html the timeout is 90 hours in each time as the setting in [19].
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2350 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

TABLE 2
Results of CTOS and its 14 Variants

Average Min. total Max. total P-value Effect

TP. Crash WC. Inv. IR Perf. CGB Total bugs imp. bugs bugs size (A12 )
CTOS(RP+RS) 113.2 6.4 1.0 1.4 1.0 0.0 9.8 33.67% 7 12 < :001 0.955
CTOS(RP+SS) 100.7 6.7 1.7 1.0 0.9 0.0 10.3 27.18% 8 12 .001 0.905
CTOS(RP+2W) 9.2 7.2 1.3 0.4 1.0 0.0 9.9 32.32% 8 11 < :001 0.960
CTOS(RPS+RS) 28.8 5.2 1.0 1.4 1.0 0.1 8.7 50.57% 7 10 < :001 1.000
CTOS(RPS+SS) 27.5 5.4 1.4 1.2 0.9 0.0 8.9 47.19% 7 11 < :001 0.980
CTOS(RPS+2W) 3.9 5.6 1.3 0.9 1.0 0.0 8.8 48.86% 6 11 < :001 0.990
CTOS(RPSv+RS) 61.4 5.7 0.9 1.3 1.0 0.1 9.0 45.56% 7 12 < :001 0.965
CTOS(RPSv+SS) 53.0 5.7 1.7 1.4 1.0 0.1 9.9 32.32% 9 11 < :001 0.980
CTOS(RPSv+2W) 5.5 6.6 1.3 1.2 1.0 0.1 10.2 28.43% 8 13 < :001 0.915
CTOS(SP+RS) 122.4 6.6 1.4 1.2 1.0 0 10.1 29.70% 9 11 < :001 0.960
CTOS(SP+2W) 8.6 6.8 1.8 0.6 1.0 0 10.2 28.43% 8 12 < :001 0.935
CTOS(SPS+RS) 117.8 5.8 1.1 1.1 1.0 0 9.0 45.56% 7 10 < :001 1.000
CTOS(SPS+SS) 102.4 6.9 2.0 0.6 1.0 0 10.5 24.76% 9 12 .001 .0930
CTOS(SPS+2W) 8.2 7.0 1.5 0.5 1.0 0 10.0 31.00% 8 12 < :001 0.955
CTOS 109.9 7.4 3.5 1.2 1.0 0 13.1 – 11 15 – –

TP.: Testing Programs, Inv. IR: Invalid IR, WC.: Wrong Code, CGB.: Code Generator Bug.

Therefore, it takes nearly 50 days to run these experiments. testing programs into the testing process, the testing effi-
Notably, the 10 runs of CTOS and its 14 variants are inde- ciency may be dramatically decreased due to the possible
pendent, i.e., the optimization sequences and testing pro- memory swapping. However, under our current experi-
grams are different in each time except the 2-way ment setting, we may only use a small fraction of the initial
combinatorial optimization sequences, which are identical 1,000 testing programs to test different optimization sequen-
in each variant of CTOS since the optimization sequences ces. This may be a threat of validity for our experiments,
generated by ACTS are constant. In addition, the testing which will be discussed in Section 6. On the other hand,
programs and optimization sequences for CTOS and its 14 regarding the baselines using the combinatorial testing tech-
variants are prepared before we carry out the experiments. nique, it generates the same set of optimization sequences
The initial number of testing programs for each experiment (i.e., 2W) each time (taking about 10 hours). The experiment
is 1,000, because we cannot know how many testing pro- reuses 2W for each run of different baselines. Thus, it may
grams can be tested before the experiments. It is rapid to bring unfair comparisons between the baselines with 2W
generate RS and RP. While the time for generating RPS and and without 2W, when the optimization sequence genera-
RPSv needs about 3 hours, since some random values of tion time is included.
configuration options may cause Csmith takes more time to Table 2 presents the experiment results of CTOS and its
generate a testing program.12 Generating SS, SP, and SPS 14 variants. The second column is the average total number
takes 3 to 6 hours in our system, respectively. Hence, com- of testing programs, and the following 5 columns are the
pared to the testing period, the time for preparing optimiza- average number of unique bugs for each type. Actually,
tion sequences and testing programs is relatively short. The many duplicate bugs can be found by CTOS and its 14 var-
most time-consuming part is to generate 2W, which takes iants. We filter out these duplicate bugs using the strategy
about 10 hours in our system. But we only need to generate described in Section 4.2. Next, the eighth column is the aver-
2W once, all the experiments use the same 2W. age total number of unique bugs for CTOS and its 14 var-
In RQ1, we do not include the time spent on generating iants. From Table 2, the numbers of testing programs for
testing programs and optimization sequences into the test- CTOS(RP + 2W), CTOS(SP + 2W), and CTOS(SPS + 2W) are
ing period for two reasons. On the one hand, we adopt this smaller than those of CTOS and other variants, since the
evaluation strategy due to the limitation of computational number of 2-way combinatorial optimization sequences is
resources. Our experiments are conducted on a computer about 11 times larger than those of other type optimization
with 16 GB of memory. Although we have tried our best to sequences. In addition, the six variants with RPS and RPSv
optimize our programs to select representative testing pro- test a small number of testing programs compared to CTOS
grams from the initial set of 100,000 testing programs in our and other variants. For example, for the variants with RS,
experiments, the selection process can consume 4-8 GB of CTOS(RPS + RS) and CTOS(RPSv + RS) only test 28.8 and
memory. Besides, we need to run 150 experiments (CTOS 61.4 testing programs on average, respectively. However,
and its 14 variants, 10 runs for each experiment) in RQ1, the numbers of testing programs of CTOS(RP + RS), CTOS
which makes us run many experiments simultaneously so (SP + RS), and CTOS(SPS + RS) are 113.3, 122.4, and 117.8,
that we can finish all experiments in nearly 50 days. For respectively. The reason is that testing programs generated
each experiment, the testing process will consume about by swarm testing may contain some complicated structures,
100M-4GB of memory. Thus, if we integrate the selection of which cause a long time to optimize them and execute the
corresponding executables. Generally, the majority of time
12. https://github.com/csmith-project/csmith/blob/master/doc/ for the whole testing process is utilized to optimize testing
p-robabilities.txt programs and execute the corresponding executables. This
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2351

time is decided by both of the corresponding program and

optimization sequence. Given a testing program, if it is com-
plicated (e.g., including many nest loops), the time for opti-
mizing and executing it could be longer than a simple
testing program. Besides, different optimization sequences
can also affect the time for optimizing and executing the
same testing program. In Table 2, the number of testing pro-
grams of CTOS is larger than CTOS(RP+SS) since there are
two experiments that CTOS tests more than 120 testing pro-
grams. Nevertheless, this result cannot suggest that CTOS
can always test more testing programs than its variants,
because the numbers of testing programs of the variants
(e.g., CTOS(RP+SS)) in some experiments are larger than
CTOS.
It is obvious from Table 2 that CTOS significantly outper-
forms its 14 variants in terms of the bug-finding capability.
CTOS can find 13.1 unique bugs on average, while the aver-
age total number of unique bugs for the best variant CTOS
(SPS+SS) is only 10.5. The ninth column shows the improve- Fig. 6. Comparison of 10 total bugs of CTOS and its 14 variants.
ment of CTOS over the baselines (i.e., the 14 variants of
CTOS) in terms of the average total number of unique bugs, some corner cases, which means that there is still room to
i.e., imp ¼ ðCTOS  baselineÞ=baseline  100%. From the further improving the selection scheme.
ninth column of Table 2, we can see that CTOS can find Additionally, most bugs found by CTOS and its variants
more unique bugs than the baselines by up to 24.76 to 50.57 are the crash bugs. However, the difference between the
percent. Specifically, CTOS can detect more crash bugs and number of crash bugs for CTOS and each variant is not very
wrong code bugs. It finds 7.4 crash bugs and 3.5 wrong large. The reason is that there are some bugs which are rela-
code bugs on average. In addition, we could observe from tively easy to be triggered by some optimization sequences.
Table 2 that CTOS and its variants with the selected optimi- For example, LLVM Bug#3962615 can be easily triggered by
zation sequences (SS) and testing programs (SP) can find the optimization sequences that contain the subsequence “.
more bugs. On the one hand, CTOS and the variants with *-early-cse-memssa.*-early-cse-memssa.*”.
SS outperform others when the testing programs are RP, Fig. 6 is the boxplot of the total bugs found during the 10
RPS, SP, and SPS, respectively. For example, CTOS(RP+SS) runs of CTOS and its 14 variants. In this boxplot, CTOS sig-
finds 10.3 bugs on average which outperforms CTOS(RP+ nificantly outperforms the variants. The columns “Min. total
RS) and CTOS(RP+2W). The only exception is RPSv, which bugs” and “Max. total bugs” in Table 2 show the minimal
makes CTOS(RPSv+2W) detect more bugs than CTOS and maximal numbers of the total bugs in the 10 runs of
(RPSv+SS). On the other hand, SP also leads CTOS, CTOS CTOS and each variant, respectively. From Table 2 and
(SP+RS), and CTOS(SP+2W) to detect more bugs than Fig. 6, we can see that the minimal and maximal numbers of
others except CTOS(RPSv+2W). For instance, CTOS(SP+RS) the total bugs of CTOS are 11 and 15 respectively, which are
finds 10.1 bugs, while there are only 9.8 and 9.0 bugs for clearly larger than those of the variants. Although in some
CTOS(RP+RS) and CTOS(SPS+RS), respectively. The reason cases the number of total bugs found by CTOS is lower than
is that the optimization sequences and testing programs the variants, the median of CTOS illustrates that the bug-find-
selected by the proposed selection scheme may have better ing capability of CTOS is better than the baselines in most
diversity. cases. In addition, we conduct the Mann-Whitney U-test with
From Table 2, almost all experiments can find the only a level of significance 0.05 on the total bugs between CTOS
one unique performance bug.13 This performance bug is and the variants according to the suggestions by Arcuri and
caused by the optimization “-newgvn”. However, CTOS and Briand [56]. The P-value (p  :001) in Table 2 shows that
the 14 variants with RP, SP, and SPS fail to detect the code CTOS performs significantly better than the variants. Fur-
generator bug in this experiment. Only some variants with thermore, we also calculate the effect size of the differences
RPS and RPSv detect one code generator bug14 that has between CTOS and the baselines using the Vargha and
been fixed. This is because testing programs generated by Delaney’s A12 statistics16 [56]. If CTOS and the baselines are
the swarm testing may cover a larger portion of input space equivalent, then A12 ¼ 0:5; if the effect of CTOS is small com-
than the other approaches due to the random mechanism of pared to the baselines, then A12 < 0:5; otherwise, A12 > 0:5.
swarm testing for constructing test configurations. From Table 2, we can see that all the effect sizes are greater
Although the proposed selection scheme aims to select than 0.9, which indicates that CTOS has a higher probability
divers testing programs, in fact it may be limited by Csmith to obtain better results than the baselines. Particularly, the
with the default configuration within the given testing values of the effect size for CTOS(RPS+RS) and CTOS
period. Hence, the proposed selection scheme may miss (SPS+RS) are 1.000. This means that the total bugs found

15. https://bugs.llvm.org/show_bug.cgi?id=39626.
13. https://bugs.llvm.org/show_bug.cgi?id=41290. 16. We use the open source code shared by Tim Menzies to calculate
14. https://bugs.llvm.org/show_bug.cgi?id=42452. A12 , https://github.com/txt/ase16/blob/master/doc/stats.md.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2352 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

TABLE 3 TABLE 5
Excluded Optimization Subsequences Buggy Optimizations for Reported Bugs

ID Regular expressions of subsequences Type Type Optimizations

1 ”.*-early-cse-memssa.*-early-cse-memssa.*” Crash Crash adce bdce consthoist correlated-propagation
2 ”.*-gvn-hoist.*-early-cse-memssa.*” Crash early-cse-memssa flattencfg gvn gvn-hoist
3 ”.*-loop-unroll.*-licm.*” Crash gvn-sink indvars inline instcombine
4 ”.*-loop-reduce.*-loop-reduce.*” Crash ipconstprop ipsccp jump-threading licm
5 ”.*-loop-rotate.*-loop-vectorize.*” Wrong code loop-deletion loop-distribute loop-extract-
6 ”.*-loop-unroll.*-loop-reroll.*” Wrong code single loop-instsimplify loop-reduce loop-
7 ”.*-memcpyopt.*-gvn.*” Wrong code rotate loop-unswitch loop-versioning-licm
8 ”.*-reg2mem.*-newgvn.*” Wrong code mem2reg memoryssa mergefunc newgvn
partial-inliner separate-const-offset-from-gep
simplifycfg slp-vectorizer sroa
TABLE 4 Invalid IR hotcoldsplit indvars loop-extract-single
Reported Bugs loop-instsimplify loop-interchange loop-
reroll loop-rotate loop-unroll-and-jam loop-
Type num. duplicate confirmed/fixed unswitch loop-versioning-licm sroa
scalarizer
Crash 57 8 16
Invalid IR 13 0 2 Wrong Code called-value-propagation constprop
Wrong Code 24 1 1 functionattrs globalopt gvn gvn-hoist
Performance 9 0 1 indvars inline instcombine ipsccp jump-
Code Generator Bug 1 0 1 threading loop-reroll loop-simplify loop-
unroll loop-vectorize newgvn structurizecfg
Total 104 9 21 Performance jump-threading licm loop-extract newgvn
slp-vectorizer
during the 10 runs of CTOS are completely larger than those Bug of Code indvars
Gen.
of CTOS(RPS+RS) and CTOS(SPS+RS).
Answer to RQ1. The experimental results demonstrate that
CTOS significantly outperforms the baselines by detecting assertion failures and segmentation faults. For these crash
24:76%
50:57% more bugs on average, which reveals the bugs, 16 of them have been confirmed or fixed. However, 8
advantage of the proposed selection scheme for selecting rep- of these crash bugs are duplicate, which are caused by the
resentative optimization sequences and testing programs. constant hoisting optimization. Although the optimization
sequences that trigger these 8 bugs are different, the back-
4.4 Answer to RQ2 trace information of these bugs is identical. Thus developers
To evaluate the bug-finding capability of CTOS in practice, treat them as duplicate bugs [46]. We also adopt this strat-
we conduct an experiment over seven months from January egy to filter out duplicate crash bugs. In addition, 24 valid
2019 to July 2019. In this experiment, we mainly test the lat- wrong code bugs have been reported, one of them is con-
est development version of LLVM, since the developers of firmed. Compared to crash bugs, the number of confirmed
LLVM fix bugs primarily in the latest development version or fixed wrong code bugs is very small. The reason is that
rather than in stable versions [17], [57]. The time of one test- the root causes of wrong code bugs are hard to be isolated
ing process is about two weeks since CTOS can test 1,000 [58]. For crash bugs, developers can leverage the backtrace
testing programs and 3,000 optimization sequences in this information to analyze the root causes of the bugs, while
period on our system. Notably, we gradually exclude some only limited information (e.g., intermediate results of com-
optimization subsequences that easily cause duplicate bugs pilers) could be used to help developers logically under-
for improving the test efficiency until the corresponding stand the root cause of a wrong code bug.
bugs are fixed. These optimization subsequences are the Apart from crash bugs and wrong code bugs, we also
most frequent pairs of optimizations in the reduced optimi- reported 13 invalid IR bugs, 9 performance bugs, and 1 code
zation sequences that triggered a mass of duplicate bugs. generator bug. For these bugs, 2 invalid IR bugs, 1 perfor-
Table 3 shows 8 regular expressions of subsequences that mance bugs, and 1 code generator bug have been confirmed
have been excluded. The first four subsequences trigger or fixed. Especially, the code generator bug that prevents the
many duplicate crash bugs, while the latter four are used to code generator from emitting physreg copy instruction18 is fixed
avoid duplicate wrong code bugs. All the detected bugs only after one day since it is critical for LLVM to generate the
have been fed back to the LLVM bug repository. In the correct assembly code. The invalid IR bugs are also important
seven months, we have reported in total 104 valid bugs for LLVM as they may reveal the design flaws of the optimiza-
within 5 types, of which 21 have been confirmed or fixed. tions. For example, LLVM bug#4172319 found by CTOS shows
Table 4 summarizes the testing results.17 that the “-scalarizer” optimization may produce the invalid IR
From Table 4, we can observe that the most reported that cannot be processed by other optimizations, since it can-
bugs are crash bugs and wrong code bugs. We have not correctly process the unreachable blocks.
detected 57 crash bugs, which are mainly caused by

17. The details of these bugs can be found on the website, https:// 18. https://bugs.llvm.org/show_bug.cgi?id=42452.
github.com/CTOS-results/LLVM-Bugs-by-Optimization-sequences. 19. https://bugs.llvm.org/show_bug.cgi?id=41723.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2353

Fig. 7. Top 5 most used optimizations for reported bugs. jump-t: jump-threading, instc: instcombine, structcfg: structurizecfg, loop-e: loop-extract,
loop-r: loop-rotate, ls-vec: load-store-vectorizer.

From Table 4, only 21 of the reported bugs are confirmed Table 5 presents the buggy optimizations of the 104
or fixed. To investigate this phenomenon, we collect 1,323 reported bugs arranged in the alphabetical order. These
unique bugs related to scalar optimizations (most optimiza- buggy optimizations are the last ones in the reduced optimi-
tions in LLVM belong to this component) from the LLVM zation sequences, since the optimizations of LLVM are
bug repository20 from October 2003 to Jun 2019. In these mostly designed to operate independently and the develop-
bugs, 828 bugs have been confirmed or fixed, and 495 bugs ers always blame the last optimizations in the reduced
are still kept as “NEW”. For the 828 confirmed or fixed sequences [46]. 47 unique optimizations have been reported
bugs, although 428 bugs are confirmed or fixed in one to be faulty. Specifically, there are 15 buggy loop related
month, the developers take more than 15 months to confirm optimizations (in bold fonts), such as loop-rotate, loop-unroll
or fix the most residual bugs. The average number of and loop-vectorize. From Table 5, we can see that the loop
months for confirming or fixing these bugs is 5.6. In addi- related optimizations are more bug-prone than other opti-
tion, 495 bugs with “NEW” status have already existed for a mizations. This result indicates that the design of loop opti-
long time, an average of 14.1 months. This indicates that the mizations may exist some flaws and should be further
overall speed to confirm or fix LLVM bugs is relatively enhanced by the developers.
slow, not just for our reported bugs. One possible reason is Fig. 7 shows the statistics of the top 5 most used optimi-
that it is hard and time-consuming to analyze and find the zations in the reduced optimization sequences for the 104
root causes of compiler bugs [58]. Especially, for a bug reported bugs. From Fig. 7f, the optimizations jump-thread-
caused by an optimization sequence, the root causes of this ing, gvn, licm, loop-rotate, and instcombine are the 5 most used
bug may lie in any optimization of this sequence. In addi- optimizations in all reduced optimization sequences. In par-
tion, the study by Sun et al. [19] also finds that the bug-fixing ticular, jump-threading appears 44 times in all reduced opti-
rate of LLVM is lower than GCC. The authors explain that mization sequences for the 104 reported bugs. This
this is due to the limited human resources since some optimization is used to turn conditional into unconditional
LLVM developers in Apple are pulled into other projects branches that can greatly improve performance for hard-
like Swift [19]. Furthermore, we also talk with three devel- ware with branch prediction, speculative execution, and
opers of an international company that has a team to prefetching. However, the code structure may be compli-
develop compilers using LLVM. We ask these three devel- cated after performing jump-threading, since it will add
opers what are the difficulties to fix an LLVM bug during some new paths and duplicate code.21 This may cause other
their development. All these three developers say that it is optimizations to produce wrong results. In Figs. 7a, 7b, and
difficult to analyze and find the root causes of compiler 7c, we can see that the optimization jump-threading is the
bugs, especially for the wrong code bugs. top 2 most used optimization for the crash bugs, invalid IR

20. https://bugs.llvm.org/. 21. http://beza1e1.tuxen.de/articles/jump_threading.html.

Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2354 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

bugs, and wrong code bugs. In addition, from Figs. 7a, 7b, duplicate bugs with high probability. This helps us to test
7c, 7d, and 7e, optimizations that change the structure of a more optimization sequences and reduce the time to analyze
program (loop-rotate changes the structure of a loop and duplicate bugs. Second, when the corresponding bugs have
structurizecfg transforms the control flow structure of a pro- been fixed, we will remove the limitation of these subsequen-
gram) are widely used in the buggy optimization sequences. ces, such that the deep bugs caused by these subsequences
This indicates that the design flaws of optimizations may be could be detected. In the future, automation techniques may
introduced by the edge cases of the structure of a program, be developed to make the summarization of subsequences
which may help developers to pay more attention to the more precise and filter out duplicate bugs to improve the test-
interactions among these optimizations when they design ing efficiency.
and implement new optimizations. Buggy Optimization Isolation Challenge. In our study, we
Answer to RQ2. Our testing efforts over seven months treat the last optimization in a reduced optimization
clearly demonstrates that CTOS is effective in detecting sequence as a buggy optimization. However, this strategy is
LLVM bugs caused by optimization sequences. In the seven not absolutely true, since the real reason for a bug may lie in
months, we reported 104 valid bugs within 5 types, of any optimization of the reduced optimization sequence.
which 21 have been confirmed or fixed. 47 unique optimiza- This indicates that the results in Table 5 may not be accu-
tions are identified to be faulty and 15 of them are loop rate. For example, an assertion fails in LLVM bug#4226423
related optimizations. when the optimizer Opt optimizes a program using “-early-
cse-memssa -die -gvn-hoist”. We then treat “-gvn-hoist” as the
buggy optimization. Nevertheless, the developers show
5 DISCUSSION
that the root cause is introduced by “-die” since it does not
Importance of Optimization Sequence. The optimization correctly preserve the information generated by “-Memo-
sequences are mainly used to improve the performance rySSA” (an analysis method in LLVM24). Hence, we may
(e.g., size, speed, and energy) of a program. Especially, the underestimate the effectiveness of CTOS in the experiments
default optimization levels (e.g., -O1, -O2, and -O3) pro- since some bugs may be wrongly labeled as duplicates.
vided by a compiler are specific optimization sequences However, we must make a tradeoff to avoid reporting too
designed by the compiler experts. Although the default many duplicate bugs to developers. In practice, it may not
optimization levels can significantly improve the program be acceptable for developers to receive hundreds of
performance, many studies (e.g., [1], [2]) have shown that reported bugs in a few days, where the majority of the bugs
the autotuning of optimization sequences helps to further are duplicates. This situation is currently difficult to be alle-
improve the performance of a program. In addition, a pro- viated. First, to the best of our knowledge, there does not
gram in different scenarios may have different performance exist a perfect method to locate the real reasons for a com-
requirements. For example, the energy consumption may piler bug. In our work, we manually validate the reduced
be more important for a program in an embedded system, optimization sequences to guarantee that the bugs cannot
while the speed of a scientific program may be sensitive on be reproduced when omitting the last optimization of the
a high-performance supercomputer. Nevertheless, these sequences. Second, the developers of LLVM also utilize the
techniques may be invalid due to the potential bugs in the same strategy to roughly determine whether the bugs
selected optimization sequences. On the other hand, com- caused by optimization sequences are duplicate. The opti-
piler developers of a new program language (e.g., Rust, mizations in LLVM are mostly designed to operate indepen-
Swift) based on LLVM need to design better optimization dently, the developers always blame the last optimization in
sequences as the default optimization levels (e.g., O1, O2, a reduced optimization sequence [46]. Thus, we believe that
and O3 in LLVM) to meet the features in the new program the results of Table 5 are reasonable. In the future, we plan
language. In this case, if there are bugs in the optimization to introduce advanced fault localization techniques [58],
sequences, compiler developers could be frustrated, thus [59], [60], [61] to address this challenge.
badly slowing down the development. Hence, it is critical to Limitation of the Selection Scheme. The experimental results
guarantee the correctness of optimization sequences. show that CTOS is effective to detect LLVM bugs caused by
Buggy Subsequences Exclusion. In the testing process, we optimization sequences. However, the selection scheme in
exclude some optimization subsequences listed in Table 3 CTOS may be limited. In our study, the selection scheme is
that can easily trigger duplicate bugs. However, the excluded based on the hypothesis that the effects of two testing pro-
subsequences do not always trigger duplicate bugs. For grams (or optimization sequences) for testing LLVM are
example, LLVM bug#3962622 can be easily triggered by the similar if they are closed to each other. Therefore, for a set
optimization sequences with first subsequence in Table 3. of testing programs (or optimization sequences), our goal is
Especially, the optimizer Opt must be crashed when any pro- to select representative testing programs (or optimization
gram is optimized using “-early-cse-memssa -early-cse-mem- sequences) such that the total distances among them are
ssa”, while the subsequence “-early-cse-memssa -gvn -early-cse- maximized. Nevertheless, the selection scheme may miss
memssa” cannot trigger this bug. Even so, we think this exclu- some corner test cases, as it is difficult to know which test-
sion strategy contributes to improving the testing efficiency. ing program (or optimization sequence) can trigger a bug
First, the excluded subsequences are manually summarized before execution. Besides, we currently use Csmith with the
from many duplicate bugs, which indicates that the optimiza- default configuration to generate the initial testing
tion sequences that contain these subsequences may trigger
23. https://bugs.llvm.org/show_bug.cgi?id=42264.
22. https://bugs.llvm.org/show_bug.cgi?id=39626. 24. https://www.llvm.org/docs/MemorySSA.html.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2355

programs, which may limit the diversities of the generated many duplicate bugs are triggered in the testing process
testing programs and affect the effectiveness of the pro- though we have leveraged the selection scheme to obtain
posed selection scheme. In future work, we will consider the representative optimization sequences and testing pro-
more advanced techniques (e.g., combine the selection grams. To alleviate this threat, we summarize the subse-
scheme with coverage information) to select representative quences (listed in Table 3) that could easily trigger
testing programs and optimization sequences to include duplicate bugs and remove the optimization sequences that
more corner cases. contain these subsequences in the next testing process until
the corresponding bugs are fixed. In addition, we utilize the
duplicate bug identification strategy described in Section 4.2
6 THREATS TO VALIDITY to identify duplicate bugs. However, this strategy may be
Threats to Internal Validity. The threats to internal validity not precise, which may influence the effectiveness of CTOS.
mainly lie in the implementations of CTOS. As mentioned in This is because the root causes for a bug can be introduced
Section 3, the vector representations of the optimization by any optimization in the reduced optimization sequence.
sequences and testing programs rely on the Doc2Vec tech- Since our strategy is also adopted by LLVM developers to
nique. Hence, the testing efficiency may be impacted by the identify duplicate bugs caused by optimizations in practice,
implementation of Doc2Vec. To reduce this threat, we adopt this strategy can still significantly reduce the negative influ-
the widely used tool Gensim [38] that has an efficient imple- ence of duplicate bugs on the experiments. In the future, we
mentation of Doc2Vec. In addition, the parameters of Doc2Vec will consider to apply advanced software fault localization
are currently set according to the documents of Gensim and techniques to improve the strategy for identifying duplicate
the suggestions by [39]. For the parameters of Algorithm 3, we bugs in our study.
set the values of these two parameters according to the hard- Second, we utilize Csmith to generate the testing pro-
ware limitation of our system. We do not investigate the grams in this study. However, Csmith has been widely used
impact of these parameters for CTOS in this paper, due to the to test LLVM for a long time, which makes LLVM, to a cer-
heavy time cost to fine-tune the parameters of CTOS. There tain extent, resistant to it. In the future, the advanced techni-
are in total 6 parameters, i.e., 4 parameters for Doc2Vec and ques (e.g., the test-program generation approach via history-
2 parameters for Algorithm 3. Assuming that each parameter guided configuration diversification [62]) may be employed
has 10 candidate values, we will get 106 parameter combina- to further improve the diversity of testing programs.
tions. The large number of parameter combinations can lead
to a long time to investigate the impact of the parameters for
7 RELATED WORK
CTOS, even the testing period is only 90 hours (as set in RQ1)
for evaluating one parameter combination. Despite this, the 7.1 Compiler Testing
experimental results illustrate that CTOS can achieve good Compiler testing is currently the most important technique
results under the parameter settings in our paper. to guarantee the quality of compilers. In the literature, the
Besides, in our experiments for RQ1, the time to obtain techniques of compiler testing fall into three categories,
testing programs and optimization sequences is not inclu- namely, Randomized Differential Testing (RDT), Different
ded in the testing period, which may cause unfair compari- Optimization Levels (DOL), and Equivalence Modulo
sons between CTOS and the baselines. For example, Inputs (EMI) [9], [15], [16], [17], [18]. For a given testing pro-
compared with a random strategy (i.e., RS and RP), COTS gram, RDT detects compiler bugs by comparing the outputs
spends 3-6 more hours in generating testing programs and of some compilers with the same specification. DOL is a var-
optimization sequences. However, we do not expect this can iant of RDT, and compares the outputs produced by the
dramatically affect the experiment results in Table 2, because same compiler with different optimization levels to deter-
the time to obtain testing programs and optimization sequen- mine whether a compiler has bugs. Most of the techniques
ces (3-6 hours) is much smaller than the testing period (i.e., [12], [13], [14] belonging to RDT and DOL use randomly
90 hours). To investigate its potential impact on the results of generated testing programs to test a compiler. Zhao et al.
CTOS, we analyze the number of bugs detected by CTOS in [12] develop a tool, called JTT, that automatically generates
the first 80 hours. That is, we exclude 10 hours from the test- testing programs to validate the EC++ embedded compiler.
ing period which are assumed to be used for CTOS to obtain In particular, Csmith [13] as the most successful random C
testing programs and optimization sequences. As presented program generator has been widely used to test C com-
in the supplemental material,25 CTOS can find the majority of pilers. Lidbury et al. [14] develop CLsmith based on Csmith
bugs (11.7 on average) in the first 80 hours; it outperforms all to generate programs for testing the OpenCL compilers.
the baselines which run in 90 hours. For example, CTOS(RP Different from RDT and DOL, EMI compares the outputs
+RS) detects 9.8 bugs on average in 90 hours, which is produced by equivalent variants of a seed program to detect
19.39 percent fewer than the number of bugs detected by compiler bugs. If an output is different from others, the com-
CTOS in the first 80 hours. Thus, we believe that CTOS out- piler then contains a bug [9]. There are three instantiations of
performs the baselines even when the time to obtain testing EMI, namely, Orion [9], Athena [15], and Hermes [17]. Orion
programs and optimization sequences is considered. tries to randomly prune unexecuted statements to generate
Threats to External Validity. The threats to external valid- variant programs [9], while Athena can delete code from or
ity mainly lie in duplicate bugs and testing programs. First, insert code into code regions that are not executed under the
inputs [15]. In contrast to Orion and Athena, Hermes [17] can
25. https://github.com/CTOS-results/LLVM-Bugs-by- generate variant programs via mutation performed on both
Optimization-sequences/blob/master/Appendix.pdf. live and dead code regions. An empirical study conducted
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2356 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

by Chen et al. [19] compares the strength of RDT, DOL, and evaluation demonstrates that CTOS significantly outper-
EMI, and reveals that DOL is more effective in detecting forms the baselines by detecting 24:76%
50:57% more bugs
compiler bugs related to optimizations. on average. Within only seven months, we have reported 104
To accelerate compiler testing, a method [63] based on valid bugs within 5 types, of which 21 have been confirmed
machine learning is proposed to predict the bug-revealing or fixed. 47 unique optimizations are identified to be faulty
probabilities of testing programs, such that the testing pro- and 15 of them are loop related optimizations.
grams with large bug-revealing probabilities can be exe- For future work, we will keep actively testing LLVM
cuted as early as possible. Recently, Chen [20] et al. present with CTOS, and report the detected bugs. Furthermore, we
a more efficient technique to predict test coverage statically plan to design more efficient compiler fuzzing techniques
for compilers, and then leverage the predicted coverage with coverage information of LLVM to further improve the
information to prioritize testing programs. reliability of compiler optimizations.
Our work is similar to DOL. However, unlike the tradi-
tional DOL which only considers the default optimization ACKNOWLEDGMENT
levels with fixed orders of optimizations, CTOS tests LLVM
We would like to thank the LLVM developers for analyz-
with arbitrary optimization sequences.
ing and fixing our reported bugs. This work was sup-
ported in part by the National Natural Science Foundation
7.2 Compiler Phase-Ordering Problem of China under Grants 61772107, 61722202, 61902181, and
The compiler phase-ordering problem aims to improve 62032004.
the performance of target programs by selecting good
optimization sequences [7], [8], [64]. Currently, two meth- REFERENCES
odologies have been proposed to resolve the compiler
[1] G. Fursin et al., “Milepost GCC: Machine learning enabled self-
phase-ordering problem. The approaches in the first cate- tuning compiler,” Int. J. Parallel Program., vol. 39, no. 3, pp. 296–
gory treat the compiler phase-ordering problem as an 327, 2011.
optimization problem and then evolutionary algorithms [2] J. Ansel et al., “OpenTuner: An extensible framework for program
autotuning,” in Proc. 23rd Int. Conf. Parallel Archit. Compilation,
are used to resolve it. For example, Kulkarni et al. [65],
2014, pp. 303–316.
[66] develop a method based on genetic algorithms for [3] S. Kulkarni and J. Cavazos, “Mitigating the compiler optimization
quickly searching effective optimization sequences. Purini phase-ordering problem using machine learning,” in Proc.
et al. [4] propose a downsampling technique to reduce the ACM Int. Conf. Object Oriented Program. Syst. Lang. Appl., 2012,
pp. 147–162.
infinitely large optimization sequence space. OpenTuner [4] S. Purini and L. Jain, “Finding good optimization sequences cov-
[2] uses the ensembles of search techniques to find opti- ering program space,” ACM Trans. Archit. Code Optim., vol. 9,
mal optimizations for a program. no. 4, pp. 56:1–56:23, 2013.
The approaches in the second category tackle the com- [5] L. G. Martins, R. Nobre, J. M. Cardoso, A. C. Delbem, and
E. Marques, “Clustering-based selection for the exploration of
piler phase-ordering problem based on machine learning compiler optimization sequences,” ACM Trans. Archit. Code
[1], [3], [6]. Most recent methods based on machine learning Optim., vol. 13, no. 1, 2016, Art. no. 8.
for compiler auto-tuning have been introduced by the sur- [6] A. H. Ashouri, A. Bignoli, G. Palermo, C. Silvano, S. Kulkarni,
and J. Cavazos, “MiCOMP: Mitigating the compiler phase-
vey [64]. Milepost [1] is a machine-learning based compiler ordering problem using optimization sub-sequences and mac-
that automatically adapts the internal optimization heuristic hine learning,” ACM Trans. Archit. Code Optim., vol. 14, no. 3,
to improve the performance. Kulkarni and Cavazos [3] pro- pp. 29:1–29:28, Sep. 2017.
pose a method based on the Markov process to mitigate the [7] D. B. Loveman, “Program improvement by source-to-source
transformation,” J. ACM, vol. 24, no. 1, pp. 121–145, 1977.
compiler phase-ordering problem. Ashouri et al. [6] leverage [8] S. R. Vegdahl, “Phase coupling and constant generation in an opti-
the optimization subsequences and machine learning to mizing microcode compiler,” ACM SIGMICRO Newslett., vol. 13,
build a predictive model. Huang et al. [67] present Auto- no. 4, pp. 125–133, 1982.
[9] V. Le, M. Afshari, and Z. Su, “Compiler validation via equivalence
Phase, a deep reinforcement learning method to tackle the modulo inputs,” in Proc. 35th ACM SIGPLAN Conf. Program. Lang.
compiler phase-ordering problem for multiple high-level Des. Implementation, 2014, pp. 216–226.
synthesis programs. [10] C. Lindig, “Random testing of C calling conventions,” in Proc. Int.
However, there is no guarantee that the programs opti- Symp. Automated Anal.-Driven Debugging, 2005, pp. 3–12.
[11] F. Sheridan, “Practical testing of a C99 compiler using output
mized by different optimization sequences are correct. No comparison,” Softw.: Practice Experience, vol. 37, no. 14, pp. 1475–1488,
systematic work has been conducted to detect compiler 2007.
bugs caused by optimization sequences. We present CTOS [12] C. Zhao, Y. Xue, Q. Tao, L. Guo, and Z. Wang, “Automated test
to mitigate this problem to further improve the reliability of program generation for an industrial optimizing compiler,” in
Proc. ICSE Workshop Autom. Softw. Test, 2009, pp. 36–43.
optimization sequences. [13] X. Yang, Y. Chen, E. Eide, and J. Regehr, “Finding and under-
standing bugs in C compilers,” ACM SIGPLAN Notices, vol. 46,
no. 6, pp. 283–294, 2011.
8 CONCLUSION [14] C. Lidbury, A. Lascu, N. Chong, and A. F. Donaldson, “Many-core
In this study, we presented CTOS, a method based on differ- compiler fuzzing,” in Proc. 36th ACM SIGPLAN Conf. Program.
Lang. Des. Implementation, 2015, pp. 65–76.
ential testing, for catching compiler bugs caused by optimi- [15] L. Vu, S. Chengnian, and S. Zhendong, “Finding deep compiler
zation sequences of LLVM. Rather than only testing bugs via guided stochastic program mutation,” in Proc. ACM SIG-
compilers with predefined optimization sequences like the PLAN Int. Conf. Object-Oriented Program. Syst. Lang. Appl., 2015,
pp. 386–399.
state-of-the-art methods, our technique validates compilers [16] L. Vu, S. Chengnian, and S. Zhendong, “Randomized stress-test-
with arbitrary optimization sequences, which significantly ing of link-time optimizers,” in Proc. Int. Symp. Softw. Testing
increases the test efficiency for detecting bugs. Our Anal., 2015, pp. 327–337.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 JIANG ET AL.: CTOS: COMPILER TESTING FOR OPTIMIZATION SEQUENCES OF LLVM 2357

[17] C. Sun, V. Le, and Z. Su, “Finding compiler bugs via live code [47] LLVM bug 40926, 2021. [Online]. Available: https://bugs.llvm.
mutation,” in Proc. ACM SIGPLAN Int. Conf. Object-Oriented Pro- org/show_bug.cgi?id=409–26
gram. Syst. Lang. Appl., 2016, pp. 849–863. [48] LLVM bug 40928, 2021. [Online]. Available: https://bugs.llvm.
[18] Q. Zhang, C. Sun, and Z. Su, “Skeletal program enumeration for org/show_bug.cgi?id=409–28
rigorous compiler testing,” in Proc. 38th ACM SIGPLAN Conf. Pro- [49] LLVM bug 40929, 2021. [Online]. Available: https://bugs.llvm.
gram. Lang. Des. Implementation, 2017, pp. 347–361. org/show_bug.cgi?id=409–29
[19] J. Chen et al., “An empirical comparison of compiler testing [50] LLVM bug 40933, 2021. [Online]. Available: https://bugs.llvm.
techniques,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 180–190. org/show_bug.cgi?id=409–33
[20] J. Chen et al., “Coverage prediction for accelerating compiler testing,” [51] LLVM bug 40925, 2021. [Online]. Available: https://bugs.llvm.
IEEE Trans. Softw. Eng., vol. 47, no. 2, pp. 261–278, Feb. 2021. org/show_bug.cgi?id=409–25
[21] LLVM Compiler Community, “LLVM language reference manual,” [52] C. Nie and H. Leung, “A survey of combinatorial testing,” ACM
2021. [Online]. Available: https://llvm.org/docs/LangRef.html Comput. Surv., vol. 43, no. 2, pp. 11:1–11:29, Feb. 2011.
[22] LLVM Compiler Community, “LLVM’s analysis and transform [53] D. R. Kuhn, R. N. Kacker, and Y. Lei, “SP 800-142. practical combi-
passes,” 2021. [Online]. Available: https://www.llvm.org/docs/ natorial testing,” Nat. Inst. Standards Technol., 2010.
Passes.html [54] A. Groce, C. Zhang, E. Eide, Y. Chen, and J. Regehr, “Swarm
[23] Clang, “Clang: A C language family frontend for LLVM,” 2021. testing,” in Proc. Int. Symp. Softw. Testing Anal., 2012, pp. 78–88.
[Online]. Available: http://clang.llvm.org/ [55] J. Chen, G. Wang, D. Hao, Y. Xiong, H. Zhang, and L. Zhang,
[24] Rust, “Rust program language,” 2021. [Online]. Available: “History-guided configuration diversification for compiler test-
https://www.rust-lang.org/ program generation,” in Proc. 34th IEEE/ACM Int. Conf. Automated
[25] Swift, “Swift program language,” 2021. [Online]. Available: Softw. Eng., 2019, pp. 305–316.
https://developer.apple.com/swift/ [56] A. Arcuri and L. Briand, “A practical guide for using statistical
[26] A. Haas et al., “Bringing the web up to speed with webassembly,” tests to assess randomized algorithms in software engineering,”
in Proc. 38th ACM SIGPLAN Conf. Program. Lang. Des. Implementa- in Proc. 33rd Int. Conf. Softw. Eng., 2011, pp. 1–10.
tion, 2017, pp. 185–200. [57] C. Sun, V. Le, and Z. Su, “Finding and analyzing compiler warn-
[27] C. Cadar, D. Dunbar, and D. Engler, “KLEE: Unassisted and auto- ing defects,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 203–213.
matic generation of high-coverage tests for complex systems pro- [58] J. Chen, J. Han, P. Sun, L. Zhang, D. Hao, and L. Zhang,
grams,” in Proc. 8th USENIX Conf. Operating Syst. Des. “Compiler bug isolation via effective witness test program gener-
Implementation, 2008, pp. 209–224. ation,” in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp.
[28] P. D. Schubert, B. Hermann, and E. Bodden, “PhASAR: An inter- Found. Softw. Eng., 2019, pp. 223–234.
procedural static analysis framework for C/C++,” in Proc. Int. [59] S. Pearson et al., “Evaluating and improving fault localization,” in
Conf. Tools Algorithms Construction Anal. Syst., 2019, pp. 393–410. Proc. 39th Int. Conf. Softw. Eng., 2017, pp. 609–620.
[29] Q. Le and T. Mikolov, “Distributed representations of sentences [60] Y. Chen et al., “Taming compiler fuzzers,” in Proc. 34th ACM SIG-
and documents,” in Proc. 31st Int. Conf. Int. Conf. Mach. Learn., PLAN Conf. Program. Lang. Des. Implementation, 2013, pp. 197–208.
2014, pp. II-1188–II-1196. [61] J. Holmes and A. Groce, “Using mutants to help developers dis-
[30] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, tinguish and debug (compiler) faults,” Softw. Test. Verification Rel.,
“Distributed representations of words and phrases and their vol. 30, no. 2, 2020, Art. no. e1727.
compositionality,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2013, [62] J. Chen, G. Wang, D. Hao, Y. Xiong, H. Zhang, and L. Zhang,
pp. 3111–3119. “History-guided configuration diversification for compiler test-
[31] W. M. McKeeman, “Differential testing for software,” Digit. Tech. program generation,” in Proc. 34th IEEE/ACM Int. Conf. Automated
J., vol. 10, no. 1, pp. 100–107, 1998. Softw. Eng., 2019, pp. 305–316.
[32] Z. Harris, “Distributional structure,” Word, vol. 10, no. 23, pp. 146–162, [63] J. Chen, Y. Bai, D. Hao, Y. Xiong, H. Zhang, and B. Xie, “Learning
1954. to prioritize test programs for compiler testing,” in Proc. IEEE/
[33] A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman, “Compilers: Prin- ACM Int. Conf. Softw. Eng., 2017, pp. 700–711.
ciples, Techniques, and Tools (2nd Edition),” Addison-Wesley [64] A. H. Ashouri, W. Killian, J. Cavazos, G. Palermo, and C. Silvano,
Longman Publishing Co., Inc., 2006. “A survey on compiler autotuning using machine learning,” ACM
[34] R. Johnson, D. Pearson, and K. Pingali, “The program structure Comput. Surv., vol. 51, no. 5, 2018, Art. no. 96.
tree: Computing control regions in linear time,” ACM SIGPLAN [65] P. A. Kulkarni, S. Hines, J. Hiser, D. B. Whalley, J. W. Davidson,
Notices, vol. 29, no. 6, pp. 171–185, 1994. and D. L. Jones, “Fast searches for effective optimization phase
[35] J. Vanhatalo, H. V€ olzer, and J. Koehler, “The refined process struc- sequences,” in Proc. ACM SIGPLAN Conf. Program. Lang. Des.
ture tree,” Data Knowl. Eng., vol. 68, no. 9, pp. 793–818, 2009. Implementation, 2004, pp. 171–182.
[36] Region graph, 2019. [Online]. Available: https://www.llvm.org/ [66] P. A. Kulkarni, S. R. Hines, D. B. Whalley, J. D. Hiser, J. W. Davidson,
doxygen/regioninfo_8h_-source.html and D. L. Jones, “Fast and efficient searches for effective optimiza-
[37] D. Pelleg and A. Moore, “X-means: Extending K-means with effi- tion-phase sequences,” ACM Trans. Archit. Code Optim., vol. 2, no. 2,
cient estimation of the number of clusters,” in Proc. 17th Int. Conf. pp. 165–198, 2005.
Mach. Learn., 2000, pp. 727–734. [67] A. Haj-Ali et al., “AutoPhase: Compiler phase-ordering for

[38] R. Rehu rek and P. Sojka, “Software framework for topic model- high level synthesis with deep reinforcement learning,” IEEE
ling with large corpora,” in Proc. LREC Workshop New Challenges 27th Annu. Int. Symp. Field-Programmable Custom Comput. Mach.,
NLP Frameworks, 2010, pp. 45–50. early access, pp. 308–308, doi: 10.1109/FCCM.2019.00049.
[39] J. H. Lau and T. Baldwin, “An empirical evaluation of doc2vec
with practical insights into document embedding generation,” in
Proc. 1st Workshop Representation Learn. NLP, 2016, pp. 78–86.
[40] D. A. Schult, “Exploring network structure, dynamics, and function He Jiang (Member, IEEE) received the PhD
using NetworkX,” in Proc. 7th Python Sci. Conf., 2008, pp. 11–15. degree in computer science from the University
[41] A. Novikov, “PyClustering: Data mining library,” J. Open Source of Science and Technology of China, Hefei,
Softw., vol. 4, no. 36, Apr. 2019, Art. no. 1230. [Online]. Available: China. He is currently a professor with the Dalian
https://doi.org/10.21105/joss.01230 University of Technology, China. He is also a
[42] A. Balestrat, “CCG: A random C code generator,” 2021. [Online]. member of the ACM and the CCF (China Com-
Available: https://github.com/Merkil/ccg/ puter Federation). He is one of the ten supervi-
[43] V. L. Dmitry Babokin and J. Regehr, “Yarpgen,” 2021. [Online]. sors for the Outstanding Doctoral Dissertation of
Available: https://github.com/intel/yarpgen the CCF in 2014. His current research interests
[44] R. Hodov an, A. Kiss, and T. Gyim othy, “Grammarinator: A gram- include search-based software engineering
mar-based open source fuzzer,” in Proc. 9th ACM SIGSOFT Int. Work- (SBSE) and mining software repositories (MSR).
shop Automating TEST Case Des. Selection Eval., 2018, pp. 45–48. His work has been published at premier venues like ICSE, SANER, and
[45] J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang, “Test- GECCO, as well as in major IEEE transactions like IEEE Transactions
case reduction for C compiler bugs,” in Proc. 33rd ACM SIGPLAN on Software Engineering, IEEE Transactions on Knowledge and Data
Conf. Program. Lang. Des. Implementation, 2012, pp. 335–346. Engineering, IEEE Transactions on Systems, Man, and Cybernetics,
[46] LLVM bug 40927, 2021. [Online]. Available: https://bugs.llvm. Part B (Cybernetics), IEEE Transactions on Cybernetics, and IEEE
org/show_bug.cgi?id=409–27 Transactions on Services Computing.
Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.
 2358 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 48, NO. 7, JULY 2022

Zhide Zhou received the BS degree in computer Jingxuan Zhang received the PhD degree in
science and technology from the Guilin University software engineering from the Dalian University
of Electronic Technology, Guilin, China, in 2013. of Technology, Dalian, China. He is a lecturer with
He is currently working toward the PhD degree at the College of Computer Science and Technol-
the Dalian University of Technology, Dalian, ogy, Nanjing University of Aeronautics and Astro-
China. He is a student member of the China nautics, China. His current research interests
Computer Federation (CCF). His current include mining software repositories and software
research interests include intelligent software data analytics.
engineering, software testing, and program anal-
ysis techniques.

Zhilei Ren received the BS degree in software

engineering and the PhD degree in computa- Xiaochen Li received the doctoral degree in soft-
tional mathematics from the Dalian University of ware engineering from the Dalian University of
Technology, Dalian, China, in 2007 and 2013, Technology, Dalian, China, in 2019 under super-
respectively. He is currently an associate profes- vision with Prof. He Jiang. He is currently a
sor with the Dalian University of Technology, research associate with Software Verification and
Dalian. His current research interests include Validation Research Group, University of Luxem-
evolutionary computation, automatic algorithm bourg, headed by Prof. Lionel Briand. His current
configuration, and mining software repositories. research interests include intelligent software
engineering and software semantic analysis. His
work has been published at premier venues like
IEEE Transactions on Software Engineering,
ACM Transactions on Software Engineering and Methodology, Empirical
Software Engineering, and ICSE. For more information, please visit
https://xiaochen-li.github.io.

" For more information on this or any other computing topic,

please visit our Digital Library at www.computer.org/csdl.

Authorized licensed use limited to: NORTHWESTERN POLYTECHNICAL UNIVERSITY. Downloaded on April 04,2024 at 02:29:43 UTC from IEEE Xplore. Restrictions apply.