This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 1

IEEE Transactions on Software Engineering (year:2018)

Reviving Sequential Program Birthmarking for

Multithreaded Software Plagiarism Detection
Zhenzhou Tian, Ting Liu, Member, IEEE, Qinghua Zheng, Member, IEEE, Eryue Zhuang, Ming Fan
and Zijiang Yang, Senior Member, IEEE

Abstract—As multithreaded programs become increasingly popular, plagiarism of multithreaded programs starts to plague the
software industry. Although there has been tremendous progress on software plagiarism detection technology, existing dynamic
birthmark approaches are applicable only to sequential programs, due to the fact that thread scheduling nondeterminism severely
perturbs birthmark generation and comparison. We propose a framework called TOB (Thread-oblivious dynamic Birthmark) that
revives existing techniques so they can be applied to detect plagiarism of multithreaded programs. This is achieved by thread-
oblivious algorithms that shield the influence of thread schedules on executions. We have implemented a set of tools collectively
called TOB-PD (TOB based Plagiarism Detection tool) by applying TOB to three existing representative dynamic birthmarks,
including SCSSB (System Call Short Sequence Birthmark), DYKIS (DYnamic Key Instruction Sequence birthmark) and JB (an
API based birthmark for Java). Our experiments conducted on large number of binary programs show that our approach exhibits
strong resilience against state-of-the-art semantics-preserving code obfuscation techniques. Comparisons against the three
existing tools SCSSB, DYKIS and JB show that the new framework is effective for plagiarism detection of multithreaded programs.
The tools, the benchmarks and the experimental results are all publicly available.

Index Terms—software plagiarism detection, multithreaded program, software birthmark, thread-oblivious birthmark

1 I NTRODUCTION software plagiarism detection a daunting task. Nev-

ertheless, researchers welcomed this challenge and
S OFTWARE plagiarism, an act of illegally copying
others’ code, severely affect both open source
communities and honest software companies. The
developed effective methods, of which software birth-
marking [7], [8] is a popular and recently well studied
recent incidents include the lawsuit against Verizon technique. A birthmark is a set of characteristics ex-
by Free Software Foundation for distributing Busybox tracted from a program that can be used to uniquely
in its FIOS wireless routers [1], and the crisis of identify the program. Depending on whether its ex-
Skype’s VOIP service for the violation of licensing traction relies on program runs, a software birthmark
terms of Joltid [2]. Unfortunately software plagiarism can be either considered static or dynamic. Static
is easy to implement but difficult to detect. A study birthmarks, generated mainly by analyzing syntactic
in 2012 [3] shows that about 5%-13% of apps in the features, are usually ineffective against semantics-
third-party app markets are copied and redistributed preserving obfuscations that can modify the syntactic
from the official Android market. The unavailability structure of a program. Besides, static birthmarks are
of source code and the existence of powerful au- easily defeated by packing techniques [9], [10] which
tomated semantics-preserving code obfuscation tech- can make processed executables rather different in
niques and tools [4]–[6] are a few reasons that make the static level. In contrast, dynamic birthmarks are
extracted based on runtime behaviors and thus are
• Z. Tian is with the Ministry of Education Key Lab For Intelligent believed to be more accurate reflections of program
Networks and Network Security (MOEKLINNS), Department of semantics and more robust against obfuscations [11]–
Computer Science and Technology, Xi’an Jiaotong University, Xi’an
710049, China, and with the School of Computer Science and Technol-
[15]. Thus in this paper we mainly focus on dynamic
ogy, Xi’an University of Posts and Telecommunications, Xi’an 710121, software birthmark methods.
China. Email: [email protected]. Despite the tremendous progress in software pla-
• T. Liu and Q. Zheng are with the Ministry of Education Key Lab For
Intelligent Networks and Network Security (MOEKLINNS), School
giarism detection technology, a new trend in software
of Electronic and Information Engineering, Xi’an Jiaotong University, development greatly threatens its effectiveness. The
Xi’an 710049, China. Email: {tingliu, qhzheng}@mail.xjtu.edu.cn. trend towards multithreaded programs is creating a
• E. Zhuang and M. Fan are with the Ministry of Education
Key Lab For Intelligent Networks and Network Security (MOEK-
gap between the current software development prac-
LINNS), Department of Computer Science and Technology, Xi’an tice and the software plagiarism detection technology,
Jiaotong University, Xi’an 710049, China. Email: {zhuangeryue, fan- as the existing dynamic approaches remain optimized
ming.911025}@stu.xjtu.edu.cn.
• Z. Yang is with the Department of Computer Science, Western Michi-
for sequential programs. Due to the perturbation
gan University, Kalamazoo, MI 49008, USA, and with the Department caused by non-deterministic thread scheduling, ex-
of Computer Science and Technology, Xi’an Jiaotong University, Xi’an isting birthmark generation and comparison are no
710049, China. Email: [email protected].
longer applicable to modern software with multiple

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on
IEEE TRANSACTIONS ONSoftware
SOFTWAREEngineering (year:2018)
ENGINEERING, VOL. X, NO. X, X 2

#include <stdio.h> TABLE 1

#include <pthread.h> Similarity scores calculated between birthmarks of
#include <semaphore.h> multiple runs of the sample program in Figure 1. The
#define N 8 column headings indicate the number of threads.
pthread_t mThread[N];
sem_t sem_a; (a) SCSSB
int tid;
void *run(void *data){ SCSSB 1 2 4 6 8
sem_wait(&sem_a); Cosine Distance 1.000 0.898 0.670 0.569 0.469
tid = (int)data; Jaccard Index 1.000 0.76 0.439 0.363 0.265
printf("hello world from thread %d\n", tid); Dice Coefficient 1.000 0.864 0.609 0.532 0.403
Containment 1.000 0.864 0.619 0.57 0.412
sem_post(&sem_a);
if(((int)data)%2==0){
(b) DYKIS
usleep(0);}
char str[25]; DYKIS 1 2 4 6 8
sprintf(str,"%04X",(int)data); Cosine Distance 1.000 1.000 0.968 0.708 0.471
return NULL;} Jaccard Index 1.000 1.000 0.874 0.437 0.311
int main(int argc, char *argv[]){ Dice Coefficient 1.000 1.000 0.926 0.6 0.459
int rc, i, j; Containment 1.000 1.000 0.984 0.6 0.482
sem_init(&sem_a, 0, 1);
printf("input a number please: \n");
scanf("%d", &i); j = i;
for (i; i < N; i++){ are all 1.0, pointing out correctly that the runs are
rc = pthread_create(&mThread[i], NULL, run, (void *)i); from identical programs. However, as the number of
if (rc) threads increase, the similarity scores quickly deteri-
printf("create thread failed. error code = %d\n", rc);} orate. As explained in Section 2, in birthmark-based
for (i = j; i < N; i++) techniques a threshold ε is used to indicate plagia-
pthread_join(mThread[i], NULL); rism. A similarity score greater than 1 − ε indicates
printf("main thread finished\n");
strong possibility of plagiarism, while a score less than
return 0;}
ε indicates the opposite. Considering typical value of
Fig. 1. A simple multithreaded program ε is between 0.15 and 0.35, SCSSB and DYKIS will not
claim plagiarism when the number of threads is 6, and
start to claim the runs are from different programs
threads. when the number of threads becomes 8.
The problems with existing dynamic birthmarks The example illustrates that the existing dynamic
can be illustrated by a multithreaded program shown birthmark based approaches are inadequate in de-
in Figure 1. The program is a test case in the WET tecting plagiarism of multithreaded programs because
project [16] with slight modifications. We apply two they neglect the effect of thread scheduling. Sequential
typical plagiarism detection algorithms on multiple program behavior is deterministically determined by
runs of this program. One is called System Call Short system inputs, including I/O, DMA, interrupts, thus
Sequence Birthmark (SCSSB) [15] and the other is a executions of highly similar programs under the same
more recent algorithm called DYKIS [17]. The details input are similar. This assumption no longer holds for
of both algorithms will be presented in Section 2. multithreaded programs since thread schedules are
If the existing approaches fail to claim plagiarism a major source of non-determinism. For a program
on different runs of the same small program, even with n threads, each executing k steps, there can
n k
without any code modifications, the capability of such be as many as (nk)!/(k!) > (n!) different thread
approaches is in doubt. interleavings, a doubly exponential growth in terms of
Dynamic birthmarks usually give quantitative mea- n and k. This indicates that two executions under the
surement between 0 and 1 to indicate the similarity same inputs can be very different, which invalidates
between two runs. A value of 1 indicates identicalness the basic assumption of existing approaches.
and 0 refers to complete difference. The measurement In this paper, we present TOB (Thread-Oblivious
is given by applying metrics, such as Cosine distance, dynamic Birthmark), a framework that can revive ex-
Jaccard index, Dice coefficient and Containment [13], isting dynamic birthmarks such as SCSSB [15], DYKIS
[15], [18], [19], on the birthmarks obtained by a pair of [17], JB [18] to handle multithreaded programs. Unlike
executions. Tables 1(a) and 1(b) show the experimental many prior approaches [20], [21], TOB operates on
results of SCSSB and DYKIS, where the column and binary executables rather than source code that is
row headings indicate the number of threads and the usually unavailable when birthmark techniques are
evaluation metrics, respectively. When there is only used to obtain initial evidence of software plagiarism.
one thread, the program becomes sequential. Without This paper extends our preliminary conference pa-
non-deterministic thread scheduling, the executions per [22], and makes the following contributions:
are deterministic. As expected, the similarity scores • To the best of our knowledge, this is the first work

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEEIEEE Transactions
TRANSACTIONS on Software
ON SOFTWARE Engineering VOL.
ENGINEERING, (year:2018)
X, NO. X, X 3

that discusses the impact of thread scheduling on p if and only if both of the following conditions are
birthmark based software plagiarism detection, satisfied:
and proposes a solution to remedy the problem. - f (p, I) is obtained only from p itself when exe-
• We apply the var-gram [23] algorithm in birth- cuting p with input I.
mark generation. As far as we know, this is the - Program q is a copy of p ⇒ f (p, I) = f (q, I).
first time this algorithm is used for such purpose.
Based on the two conceptual descriptions, various
Our experiments confirm its effectiveness.
implementable birthmarks have been developed by
• We have implemented a set of tools collectively
mining characteristics from different aspects of the
called TOB-PD (TOB based Plagiarism Detection
program, of which the SCSSB [15] extracted from
tool) by integrating the principle of TOB with
system calls, the DYKIS [13] extracted from executed
existing algorithms, including SCSSB [15],
instructions, the JB [18] extracted from executed Java
DYKIS [17] and JB [18]. The tools as well as
APIs are several representative dynamic birthmarks.
the source codes are publicly available at website:
The two conceptual definitions of software birth-
http://labs.xjtudlc.com/labs/wlaq/TAB-PD/site.
marks require that if two programs are in copy rela-
•Our experiments on 418 versions of 35 different
tion, their birthmarks should be the exactly the same.
multithreaded programs show that the new tools
But due to many practical issues in implementing
are highly effective in detecting plagiarism and
specific birthmarks, even if q is a copy of p, their
are resilient to most state-of-the-art semantics-
corresponding birthmarks are not identical. Thus in
preserving obfuscation techniques implemented
the literature of software birthmarking, the plagiarism
in tools such as SandMark [4], DashO [24] and
of two programs is decided by a threshold ε and
UPX. All benchmarks and the experimental data
a function sim that computes the similarity score
can also be downloaded from our website.
between their birthmarks. The range of a similarity
The remainder of the paper is organized as follows. score is between 0 and 1. Although a value of 0.25 was
Necessary background on software birthmarks are typically used as the threshold in previous studies,
described in Section 2. In Section 3 we present the other values were used as well and we found that
TOB framework that revives existing birthmarks. In the choice was quite arbitrary. Thus in our work, we
Section 4 the approaches for comparing the TOB- do not set ε to a particular value. Instead we analyze
revived birthmarks are discussed. The system design its impact on the performance under a wide range of
and implementation details are described in Section 5. values. Let p and pB be the plaintiff program and its
Section 6 presents the empirical study, including the birthmark, and q and qB be the defendant program
evaluation on resilience and credibility of the thread- and its birthmark. The plagiarism is decided with
oblivious birthmarks, the comparison with traditional Equation 1, which gives a conceptual definition of sim
SCSSB and the integration of TOB with DYKIS and that returns a three-value result: positive, negative or
JB. In Section 7 we discuss threats to the validity of inconclusive.
our approaches. Section 8 reviews related work and
finally we conclude the paper in Section 9. 
 > 1 − ε positive : q is a copy of p
sim (pB , qB ) = < ε negative : q is not a copy of p
2 P RELIMINARIES 
otherwise inconclusive
2.1 Birthmark based Plagiarism Detection (1)
A high quality birthmark manifests in that the ratio
A software birthmark, whose classical definitions are of incorrect classifications should be low enough for a
as depicted in Definition 1 and Definition 2, is a set certain ε. However, false negative is more intolerable
of characteristics extracted from a program statically than false positive, since birthmarking technique is
or dynamically, that reflects intrinsic properties of the not a proving techniques but rather a detecting tech-
program and that can be used to identify the program nique of suspected copies [14], [15], [17], [26]. Gener-
uniquely. ally in the literature, the following two properties of
Definition 1: Software Birthmark [8]. Let p be a a birthmark should be satisfied to make it valid. We
program and f be a method for extracting a set of refer to the definitions [17] restated from the original
characteristics from p. We say f (p) is a birthmark of descriptions of Myles [27] and Choi [19].
p if and only if both of the following conditions are Property 1: Resilience. Let p be a program and q be a
satisfied: copy of p generated by applying semantics-preserving
- f (p) is obtained only from p itself. code transformations τ . A birthmark is resilient to τ
- Program q is a copy of p ⇒ f (p) = f (q). if sim (pB , qB ) > 1 − ε.
Definition 2: Dynamic Software Birthmark [25]. Let Property 2: Credibility. Let p and q be independently
p be a program and I be an input to p. Let f (p, I) be a developed programs that may accomplish the same
set of characteristics extracted from p when executing task. A birthmark is credible if it can differentiate the
p with I. We say f (p, I) is a dynamic birthmark of two programs, that is sim (pB , qB ) < ε.

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on
IEEE TRANSACTIONS ONSoftware
SOFTWAREEngineering (year:2018)
ENGINEERING, VOL. X, NO. X, X 4

2.2 Three Representative Dynamic Birthmarks

2.2.1 SCSSB
SCSSB (System Call Short Sequence Birthmark) [15] is
a dynamic birthmark extracted from executed system
calls that are believed to be a fundamental runtime
indicator of program behavior. Modification of system
calls usually leads to incorrect programs, and there-
fore, a birthmark generated from sequence of system Fig. 2. A time window of executions between two
calls can be used to uniquely identify a program even threads of a multithreaded program
after it has been modified. In SCSSB, the sets of k-
length system call sequence is treated as the birth-
marks. The birthmark scales well, and their experi- program non-deterministic even under a fixed input.
mental results show resilience against either evasion The classical definition of dynamic software birth-
techniques enabled by different compilers or powerful mark is no longer correct because f (p, I) 6= f (q, I)
obfuscations provided in Sandmark [4]. However, even if q is a copy of p. In the following we give a
as non-deterministic thread scheduling perturbs the definition suitable for multithreaded programs.
order of system calls, SCSSB becomes ineffective in Definition 3: Thread-Oblivious dynamic
handling multithreaded programs. This is confirmed Birthmark. Let p, q be two multithreaded programs.
by our empirical study in Section 6. Let I be an input and s be a thread schedule to p and
q. Let f (p, I, s) be a set of characteristics extracted
2.2.2 DYKIS
from p when executing p with I and schedule s. We
DYKIS (DYnamic Key Instruction Sequence birth- say f (p, I, s) is a dynamic birthmark of p if and only
mark) [17] is extracted from executed instructions of if both of the following conditions are satisfied:
a binary executable. Rather than taking the whole
- f (p, I, s) is obtained only from p itself when
instructions for birthmark generation, the authors pro-
executing p with input I and thread schedule s.
pose the concept of key instructions. Specifically, they
- Program q is a copy of p ⇒ f (p, I, s) = f (q, I, s).
treat an instruction as key instruction if it is both
value-updating and input-correlated, where value- Similar to Definition 1 and Definition 2, Definition 3
updating refers to that the execution of an instruction provides an abstract guideline without considering
will generate new values rather than transfer values, any implementation details. In practice it is almost
and input-correlated refers to that the execution of an impossible to predetermine a thread schedule and
instruction will introduce new taint labels (dynamic enforce the same thread scheduling across multiple
taint analysis is conducted to acquire this). DYKIS runs, especially for the programs that have been
is then extracted from a key instruction sequence of obfuscated or even independently developed [28],
length k. Operating directly on the assembly instruc- [29]. Therefore instead of enforcing thread schedules
tions, DYKIS is able to detect cross-platform plagia- in our algorithms, we try to shield the influence
rism. Also, the experimental evaluation shows that of thread schedules on executions. That is, to make
it is resilient against various obfuscation techniques. a birthmark thread-oblivious, we must ensure that
However, suffering from the same issue as SCSSB, ∀s1 ,s2 ∈S , f (p, I, s1 ) ≈ f (p, I, s2 ), where S denotes the
DYKIS cannot be applied for plagiarism detection of set of all possible thread schedules of program p.
multithreaded programs.
3.1 Intuition
2.2.3 JB
Figure 2 depicts a typical execution snippet between
Different from SCSSB and DYKIS that operate on
two possible threads of a multithreaded program. De-
binary executables, JB [18] is a dynamic birthmark
spite there can be many different thread interleavings,
for Java programs. It observes short sequences of
the order of the events happened in each single thread
Java Standard API calls received by individual ob-
seems not affected by thread scheduling. That is, if e1
jects. Similar to both SCSSB and DYKIS, an API call
happens before e2 in an execution, the order of two
trace is chopped into a set of short sequences to
events from the same thread will remain the same
generate birthmarks. Organizing traces on a per-object
even under different thread interleavings.
basis makes JB less susceptible to thread scheduling.
Based on the observation, we project traces on
However, we will demonstrate that it can be further
individual threads, and then check whether two traces
improved with our TOB framework.
match each other after the projection. Consider the
system call traces comprised of eight threads that
3 T HREAD -O BLIVIOUS B IRTHMARKS correspond to the case as indicated by the last column
As illustrated by the example in Figure 1, thread in Table 1(a). As depicted in Figure 3, almost identical
scheduling makes the behavior of a multithreaded traces are observed (except subtle differences on the

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE
IEEE TransactionsONonSOFTWARE
TRANSACTIONS Software ENGINEERING,
Engineering (year:2018)
VOL. X, NO. X, X 5

Thread ID Thread Slice Thread Slice Thread ID

3 311, 240, 4, 240, 162, 219, 1 311, 240, 4, 219, 1 3'
scores calculated without and with the trace refining.
2 311, 4, 240, 162, 219, 1 311, 4, 240, 162, 219, 1 2' As indicated by the scores that are all near 1.0 in
1 311, 240, 4, 240, 219, 1 311, 240, 4, 240, 219, 1 1' the white columns, our hypothesis applies to the real
0 45, 192, 5, , 91, 4, 252 45, 192, 5, , 91, 4, 252 0'
world multithreaded programs.
7 311, 240, 4, 219, 1 311, 4, 162, 219, 1 7'
6 311, 240, 4, 240, 162, 219, 1 311, 4, 219, 1 6' Considering that more threads usually lead to more
5 311, 240, 4, 240, 219, 1 311, 4, 219, 1 5' complex thread interleaving, we vary the number of
4 311, 240, 4, 240, 219, 1 311, 4, 162, 219, 1 4' threads that each program starts with. The left figure
in Figure 4 depicts how the average similarity scores
Fig. 3. Trace comparison on the thread level. System
change for each program as the number of threads
call No. is used to represent each system call instance.
increases. As it shows, the scores calculated with
the TOB-revived birthmarks are all very high. There
is also no significant variation across the x-axis. It
system call futex numbered 240) after projecting the
indicates that the number of threads has negligible
traces on individual threads. Futex is called only when
effect on the hypothesis.
it is likely that the program has to block for a longer
In the third experiment, we validate whether the
time until the condition becomes true. Its occurrences
hypothesis holds under different workloads. The
show intrinsic randomness under different executions,
benchmark programs all come with input of dif-
as futex is essentially designed to reduce the number
ferent sizes, each representing a different workload.
of system calls so as to improve performance. Thus
Specifically, we use five levels of inputs, arranged in
in our implementation, as described in Section 5, we
ascending order in terms of the workloads, to drive
treat them as noises. After refining the traces, exact
the executions. The five levels include Test, Simdev,
match among all the traces under the same inputs for
Simsmall, Simmedium and Simlarge. There is a large
the sample program can be observed.
span between different levels of inputs. For example,
To validate whether the hypothesis (that projected
the size of the Test input for dedup is just 6 byte,
events within individual threads are relatively
while the size of its Simlarge input is 184MB. Thus, the
stable under different thread interleavings) also
inputs provided by the benchmarks can properly test
holds among real world multithreaded programs,
the interleaving situation of a multithreaded program
we further investigated in detail the traces of ten
under different levels of workloads. The right figure
multithreaded programs, including blackscholes,
in Figure 4 summarizes the results. As it shows, the
bodytrack, fludanimate, canneal, dedup,
average similarity scores are all very high regardless
ferret, freqmine, streamcluster, swaptions,
of the workload, and do not exhibit significant differ-
and x264, from the PARSEC benchmark suite [30].
ences. This indicates that the effect of different levels
These programs are typical complex multithreaded
of workload on the hypothesis is trivial.
programs covering different domains, such as
Finally, the thread interleaving can be affected by
computer vision, video encoding, financial analytic,
scheduling policies. There are basically three schedul-
animation physics and image processing. They come
ing policies supported by the Linux kernel, including
with inputs of different sizes, and allow different
SCHED OTHER, SCHED FIFO (realtime first-in-first-
number of threads to start with.
out) and SCHED RR (realtime round-robin). Since all
In our initial empirical study, we run each pro-
the benchmark programs adopt the default scheduling
gram with four threads 1 under the same inputs
policy (SCHED OTHER), we make slight modifica-
multiple times. After projecting the traces on indi-
tions to the source code in order to support the
vidual threads, traces executed by blackscholes,
other two scheduling policies. Table 3 summarizes the
fludanimate, canneal and streamcluster show
average scores with respect to different scheduling
perfect match. The projected traces executed by other
policies. As the data shows, all the scores are close
programs show exact match under some inputs, while
to 1.0 and score variations between different schedul-
present subtle differences under some other inputs.
ing policies are trivial. The results indicate that the
After manual examination, we conclude that the sub-
scheduling policy has little effect on the hypothesis.
tle differences are due to the fact that a thread’s behav-
Based on these observations, we propose a solution
ior may be affected by other threads. In Section 5 we
to shield the influence of thread scheduling in birth-
discuss this issue further and provide an optimization
mark generation. Figure 5 depicts our approach of
to alleviate the problem. To give a more intuitive
reviving existing birthmarks. Given a multithreaded
view of the matching, we compute the average of
execution trace, TOB projects the trace on individual
similarity scores between the traces obtained under
threads. Each thread slice is insensitive to thread
same inputs, with the TOB-revived SCSSBs (to be
scheduling so we can create a birthmark using exist-
discussed in the following sections) for each program.
ing techniques. Then TOB uses two models to com-
The gray and white columns in Table 2 summarize the
bine individual thread slice birthmarks into a thread-
1. According to the document of the PARSEC benchmark suite, oblivious birthmark for the multithreaded execution.
the actual number of threads can be higher. We use the rest of this section to explain our approach

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 6

TABLE 2
Average of the similarity scores between the traces obtained under same inputs. For simplicity, we use SA and
SS to denote the TOB-revived birthmarks. The gray and white columns summarize the scores calculated
without and with trace refining, respectively.

Cosine Distance Jaccard Index Dice Coefficient Containment

SA SS SA SS SA SS SA SS
blackscholes 0.990 1.000 0.989 1.000 0.935 1.000 0.936 1.000 0.965 1.000 0.965 1.000 0.970 1.000 0.970 1.000
bodytrack 0.995 1.000 0.992 1.000 0.873 1.000 0.873 1.000 0.929 1.000 0.921 1.000 0.937 1.000 0.944 1.000
canneal 0.972 1.000 0.979 1.000 0.944 1.000 0.948 1.000 0.960 1.000 0.964 1.000 0.963 1.000 0.968 1.000
dedup 0.941 0.991 0.819 0.989 0.568 0.991 0.502 0.989 0.714 0.994 0.619 0.989 0.738 0.997 0.672 0.989
ferret 0.896 1.000 0.860 1.000 0.579 0.995 0.727 1.000 0.703 0.998 0.783 1.000 0.720 1.000 0.827 1.000
fludanimate 0.943 1.000 0.950 1.000 0.906 1.000 0.924 1.000 0.924 1.000 0.935 1.000 0.927 1.000 0.938 1.000
freqmine 0.963 0.997 0.973 0.996 0.766 0.932 0.785 0.950 0.857 0.962 0.872 0.971 0.864 0.965 0.882 0.975
streamcluster 0.884 0.996 0.885 0.998 0.800 0.993 0.878 0.998 0.839 0.995 0.881 0.998 0.846 0.996 0.884 0.998
swaptions 0.989 0.992 0.983 0.988 0.922 0.978 0.918 0.988 0.956 0.986 0.943 0.988 0.966 0.993 0.953 0.988
x264 0.845 0.999 0.889 0.999 0.795 0.998 0.866 0.999 0.820 0.999 0.877 0.999 0.829 0.999 0.888 0.999

1 1
0.9 0.9
Containment Similarity Score

Containment Similarity Score

0.8 0.8
0.7 0.7
0.6 0.6
0.5 0.5
0.4 blackscholes bodytrack canneal 0.4 blackscholes bodytrack canneal
0.3 dedup ferret fluidanimate 0.3 dedup ferret fluidanimate
0.2 freqmine streamcluster swaptions 0.2 freqmine streamcluster swaptions
0.1 x264 0.1 x264

0 0
1 2 4 6 8 10 15 30 50 Test Simdev Simsmall Simmedium Simlarge
Number of Threads Specified Different Workloads

Fig. 4. Validation of the hypothesis under different number of threads and different workloads. Since similar
results are observed, only Containment scores measured for SA birthmarks are given here to save space.

TABLE 3
Validation of the hypothesis under different scheduling policies. To save space, only results measured with SA
birthmarks are given here.

Cosine Jaccard Dice Containment

OTHER FIFO RR OTHER FIFO RR OTHER FIFO RR OTHER FIFO RR
blackscholes 1.000 1.000 0.999 1.000 1.000 0.999 1.000 1.000 0.999 1.000 1.000 0.999
bodytrack 1.000 1.000 1.000 0.994 0.989 0.990 0.997 0.994 0.995 0.999 0.997 0.999
canneal 1.000 0.999 0.999 0.999 0.998 0.998 0.999 0.999 0.998 0.999 0.999 0.999
dedup 0.993 0.980 0.992 0.973 0.960 0.976 0.983 0.971 0.985 0.992 0.979 0.992
ferret 0.999 0.999 0.999 0.986 0.989 0.985 0.993 0.994 0.992 0.997 0.998 0.997
fluidanimate 1.000 1.000 1.000 0.999 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000
freqmine 0.996 0.995 0.996 0.912 0.902 0.894 0.951 0.945 0.940 0.954 0.948 0.944
streamcluster 0.996 0.996 0.998 0.990 0.990 0.994 0.993 0.993 0.996 0.996 0.996 0.998
swaptions 0.994 0.993 0.996 0.986 0.987 0.990 0.991 0.991 0.993 0.994 0.995 0.997
x264 1.000 1.000 0.999 0.998 0.998 0.998 0.999 0.999 0.999 1.000 0.999 0.999

in details. Note that although in this paper we fo- trace with thread identifier. It then project the trace on
cus on birthmarks in set format, birthmarks in other thread identifiers to obtain sub-traces, each of which
formats such as sequences or graphs can also utilize belongs to a single thread. As a result, the birthmarks
TOB to handle multithreaded programs. In the latter extracted from the sub-traces can remain same even
case appropriate algorithms for thread slice birthmark under different thread schedules.
generations and comparisons need to be developed.
Formally, let an execution trace of program p un-
der input I be trace(p, I) = he1 , e2 , · · · , en i. Each
3.2 Birthmark for Individual Threads recorded event ei is an instruction, a system call
In order to shield the influence of non-deterministic or an API, along with its thread identifier that
scheduling, TOB annotate each event in an execution is denoted as ei .tid. We define its projection on

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions
IEEE TRANSACTIONS on Software
ON SOFTWARE Engineering
ENGINEERING, (year:2018)
VOL. X, NO. X, X 7

The two models produce thread-oblivious birthmarks

birthSA (p, I) and birthSS (p, I) from thread slice birth-
marks. We also use birth(p, I) to represent a birthmark
if we do not care whether it is obtained by SA or SS.
Definition 5: Slice Aggregation Model.
The slice aggregation model is a map
f
→ birthSA (p, I), where
f : {birth (p, I, t) |0 ≤ t < m} −
thread-oblivious
m is the number of threads and birthSA (p, I) =
{(g, f req(g, I))|g ∈ ∪ gram(p, I, t), f reg(g, I) =
Fig. 5. TOB framework that revives traditional birth- P 0≤t<m

mark to thread-oblivious birthmark (f req(g, I, t))}.

0≤t<m
Definition 6: Slice Set Model. The slice set model is
f
thread t to be an ordered subsequence slice (p, I, t) = → birthSS (p, I),
a map f : {birth (p, I, t) |0 ≤ t < m} −
hei |ei ∈ trace (p, I) ∧ ei .tid = ti. The projections of all where m is the number of threads and birthSS (p, I) =
the threads appearing in the trace form a partition {(t, birth(p, I, t)) |0 ≤ t < m}
of trace(p, I), and we refer to each subsequence
slice(p, I, t) as a thread slice. 4 P LAGIARISM D ETECTION WITH TOB-
TOB further abstract each subsequence into a set of R EVIVED B IRTHMARKS
short sequences. Two methods are used to partition a
sequence. One is the typical k-gram algorithm that is The approaches provided by TOB framework lead
widely used in birthmark generation literature [13], to four types of thread-oblivious birthmarks, i.e., SA
[15], [18], [27]. which partitions a sequence with a birthmarks and SS birthmarks extracted with k-gram
length k windows, generating a set of fixed-length or var-gram algorithms, when reviving a traditional
short sequences called k-grams. The other is the var- dynamic birthmark. In this section, we describe how
gram algorithm [23] that divides a subsequence into to detect plagiarism on top of these birthmarks.
variable-length short sequences called var-grams. var-
gram algorithm is widely used in behavior pattern 4.1 Similarity Calculation
mining, and is believed to be more natural description Different similarity calculation methods should be
of program behavior [31], [32]. In this paper, it is adopted depending on the specific format of birth-
applied for the first time to extract birthmarks. marks. State-of-the-art birthmarks mainly exist in
Similar to the typical dynamic birthmarks such as three forms: sequences, sets and graphs. For birth-
SCSSB, DYKIS, and JB, the set of key-value pairs marks in sequence format, their similarity can be
is taken as the birthmark for each thread slice. The computed with pattern matching methods, such as
keys and values consist of all unique grams and their calculating the longest common subsequences [33]
corresponding frequencies, respectively. Definition 4 [34]. Birthmarks in set form are usually generated by
gives the definition of thread slice birthmark. dividing sequences into short subsequence to make
Definition 4: Thread Slice Birthmark. Given a comparisons efficient. SCSSB, DYKIS, as well as many
thread slice slice(p, I, t), we treat the key-value pair other birthmark methods [8], [13], [18], [27], [35],
set birth (p, I, t) = {(g, f req (g, I, t))} as the thread [36] utilize such principle. Then various methods
slice birthmark of slice(p, I, t), where g is a unique widely used in the field of information retrieval are
gram and f req(g, I, t) is its frequency in gram(p, I, t). adopted for calculating the similarity between sets,
including Dice coefficient [19], Jaccard index [18],
3.3 Thread-Oblivious Birthmark Generation and Cosine distance [13]. Computing the similarity
With the availability of birthmark for each individual of graphs is relatively more complex. It is conducted
thread, TOB provides two models, Slice Aggregation by either graph monomorphism [11] or isomorphism
(SA) and Slice Set (SS), to generate thread-oblivious algorithms [20], or by translating a graph into a vector
birthmarks for multithreaded programs. Defined in using algorithms such as random walk with restart
Definition 5, SA generates birthmarks by aggregating [37], or possibly by mapping graphs to values, with
all thread slice birthmarks into a single set of key- techniques such as the Centroid [38], [39] and struc-
value pairs, where the keys are the unique k-grams ture measurement [40]. Unfortunately these existing
or var-grams in any of the thread slice birthmark, methods cannot be directly applied for comparing
and the values are frequencies of corresponding k- thread-oblivious birthmarks.
grams or var-grams. If a key is owned by multiple
thread slice birthmarks, its frequency is the sum of the 4.1.1 Similarity Calculation for SA Birthmarks
individual frequencies. As described in Definition 6, According to its definition, birthSA (p, I) is in the
SS treats the thread identifiers as the keys and their format of key-value pair set, therefore similarity com-
corresponding thread slice birthmarks as the values. putation methods such as Cosine distance, Jaccard

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 8

index, Dice coefficient and Containment can be used2 .

However, two different thread-oblivious birthmarks
may be considered identical because these metrics
do not consider frequency of the elements. In order
to address this issue we add a factor θ to each of
the traditional metric to take into consideration the
frequencies of the keys in a birthmark.
Given a SA birthmark A =
{(k1 , v1 ) , (k2 , v2 ) , · · · , (kn , vn )}, let kSet(A) be
the set of keys in A. That is, kSet (A) =
{k1 , k2 , · · · , kn }. Given a second SA birthmark
B = {(k10 , v10 ) , (k20 , v20 ) , · · · , (km
0 0
, vm )}, let
U = kSet (A) D∪ kSet (B). WeE convert set U to
→
vector U = k100 , k200 , · · · , k|U
00
by assigning an Fig. 6. Schematic diagram of bipartite matching mod-
|
elling for birthmarks generated by the SS model
arbitrary order to the elements in U . Let vector
→
−

A = a1 , a2 , · · · , a|u| , where
vi , if (ki00 , vi ) ∈ A

ai = Since the main purpose of software birthmarking
0, if (ki00 , vi ) ∈ /A is to detecting rather than proving plagiarism, false
→
−
 negative is more serious than false positive. Con-
Likewise we define B = b1 , b2 , · · · , b|U | . And we sidering that further investigation can be conducted
calculate θ as: once plagiarism is detected, we calculate the maximal
→  →
min A , B 
    similarity between two SS birthmarks. To achieve
θ= →  → this, we reduce the problem of calculating similar-
max A , B 
    ity between SS birthmarks into finding a maximum
→ weighted bipartite matching as illustrated in Figure 6.
−  r P 2
→
−  r P 2
 
where  A  = ai , and  B  = bi . In particular, each node marked by a thread identifier
→
ai ∈ A
→
bi ∈ B
corresponds to a thread slice birthmark of the thread,
Thus the modified metrics are defined as following: and a weighted edge denotes the similarity between
→ →
two thread slice birthmarks.
Ex − Cosine (A, B) = A • B
→→ × θ; Formally, let A= {(t1 , birth (p, I, t1 )) , · · · ,
 A  B 
|A∩B| (tm , birth (p, I, tm ))} and B= {(t01 , birth (p0 , I, t01 )) ,
Ex − Jaccard (A, B) = |A∪B| × θ; · · · , (t0n , birth (p0 , I, t0n ))} be two SS birthmarks. A
2|A∩B|
Ex − Dice (A, B) = |A|+|B| × θ; m × n similarity matrix is generated by comparing
Ex − Containment (A, B) = |A∩B| |A| × θ; between every pair of thread slice birthmarks in
A and B using either of the four metrics defined
The similarity of two SA birthmarks, simc (A, B), can in Section 4.1.1. Note that the definition can be
be calculated with Ex-Cosine(A, B), Ex-Jaccard(A, B), applied to thread slice birthmarks directly. For
Ex-Dice(A, B) or Ex-Containment(A, B), by specifying example, when Ex-Jaccard is chosen, simc (t1 , t01 ) =Ex-
c to either of the four metrics. Jaccard(birth(p, I, t1 ), birth(p0 , I, t01 )).

4.1.2 Similarity Calculation for SS Birthmarks

simc (t1 , t01 ) simc (t1 , t0n )
 
...
The SS birthmarks are also in the format of key-value  simc (t2 , t01 ) ... simc (t2 , t0n ) 
pair set. But unlike SA birthmarks where the keys are simM atr (A, B) = 

.. .. ..


grams and values are corresponding frequencies, each  . . . 
key-value pair in SS birthmarks consists of a thread simc (tm , t01 ) ... simc (tm , t0n )
identifier and its thread slice birthmark. Although
the keys, i.e. thread identifiers, have clear physical A valid match can be found by applying
meaning, such definition actually makes it difficult any maximum weighted bipartite matching
to compare SS birthmarks. This is because a thread algorithms, formally denoted as M axM atch (A, B) =
identifier is just a label that is assigned by another {(u1 , v1 ) , (u2 , v2 ) , · · · , (ul , vl )} where l = min (m, n),
forking thread during dynamic execution. This means ui ∈ kset (A), vi ∈ kset (B), ui 6= uj if i 6= j,
thread identifies can vary across different runs of Pl
same or difference programs. vi 6= vj if i 6= j, and simc (ui , vi ) has the
i
maximum value among all possible matchings.
2. All of the four metrics were used to compute birthmark
similarities. We implemented all of them after slight modification Finally, the similarity of two SS birthmarks are
in our prototype. calculated with the following formula: sim (A, B) =

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 9

simc (ti ,t0j )×(cnt(ti )+cnt(t0j ))

P
(ti ,t0j )∈M axM atch(A,B) each line corresponds to a system call record. A sys-
m n ,
P
cnt(ti )+
P
cnt(t0j ) tem call record contains a thread identifier, a system
i=1 j=1
call number with which we uniquely identify each
where
cnt
 (ti ) = |kSet

(birth (p, I, ti ))|, and
system call, the name of the system call, parameters
cnt t0j = kSet birth p0 , I, t0j .
when it is called, and its return value. These attributes
are separated with “#”.

4.2 Plagiarism Detection

5.2 Pre-Processor
The purpose of extracting birthmarks and calculating
The raw sequences extracted by sysTracer are not
their similarity is to eventually determine whether
appropriate for birthmark generation yet. The pre-
there exists plagiarism. False negatives are possible
processor needs to parse the system call records and
due to sophisticated code obfuscation techniques that
eliminate unnecessary information. First, failed calls
camouflage stolen software. One of our goals is to
are eliminated from the system call sequence because
make our approach resilient to these techniques and
they do not affect the behavior characteristics of a
tools. False positives, are also possible, even though
program [14], [15]. This is accomplished by checking
executions faithfully reveal program behavior un-
the return value attribute of each system call record.
der a particular input. For example, two indepen-
In addition, as discussed in Section 3.1, the nature
dently developed programs adopting standard error-
of system call futex determines that its occurrences
handling subroutines may exhibit identical behavior
vary across multiple executions. Besides, the memory
under error-inducing inputs [17]. In order to alleviate
management system calls such as mmap, brk etc. also
this problem, the calculation of the similarity score
show intrinsic randomness across different runs. We
between two programs is based on various birthmarks
treat all the records corresponding to these system
obtained under multiple inputs.
calls as noises, and remove them from the sequence.
Let p and q be the plaintiff and defendant. Given a
We know that for multithreaded programs multiple
set of inputs {I1 , I2 , · · · , In } to drive the execution of
executions under the same input may vary signifi-
the programs, we obtain n pair of birthmarks (for each
cantly, which is the reason that we are focusing on
birthmark type) {(A1 , B1 ) , (A2 , B2 ) , · · · , (An , Bn )}.
more stable thread slices for our birthmark generation.
The similarity score between programs p and q is
Pn However, even thread slices may vary due to the facts
calculated by sim (p, q) = sim (Ai , Bi ) n. That is, that a thread’s behavior may be affected by other
i=1
the existence of plagiarism between p and q is decided threads and there exist other random factors like OS-
by the average of similarity scores of their birthmarks state related operations. In particular, a system call
and the threshold ε. that has been executed in one thread in the first run
may appear in another thread in the second run.
Since such randomness are unavoidable, we execute
5 D ESIGN AND I MPLEMENTATION a program multiple times under the same input to
We have applied the TOB framework to revive three obtain a series of traces. We then select two most
typical birthmarks, including SCSSB [14], DYKIS [17] similar traces for plaintiff and defendant within the
and JB [18]. This leads a set of tools that are collec- TraceSelector, implemented as a sub-module of
tively called TOB-PD. All the tools follow the same the pre-processor. In our implementation, by default
workflow as depicted in Figure 7, except for some TraceSelector select traces based on their Ex-
differences in the implementation of each module. For Containment similarity, but other similarity metrics
example, the tracer for DYKIS is implemented on top can be used as well. Effectiveness of this optimization
of PIN [41] to monitor executed instructions of binary will be evaluated in Section 6.3.4.
executables, while the tracer for JB is implemented
with ASM [42] to monitor API calls of Java programs. 5.3 Birthmark Generator
To avoid redundancy, in this paper we mainly discuss
Following the procedure depicted in Figure 5, the
the implementation details of applying TOB to revive
birthmark generator performs thread slicing on a pre-
SCSSB. We name the enhanced thread-oblivious birth-
processed trace, generates thread slice birthmarks and
marks SCSSBSA and SCSSBSS accordingly.
then produce the thread-oblivious birthmarks using
the SA and SS models. Both k-gram and var-gram
5.1 Tracer algorithms are implemented. There are several algo-
The tracer for SCSSB is called sysTracer, which is rithms for mining var-grams and we choose Teire-
implemented as a PIN [41] plugin. By instrumenting sias [23], a well-known algorithm initially developed
each call point to recognize the system calls and for discovering rigid patterns in unaligned biological
collect required information, sysTracer produces sequences. It outperforms other var-gram algorithms
a system call sequence for each program run. The [43], [44] in that it is capable of discovering all patterns
format of the sequence is illustrated in Figure 8, where without enumerating the solution space.

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 10

Plaintiff and
Birthmark Similarity Decision
Defendant Tracer Pre-Processor
Generator Calculator Maker

Fig. 7. Overview of the TOB based plagiarism detection tool TOB-PD

2#4#__NR_write#(0x6, 0xb2fcd0ef, 0x1, 0x90307d0, 0x1, 0xb2fcd064)#0x1

2#162#__NR_nanosleep#(0xb2fcd340, 0x0, 0x1bbc, 0xb1367010, 0x1bbc, 0xb2fcd248)#0x2 Jshrink, ProGuard and RetroGuard,
4#102#__NR_socketcall#(0x9, 0xb13570c0, 0xb41bfff4, 0x0, 0xb05004bc, 0xb13570a8)#0x14
4#3#__NR_read#(0x5, 0xb1357202, 0xa, 0x9030748, 0xa, 0xb1357164)#0x1
that provide strong obfuscations.
4#168#__NR_poll#(0xb0500498, 0x2, 0xffffffff, 0x0, 0xb5dfbff4, 0xb1357160)#0xffffffff – S3 : We utilize packing tool UPX to obfuscate
binaries.
Fig. 8. Instances of elements comprising the system – S4 : We generate programs that adopt differ-
call sequence ent scheduling policies.
• We evaluate the credibility of our methods with
independently developed programs in the same
5.4 Similarity Calculator & Decision Maker
and different categories.
These two modules implement the algorithms dis- • All the programs are executed with multiple in-
cussed in Sections 4.1 and 4.2, respectively. In the puts that represent different workloads. And a
similarity calculator, scores are computed with each program is executed four times for each input.
of the four similarity metrics for SA birthmarks. For All the execution traces are collected on a desktop
SS birthmarks, we use each of the four metrics to (with Intel(R) [email protected] CPU and 2.0GB
calculate the weight of the edges in the bipartite Memory) that runs the Linux (Ubuntu 12.04)
graph. This produces four SimM atr correspondingly. system.
For each SimM atr, the Kuhn-Munkres algorithm [45] With these settings, we mainly evaluate the re-
is used to find a maximum weighted bipartite match- silience and credibility of the thread-oblivious birth-
ing. The decision maker decides plagiarism by the marks on the specific TOB implementation SCSSBSA
average value of multiple similarity scores against a and SCSSBSS . The overall quality of the TOB-revived
predefined threshold ε that varies between 0 and 0.5. birthmarks are further compared with the original
SCSSB [15] and between each other3 , with respect
6 E XPERIMENTS AND E VALUATION to three performance metrics. Finally, we illustrate
the effectiveness and easiness of applying the TOB
We have conducted extensive experiments for eval- framework to two other typical dynamic birthmarks.
uating the effectiveness of our TOB method. Table 4 It should be noted that, for birthmarks generated
lists the names and some other basic information of with the k-gram algorithm, different values of k lead
our benchmarks. The columns under #Ver give the to different birthmarks even for the same execution
number of versions of each program, where Column trace. In previous studies [15], [17], [18], [22] where
Total gives the total number of versions including the k-gram is also used to generate birthmarks, setting
original program and its obfuscated versions, while the value of k to 4 or 5 is believed to be a proper
the other four columns S1 , S2 , S3 and S4 give the compromise between accuracy and efficiency. To be
number of obfuscated versions generated with differ- fair, when reviving these traditional birthmarks and
ent obfuscation strategies. The meaning of S1 , S2 , S3 comparing with them, we adopt the same k values.
and S4 are explained below. Column Size gives the
number of kilobytes in the largest version, with its
version number listed in Column Version. Following 6.1 Validation of Resilience Property
is a summary of our testing environment. 6.1.1 Resilience to Different Compilers and Optimiza-
• The benchmarks consist of 35 mature multi- tion Levels
threaded software implemented in C/C++ or Different compilers and optimization levels usually
Java. lead to different binaries from the same source code.
• We process the programs with the following ob- We firstly validate whether the revived birthmarks
fuscation strategies for evaluating the resilience can handle such relatively weak semantics-preserving
of our methods. code transformations. Specifically, we choose the ten
– S1 : We use two different compliers gcc and
3. Note that not all programs can start execution with a specified
llvm with various optimization levels to number of threads. Even if the number of threads can be specified,
compile source code. there is no guarantee that certain number can be maintained when
– S2 : We apply publicly available code program runs. Thus, for the sake of fairness, when comparing
the traditional and TOB-revived birthmarks, we do not manually
obfuscators, including Sandmark, Zelix specify but let a program itself to determine the number of threads
KlassMaster, Allatori, DashO, to handle different workloads.

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 11

TABLE 4
Benchmark programs

#Ver #Ver
Name Size(kb) Version Name Size(kb) Version
Total S1 S2 S3 S4 Total S1 S2 S3 S4
pigz 294 2.3 23 19 - 1 2 luakit 153.4 d83cc7e 1 - - - -
lbzip 113.3 2.1 1 - - - - midori 347.6 0.4.3 1 - - - -
lrzip 219.2 0.608 1 - - - - seaMonkey 760.9 2.21 1 - - - -
pbzip2 67.4 1.1.6 1 - - - - Daisy 201.9 SIR 37 - 35 1 -
plzip 51 0.7 1 - - - - Elevator 92.1 SIR 44 - 42 1 -
rar 511.8 5.0 1 - - - - Groovy 59.5 SIR 44 - 42 1 -
cmus 271.6 2.4.3 1 - - - - Pool 205.7 SIR 30 - 28 1 -
mocp 384 2.5.0 1 - - - - blackscholes 23 Parsec3.0 23 19 - 1 2
mp3blaster 265.8 3.2.5 1 - - - - bodytrack 3,368 Parsec3.0 13 9 - 1 2
mplayer 4,300 r34540 1 - - - - fludanimate 126.6 Parsec3.0 23 19 - 1 2
sox 55.2 14.3.2 1 - - - - canneal 414.7 Parsec3.0 23 19 - 1 2
arora 1,331 0.11 1 - - - - dedup 388 Parsec3.0 23 19 - 1 2
chromium 80,588 28.0.1500.71 1 - - - - ferret 2,150 Parsec3.0 23 19 - 1 2
dillo 610.9 3.0.2 1 - - - - freqmine 227.6 Parsec3.0 23 19 - 1 2
Dooble 364.4 0.07 1 - - - - streamcluster 103 Parsec3.0 23 19 - 1 2
epiphany 810.9 3.4.1 1 - - - - swaptions 94 Parsec3.0 23 19 - 1 2
firefox 59,904 24.0 1 - - - - x264 896.3 Parsec3.0 23 19 - 1 2
konqueror 920.1 4.8.5 1 - - - -

TABLE 5 var-gram algorithm are marked as SA V and SS V.

Statistical differences of the pigz versions generated It can be observed that the majority scores are in
with different compilers and optimization levels the 90%-above region for either kind of birthmark
regardless of the similarity metrics. These observa-
Size(Kb) #Functions #Instructions #Blocks #Calls
tions indicate that the thread-oblivious birthmarks
Max. 295 415 22178 3734 2376 exhibit strong resilience against obfuscations caused
Min. 84 342 13860 2672 1031
Avg. 151.75 380.25 16269 3068.9 1206.8 by different compilers and optimization levels.
Stdev. 60.53 23.4 2679 286.58 280.9
6.1.2 Resilience to Obfuscation Tools
In this section, we evaluate the resilience of our
PARSEC benchmark4 programs and a compression methods against advanced obfuscation techniques.
program pigz as the experimental subjects. Two com- In particular, we use the Java bytecode obfuscation
pilers llvm3.2 and gcc4.6.3 are used to compile tool SandMark [4] to generate a group of obfuscated
the source code of each program with different op- versions, which are then converted to x86 executables
timization levels (-O0, -O1, -O2, -O3 and -Os) and by GCJ [46]. Since Sandmark can only obfuscate Java
the debug option (-g) switched on or off. Such setup programs, thus the 4 multithreaded Java programs
leads to 20 different executables for each experimental from the SIR Benchmark Suite6 , including Daisy,
subject5 . Table 5 gives the statistical differences on Elevator, Groovy and Pool are used as the ex-
the size, number of functions, number of instructions, perimental subjects. We design similar experiments as
number of blocks and number of function calls be- those conducted in [14], [17] to measure the resiliency
tween the 20 binaries of pigz. The data indicate that of the thread-oblivious birthmarks against single ob-
even weak code transformations can make significant fuscations, where only one obfuscation technique is
differences to the produced binaries. applied at a time, and multiple obfuscations, where
Pair-wise similarity is calculated between the bi- multiple obfuscation techniques are applied at the
naries of each program with our methods. Figure 9 same time.
illustrates the distribution of the similarity scores. a) Single obfuscation
There are four subfigures, each corresponding to one We apply the 39 obfuscation techniques implemented
the four metrics utilized for similarity computation. in Sandmark on each program one at a time and
In each subfigure, the horizontal axis represents the generate a series of obfuscated versions. In order to
thread-oblivious birthmarks, the vertical axis repre- ensure correctness of these transformations, all obfus-
sents the percentage of birthmark pairs belonging to cated versions are tested with a set of inputs and ver-
each similarity range as specified in the legend. For sions with wrong outputs are eliminated. We finally
simplicity, we mark SA and SS birthmarks generated obtained 116 successfully obfuscated versions. There
with k-gram algorithm as SA K and SS K in the are 40 failed transformations. The failures are because
figures. Correspondingly birthmarks generated with certain obfuscation techniques cannot be applied, GCJ
fails to compile obfuscated versions, or obfuscated
4. http://parsec.cs.princeton.edu/ versions no longer give correct outputs.
5. Ten binaries are generated for bodytrack, since we fail to
compile its source code with llvm. 6. http://sir.unl.edu/content/sir.php

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 12

Fig. 9. Similarity distribution graph for birthmarks of the copies generated with different compilers and
optimization levels

b) Multiple obfuscations from illegal modification and cracking. Such tech-

As indicated by the 40 failures in the prior section, niques may be used to evade plagiarism detection.
transformations with a single obfuscation are not This strategy can defeat most static birthmarks as it
always successful. Not surprisingly, applying multiple significantly modifies the syntax of a program.
obfuscations simultaneously can significantly increase The only publicly available packing tool we know
the failure rate. To facilitate our experiments, we that handles ELF, the executable file format under
adopt the method used in [14], [17] where the obfus- Linux, is UPX13 . We process the previously used bina-
cators in SandMark are classified into two categories: ries with UPX. Similarity scores are calculated between
data obfuscators and control obfuscators. Only the birthmarks of the original programs and their corre-
obfuscators in the same category are applied simul- sponding UPX-packed versions. The results show that
taneously to the same experimental subject. the majority scores are above 70%. It indicates that
Besides the SandMark obfuscators, six other com- the thread-oblivious birthmarks are resilient against
mercial and open source obfuscation tools, includ- packing techniques.
ing Zelix KlassMaster7 , Allatori8 , DashO9 ,
6.1.4 Resilience to Different Scheduling Policies
JShrink10 , ProGuard11 and RetroGuard12 that
support layout, data and control flow obfuscations A possible approach that plagiarized program adopts
are also selected. Under the requirement of semantic to evade detection is to exploit different schedul-
equivalence between the original and the transformed ing policies different from the original program. As
programs, we turn on as many obfuscators as possible shown in Section 3.1, the scheduling policy a multi-
for each tool. Together with the two categories of threaded program adopts has insignificant effect on
SandMark, such setup leads to eight deeply obfus- the executions if viewed at the thread level. Thus,
cated versions for each experimental subject. even if the defendant program adopts a different
The similarity scores are calculated between the policy, we can simply compare the plaintiff with
birthmarks of each original program and all its obfus- the defendant by trying all possible policies. This is
cated versions. According to the experimental results, doable as the source code of the plaintiff is always
most scores locate in the 70%-above region. In the available, by modifying which we can set different
cases where Ex-Cosine metric is used, most scores policies. But we believe our methods still work even if
are in 90%-100% interval. These observations give a we do not adjust the scheduling policy of the plaintiff
strong evidence that the thread-oblivious birthmarks to the same as the defendant.
are resilient to the semantics-preserving code obfus- To validate it, we generate three versions that adopt
cations. SCHED OTHER, SCHED FIFO and SCHED RR, re-
spectively, for the previously used C/C++ programs.
6.1.3 Resilience to Packing Tools We use the version that adopts the default policy as
the plaintiff. We process the other two versions that
The packing tools [9], [10], which implement various adopt FIFO and RR policy with UPX, and take them
binary obfuscation techniques as well as compression as the the defendants. The experimental results that
and encryption techniques, are widely used to hide all scores are above 80% indicate that the TOB-revived
the maliciousness of malware or to protect software birthmarks are not affected by scheduling policies.
7. http://www.zelix.com/klassmaster
6.2 Validation of Credibility Property
8. http://www.allatori.com
9. https://www.preemptive.com/products/dasho Credibility of a birthmark is evaluated by its
10. //www.e-t.com/jshrink.html capability of distinguishing independently developed
11. http://proguard.sourceforge.net
12. http://java-source.net/open-source/obfuscators/retroguard 13. http://upx.sourceforge.net/

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 13

programs. Three types of widely used multithreaded 6.3.1 Performance Evaluation with Respect to URC
Linux applications are selected as our experimental Resilience and credibility reflect from different aspects
subjects, including 6 compression software (lbzip2, the qualities of a birthmark. URC (Union of Resilience
lrzip, pbzip2, pigz, plzip and rar), 10 web and Credibility) [47], defined below, is a metric pro-
browsers (arora, chromium, dillo, Dooble, posed for evaluating the overall performance of birth-
luakit, midori, epiphany, firefox, marks that considers both aspects.
konqueror and seaMonkey), and 5 audio players
(cmus, mocp, mp3blaster, mplayer and sox). R×C
URC = 2 × (2)
Firstly, we validate whether the TOB-revived birth- R+C
marks can distinguish programs in different cate- In the definition R represents the ratio of correctly
gories. Similarity scores between the 6 compression classified pairs where there exists plagiarism and C
programs and the 5 audio players are computed. represents the ratio of correctly classified pairs where
According to the experimental results, the majority there is no plagiarism. The value of URC ranges from 0
of the scores are below 10%. These data indicate that to 1, with higher value indicating a better birthmark.
thread-oblivious birthmarks have strong credibility in Let EP be the set of pairs of programs such that
distinguishing independently developed programs. ∀ (p, q) ∈ EP , q is a copy of p, and JP be the set of
Distinguishing programs in the same category is pairs such that ∀ (p, q) ∈ JP , a plagiarism detection
more challenging because they may overlap greatly method believes that q copies p. Similarly, let EI
in their functionality. Figure 10 depicts the similarity be the set of pairs such that ∀ (p, q) ∈ EI, q and
score distribution for the ten web browsers. It can p are independent, and JI be the set of pairs that
be observed that about 90% of the scores are below are deemed independent by a plagiarism detection
30%. Also as illustrated by Columns Avg in Table 6, method. R and C are formally defined as:
the average scores are all around 0.1. Similar results
are observed between the compression software and |EP ∩ JP | |EI ∩ JI|
R= and C = (3)
between the audio players. |EP | |EI|
There are several similarity scores above 40%. This As indicated by Equation 1, the detection result
is because some of the browsers share the same depends on the value of threshold ε. Therefore in the
layout engine. Our manual inspection discovers that experiments we vary the value of ε from 0 to 0.5. Note
five of the browsers (arora, Dooble, epiphany, that ε cannot be greater than 0.5, otherwise plagiarism
luakit and midori) are all Webkit-based. In order can be claimed to exist and non-exist at the same time.
to observe the effect of overlapped functionality on Figure 11 shows the results. In each subfigure, the
thread-oblivious birthmarks, we give the average sim- data for SCSSB as well as its TOB-revived SA and
ilarity scores between these Webkit-based browsers in SS birthmarks are depicted by the lines marked with
Column Avg+, and the average similarity scores be- square, triangle and circle symbols, respectively.
tween Webkit-based and non-Webkit-based browsers It can be observed from the figures that the SA
in Column Avg-. It can be observed that the values and SS birthmarks do not exhibit significant difference
in Column Avg+ are 3 to 5 times greater than those regardless of similarity metrics. Meanwhile, both have
values in Column Avg-. Since the goal is to detect greater URC values than SCSSB’s across the x-axis. It
whole program plagiarism, we believe the experimen- can also be observed that SCSSB’s curves are closer
tal results show strong credibility for real-world appli- to the curves of its TOB-revived versions when Ex-
cations where certain libraries are shared. If there exist Cosine is adopted, indicating similarity calculation us-
trivial programs that simply calls the same third-party ing such metric is less sensitive to thread scheduling.
functions, it is hard to give a conclusive judgment Moreover, the curves of var-gram generated birth-
even with manual examination. marks are above the corresponding curves of k-gram
generated birthmarks, especially for SCSSB. This indi-
cates the superiority of applying var-gram algorithm
6.3 Comparison with Traditional Birthmarks to birthmark generation.
This section compares the overall performance of the
TOB-revived SCSSBs against the original SCSSB. We 6.3.2 Performance Evaluation with F-Measure and
utilize the three evaluation metrics adopted in [17], MCC
including URC, F-Measure and MCC. URC measures As explained in work [17], URC mainly measures the
resilience and credibility, while the other two are more rate of correct classifications, while inconclusiveness is
comprehensive metrics introduced for amending the considered as incorrect classification. Thus, URC gives
problem of URC that focuses only on the rate of better results with higher value of ε in Figure 11.
correct classifications. All the comparison pairs of As the value of ε increases, the chance of inconclu-
programs from Section 6.1 to Section 6.2 are taken as siveness becomes smaller, leading to less incorrect
the experimental subjects. classifications.

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
IEEE Transactions on Software Engineering (year:2018) Transactions on Software Engineering

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 14

Fig. 10. Similarity distribution graph for birthmarks of the web browsers

TABLE 6
Credibility evaluation of the thread-oblivious birthmarks using software in the same category

K-GRAM VAR-GRAM
SA SS SA SS
Avg Avg+ Avg- Avg Avg+ Avg- Avg Avg+ Avg- Avg Avg+ Avg-
Ex-Cosine 0.137 0.334 0.078 0.133 0.314 0.079 0.075 0.156 0.041 0.078 0.167 0.041
Ex-Jaccard 0.090 0.213 0.046 0.072 0.163 0.034 0.068 0.128 0.042 0.068 0.132 0.040
Ex-Dice 0.134 0.322 0.078 0.103 0.238 0.056 0.100 0.189 0.069 0.092 0.173 0.060
Ex-Containment 0.166 0.364 0.111 0.135 0.281 0.087 0.128 0.208 0.097 0.106 0.186 0.068

Fig. 11. Performance evaluation with respect to URC. The left four figures depict the curves for birthmarks
generated with k-gram, the right four figures depict the curves for birthmarks generated with var-gram

To address the problem, similarly as in work [17], of P recision and Recall:

the birthmark methods are further compared against 2 × P recision × Recall
two other metrics, F-Measure and MCC (Matthews F-Measure = (5)
P recision + Recall
Correlation Coefficient) [48]. However, these two met-
where P recision and Recall are defined as following:
rics cannot be directly applied as they mainly measure
binary classifications. Thus, the definition of sim is re- |EP ∩ JP | |EP ∩ JP |
P recision = and Recall =
vised as following by removing the inconclusiveness: |JP | |EP |
MCC, defined below, is regarded as one of the best
metrics that evaluate true and false positives and

≥1−ε q is a copy of p negatives by a single value.
sim (pB , qB ) =
<1−ε q is not a copy of p TP × TN − FP × FN
MCC = p
(4) (T P + F P ) (T P + F N ) (T N + F P ) (T N + F N )
F-Measure is based on the weighted harmonic mean (6)

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 15

IEEE Transactions on Software Engineering (year:2018)

T P , T N , F P and F N are the number of true pos- better distinct similarity scores. In particular, the TOB-
itives, true negatives, false positives and false neg- revived birthmarks achieve 100% detection accuracy
atives, respectively, that can be computed with the at ε = 0.34, where neither false positives nor false
following formulas: negatives are observed.
T P = |EP ∩ JP | ; F N = |EP ∩ JI| 6.3.3 Comparing the Birthmarks with AUC Analysis
F P = |EI ∩ JP | ; T N = |EI ∩ JI| As discussed above, the birthmark methods exhibit
different performance under different thresholds. To
Figure 12 depicts the experimental results with give an intuitive comparison, we compute the AUC
respect to F-Measure and MCC, respectively. Over- (Area Under the Curve) values for each method with
all, similar results are observed as in the evaluation respect to the URC, F-Measure and MCC metrics.
against URC. The TOB-revived birthmarks almost al- The AUC value gives a proper overall performance
ways outperform SCSSB across the whole x-axis. summary for each birthmark method. A larger value
More specifically, consider the curves summarizing of AUC indicates better birthmark quality. The results
the results with respect to F-Measure. The TOB- are summarized in the white areas of Table 7. It can
revived birthmarks outperform SCSSB mainly in the be observed that the AUC values of the TOB-revived
right region with relatively small thresholds. With rel- birthmarks are all larger than those of SCSSB’s.
atively large thresholds the performance is very simi- We quantify the performance gains P erGain by tak-
lar. Taking the upper-left figure that depicts results for ing the original SCSSB as baseline. The quantification
k-gram generated birthmarks and calculated with Ex- indicates the improvement of each thread-oblivious
Containment as an example, SCSSB performs almost birthmark against the original birthmark, with respect
as well as its TOB-revived ones until the value of ε to the same similarity metric and the same perfor-
becomes smaller than 0.55. But its F-Measure value mance evaluation metric.
decreases sharply when adopting a smaller threshold.
AU Ctob − AU Corg
To see the reason for the sharp decrease, we check the P erGain = × 100%
specific Precision, Recall and F-Measure values under AU Corg
different thresholds. According to the data, there is , where AU Ctob and AU Corg represent the AUC value
almost no difference between the Precision values of of the thread-oblivious birthmark and the original
SCSSB and its TOB-revived ones under all thresholds. birthmark respectively. For example, the P erGain
Also, the Recall value of SCSSB is almost identical value with respect to Ex-Containment similarity and
with its TOB-revived ones, until it decreases sharply URC metric for SCSSBSA generated with k-gram is:
from threshold 0.5.
0.822 − 0.56
The reason for the sharp decrease on Recall value × 100% = 47%
0.56
is the following. Due to the combined impacts from
obfuscations and thread interleavings, the SCSSB of The average and maximal performance gains are
plagiarized pairs are greatly affected, which leads to summarized in the last row of Table 7. As the data in-
low similarity scores. As the Recall values of SCSSB dicate, TOB-revived birthmarks improve the original
indicate, only about 58% scores are above 0.75, and birthmark. The maximum performance gains happen
only about 6% scores are above 0.9. But for the for those SS birthmarks generated with k-gram and
TOB-revived ones, there are about 90% scores that calculated with the Ex-Jaccard similarity, where 129%,
are above 0.9. Such results indicate that the thread- 46% and 94% improvements are obtained with respect
oblivious birthmarks are resilient to obfuscations and to URC, F-Measure and MCC metrics, respectively.
thread interleavings. Additionally, it can be observed that the AUC values
Figure 13 shows that SCSSB can differentiate pla- based on var-gram are larger than those based on k-
giarized pairs and independent pairs even though the gram, indicating the superiority of applying var-gram
range of similarity scores is small. This explains the algorithm to birthmark generation.
almost identical Precision values between SCSSB and
its TOB-revived ones. In birthmark based plagiarism 6.3.4 Evaluation of the TraceSelector Optimization
detection literature, plagiarism is determined by the As mentioned in Section 5.2, we perform an opti-
similarity score and a threshold. Unfortunately, with- mization that chooses two most similar sequences
out abundant real-world plagiarism samples, deciding from plaintiff and defendant programs to reduce the
a threshold value is an arbitrary decision. In the ideal randomness of thread interleaving. In this section, we
but unrealistic case, we hope the similarity scores evaluate the impact of such optimization.
for plagiarized pairs are all 1.0, and for independent To simulate the worst case caused by thread inter-
pairs are all 0. Thus, we believe the greater the dif- leavings, two least similar traces are selected from the
ference between the two type of scores, the better a executions of the plaintiff and defendant programs.
birthmark method is. As indicated by the boxplots in The gray areas in Table 7 summarize the AUC values
Figure 13, the TOB-revived birthmarks exhibit much without optimization. By comparing with the values

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEE
IEEETRANSACTIONS
Transactions ON
on SOFTWARE ENGINEERING,
Software Engineering VOL. X, NO. X, X
(year:2018) 16

Fig. 12. F-Measure and MCC curves for the birthmark methods. The upper-left four figures depict the F-Measure
curves for birthmarks generated with k-gram under each similarity metric, the upper-right four figures depict the
F-Measure curves for birthmarks generated with var-gram, the bottom-left four figures and the bottom-right four
figures similarly depict the MCC curves for birthmarks generated with k-gram and var-gram respectively.

1 1 1
0.9 0.9 0.9
0.8 0.8 0.8
0.7 0.7 0.7
Similarity Score

Similarity Score

Similarity Score

0.6 0.6 0.6

0.5 0.5 0.5
0.4 0.4 0.4
0.3 0.3 0.3
0.2 0.2 0.2
0.1 0.1 0.1
0 0 0
Plagiarized Pairs Independent Pairs Plagiarized Pairs Independent Pairs Plagiarized Pairs Independent Pairs

(a) SCSSBSA (b) SCSSBSS (c) SCSSB

Fig. 13. Boxplots that summarize the statistical distribution of similarity scores corresponding to the very upper-
left figure in Figure 12

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEEIEEE Transactions
TRANSACTIONS on Software
ON SOFTWARE EngineeringVOL.
ENGINEERING, (year:2018)
X, NO. X, X 17

TABLE 7
AUC analysis results
(a) With respect to URC evaluation metric

K-GRAM VAR-GRAM
SCSSBSA SCSSBSS SCSSB SCSSBSA SCSSBSS SCSSB
Ex-Containment 0.878 0.822 0.874 0.837 0.45 0.56 0.875 0.856 0.884 0.875 0.58 0.749
Ex-Cosine 0.883 0.834 0.873 0.835 0.787 0.799 0.895 0.884 0.897 0.884 0.624 0.821
Ex-Dice 0.885 0.845 0.88 0.857 0.426 0.543 0.889 0.879 0.895 0.889 0.538 0.739
Ex-Jaccard 0.878 0.864 0.88 0.872 0.236 0.38 0.903 0.899 0.909 0.903 0.462 0.726
PerGain(%) - 59\127 - 61\129 - - - 16\24 - 17\24 - -

(b) With respect to F-Measure evaluation metric

K-GRAM VAR-GRAM
SCSSBSA SCSSBSS SCSSB SCSSBSA SCSSBSS SCSSB
Ex-Containment 0.978 0.98 0.976 0.979 0.72 0.782 0.993 0.994 0.994 0.995 0.778 0.918
Ex-Cosine 0.989 0.989 0.986 0.988 0.92 0.938 0.993 0.993 0.992 0.993 0.818 0.959
Ex-Dice 0.972 0.974 0.971 0.976 0.703 0.769 0.991 0.992 0.992 0.993 0.753 0.9
Ex-Jaccard 0.96 0.966 0.964 0.968 0.579 0.663 0.988 0.99 0.99 0.992 0.698 0.867
PerGain(%) - 26\46 - 26\46 - - - 9\14 - 9\14 - -

(c) With respect to MCC evaluation metric

K-GRAM VAR-GRAM
SCSSBSA SCSSBSS SCSSB SCSSBSA SCSSBSS SCSSB
Ex-Containment 0.823 0.802 0.821 0.808 0.483 0.51 0.894 0.894 0.903 0.904 0.389 0.619
Ex-Cosine 0.889 0.876 0.88 0.874 0.734 0.744 0.892 0.894 0.893 0.893 0.468 0.774
Ex-Dice 0.795 0.782 0.793 0.788 0.461 0.496 0.879 0.883 0.894 0.896 0.371 0.586
Ex-Jaccard 0.728 0.73 0.737 0.739 0.338 0.381 0.868 0.877 0.884 0.885 0.318 0.5
PerGain(%) - 56\92 - 57\94 - - - 47\75 - 48\77 - -

between the gray columns without optimization and the similarity scores for both plagiarized pairs and
white columns with optimization, we can see that independent pairs. It means that the optimization
the performance of the original SCSSB always gets enhances the resilience but weakens the credibility,
improved after the optimization. We use the following as larger similarity scores bring not only less false
equation to quantify the improvement achieved by the negatives but also more false positives. Consider the
optimization: two bold font values -6.4 and 24.4 in Table 8, which are
AU Copt − AU CnoOpt the OptGain values for SCSSBSA and SCSSB (gener-
OptiGain = × 100% ated with k-gram and measured with Ex-Containment
AU CnoOpt
similarity) with respect to URC. Table 9 gives their
As shown by the OptGain values in Table 8, the corresponding resilience (reflected by R in equation 3),
performance gains achieved by the optimization are credibility (reflected by C in equation 3) and URC val-
significant. Conclusion can be drawn that such opti- ues. The gray columns summarize the values without
mization helps to a large extent alleviate the problem the optimization, and the white columns summarize
of SCSSB in applying to multithreaded programs. the values with the optimization.
Yet as indicated by the data in Table 7, it is still
not adequate to handle the disturbance of thread As the data show, resilience of both SCSSB and
interleavings. This also demonstrates the significant SCSSBSA is enhanced after the optimization, while
impact thread interleaving could enforce on tradi- credibility of both birthmarks is weakened. However,
tional SCSSB. On the other hand, the improvement the degree for the resilience promotion of SCSSB
for the TOB-revived ones are much less significant, are rather obvious compared with the degree of its
indicating much less impact of thread interleaving credibility reduction, leading to a significant increase
on thread-oblivious birthmarks. Besides, as shown in its URC values. On the other hand, the degree for
by the negative values, the optimization sometimes credibility reduction of SCSSBSA are more obvious
can make the overall performance of TOB-revived than the degree of its resilience promotion, resulting
birthmarks worse. in minor decrease in its URC values. Similarly, the
The optimization, which pre-select more similar ex- negative OptGain values in terms of MCC can also
ecution traces of the plaintiff and defendant, improve be explained, as the number of false positives the

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 18

IEEE Transactions on Software Engineering (year:2018)

TABLE 8
OptGain values for the TraceSelector optimization

URC F-Measure MCC

SCSSBSA SCSSBSS SCSSB SCSSBSA SCSSBSS SCSSB SCSSBSA SCSSBSS SCSSB
Ex-Containment -6.4 -4.2 24.4 0.2 0.3 8.6 -2.6 -1.6 5.6
K- Ex-Cosine -5.5 -4.4 1.5 0.0 0.2 2.0 -1.5 -0.7 1.4
GRAM Ex-Dice -4.5 -2.6 27.5 0.2 0.5 9.4 -1.6 -0.6 7.6
Ex-Jaccard -1.6 -0.9 61.0 0.6 0.4 14.5 0.3 0.3 12.7
Ex-Containment -2.2 -1.0 29.1 0.1 0.1 18.0 0.0 0.1 59.1
V- Ex-Cosine -1.2 -1.4 31.6 0.0 0.1 17.2 0.2 0.0 65.4
GRAM Ex-Dice -1.1 -0.7 37.4 0.1 0.1 19.5 0.5 0.2 58.0
Ex-Jaccard -0.4 -0.7 57.1 0.2 0.2 24.2 1.0 0.1 57.2

optimization brings are larger than the number of 6.5.1 Reviving DYKIS with TOB
false negatives it reduces. Birthmarking is a detect- For DYKIS [17], we use the 20 pigz binaries gener-
ing technique of suspected plagiarisms rather than ated with different compilers and optimization levels
a proving technique [14], [15], [17], false negative is as the experimental subjects. Figure 14 illustrates the
more critical than false positive. Thus, we believe the distribution graph of the similarity scores calculated
optimization is proper and necessary. between the birthmarks of the 20 pigz binaries. As
it shows, all the scores of thread-oblivious birthmarks
6.4 Performance of the Analysis are above 70%, while for DYKIS there are quite a num-
This section presents the analysis performance of ber of scores below 70%. It indicates the effectiveness
SCSSB and its TOB-revived ones. Both the birthmark of applying the TOB framework on DYKIS.
methods involve basically two phases: the dynamic
analysis phase that traces program executions (Phase 6.5.2 Reviving JB with TOB
I) and the static detection phase that extracts birth- For JB [18], the 4 Java programs as well as their
marks and calculates similarities (Phase II). 149 single and deep obfuscated versions are used
For Phase I, our measurement indicates that on av- as the experimental subjects. Similarity scores are
erage a program is observed to become about 3 times calculated between the original Java programs and
slower once PIN and our tracer plugin are attached. their obfuscated versions. No significant differences
The tracing overhead is observed smaller for larger are observed between the original JB and its TOB-
inputs. It is because the overhead imposed by PIN’s revived ones. This is because JB is extracted from
runtime environment alone (that is runs a program API call sequences at object level. Similar to the TOB
using PIN without any instrumentation or analysis) framework that slices traces by thread, JB essentially
become trivial compared to the total runtime. slices traces by Java objects that greatly mitigates
For Phase II, no significant difference on calcu- the effect of thread scheduling. However, JB is only
lation overhead is observed between the methods. applicable to Java programs, while our TOB frame-
Specifically, for k-gram generated birthmarks, Phase work can be applied to transforming existing dynamic
II takes on average 54ms, 58ms and 67ms for process- birthmarks, including JB, into thread-oblivious ones.
ing a trace pair for SCSSBSA , SCSSBSS and SCSSB, In addition, the larger average and minimum scores
respectively. For var-gram generated birthmarks, the as summarized in Table 10 show that the TOB-revived
corresponding average time are 2.43s, 2.44s, and 2.43s, birthmarks indeed improve the original JB, although
where the variable-pattern mining costs the most not significantly.
time.
7 T HREATS TO VALIDITY
6.5 Applying TOB to Other Birthmarks Dynamic birthmarks are extracted from execution
In this section, we further demonstrate the application traces, therefore execution monitoring is necessary. It
of TOB on two other representative dynamic birth- is an undeniable fact that the monitoring itself may
marks. One is DYKIS [17] that is based on executed affect the thread interleavings during the execution of
key instructions, and the other one is JB [18] that a multithreaded program. Many other factors such as
is based on executed APIs of a Java program. For workload, scheduling policies, and runtime environ-
simplicity, we use the k-gram algorithm only. We ments affect the interleavings as well. However, we do
use DYK TR, DYK SA, and DYK SS to represent the not believe these issues cause an unfair comparison
original DYKIS and its TOB-revived versions using SA against traditional birthmarks.
and SS models, respectively. Similarly we have JB TR, For a multithreaded program, it is possible that
JB SA, and JB SS for JB. the effect of scheduling causes it to execute different

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE Transactions on Software Engineering (year:2018)
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 19

TABLE 9
Impact analysis of the optimization to birthmark credibility and resilience

SCSSBSA generated with k-gram SCSSB generated with k-gram

Threshold
Resilience Credibility URC Resilience Credibility URC
0 0 0 0 0 0 0 0 0 0 0 0 0
0.05 0.84 0.86 0.63 0.5 0.72 0.63 0.02 0.02 0.66 0.53 0.04 0.04
0.1 0.88 0.9 0.8 0.66 0.84 0.76 0.02 0.06 0.82 0.63 0.04 0.11
0.15 0.94 0.95 0.89 0.71 0.91 0.81 0.03 0.21 0.89 0.71 0.05 0.32
0.2 0.95 0.95 0.91 0.77 0.93 0.85 0.19 0.43 0.96 0.79 0.31 0.56
0.25 0.95 0.95 0.93 0.8 0.94 0.87 0.37 0.58 0.97 0.84 0.53 0.69
0.3 0.99 1 0.96 0.84 0.98 0.91 0.45 0.71 0.97 0.94 0.62 0.81
0.35 1 1 0.97 0.91 0.98 0.95 0.61 0.74 0.98 0.96 0.75 0.84
0.4 1 1 0.98 0.95 0.99 0.97 0.73 0.81 0.99 0.96 0.84 0.88
0.45 1 1 0.99 0.96 0.99 0.98 0.8 0.81 0.99 0.96 0.84 0.88
0.5 1 1 1 0.97 1 0.98 0.81 0.84 0.99 0.99 0.89 0.91

Fig. 14. Similarity distribution graph for DYKIS and its TOB-revived thread-oblivious ones

TABLE 10
Effectiveness of applying the TOB framework to JB

JB SA JB SS JB TR
Avg. Max. Min. Avg. Max. Min. Avg. Max. Min.
Ex-Cosine 0.983 1.000 0.554 0.983 1.000 0.602 0.983 1.000 0.555
Ex-Jaccard 0.970 1.000 0.530 0.979 1.000 0.618 0.963 1.000 0.470
Ex-Dice 0.976 1.000 0.503 0.978 1.000 0.527 0.964 1.000 0.453
Ex-Containment 0.986 1.000 0.545 0.985 1.000 0.596 0.978 1.000 0.508

paths across multiple runs even under the same input. the experiments, despite large number of executions,
In such cases, the execution traces of multiple runs the inputs still constitute only a small proportion of
apparently can become different even after projection the whole input space. This is the fundamentally chal-
on individual threads, which may lead to the failure lenge for all dynamic birthmarks. One way to alleviate
of our methods. But as indicated by our experiments the concerns is to combine with testing techniques. We
conducted on the various types of real multithreaded take it as one of our future work.
programs, the thread-oblivious birthmarks always il-
lustrate good performance. Thus we believe such Effectiveness of the TOB framework is mainly
cases rarely happen in practical programs. Besides, evaluated on whole program plagiarism detec-
as discussed in Section 5 and Section 6.3.4, we adopt tion, where a complete program is copied and
an optimization that select two most similar traces then disguised through various automatic semantics-
from plaintiff and defendant for further birthmark preserving transformations. One problem plagiarism
generation. Thus even if the mentioned cases happen, detection researches face is the lack of real-world
the optimization helps alleviate the problem. plagiarism cases [49], [50]. In recent years, whole
program plagiarism on mobile markets starts to rise,
The thread-oblivious birthmarks improve upon tra- and many of the stolen apps have been processed with
ditional birthmark with TOB framework. Thus they obfuscation techniques to evade plagiarism detection.
suffer the same limitation of dynamic birthmarks in According to a recent study [51], about 5%-13% of
exhaustively covering all behaviors of a program. In apps in the third-party app markets are copied and

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEE
IEEE Transactions
TRANSACTIONS ONon SoftwareENGINEERING,
SOFTWARE Engineering (year:2018)
VOL. X, NO. X, X 20

redistributed from the official Android market. This modules in Java packages. Yet this method suffered
potentially provides rich real-world plagiarism cases. high time consumption since a mass of traces can be
Yet, identifying potential pairs of apps that plagiarism extracted if the CFG is complex. Hemel et al. [58] sug-
may exist is extremely labor-intensive. Also, tracing gested three methods to find potential cloned binaries
apps needs nontrivial efforts because our tracers sup- within a program repository by simply treating bi-
port the monitoring of binary executables and Java naries as normal files. Specifically, similarity between
bytecodes. Thus, we take it as one of our future work. two binaries were evaluated by calculating the ratio of
Besides whole program plagiarism, there exists shared string literals, by calculating the compression
many cases that only part or a library of a program is ratio, and by computing binary deltas. Since no syn-
copied. The main problem of using dynamic birth- tactic or semantic attributes of binary executables are
marks to detect partial plagiarism is that they are considered, efficiency is assured but low detection ac-
mainly based on the similarity of program executions. curacy is expected. Lim used control flow information
Therefore, if there is only a small portion of code that reflected runtime behaviors to supplement static
being copied, these approaches give low similarity approaches [59]. Recently he proposed to analyze
scores. Improved upon existing dynamic birthmarks, stack flows obtained by simulating operand stack
the thread-oblivious birthmarks suffer the same prob- movements to detect copies [60]. But they are only
lem. A straightforward solution is to instrument only available to Java programs. An obfuscation-resilient
the suspicious part. But this requires manual efforts method based on longest common subsequence of
and domain knowledge. semantically equivalent basic blocks was proposed by
Luo et. al. [61]. They utilized symbolic execution to
8 R ELATED W ORK extract from basic blocks symbolic formulas, whose
pair-wise equivalence are compared via a theorem
Broadly speaking, the research areas related to our
prover. Being static analysis method, accuracy can not
work include software watermarking [25], [52] which
be assured since it has difficulty in handling indirect
protects software copyrights and detects piracy, pla-
branches. In addition, symbolic execution combined
giarism detection, as well as code clone detection [53],
with theorem proving is not scalable.
[54] and malware identification [55], [56] that detect
There are also some work focusing on detecting pla-
clones or maliciousness by characterizing software
giarism for smartphone applications. DroidMOSS [3]
with features. In this section we focus on the discus-
detects plagiarism by applying fuzzing hashing on in-
sion of birthmark based software plagiarism detection
struction sequences. Yet simple obfuscations, such as
techniques. Works targeting source code will not be
noise injection can evade the detection of DroidMOSS,
discussed here, and there have already been many
since no semantic information is used. DNADroid
mature detection systems and tools [20], [21], [57].
[62] achieves plagiarism detection by constructing
and comparing program dependence graphs between
8.1 Static birthmark based software plagiarism methods. Since considering data dependencies, this
detection method are more robust. ViewDroid [51] proposes
Myles and Collberg [27] proposed k-gram based static the feature view graph birthmark by capturing users’
birthmarks, where sets of Java bytecode sequences of navigation behaviors. But it’s vulnerable to dummy
length k are taken as the birthmarks. The similarity view insertion and encryption attacks.
between two birthmarks was calculated through set
operations that ignore the frequency of elements in the 8.2 Dynamic birthmark based software plagia-
set. Although being more robust than birthmarks pro- rism detection
posed Tamada [8], the birthmarks were still vulnera- Myles and Collberg [7] suggested the whole program
ble to code transformation attacks. Weighted k-gram path (WPP) birthmark generated by compressing a
based static birthmarks [47] improved upon Myles whole dynamic control flow trace into a directed
and Collberg’s [27] by taking the frequency of each acyclic graph form to uniquely identify program.
k-length operation code sequence into consideration. Even with compression the method does not scale,
However, the improvement in detection ability seems and it’s susceptible to various loop transformations.
minor while introducing extra cost in computing Schuler [18] treated Java standard API call sequences
change rate of k-gram frequencies. A static birthmark at object level as dynamic birthmarks for Java pro-
based on disassembled API calls from executables is grams. Such approach exhibited better performance
put forward by Seokwoo et al. [19] to detect plagia- than WPP birthmark, but they also pointed out that
rism of windows applications. The requirement for their method was affected by thread scheduling. Sim-
de-obfuscating binaries before applying their method ilar principle was applied in Tamada’s work [63],
is too restrictive and thus reduces its availability. Park where API calls of windows executables executed dur-
[12] proposed a static birthmark by extracting all ing runtime were used to derive two kind birthmarks:
possible sequences of object instructions from a CFG Sequence of API Function Calls (EXESEQ) and Fre-
of each method, and applied it for detecting common quency of API Function Calls (EXEFREQ). Apparently

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering
IEEE
IEEE Transactions
TRANSACTIONS ONon SoftwareENGINEERING,
SOFTWARE Engineering (year:2018)
VOL. X, NO. X, X 21

API based birthmarks are all language dependent. To beneficial for researchers to conduct experiments and
address the problem Wang et al. [15] proposed System present their findings. The benchmarks, the TOB-PD
Call Short Sequence birthmark (SCSSB), that treat the tools as well as the experimental data are all available.
sets of k-length system call sequences as birthmarks. Our work addresses the challenges of applying
However, as we illustrated it has limited applicability dynamic birthmark based approaches for whole pro-
to multithreaded programs. gram plagiarism detection of multithreaded software.
Liu et al. [34], [49] suggested to characterize soft- As far as we know, this is the first work that discusses
ware with core values and applied it to software and the impact of thread scheduling on birthmark based
algorithm plagiarism detection. Tian et al. [17], [64] plagiarism detection, and the first work that propose
proposed the DYKIS birthmark based on dynamic thread-oblivious birthmarks for solving the problem
key instruction sequences. By introducing dynamic systemically. In recent years, whole program plagia-
taint analysis into birthmark generation, these birth- rism of mobile apps has becomes a serious problem.
mark methods were resilient to various semantics- About 5% to 13% of apps in third-party app markets
preserving code transformations. LoPD [65], a pro- are copied and redistributed from the official Android
gram logic based approach was designed for soft- market. We plan to conduct case studies and optimize
ware plagiarism detection by leveraging symbolic our approaches for this domain.
execution and weakest precondition reasoning to find
semantic dissimilarities. Despite these methods are ACKNOWLEDGEMENT
resilient, they all suffer the scalability problem, since
The research was supported in part by National
they all operates on the instruction granularity, and
Key Research and Development Program of China
either taint analysis or symbolic execution with con-
(2016YFB1000903), National Science Foundation of
straint solving is computational non-trivial.
China under grants (91418205, 61472318, 61532015,
By integrating data flow and control flow depen-
61532004, 61672419, 61632015, 61602369, 71501156,
dency analysis, Wang et al. [14] proposed a system
61373116), Fok Ying-Tong Education Foundation
call dependency graph based birthmark, and graph
(151067), Ministry of Education Innovation Research
isomorphism is utilized for calculating similarity be-
Team (IRT13035), Science and Technology Project
tween birthmarks. Patrick et al. [11] proposed a heap
in Shaanxi Province of China (2016KTZDGY04-01,
graph birthmark for JavaScript utilizing heap mem-
2016GY-092), and the Fundamental Research Funds
ory analysis, and graph monomorphism algorithm
for the Central Universities. Any opinions, findings,
was applied for similarity computation. But to be
and conclusions expressed in this material are those
effective, these graph based birthmarks require that
of the authors and do not necessarily reflect the views
the programs under protection to have prominent
of the funding agencies. T. Liu is the corresponding
referencing structures. Also, since graph isomorphism
author.
and monomorphism algorithms are NP-complete in
general, several thousand nodes will make the meth-
ods impractical to use. R EFERENCES
[1] [Online]. Available: http://sourceauditor.com/blog/tag/
lawsuits-on-open-source/.
9 C ONCLUSION [2] [Online]. Available: http://www.martinsuter.net/blog/2009/
08/skype-joltid-licensing-dispute-epic-ma-screwup.html.
As multithreaded software become increasingly more [3] W. Zhou, Y. Zhou, X. Jiang, and P. Ning, “Detecting repack-
popular, current dynamic software plagiarism detec- aged smartphone applications in third-party android market-
places,” in Proc. ACM Conf. Data and Application Security and
tion technology geared toward sequential programs Privacy (CODASPY ’12), 2012, pp. 317–326.
are no longer sufficient. This paper fills the gap [4] C. Collberg, G. Myles, and A. Huntwork, “Sandmark-a tool
by proposing a thread-oblivious software plagiarism for software protection research,” IEEE Security and Privacy,
vol. 1, no. 4, pp. 40–49, 2003.
detection framework (TOB) that revives existing dy- [5] Z. Wu, S. Gianvecchio, M. Xie, and H. Wang, “Mimimorphism:
namic software birthmarks. We have developed a set A new approach to binary code obfuscation,” in Proc. ACM
of tools collectively called TOB-PD by applying the Conf. Computer and Communications Security (CCS ’10). ACM,
2010, pp. 536–546.
TOB framework to three typical dynamic birthmarks, [6] C. Linn and S. K. Debray, “Obfuscation of executable code to
including SCSSB, DYKIS and JB. The extensive ex- improve resistance to static disassembly,” in Proc. ACM Conf.
periments conducted on 418 versions of 35 different Computer and Communications Security (CCS ’03), 2003, pp. 290–
299.
programs show that the proposed approaches are not [7] G. Myles and C. S. Collberg, “Detecting software theft via
only accurate in detecting plagiarism of multithreaded whole program path birthmarks,” in Proc. Int. Conf. Information
programs but also robust against most state-of-the- Security (ISC ’04), 2004, pp. 404–415.
[8] H. Tamada, M. Nakamura, and A. Monden, “Design and
art semantics-preserving obfuscation techniques. In evaluation of birthmarks for detecting theft of Java programs,”
addition, a suite of benchmarks of multithreaded in IASTED Conf. on Software Engineering (IASTEDSE ’04), 2004,
programs are complied. We believe there will be more pp. 569–574.
[9] K. A. Roundy and B. P. Miller, “Binary-code obfuscations in
research on plagiarism detection for multithreaded prevalent packer tools,” ACM Computing Surveys, vol. 46, no. 4,
programs. The existence of such benchmarks will be 2013.

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
IEEE Transactions on Software Engineering (year:2018) Transactions on Software Engineering

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 22

[10] M.-J. Kim, J.-Y. Lee, H.-Y. Chang, S. Cho, and P. A. Wilsey, “De- length grams,” in Proc. Int. Conf. Very Large Data Bases (VLDB
sign and Performance Evaluation of Binary Code Packing for ’07). VLDB Endowment, 2007, pp. 303–314.
Protecting Embedded Software against Reverse Engineering,” [33] Y.-C. Jhi, X. Wang, X. Jia, S. Zhu, P. Liu, and D. Wu, “Value-
in IEEE Int. Symp. Object/Component/Service-Oriented Real-Time based program characterization and its application to software
Distributed Computing (ISORC ’10), 2010, pp. 80–86. plagiarism detection,” in Proc. Int. Conf. Softw. Eng. (ICSE ’11),
[11] P. P. F. Chan, L. C. K. Hui, and S.-M. Yiu, “Heap graph based 2011, pp. 756–765.
software theft detection,” IEEE Trans. Information Forensics and [34] F. Zhang, Y. chan Jhi, D. Wu, P. Liu, and S. Zhu, “A first step
Security, vol. 8, no. 1, pp. 101–110, 2013. towards algorithm plagiarism detection,” in Proc. Int. Symp.
[12] H. Park, H. il Lim, S. Choi, and T. Han, “Detecting common Software Testing and Analysis (ISSTA ’12), 2012, pp. 111–121.
modules in Java packages based on static object trace birth- [35] Z. Tian, T. Liu, Q. Zheng, F. Tong, M. Fan, and Z. Yang,
mark,” Computer Journal, vol. 54, no. 1, pp. 108–124, 2011. “A new thread-aware birthmark for plagiarism detection of
[13] Z. Tian, Q. Zheng, T. Liu, and M. Fan, “DKISB: Dynamic multithreaded programs,” in Int. Conf. Software Engineering
key instruction sequence birthmark for software plagiarism Companion (ICSE ’16), 2016, pp. 734–736.
detection,” in IEEE Int. Conf. High Performance Computing and [36] Z. Tian, T. Liu, Q. Zheng, M. Fan, E. Zhuang, and Z. Yang, “Ex-
Communications. (HPCC ’13), 2013, pp. 619–627. ploiting thread-related system calls for plagiarism detection of
[14] X. Wang, Y.-C. Jhi, S. Zhu, and P. Liu, “Behavior based multithreaded programs,” Journal of Systems and Software, vol.
software theft detection,” in Proc. ACM Conf. Computer and 119, pp. 136–148, 2016.
Communications Security (CCS ’09). ACM, 2009, pp. 280–290. [37] D.-K. Chae, J. Ha, S.-W. Kim, B. Kang, and E. G. Im, “Software
[15] X. Wang, Y. Jhi, S. Zhu, and P. Liu, “Detecting software theft plagiarism detection: a graph-based approach,” in Pro. ACM
via system call based birthmarks,” in Annual Computer Security Int. Conf. Information and Knowledge Management (CIKM ’13).
Applications Conference (ACSAC ’09), 2009, pp. 149–158. ACM, 2013, pp. 1577–1580.
[16] X. Zhang and R. Gupta, “Whole execution traces,” in Proc. [38] K. Chen, P. Liu, and Y. Zhang, “Achieving accuracy and
Annual IEEE/ACM Int. Symp. Microarchitecture (MICRO ’04). scalability simultaneously in detecting application clones on
IEEE Computer Society, 2004, pp. 105–116. android markets,” in Proc. Int. Conf. Sof. Eng.(ICSE’14). New
[17] Z. Tian, Q. Zheng, T. Liu, M. Fan, E. Zhuang, and Z. Yang, York, NY, USA: ACM, 2014, pp. 175–186.
“Software plagiarism detection with birthmarks based on [39] K. Chen, P. Wang, L. Y., W. X., N. Zhang, H. H., Z. W.,
dynamic key instruction sequences,” IEEE Trans. Software En- and L. P., “Finding unknown malice in 10 seconds: Mass
gineering, vol. 41, no. 12, pp. 1217–1235, 2015. vetting for new threats at the google-play scale,” in USENIX
[18] D. Schuler, V. Dallmeier, and C. Lindig, “A dynamic birthmark Security Symposium (USENIX Security’15), Washington, D.C.,
for Java,” in Proc. IEEE/ACM Int. Conf. Automated Software USA, 2015, pp. 659–674.
Engineering (ASE ’07), 2007, pp. 274–283. [40] Y. Qu, X. Guan, Q. Zheng, T. Liu, L. Wang, Y. Hou, and
[19] S. Choi, H. Park, H. il Lim, and T. Han, “A static api birth- Z. Yang, “Exploring community structure of software call
mark for windows binary executables,” Journal of Systems and graph and its applications in class cohesion measurement,”
Software, vol. 82, no. 5, pp. 862–873, 2009. Journal of Systems and Software, vol. 108, pp. 193–210, 2015.
[20] C. Liu, C. Chen, J. Han, and P. S. Yu, “GPLAG: detection of [41] C.-K. Luk, R. S. Cohn, R. Muth, H. Patil, A. Klauser, P. G.
software plagiarism by program dependence graph analysis,” Lowney, S. Wallace, V. J. Reddi, and K. M. Hazelwood, “Pin:
in Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data building customized program analysis tools with dynamic
Mining (KDD ’06), 2006, pp. 872–881. instrumentation,” in Proc. ACM SIGPLAN Conf. Programming
[21] L. Prechelt, G. Malpohl, and M. Philippsen, “Finding plagia- Language Design and Implementation (PLDI ’05), 2005, pp. 190–
risms among a set of programs with jplag,” Journal of Universal 200.
Computer Science, vol. 8, no. 11, pp. 1016–1038, 2002. [42] [Online]. Available: ASM, http://asm.ow2.org/.
[22] Z. Tian, Q. Zheng, T. Liu, M. Fan, X. Zhang, and Z. Yang, [43] I. Jonassen, J. F. Collins, and D. G. Higgins, “Finding flexi-
“Plagiarism detection for multithreaded software based on ble patterns in unaligned protein sequences.” Protein Science,
thread-aware software birthmarks,” in Proc. Int. Conf. Program vol. 4, no. 8, pp. 1587–1595, 1995.
Comprehension (ICPC ’14), 2014, pp. 304–313. [44] M.-F. Sagot and A. Viari, “A double combinatorial approach to
[23] I. Rigoutsos and A. Floratos, “Combinatorial pattern discovery discovering patterns in biological sequences,” in Combinatorial
in biological sequences: The teiresias algorithm.” Bioinformat- Pattern Matching, ser. Lecture Notes in Computer Science.
ics, vol. 14, no. 1, pp. 55–67, 1998. Springer Berlin Heidelberg, 1996, vol. 1075, pp. 186–208.
[24] [Online]. Available: DashO, [45] H. Kuhn, “The hungarian method for the assignment prob-
https://www.preemptive.com/products/dasho. lem,” Naval Research Logistics, vol. 52, no. 1, pp. 7–21, 2005.
[25] C. S. Collberg, E. Carter, S. K. Debray, A. Huntwork, J. D. Kece- [46] [Online]. Available: GCJ, https://gcc.gnu.org/java/.
cioglu, C. Linn, and M. Stepp, “Dynamic path-based software [47] X. Xie, F. Liu, B. Lu, and L. Chen, “A software birthmark based
watermarking,” in Proc. ACM SIGPLAN Conf. Programming on weighted k-gram,” in IEEE Int. Conf. Intelligent Computing
Language Design and Implementation (PLDI ’04), 2004, pp. 107– and Intelligent Systems (ICIS ’10), 2010, pp. 400–405.
118. [48] B. W. Mathews, “Comparison of the predicted and observed
[26] Z. Tian, T. Liu, Q. Zheng, F. Tong, D. Wu, S. Zhu, and K. Chen, secondary structure of T4 phage lysozyme,” Biochimica Et
“Software plagiarism detection: A survey,” Journal of Cyber Biophysica Acta (bba) - Protein Structure, vol. 405, pp. 442–451,
Security, vol. 3, pp. 52–76, 2016. 1975.
[27] G. Myles and C. Collberg, “K-gram based software birth- [49] Y.-C. Jhi, X. Jia, X. Wang, S. Zhu, P. Liu, and D. Wu, “Program
marks,” in Proc. ACM Symp. Applied Computing (SAC ’05), 2005, characterization using runtime values and its application to
pp. 314–318. software plagiarism detection,” IEEE Trans. Software Engineer-
[28] M. Olszewski, J. Ansel, and S. Amarasinghe, “Kendo: efficient ing, vol. 41, no. 9, pp. 925–943, 2015.
deterministic multithreading in software,” ACM SIGPLAN [50] L. Luo, J. Ming, D. Wu, P. Liu, and S. Zhu, “Semantics-based
Notices, vol. 44, no. 3, pp. 97–108, 2009. obfuscation-resilient binary code similarity comparison with
[29] H. Cui, J. Wu, J. Gallagher, H. Guo, and J. Yang, “Efficient applications to software and algorithm plagiarism detection,”
deterministic multithreading through schedule relaxation,” in IEEE Transactions on Software Engineering, vol. PP, no. 99, pp.
Proc. ACM Symp. Operating Systems Principles (SOSP ’11). 1–1, 2017.
ACM, 2011, pp. 337–351. [51] F. Zhang, H. Huang, S. Zhu, D. Wu, and P. Liu, “ViewDroid:
[30] C. Bienia, “Benchmarking modern multiprocessors,” Ph.D. Towards obfuscation-resilient mobile application repackaging
dissertation, Princeton University, January 2011. detection,” in Proc. ACM Conf. Security and Privacy in Wireless
[31] A. Wespi, M. Dacier, and H. Debar, “Intrusion detection using and Mobile Networks (WiSec ’14), 2014, pp. 25–36.
variable-length audit trail patterns,” in Int. Symp. Recent Ad- [52] C. S. Collberg and C. Thomborson, “Watermarking, tamper-
vances in Intrusion Detection (RAID ’00). Springer, 2000, pp. proofing, and obfuscation-tools for software protection,” IEEE
110–129. Trans. Software Engineering, vol. 28, no. 8, pp. 735–746, 2002.
[32] C. Li, B. Wang, and X. Yang, “Vgram: Improving performance [53] T. Kamiya, S. Kusumoto, and K. Inoue, “CCFinder: A multi-
of approximate queries on string collections using variable- linguistic token-based code clone detection system for large

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSE.2017.2688383, IEEE
Transactions on Software Engineering

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. X, NO. X, X 23

scale source code,” IEEE Trans. Software Engineering, vol. 28, Qinghua Zheng received the B.S. degree in
no. 7, pp. 654–670, 2002. computer software in 1990, the M.S. degree
[54] H. Kim, Y. Jung, S. Kim, and K. Yi, “MeCC: memory in computer organization and architecture in
comparison-based clone detector,” in Proc. Int. Conf. Softw. Eng. 1993, and the Ph.D. degree in system engi-
(ICSE ’11), 2011, pp. 301–310. neering in 1997 from Xi’an Jiaotong Univer-
[55] M. D. Preda, M. Christodorescu, S. Jha, and S. Debray, “A sity, China. He was a postdoctoral researcher
semantics-based approach to malware detection,” ACM Trans. at Harvard University in 2002. He is currently
Programming Languages and Systems, vol. 30, no. 5, pp. 25:1– a professor in Xi’an Jiaotong University, and
25:54, 2008. the dean of the Department of Computer Sci-
[56] S. Chaki, C. Cohen, and A. Gurfinkel, “Supervised learning ence. His research areas include computer
for provenance-similarity of binaries,” in Proc. ACM SIGKDD network security, intelligent e-learning theory
Int. Conf. Knowledge Discovery and Data Mining (KDD ’11), 2011, and algorithm, multimedia e-learning, and trustworthy software.
pp. 15–23.
[57] G. Cosma and M. Joy, “An Approach to Source-Code Pla-
giarism Detection and Investigation Using Latent Semantic
Analysis,” IEEE Trans. Computers, vol. 61, pp. 379–394, 2012.
[58] A. Hemel, K. T. Kalleberg, R. Vermaas, and E. Dolstra, “Find-
ing software license violations through binary code clone
detection,” in Proc. Working Conf. Mining Software Repositories
(MSR ’11), 2011, pp. 63–72.
[59] H. il Lim, H. Park, S. Choi, and T. Han, “A method for
detecting the theft of Java programs through analysis of the Eryue Zhuang received the B.S. degree in
control flow information,” Information and Software Technology, software and microelectronics from North-
vol. 51, no. 9, pp. 1338–1350, 2009. western Polytechnical University, China, in
[60] H. il Lim and T. Han, “Analyzing stack flows to compare 2014. She is currently working toward the
Java programs,” IEICE Trans. Information and Systems, vol. 95- M.S. degree in the Department of Computer
D, no. 2, pp. 565–576, 2012. Science and Technology at Xi’an Jiaotong
[61] L. Luo, J. Ming, D. Wu, P. Liu, and S. Zhu, “Semantics-based University, China. Her research interests in-
obfuscation-resilient binary code similarity comparison with clude trustworthy software and user behavior
applications to software plagiarism detection,” in Proc. ACM analysis.
SIGSOFT Symp. Found. Softw. Eng. (FSE ’14), 2014, pp. 389–400.
[62] J. Crussell, C. Gibler, and H. Chen, “Attack of the clones: De-
tecting cloned applications on android markets,” in Proc. Eur.
Symp. Research in Computer Security (ESORICS ’12). Springer,
2012, pp. 37–54.
[63] H. Tamada, K. Okamoto, M. Nakamura, A. Monden, and K.-
i. Matsumoto, “Dynamic software birthmarks to detect the
theft of windows applications,” in Int. Symp. Future Software
Technology (ISFST ’04), 2004, pp. 1–6.
[64] Z. Tian, Q. Zheng, M. Fan, E. Zhuang, H. Wang, and T. Liu,
“DBPD: A dynamic birthmark-based software plagiarism de- Ming Fan received the B.S. degree in com-
tection tool,” in Int. Conf. Software Engineering and Knowledge puter science and technology from Xi’an
Engineering (SEKE ’14), 2014, pp. 740–741. Jiaotong University, China, in 2013. He is cur-
[65] F. Zhang, D. Wu, P. Liu, and S. Zhu, “Program logic based rently working toward the Ph.D. degree in the
software plagiarism detection,” in IEEE Int. Symp. Software Department of Computer Science and Tech-
Reliability Engineering (ISSRE ’14), 2014, pp. 66–77. nology at Xi’an Jiaotong University, China.
His research interests include trustworthy
software and malware detection of Android
Apps.

Zhenzhou Tian received the B.S. degree

and Ph.D. degree in computer science and
technology from Xi’an Jiaotong University,
China, in 2010 and 2016, respectively. He
is currently a lecturer in the School of Com-
puter Science and Technology at Xi’an Uni-
versity of Posts and Telecommunications.
His research interests include trustworthy
software, software plagiarism detection, and
Zijiang Yang is a professor in computer
software behavior analysis.
science at Western Michigan University. He
holds a Ph.D. from the University of Penn-
sylvania, an M.S. from Rice University and
a B.S. from the University of Science and
Technology of China. Before joining WMU he
was an associate research staff member at
Ting Liu received his B.S. degree in informa- NEC Labs America. He was also a visiting
tion engineering and Ph.D. degree in system professor at the University of Michigan from
engineering from School of Electronic and 2009 to 2013. His research interests are in
Information, Xi’an Jiaotong University, Xi’an, the area of software engineering with the
China, in 2003 and 2010, respectively. Cur- primary focus on the testing, debugging and verification of software
rently, he is an associate professor of the systems. He is a senior member of IEEE.
Systems Engineering Institute, Xi’an Jiao-
tong University. His research interests in-
clude smart grid, network security and trust-
worthy software.

0098-5589 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.