Dimensions in Program Synthesis

[Invited Talk Paper]

Sumit Gulwani
Microsoft Research, Redmond, WA, USA
[email protected]

Abstract intent, the space of programs over which it searches, and the search
Program Synthesis, which is the task of discovering programs technique it employs. (i) The user intent can be expressed in the
that realize user intent, can be useful in several scenarios: en- form of logical relations between inputs and outputs, input-output
abling people with no programming background to develop util- examples, demonstrations, natural language, and inefficient or re-
ity programs, helping regular programmers automatically discover lated programs. (ii) The search space can be over imperative or
tricky/mundane details, program understanding, discovery of new functional programs (with possible restrictions on the control struc-
algorithms, and even teaching. ture or the operator set), or over restricted models of computations
This paper describes three key dimensions in program synthesis: such as regular/context-free grammars/transducers, or succinct log-
expression of user intent, space of programs over which to search, ical representations. (iii) The search technique can be based on
and the search technique. These concepts are illustrated by brief de- exhaustive search, version space algebras, machine learning tech-
scription of various program synthesis projects that target synthesis niques (such as belief propagation or genetic programming), or log-
of a wide variety of programs such as standard undergraduate text- ical reasoning techniques. Most logical reasoning techniques in-
book algorithms (e.g., sorting, dynamic programming), program volve two main steps: constraint generation, and constraint solv-
inverses (e.g., decoders, deserializers), bitvector manipulation rou- ing. (a) Constraint generation can be invariant-based, path-based,
tines, deobfuscated programs, graph algorithms, text-manipulating or input-based. (b) Constraint solving of resultant second-order
routines, mutual exclusion algorithms, etc. quantified formulas typically involves reducing second-order un-
knowns to first-order unknowns (by use of templates), and elimi-
Categories and Subject Descriptors D.1.2 [Programming Tech- nating universal quantifiers (by use of techniques such as Farkas
niques]: Automatic Programming; I.2.2 [Artificial Intelligence]: lemma, cover algorithms, sampling), and then solving the resultant
Automatic Programming – Program Synthesis; F.3.1 [Logics and first-order quantifier-free constraints using off-the-shelf SAT/SMT
Meanings of Programs]: Specifying and Verifying and Reasoning solvers.
about Programs In this paper, we illustrate the above concepts by brief descrip-
General Terms Languages, Theory tion of various program synthesis projects that target synthesis of
a wide variety of programs such as standard undergraduate text-
Keywords Deductive Synthesis, Inductive Synthesis, Program- book algorithms (e.g., sorting, dynamic programming), program
ming by Examples, Programming by Demonstration, SAT Solving, inverses (e.g., decoders, deserializers), bitvector manipulation rou-
SMT Solving, Machine Learning, Probabilistic Inference, Belief tines, deobfuscated programs, graph algorithms, text-manipulating
Propagation, Genetic Programming routines, mutual exclusion algorithms, etc.
This paper is organized as follows. We start by describing a
1. Introduction few applications where program synthesis would be feasible and
Program Synthesis is the task of discovering an executable pro- valuable (Section 2). We then describe the three dimensions in pro-
gram from user intent expressed in the form of some constraints. gram synthesis: user intent (Section 3), search space (Section 4),
Unlike compilers, which take as input programs written in a struc- and search technique (Section 5). We then describe the class of
tured language and mostly perform syntax-directed translations, search techniques based on logical reasoning in more detail. These
synthesizers can accept a variety and mixed form of constraints techniques typically have two main steps: Constraint Generation
(such as input-output examples, demonstrations, logical relations (described in Section 6) and Constraint Solving (described in Sec-
between inputs and outputs, natural language, partial or inefficient tion 7). Finally, we conclude by describing some open research
programs), and mostly perform some kind of search over some questions and mentioning how different areas of computer science
space of programs. can play a role in realizing the revolutionary potential of program
A synthesizer is typically characterized by three key dimen- synthesis technology (Section 8).
sions: the kind of constraints that it accepts as expression of user
2. Applications
Program Synthesis has the potential to influence various classes of
Permission to make digital or hard copies of all or part of this work for personal or users in the technology pyramid ranging from algorithm design-
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
ers (Section 2.1) and regular programmers (Sections 2.4-2.6) at the
on the first page. To copy otherwise, to republish, to post on servers or to redistribute top of the pyramid to end-users (Section 2.2) and students (Sec-
to lists, requires prior specific permission and/or a fee. tion 2.3) at the bottom of the pyramid. Synthesis technology can
PPDP ’10 Hagenberg, Austria also be useful for designers of cyber-physical systems, which are
Copyright ⃝ c ACM [to be supplied]. . . $10.00 being increasingly deployed in transportation, health-care and other
societal applications [27, 34]. We discuss below few applications of to manage/manipulate their data, such as spreadsheets, databases,
program synthesis. browsers, and scripting platforms, are not written with their partic-
ular needs in mind. These programs attempt to do an average job
2.1 Discovery of New Algorithms for meeting average needs and are far from doing a perfect job for
Finding a new algorithmic solution for a given problem requires personalized task-oriented needs.
human ingenuity: “The process of preparing programs for a digital Program synthesis can play an extremely useful role for end-
computer is especially attractive, not only because it can be eco- users who can easily illustrate their intent by input-output examples
nomically and scientifically rewarding, but also because it can be an or traces, but find it difficult to write programs. A good start in
aesthetic experience much like composing poetry or music” [40]. this direction has been made by Programming by demonstration
Use of computational techniques to discover new algorithmic in- systems that enable users to write programs by demonstrating the
sights can be the ultimate application of program synthesis. program on one or more concrete examples [13, 44].
One domain of algorithms that has been shown amenable to
automated synthesis is the class of bitvector algorithms [39, 74]. 2.3 Teaching
These algorithms “typically describe some plausible yet unusual The importance of automation in teaching cannot be understated.
operation on integers or bit strings that could easily be programmed Most public schools, especially in developing countries, struggle to
using either a longish fixed sequence of machine instructions or a find good teachers. Many parents spend several man-years helping
loop, but the same thing can be done much more cleverly using just their kids with their homeworks and imparting knowledge. If this
four or three or two carefully chosen instructions whose interac- motivation is too earthly, consider the role that educational robots
tions are not at all obvious until explained or fathomed” [74]. Such play in the frozen embryos approach to inter-stellar travel.1
programs can be quite unintuitive and extremely difficult for aver- Program synthesis can play a revolutionary role in automating
age, or sometimes even expert, programmers to discover methodi- the teaching landscape. In particular, program synthesis technology
cally. Initial work [2, 52, 21] on automated synthesis of such algo- can help with automated problem solving. Manna and Waldinger
rithms used brute-force search, while recent work [23, 33] uses log- observed that the ability to construct proofs of theorems can yield a
ical reasoning based techniques (powered by underlying SAT/SMT program to solve the corresponding problem [51]. Since then, many
solvers) and scales much better. new program synthesis approaches have emerged that can also be
Another domain of algorithms that has been shown amenable to used to classroom problem solving.
automated synthesis is that of mutual exclusion algorithms, which As part of this effort, the author is currently working on a
guarantee mutually exclusive access to a critical section among a technique to automatically synthesize high-school geometrical con-
number of competing processes [17]. The essence of mutual ex- structions such as construction of a regular hexagon given a side.
clusion algorithms is the code before and after the critical sec- Another example of class of problems (taught in an undergraduate
tion that together ensures properties such as mutual exclusion, class) whose solutions can be automated (using a current synthesis
deadlock-freedom, and possibly starvation-freedom. Mutual exclu- technology [4]), is that of construction of finite automatas such as
sion algorithms differ based on number and size of shared/local the smallest automata that accepts all strings with even number of
variables, number of commands in the entry/exit sections, the kind 1s but not more than 4 consecutive 0s.
of conditionals used, and whether starvation-freedom is desired or Among other teaching tasks that can be automated (besides
not. Initial work [5] for automatically discovering such algorithms problem solving) using synthesis technology are construction of
used brute-force search (carefully trimmed by various optimiza- problems of a measured difficulty, and interactive grading with
tions) and successfully discovered many new algorithms (and re- explanations of any errors and fixes. The latter task of pointing
discovered some known ones). Recent work [38] uses a novel tech- errors and suggesting fixes might benefit from the work on bug
nique based on combining genetic programming and model check- localization and automated debugging in the program analysis and
ing. It reports performance better than brute-force methods, but re- software engineering community.
quires carefully designed fitness functions.
2.4 General Purpose Programming Assistance
2.2 Automating Repetitive Programming for End-Users
Since synthesis is a very hard problem, fully automated synthesis
Computational devices have become accessible to people at large of large pieces of software might not be possible in the near fu-
at an impressive rate. Most computer users are end-users, who are ture. However, an effective use of synthesis technology for general
at the end of the process of computer programming, far removed purpose programming can be its role as a programmer’s assistance.
from the programmer. These business end-users have a myriad of Synthesis can be used to synthesize small program fragments from
diverse backgrounds and include commodity traders, graphic de- higher-order logical specifications [43, 31]. Synthesis can be used
signers, chemists, human resource managers, finance pros, market- to find tricky/mundane implementation details after human insight
ing managers, underwriters, compliance officers, and even mail- has been expressed in the form of a partial program [65]. Various
room clerks – they are not professional programmers, but they need interesting forms of interactivity can also be employed between the
to create small, often one-off, applications to support business func- programmer and the synthesizer in a semi-automated software de-
tions [22]. velopment process [64, 6, 72].
Graphical user interfaces have been designed to make it easier Another interesting application of synthesis technology in the
for these end-users to use computers. However, they are still not software development process can be to assist with the debugging
ideal since users struggle to find the correct feature or succession process, as explained below.
of commands to use from a maze of features to accomplish their
task. Figuring out the right parameters to provide as trigger/input 1 Wikipedia defines it as: “A robotic space mission carrying some number
to the feature is also sometimes difficult. More significantly, pro-
of frozen early stage human embryos is another theoretical possibility. This
gramming is still required to perform tedious and repetitive tasks. method of space colonization requires, among other things, the development
Examples of such repetitive tasks include: transforming lists of of a method to replicate conditions in a uterus, the prior detection of a
addresses from one format to another, extracting data from sev- habitable terrestrial planet, and advances in the field of fully autonomous
eral web pages into a single document, renaming files in a direc- mobile robots and educational robots which would replace human parents.”
tory, managing bibliographies, etc. The programs that end-users use on http://en.wikipedia.org/wiki/Interstellar travel#Frozen embryos.
Automated Debugging Debugging is a time-consuming and dif- 2.6 Program Understanding
ficult process that is today mostly done manually, often requiring A given program may often be non-trivial to understand if it is,
hours to fix a single bug, and unnecessary taxation of brain power, for example, obfuscated, too low-level, or missing documenta-
better saved for less mundane tasks. Debugging can be phrased as a tion. Program synthesis technology can help in program under-
program synthesis problem after identification of a region of code standing by translating such a given piece of code to a semanti-
that is suspected of incorrect behavior, and a description of the de- cally equivalent code written in some target language that is more
sired behavior, possibly in the form of input-output examples or readable/understandable/higher-level. The given program acts as a
symbolic relations between values of variables at the beginning and specification of the desired equivalent program in the target lan-
end of the identified region of code. The buggy region of code may guage.
be identified by the programmer [36] or may be guessed to be those One instance of program understanding arises in the context
regions of code that are executed on the failing runs, but not on the of malware deobfuscation, where the challenge is to understand
passing runs [75]. what the malicious code is doing. The need for deobfuscation tech-
The debugging application also provides for some unique ad- niques has arisen in recent years, especially due to an increase in the
vantages that can be exploited by program synthesis techniques, amount of malicious, and mostly obfuscated, code (malware) [70].
especially those based on machine learning techniques. Currently, human experts use decompilers and manually deobfus-
The probabilistic inference based program synthesis technique cate the resulting code (see, e.g., [59]). Clearly, this is a tedious
described in [36] can take advantage of the existence of initial task that could benefit from automated tool support. The program
buggy piece of code. The core idea of this technique is to model synthesis technique described in [33] has been applied to malware
the program as a graph consisting of instructions and states as deobfuscation by deobfuscating examples drawn from and inspired
potentially unknown variables, connected by constraint nodes, and by the Conficker [59] and MyDoom [54] viruses.
then use belief propagation to infer both the intermediate program A related instance is in the context of reverse engineering of a
states and the instructions that satisfy all the constraints. In the binary, when the source code is not available. Program synthesis
debugging scenario, the beliefs about instructions can be initialized techniques for learning programs from traces are well suited to
based on the initial buggy piece of code, and belief propagation be applied in this context since the program state can easily be
is performed (which leads to altering the code) until the resultant observed even though the source code is not [47].
code satisfies the input-output constraints. It is shown that such an Another instance arises in the context of software maintenance
initialization speeds up inference. This technique has been used to tasks (such as bug fixing or refactoring), where the challenge is
fix small bugs involving up to six incorrect instructions. These bugs to first understand a given piece of poorly documented low-level
may be simple ones like use of a wrong variable name, resetting code, possibly by translating it into some higher level representa-
a variable to zero at the wrong location, inverting the order of tion involving, for example, sets and second-order logic. This is an
two instructions, forgetting to increase a counter (or decreasing application of synthesis technology that is an antithesis of its more
it instead), etc, or more complex ones that are best addressed by traditional application of converting programs written using higher-
completely replacing a short code fragment. order primitives such as sets and higher-order quantifiers into effi-
The genetic programming based program synthesis technique cient code (as is done in [31]). The QuickSpec tool [12] takes an-
described in [75] takes advantage of the hypothesis that a missing other interesting approach in this context and generates algebraic
important functionality in a program can be copied and adapted specifications for modules consisting of pure functions. It has been
from another location in the program. The core idea of this genetic applied to understanding of a heap library for Haskell and a fixed-
programming technique is to evolve program variants until one is point arithmetic library for Erlang.
found that both retains required functionality and also avoids the
defect in question. This technique has been used to automatically
generate repairs for ten C programs totaling 63,000 lines of code. 3. First Dimension: User Intent
The most important dimension in program synthesis from the per-
spective of the user is the mechanism for describing intent. There
2.5 Synthesis of Program Inverses
are various choices possible that have been briefly explained be-
The problem of program inversion is to derive a program 𝑃 −1 that low: logical specifications, natural language, input-output exam-
negates the computation of a given program 𝑃 . More formally, it is ples, traces, and higher-order, inefficient or partial programs. A par-
the problem of inverting an injective program 𝑃 by finding another ticular choice may be more suited in a given scenario depending on
program 𝑃 −1 that is its left-inverse. This problem arises naturally the underlying task as well as on the technical background of the
in paired computations such as compression/decompression, en- user. An important open research question is to identify how to
cryption/decryption, serialization/deserialization, insert/delete op- combine these various choices in a unified interface.
erations on data structures, transactional memory rollback, bidirec-
tional programming, and even client-server applications. Given the 3.1 Logical Specifications
prevalence of program inverses and the cost associated with main-
taining two closely related programs, automatic program inversion A logical specification is a logical relation between inputs and
can be beneficial in ensuring correctness and maintainability. outputs of a program. It can act as a precise and succinct form of
Initial work on deriving program inverses used proof-based [16] functional specification of the desired program.
or grammar-based [18] approaches. Recent work [67] phrases the For example, the logical specification for a sorting algorithm
program inversion problem as a finite synthesis problem, and uses would be the following logical relation that asserts that the output
a novel path-based inductive synthesis technique (described in Sec- array 𝐵 is sorted (Eqn 1) and is a permutation of the input array 𝐴
tion 6.2) to invert a larger class of programs than was possible of size 𝑛 (Eqn 2).2
before. In particular, it can synthesize inverses for compressors
(e.g., LZ77), packers (e.g., UUEncode), and arithmetic transform-
ers (e.g., image rotations). The inverses for these non-trivial pro- 2 Eqn 2 correctly states the permutation requirement under the assumption
grams range from 5 to 20 lines of code, and are automatically syn- that all elements in the input array are distinct, which is assumed for
thesized in a median time of 40 seconds. simplicity.
 𝑥=10101100
↓ 𝑥 = 10101100
𝑦=10101000 ↓ ∃𝑖 { 0 ≤ 𝑖 ≤ 𝑛 ∧ (∀𝑘. 0 ≤ 𝑘 < 𝑖 =⇒ 𝑥[𝑘] = 0)
𝑥−1 = 10101011 ∧ (𝑖 = 𝑛 ∨ (𝑖 < 𝑛 ∧ 𝑥[𝑖] = 1 ∧ 𝑦[𝑖] = 0))
(a)
𝑥 &(𝑥 − 1) = 10101000 ∧ (∀𝑘. (0 ≤ 𝑘 < 𝑛 ∧ 𝑘 ∕= 𝑖) =⇒ 𝑦[𝑘] = 𝑥[𝑘]) }
𝑦 := 𝑥&(𝑥 − 1)

(b) (c) (d)

Figure 1. Consider the problem of masking off the rightmost significant one-bit in a given bitvector. (a) describes an example input-output
pair (𝑥, 𝑦). (b) describes a 2-step program to solve the problem. (c) describes the working of the 2 step program. (d) describes the intent
using a logical relation between input bitvector 𝑥 and output bitvector 𝑦. For notational convenience, we represent an 𝑛-bit bitvector as an
array of 𝑛 bits, with the bit positions starting from 0 and numbered from right to left.

𝑥=10101100 TurnOffRightMostOnes(𝑥)
↓ ∃𝑖, 𝑗. { 0 ≤ 𝑖, 𝑗 < 𝑛 ∧ (∀𝑘. 𝑗 ≤ 𝑘 ≤ 𝑖 =⇒ 𝑥[𝑘] = 1) 𝑖 := 0;
𝑦=10100000 ∧ (∀𝑘. 0 ≤ 𝑘 < 𝑗 =⇒ 𝑥[𝑘] = 0) while(𝑥[𝑖]==0 ∧ 𝑖 < 𝑛)
∧ (𝑗 ≤ 𝑖 ∨ 𝑗 = 𝑛 − 1) 𝑖 := 𝑖+1;
(a) while(𝑥[𝑖]==1 ∧ 𝑖 < 𝑛)
∧ (∀𝑘. 𝑖 < 𝑘 < 𝑛 =⇒ 𝑥[𝑘] = 𝑦[𝑘]) 𝑥[𝑖] := 0; 𝑖 := 𝑖+1;
𝑦 := 𝑥&(1+(𝑥∣(𝑥−1))) ∧ (∀𝑘. 0 ≤ 𝑘 ≤ 𝑖 =⇒ 𝑦[𝑘] = 0) } return 𝑥;

(b) (c) (d)

Figure 2. Consider the problem of masking off the rightmost contiguous sequence of 1’s in a given bitvector. (a) describes an example
input-output pair (𝑥, 𝑦). (b) describes a 4-step program to solve the problem. (c) describes the intent using a logical relation between input
bitvector 𝑥 and output bitvector 𝑦, both of which are of size 𝑛. (d) describes the intent using an inefficient program.

the need of end-users who interact with databases, but are intimi-
dated by the idea of using languages such as SQL [3, 49].
∀𝑘. (0 ≤ 𝑘 < 𝑛 − 1) =⇒ (𝐵[𝑘] ≤ 𝐵[𝑘 + 1]) (1) One disadvantage with natural language is that it can be ambigu-
∧ ∀𝑘 ∃𝑗. (0 ≤ 𝑘 < 𝑛) =⇒ (0 ≤ 𝑗 < 𝑛 ∧ 𝐵[𝑗] = 𝐴[𝑘]) (2) ous. This issue can potentially be resolved by using the concept of
paraphrasing where the system interacts with the user to resolve
Note that the above specification of sorting property does not sug-
any ambiguity in the user-provided description [29].
gest in any way how to efficiently implement a sorting algorithm.
The approach described in [69] can discover algorithms from log-
ical specifications as above. In particular, given the above sorting 3.3 Input-Output Examples
specification, it discovers five sorting algorithms: InsertionSort, Se- In several scenarios, input-output examples can act as the simplest
lectionSort, BubbleSort, MergeSort, and QuickSort3 . form of specification, with relatively little chances of error. Quite
As another example, consider the logical specifications pro- contrary to what it may seem initially, input-output examples, in
vided in Figure 1(d) and Figure 2(c) for the corresponding bitvec- conjunction with interactive rounds, can often play the role of a
tor tasks illustrated in Figure 1(a) and Figure 2(a) respectively. The full functional specification.
technique described in [23] is specialized to discovering efficient It is natural to ask what prevents the synthesizer from synthesiz-
bitvector tricks (as in Figure 1(b) and Figure 2(b)) from such logi- ing a trivial program that simply performs a table lookup as follows,
cal specifications. when provided with the set {(𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ), . . . , (𝑥𝑛 , 𝑦𝑛 )} of
Several synthesis systems accept user intent in the form of input-output pairs.
logical specifications [51, 69, 23, 31]. Compared to other forms of
specifications such as input-output examples and demonstrations, switch 𝑥
logical relations require additional knowledge of logic and might be case 𝑥1 : return 𝑦1 ;
harder to get right, and may not be a preferred form of specification case 𝑥2 : return 𝑦2 ;
for end-users. .
.
.
3.2 Natural Language case 𝑥𝑛 : return 𝑦𝑛 ;
Given advances in natural language processing, it is possible to map The restriction on the search space is the first line of defense against
natural language sentences into logical representations [77]. Natu- allowing such trivial solutions. In particular, the search space might
ral language can be used as a substitute for logical relations, and permit only a bounded number of statements or conditionals.
end-users might find it very valuable. In particular, natural language Another concern with input-output examples is the selection cri-
interfaces have been designed to query databases to accommodate terion and the number of input-output examples. In other words,
identifying what constitutes a good input-output example, and how
3 One might ask why the system described in [69] only discovers these many examples should the user provide? The user may start by pro-
five sorting algorithms. This is because the underlying search space was viding few input-output examples (possibly a couple of examples
expressive enough to represent only these algorithms. for each of the corner cases) and then add more input-output exam-
 User Oracle
Input → Output Program 1 Program 2 Distinguishing Input ?
01011 → 01000 (𝑥 + 1) & (𝑥 − 1) (𝑥 + 1) & 𝑥 00000 ?
00000 → 00000 −(¬𝑥) & 𝑥 (((𝑥& − 𝑥) ∣ − (𝑥 − 1))&𝑥) ⊕ 𝑥 00101 ?
00101 → 00100 (𝑥 + 1)&𝑥 ⋅⋅⋅ 01111 ?
01111 → 00000 ⋅⋅⋅ ⋅⋅⋅ 00110 ?
00110 → 00000 ⋅⋅⋅ ⋅⋅⋅ 01100 ?
01100 → 00000 ⋅⋅⋅ ⋅⋅⋅ 01010 ?
01010 → 01000 (((𝑥 − 1)∣𝑥) + 1)&𝑥 None Program is
(((𝑥 − 1)∣𝑥) + 1)&𝑥

Figure 3. An illustration of the synthesizer driven interaction model for synthesis from input-output examples of the task illustrated in
Figure 2(a). Program 1 and Program 2 are two semantically different programs generated by the synthesizer that are consistent with the past
set of input-output pairs provided by the user. The synthesizer also produces a distinguishing input on which the two programs yield different
results, and asks the user for the output corresponding to the distinguishing input. The process is repeated until the synthesizer can find at
most one program.

ples in each interactive round. The interaction may either be driven with the updated set of input-output pairs, the synthesizer finds two
by the user or by the synthesizer, as explained below. different programs −(¬𝑥) & 𝑥 and (((𝑥&−𝑥) ∣ −(𝑥−1))&𝑥)⊕𝑥
and a distinguishing input 00101. It then asks the user for the out-
User Driven Interaction The user may inspect the program re- put for 00101. The newly added pair (00101, 00100) rules out
turned by the synthesizer, either by studying the program itself, or (((𝑥& − 𝑥) ∣ − (𝑥 − 1))&𝑥) ⊕ 𝑥. Note that at this stage, the pro-
by testing it on several other inputs. If the user finds any discrep- gram (𝑥 + 1)&𝑥 remains a candidate, since it was not ruled out in
ancy in the behavior of the program and the expected behavior on the earlier iterations. In next four iterations, the synthesizer driven
some new input, the user may repeat the synthesis process after interaction leads to four more input-output pairs: (01111, 00000),
adding the new input-output pair to the previous set of input-output (00110, 00000), (01100, 00000) and (01010, 01000). The seman-
examples. tically unique program generated from the resulting set of input-
output pairs is the desired program: (((𝑥 − 1)∣𝑥) + 1)&𝑥.
Synthesizer Driven Interaction Given a set of input-output pairs,
the synthesizer searches for programs that map each input in the Number of Interactive Rounds A concern in an interactive pro-
given set to the corresponding output. The number of such pro- cess might be the number of interactive rounds required. In the
grams may usually be unbounded, if the search space consists of worst case, the number of interactive rounds required might be the
all possible programs. However, since the search space is usually size of the input space. However, this may not happen in prac-
restricted, the number of such programs may either be 0, 1, or tice because the search would typically be restricted to space of
more than 1. If the synthesizer is unable to find any such program programs with low Kolmogorov (descriptive) complexity [48], or
over the search space, the synthesizer declares failure. If the syn- small teaching dimension [20]. In fact, experimental results for syn-
thesizer finds exactly 1 program, the synthesizer declares success thesis of bitvector programs indicate that the number of interactive
and presents the program to the user. rounds required is roughly equal to the number of instructions in
If the synthesizer finds at least two programs 𝑃1 and 𝑃2 , both of the synthesized program [33].
which map each input in the given set to the corresponding output,
the synthesizer declares the user specification to be partial. It then 3.4 Traces
generates a distinguishing input, an input on which the two pro- A trace is a detailed step-by-step description of how the program
grams 𝑃1 and 𝑃2 yield different results, and asks the user to provide should behave on a given input. A trace is a more detailed descrip-
the output corresponding to the distinguishing input. The synthesis tion than an input-output example since it also illustrates how a
process is then repeated after adding this new input-output pair to specific input should be transformed into the corresponding output
the previous set of input-output examples. This interaction model as opposed to just describing what the output should be. This input
is described in [33]. model can be used when the user already knows how to perform the
For example, consider the task of synthesizing the bitvector pro- task, and the goal is to create a program to perform the task. Traces
gram that masks off the rightmost contiguous sequence of 1s in are an appropriate model for programming by demonstration sys-
the input bitvector (i.e., the problem described in Figure 2). One tems for end-users [13], where the intermediate states resulting
would agree that it is easier to provide input-output examples for from the user’s successive actions on a user interface constitute a
the task than write down the logical specification described in Fig- valid trace. Traces may also be easier to provide than writing a pro-
ure 2(c). The synthesizer driven input-output interaction process is gram in the context of general purpose programming [7]. Traces
illustrated in Figure 3. The user may start out by providing one are also readily available for reverse engineering scenarios [47].
input-output example (01011, 01000) for the desired program. The From the perspective of the synthesizer, traces are preferable to
synthesizer generates a candidate program (𝑥 + 1) & (𝑥 − 1) that input-output examples since the former contains more information.
is consistent with the input-output pair (01011, 01000). Then, it The synthesizer needs to generalize from given demonstrations or
checks whether a semantically different program also exists and traces such that the generalized trace or the program can work
comes up with an alternative program (𝑥 + 1) & 𝑥 and a distin- for other inputs as well. This is a relatively easier challenge than
guishing input 00000 that distinguishes the two programs, and asks searching for a program that is consistent with a given input-output
the user for the output for the distinguishing input. The newly ob- example.
tained input-output pair (00000, 00000) rules out one of the can- From the perspective of the user, providing demonstrations in
didate programs, namely, (𝑥 + 1) & (𝑥 − 1). In the next iteration, general may be more taxing than providing input-output examples.
However, in certain scenarios, demonstrations may be more prefer- various tasks on standard architectures. Such programming prob-
able than input-output examples. This happens when the output is lems often arise while developing low level embedded code, net-
not easy to compute. For example, suppose the user wants to syn- work applications or in other domains where bit-level manipula-
thesize a program for computing factorial. A demonstration for in- tion is needed. They are of great significance for people who write
put 7 can be the string “7 × 6 × 5 × 4 × 3 × 2” or the recursive optimizing compilers or high-performance code as these code-
trace “7 × factorial(6)”, which is easier to describe than providing fragments can be used to speed up the inner loop of some integer
the final simplified output 5040. or bit-fiddly computation. Bitvector tricks are also helpful for de-
signing specialized hardware.
3.5 Programs For example, consider the problem of masking off the right-
Programmers might sometimes find a programming language as the most significant 1-bit in a bitvector, as shown in Figure 1(a). An
best means of specifying their intent. This happens quite trivially approach that comes to mind immediately is to use a loop to iterate
for certain applications such as deobfuscation (Section 2.6) and over the bitvector from right to left, checking each bit one by one,
synthesis of program inverses (Section 2.5), where the program to until a 1-bit is found, and then convert that bit to 0. The problem
be deobfuscated or inverted respectively forms the specification. with this approach is that it involves too many steps (worst-case
However, even for applications such as discovery of new algo- linear in the number of bits in the bitvector), and additionally each
rithms, some people might find it easier to write the specification step involves use of conditionals. However, the desired task can be
as an inefficient program than a logical relation. For example, con- achieved by computing 𝑥&(𝑥 − 1) (which involves composing the
sider the bitvector problem described in Figure 2, and contrast its bitwise & operator and the arithmetic subtraction operator in an
logical representation shown in Figure 2(c) with its program repre- unintuitive manner), as illustrated in Figure 1(c). [23] describes a
sentation shown in Figure 2(d). The latter might be more appealing system for synthesis of bitvector programs from logical specifica-
to programmers. tions while [33] describes a system for bitvector program synthesis
using the synthesizer driven input-output interaction model.
4. Second Dimension: Search Space 4.1.2 Control Structure
The second dimension in program synthesis is the space of pro- The control structure of the program may be restricted to a given
grams over which the desired program will be searched. This looping template [69], a program with bounded number of state-
choice is made by the developer of the synthesizer and may op- ments [5], a partial program with holes [65], or even to loop-free
tionally be restricted by the user of the synthesizer. programs [23, 33, 76, 46, 32, 43, 50]. We discuss the choice of
The developer of the synthesizer needs to strike a good balance loop-free programs in some detail below.
between expressiveness and efficiency of the search space. On one
hand, the space of the programs should be large/expressive enough Loop-free Programs The class of straight-line programs, or more
to include programs that users care about. While on the other hand, generally, loop-free programs, parametrized by the set of opera-
the space of the programs should be restrictive enough so that it tors/components used, can often express a wide range of useful
is amenable to efficient search, and it should be over a domain of computations. Following are some interesting examples of such
programs that are amenable to efficient reasoning. programs.
The user of the synthesizer can optionally restrict the search
space to obtain programs with specific resource usage. For exam- ∙ Bitvector algorithms [23, 33]. The set of components includes
ple, the user might desire a loop-free program, or a program whose arithmetic operators and bitwise operators.
memory usage does not exceed a specified amount. ∙ Text-editing programs [76, 46]. The set of components includes
Broadly speaking, the search space can be over the space of basic editing commands (available in a text-editor) such as
(Turing-complete) programs, or restricted form of computational insert, locate, select and delete.
models such as grammars or logics.
∙ Geometrical constructions [32]. The set of components includes
4.1 Programs geometrical constructors such as ruler and compass.
The space of programs can be qualified by at least two attributes: ∙ Unbounded data type manipulation [43]. The set of components
(i) the operators used in the program, and (ii) the control structure include those that the underlying decision procedure can reason
of the program. about such as linear arithmetic operators and set operators.
∙ API call sequences [50]. The set of components includes the
4.1.1 Operators API calls.
The choice of operators can be restricted to comparison operators
(sufficient for algorithms such as sorting), arithmetic operators, Even though these programs may not involve loops, they may
APIs exported by a given library, combination of arithmetic and still be challenging to synthesize. The space of loop-free programs
bitwise operators etc. We discuss the choice of combination of is still huge: the number of possible programs is exponential in
arithmetic and bitwise operators in some detail below. the number of components in the program, and hence brute-force
search methods do not usually scale. The problem of synthesis of
Arithmetic + Bitwise Operators We use the term bitvector pro- straight-line programs may be likened to the problem of solving a
grams to refer to those programs that involve combination of the Jigsaw puzzle.
following two kinds of operators:
∙ Arithmetic Operators such as Addition +, Subtraction −, Mul- 4.2 Grammars
tiplication ∗, Division /. There has been a lot of work in learning various kinds of grammars
∙ Bitwise Operators such as bitwise-and &, bitwise-or ∣, bitwise- such as regular expressions [55, 63], DFAs [8, 4], NFAs [10],
context-free grammars [15], regular transducers [73, 56]. Learning
xor ⊕, left-shift, right-shift, rotate, etc.
grammars has a wide range of applications ranging over robotics
Bitvector programs (also described in Section 2.1) are useful be- and control systems, pattern recognition, computational linguistics,
cause they often provide the most efficient way to accomplish computational biology, data compression, data mining, etc. [14].
 Learning regular expressions, in particular, is important for at partial order that allows more efficient representation of the version
least two scenarios: space by the boundary sets representing the most specific and most
∙ Regular expressions are a key component of text-editing pro-
general functions in the space.
Lau et.al. later extended Mitchell’s version space concept to ver-
grams, where they are used to select substrings inside a piece of
sion space algebra for learning more complex functions that have
text. Learning such regular expressions [55, 63] is a key step of
any range [46]. The basic idea is to build up a complex version
learning text-editing programs.
space by composing together version spaces containing simpler
∙ Regular expressions can represent the control structure of a functions as opposed to defining a single version space containing
program. Learning such regular expressions (as done in [60]) the entire function space. Just as two functions can be composed to
can increase the usability of general program synthesis schemes create a new function, two version spaces can be composed to cre-
that assume that the control structure has been provided, for ate a new version space, containing functions that are composed
example, in the form of a sketch [65] or a scaffold [69]. from the functions in the original version spaces. As Pardowitz
et.al. point out, this allows to represent hypotheses on programs
4.3 Logics and underlying loop structures in a hierarchical manner [57]. They
Logical representations, because of their succinctness, can serve are constructed out of atomic or other compound hypotheses, re-
as good target languages for program synthesis. In particular, the sembling the tree-like structure of the syntax-tree of a program.
class of first order logic together with fixed point equals the class
of PTIME algorithms over ordered structures such as graphs, trees, 5.3 Machine Learning Based Techniques
strings [30]. Hence, this class and also some of its useful subclasses We now discuss two very different techniques from the machine
(such as those with a fixed quantifier depth) can serve as good target learning community that have applied to program synthesis.
languages for synthesizing efficient graph/tree algorithms [31].
5.3.1 Probabilistic Inference
5. Third Dimension: Search Technique A program can be modeled as a graph consisting of instructions and
states, connected by constraint nodes. Each constraint node estab-
In this section, we provide a high-level description of some general lishes the semantics of some instruction by relating the instruction
techniques that have been used for program synthesis. The appli- with the state immediately before the instruction and the state im-
cability and the specifics of these techniques may depend on the mediately after the instruction. Such a modeling allows for use of a
search space of the programs and may also depend on the form of probabilistic inference technique known as belief propagation [58]
input constraints. to infer states and/or instructions. [24] uses this modeling to infer
abstract states (i.e., invariants) given the program instructions for
5.1 Brute-force Search
the purpose of program verification. [36] uses this modeling to infer
The brute-force search technique refers to the process of enumerat- both the instructions and concrete states (before and after each in-
ing the programs in the search space in some order and checking for struction) that satisfy a given set of concrete input-output examples
each program whether it satisfies the input constraints. There have for the purpose of program synthesis. [36] applies this technique to
been few success stories of using brute-force search to discover synthesis of imperative program fragments that execute polynomial
new algorithms: mutual-exclusion algorithms [5] and bitvector al- computations and list manipulations.
gorithms [2, 52, 21].
Brute-force search has also been experimented with in the con- 5.3.2 Genetic Programming
text of functional programs. [37] presents a system that searches Genetic programming is a computational method inspired by bio-
for desired small functional programs by generating a sequence logical evolution, which discovers computer programs tailored to
of type-correct programs in a systematic and exhaustive manner a particular task [42]. It maintains a population of individual pro-
and evaluating them against given specifications. The QuickSpec grams. Computational analogs of biological mutation and crossover
tool [12] automatically generates algebraic specifications for mod- produce program variants. Mutation introduces random changes,
ules consisting of pure functions for the purpose of program under- while crossover facilitates sharing of useful pieces of code between
standing. programs being evolved. The referential transparency of functional
Brute-force search is the simplest search strategy, but is more of- programs, such as Lisp, makes crossover possible, as the state does
ten than not prohibitively expensive. Each of the above-mentioned not impact the evaluation of the exchanged subexpressions. In case
system makes brute-force search feasible by implementing various of imperative programs, the meaning of a code fragment depends
optimizations for pruning the search space. on the state in which the fragment is executed, hence there is no ref-
erential transparency and the crossover can produce faulty code or
5.2 Version Space Algebra dead code. These problems may be avoided by discarding variants
Version space algebra is another relatively simple, but efficient, that do not compile or using a post-processing step to remove dead
concept that has been used for program synthesis. Version space al- code. Each variant’s suitability is evaluated using a user-defined
gebras have been used for learning repetitive robot programs [57], fitness function, and successful variants are selected for continued
shell scripts [45], text-editing programs [46], and imperative evolution. The success of a genetic programming based system cru-
Python programs [47]. cially depends on the fitness function, which require non-trivial cre-
It is based on the notion of version space that was introduced ativity.
by Mitchell who proposed a general search technique for discover- Genetic programming has been used to discover mutual exclu-
ing boolean functions given positive/negative examples and a lan- sion algorithms [38] and to fix bugs in imperative programs [75].
guage/space over which to search for the boolean function [53]. Genetic programming may be an interesting alternative to con-
The basic idea is to maintain a set of all boolean functions in the sider for program synthesis when other conventional techniques fail
language that could be the desired unknown function (referred to to provide solutions, or when an approximate solution is accept-
as version space), i.e., those that correctly classify the given train- able. An interesting prospect is to develop techniques that combine
ing inputs, and then iteratively refine the hypothesis as more data the strengths of genetic programming with that of logical reason-
becomes available. The notion of generality of functions imposes a ing [35, 38].
5.4 Logical Reasoning Based Techniques Constraint Constraint Sophisti- Coverage
Generation Size cation of
This class of techniques reduce the program synthesis problem
Methodology Constraints
to that of solving a SAT/SMT formula and let an off-the-shelf
SAT/SMT solver efficiently explore the search space. The idea Invariant-based Smallest Most Full
is to exploit the recent advances made in the Satisfiability (SAT) Path-based Medium Medium Medium
and Satisfiability Modulo Theory (SMT) solving technology4 to Input-based Largest Least Least
efficiently explore the search space of programs. The reduction
typically involves two main steps: Figure 4. This figure illustrates the trade-off between the follow-
1. Constraint Generation, which generates a logical constraint, ing aspects for various constraint generation methodologies: size of
referred to as the synthesis constraint, such that the solution generated constraints, sophistication of generated constraints, and
to the constraint yields the desired program. This is mostly a level of coverage achieved.
syntactic process.
2. Constraint Solving for solving the synthesis constraint. This
involves reducing the generated constraint to a corresponding
SAT/SMT constraint that can be solved using off-the-shelf con- The occurrence of 𝑆, as a formula in the above equation, refers
straint solvers. to its transition system representation that assigns to fresh version
We next describe these two steps in a little detail. We rather present 𝑥1 of original program variables 𝑥. 𝐼1 and 𝑟1 refers to the formula
a simplistic view; for more details, we refer the reader to work obtained from 𝐼 and 𝑟 respectively by replacing variables 𝑥 by 𝑥1 .
on template based program synthesis [69, 67, 71] and component Eqns 1 and 2 encode that 𝐼 is an inductive invariant of the
based program synthesis [23, 33]. The former requires a back- while loop (Eqn 1 encodes the base case while Eqn 2 encodes the
ground into template based program verification [68, 25, 26, 66, 27] inductive case). Eqn 3 encodes that 𝐼 is strong enough to imply the
with the connection that the synthesis problem is a more general postcondition Post. Eqns 4 and 5 encode that 𝑟 is a ranking function
form of the verification problem. of the loop, i.e., the loop terminates.

6. Constraint Generation 6.2 Path-based

We illustrate the principle of constraint generation by briefly de- This methodology (described in [67]) generates a synthesis con-
scribing few methodologies for the special case when the user in- straint that asserts that the unknown program should be such that it
tent is specified in the form of a precondition and postcondition pair behaves correctly on all inputs that exercise a given set of paths. It
(Pre, Post), which is a form of a logical relation between inputs and combines ideas from symbolic execution based testing and con-
outputs, and the search space consists of a single while loop with straint based program analysis. It has been applied to synthesis
guard 𝑐 and body 𝑆. Figure 4 briefly summarizes the relative merits of program inverses (see Section 2.5 for examples) and client-
and challenges of these methodologies. server applications [67]. Unlike the invariant-based methodology,
These methodologies can be extended to generate constraints the path-based methodology does not guarantee that the generated
for other forms of intent specifications, and more general forms of program always meets the desired pre/post specification, and hence
program structures. Other methodologies may also be possible. requires an external verifier. On the other hand, it leads to simpler
constraints that avoid quantification over invariants. The language
6.1 Invariant-based of invariants is usually far more sophisticated than the program con-
structs (guards and statements) and usually itself involves quanti-
This methodology (described in [69]) generates a synthesis con- fiers.
straint that asserts that the unknown program should be such that ⎛ ⎞
it behaves correctly on all inputs. It is inspired by proof-based
(Pre0 ∧ ¬𝑐0 ) ⇒ Post0 (6)
deductive synthesis [51]. It has been applied to synthesis of sev- ⎜ ⎟
eral undergraduate textbook algorithms such as sorting, dynamic ⎜
⎜ ∧ (Pre0 ∧ 𝜙1 ∧ ¬𝑐1 ) ⇒ Post1 (7)⎟
⎟
programming, Strassen’s matrix multiplication, Bresenham’s line ⎜ ∧ (Pre0 ∧ 𝜙2 ∧ ¬𝑐2 ) ⇒ Post2 (8)⎟
drawing [69]. Unlike inductive synthesis based techniques that re- ∃𝑐, 𝑆 ∀𝑥0 , . . , 𝑥𝑛 ⎜ ⎟
⎜ .. ⎟
quire an external verifier (in a refinement loop) for checking cor- ⎜
⎜ .
⎟
⎟
rectness of the generated program, this methodology guarantees ⎝ ⎠
∧ (Pre0 ∧ 𝜙𝑛 ∧ ¬𝑐𝑛 ) ⇒ Post𝑛 (9)
correctness of the generated solution by integrating verification as
part of synthesis.
⎛ ⎞
Pre ⇒ 𝐼 (1) where 𝜙𝑖 ≡ (𝑐0 ∧𝑆0 ∧𝑐1 ∧𝑆1 ∧. . .∧𝑐𝑖−1 ∧𝑆𝑖−1 ). 𝑐𝑖 and Post𝑖
⎜ ⎟ are obtained from 𝑐 and Post respectively by replacing program
⎜ ∧ (𝐼 ∧ ¬𝑐) ⇒ Post (2)⎟
∃𝐼, 𝑟, 𝑐, 𝑆 ∀𝑥, 𝑥1 ⎜
⎜ ⎟ variables 𝑥 by 𝑥𝑖 . 𝑆𝑖 is obtained from 𝑆 by replacing variables 𝑥
⎜ ∧ (𝐼 ∧ 𝑐 ∧ 𝑆) ⇒ 𝐼1 (3)⎟
⎟ and 𝑥1 by 𝑥𝑖 and 𝑥𝑖+1 respectively.
⎝ ∧ 𝐼 ⇒ (𝑟 ≥ 0) (4)⎠ Eqn 6 asserts that the condition 𝑐 and loop body 𝑆 are such that
∧ (𝐼 ∧ 𝑐 ∧ 𝑆) ⇒ (𝑟1 < 𝑟) (5) the while loop satisfies the pre-post specification for all inputs 𝑥
that do not go through the loop body. Eqns 7, 8, and 9 assert the
same thing for all inputs that go through one, two, and n iterations
respectively of the loop. A larger number of such constraints (i.e., a
4 SMT solving is an extension of SAT solving technology to work with larger value of 𝑛) ensures more correctness, and in the limit would
theory facts, rather than just propositional facts. In fact, there is a SMT guarantee full correctness. The hope is that few such constraints
solving competition that is now held every year, and it has stimulated would constrain the search space enough that it would lead to a
improvement in solver implementations [1]. correct solution.
6.3 Input-based 7.2 Universal Quantifier Elimination
This methodology generates a synthesis constraint that asserts that The synthesis constraint also contains universal quantification over
the unknown program should be such that it behaves correctly the program variables. We describe below few options for eliminat-
on a given set of inputs. It is used when the input constraints ing universal quantifiers.
are in the form of input-output examples (as in oracle guided
synthesis applied to bitvector algorithms and deobfuscation [33]), 7.2.1 Domain-specific Methods
or in a counterexample guided iterative synthesis technique like Farkas Lemma is a beautiful concept for reasoning about the theory
sketching [65]. This methodology provides even less coverage than of linear arithmetic and can be used to translate universal quanti-
the path-based methodology, but has the advantage of generating fiers into existential quantifiers [25]. For the theory of predicate ab-
constraints that are simplest to solve. In particular, there is no straction, we can use cover algorithms to eliminate universal quan-
quantification over invariants, and no universal quantification. tifiers [26, 66].
∃𝑐, 𝑆 (𝜓(𝑃1 , 𝑄1 ) ∧ . . . ∧ 𝜓(𝑃𝑘 , 𝑄𝑘 )) (10) 7.2.2 Sampling
where 𝜓(𝑃, 𝑄) is as follows: Sampling is a generic method for approximating a universally
⎛ ⎞ quantified formula of the form ∀𝑥 𝜙(𝑥) by 𝜙(𝑃1 ) ∧ 𝜙(𝑃2 ) ∧ . . . ∧
(¬𝑐0 ∧ 𝑥0 = 𝑃 ∧ 𝑥0 = 𝑄) (11)⎟ 𝜙(𝑃𝑘 ) for few values 𝑃1 , . . . , 𝑃𝑘 . The values 𝑃1 , . . . , 𝑃𝑘 can be
chosen using various means described below.
⎜
⎜ ∨ (𝜙1 ∧ ¬𝑐1 ∧ 𝑥0 = 𝑃 ∧ 𝑥1 = 𝑄) (12)⎟
⎜ ⎟
⎜ ∨ (𝜙 ∧ ¬𝑐 ∧ 𝑥 = 𝑃 ∧ 𝑥
2 2 0 2 = 𝑄) (13)⎟
⎟ Counterexample Driven This is a general approach that requires
∃𝑥0 , . . , 𝑥𝑛 ⎜
⎜
⎟ an external verifier that either validates the correctness of a given
..
solution to the synthesis constraint against the user intent, or pro-
⎜ ⎟
⎜ . ⎟
⎜
⎝ ∨ (𝜙𝑛 ∧ ¬𝑐𝑛 ∧ 𝑥0 = 𝑃 ∧ 𝑥𝑛
⎟ vides a new counterexample. The new counterexample yields a
= 𝑄) (14)⎠ new value 𝑃𝑖 . This approach forms the basis of counterexample
∨ (𝜙𝑛 ∧ 𝑐𝑛 ) (15) guided iterative synthesis technique [19, 62], which involves choos-
ing some initial set of test values for the universally quantified vari-
𝜙𝑖 and 𝑐𝑖 are as defined in Section 6.2. Eqn 10 asserts that the ables and then solving for the existentially quantified variables in
unknown program should be such that it satisfies the input-output the resulting constraint using SMT solvers. If the solution for the
pairs (𝑃1 , 𝑄1 ), . . . , (𝑃𝑘 , 𝑄𝑘 ). The formula 𝜓(𝑃, 𝑄) asserts that if existentially quantified variables works for all choices of univer-
the unknown program is executed in the initial state 𝑃 , then its sally quantified variables, then a solution has been found. Else,
final state satisfies 𝑄, if it terminates within 𝑛 iterations of the loop a counterexample is discovered and the process is repeated after
body. A larger value of 𝑛 would thus achieve more coverage. The adding the counterexample to the set of test values for the univer-
conjuncts 𝜙𝑖 ∧ ¬𝑐𝑖 (in Eqns 12-14) encode that the program should sally quantified variables. This approach is used in [65, 23, 31].
terminate in exactly 𝑖 iterations of the loop, while the conjuncts Random In this approach, 𝑃𝑖 ’s are simply chosen randomly. This
𝑥0 = 𝑃 ∧ 𝑥𝑖 = 𝑄 encode that the initial state should be 𝑃 and the approach may work well for synthesis of programs whose correct-
output state should be 𝑄. Eqn 15 encodes that the program executes ness can be verified using randomly chosen inputs (with high prob-
more than 𝑛 loop iterations. ability over the choice of those inputs). In other words, this ap-
proach works well for programs for which a few randomly cho-
sen inputs can act as a full functional specification of the program
7. Constraint Solving (with high probability over the choice of the random inputs). This
includes polynomials [61] and free boolean graphs [9].
The synthesis constraints generated (in Section 6) are second-order
logic formulas with universal quantification. Unfortunately, these Biased Not all inputs in the input space are equally important.
constraints cannot be solved using off-the-shelf constraint solvers In certain domains, combining some form of bias with random-
(such as SAT/SMT solvers) because of two challenges: presence of ness works better compared to simply using pure randomness. For
second-order unknowns, and presence of universal quantification. example, a program may take an integer input 𝑖, but have the same
Next, we describe few techniques for eliminating second order behavior for all 𝑖 > 5 and have interesting behaviors only on values
unknowns (Section 7.1) and universal quantification (Section 7.2). 0 ≤ 𝑖 ≤ 5. For many applications, the user knows apriori which
This allows for use of off-the-shelf constraint solvers to solve the inputs are more crucial in defining the overall program.
resultant constraints, which yields the desired program. Biased sampling works well for synthesis of bitvector programs,
which combine fixed-width arithmetic operations with bitwise op-
erations [33]. The input space for bitvector programs consists of all
7.1 Reducing Second-order Unknowns to First-order (tuples of) bitvectors of a certain bit width. It is well-known that,
Unknowns for a very large class of commonly-used bitvector functions, the
The key idea here is to assume some template structure for the var- rightmost bits influence the output more than the leftmost bits.
ious second-order unknowns (e.g., invariants, conditional guards,
P ROPERTY 1 (See [74], Chapter 2). A function mapping bitvec-
and assignment statements) in the synthesis constraint. The tem-
tors to bitvectors can be implemented with add, subtract, bitwise
plates are such that the holes/unknowns in the templates are only
and, bitwise or, and bitwise not instructions if and only if each bit
first-order entities. This reduces the second-order constraints to
of the output depends only on bits at and to the right of that bit in
first-order constraints over the unknown parameters of the tem-
each input operand.
plates. Examples of templates for invariants include boolean com-
bination of arithmetic constraints [25, 27], boolean combination of This suggests biasing the sampling so that there is more variety on
predicates from a given set [26], quantified formulas over a given the rightmost bits. For example, if 4 values can be selected, then
set of predicates [66]. Examples of templates for guards and state- this should include bitvectors whose rightmost two bits include all
ments include those that involve arithmetic and predicate abstrac- possible four combinations: 00, 01, 10, and 11, while the leftmost
tion [69, 67, 71]. bits can be kept random.
Basis For some class of programs, the behavior of a program is Chaudhuri, Jeff Foster, Neil Immerman, Susmit Jha, Nebojsa Jojic,
fully specified by its behavior on a certain subset of its inputs, re- Shachar Itzhaky, Francesco Logozzo, Madhusudan Parthasarthy,
ferred to as basis inputs. For example, the behavior of a compari- Chris Quirk, Mooly Sagiv, Sanjit Seshia, Rishabh Singh, Armando
son machine (programs that only use comparison operators) is de- Solar-Lezama, Saurabh Srivastava, Nikhil Swamy, Ashish Tiwari,
termined by its behavior on boolean arrays [41]. The behavior of Ramarathnam Venkatesan, Ben Zorn.
functions over a vector space is determined by their behavior over
any basis of the vector space. For such programs, we can restrict
sampling to the basis inputs. References
[1] Satisfiability modulo theories competition (SMT-COMP). http:
//www.smtcomp.org/2009/index.shtml.
8. Conclusion
[2] The Aha! (a hacker’s assistant) superoptimizer, 2008. Download:
Program Synthesis has the potential to bring about a revolution http://www.hackersdelight.org/aha.zip, Documentation:
in computing. Computing has become accessible to people at an http://www.hackersdelight.org/aha/aha.pdf.
impressive rate over the last few years. However, the fundamental
[3] I. Androutsopoulos, G. Ritchie, and P. Thanisch. Natural language
programming model has not changed much since the last several interfaces to databases — an introduction. Natural Language
years. Program Synthesis has the potential to make the computer Engineering, 1(1):29–81, 1995.
understand our “what” language, as opposed to us interacting with
the computer in its language of step-by-step detailed instructions [4] D. Angluin. Learning regular sets from queries and counterexamples.
Inf. Comput., 75(2):87–106, 1987.
on “how” to accomplish the desired task.
The true success of program synthesis would require a multi- [5] Y. Bar-David and G. Taubenfeld. Automatic discovery of mutual
disciplinary effort for each of the dimensions in program synthesis. exclusion algorithms. In DISC, pages 136–150, 2003.
For the first dimension of understanding user intent, Human Com- [6] S. Barmany, R. Bodik, S. Chandra, J. Galensony, D. Kimelman,
puter Interface can help in identifying the most natural interface for C. Rodarmory, and N. Tungy. Programming with angelic nondeter-
solving problems in a particular domain. Natural Language Pro- minism. In POPL, pages 339–351, 2010.
cessing can also play a key role since natural language could be [7] A. W. Biermann, R. I. Baum, and F. E. Petry. Speeding up
an integral part of most interfaces. The second dimension of se- the synthesis of programs from traces. IEEE Trans. Computers,
lecting useful space/domain of programs for synthesis (which is 24(2):122–136, 1975.
usually limited by availability of reasoning technology for those [8] A. W. Biermann and J. A. Feldman. On the synthesis of finite-state
programs) can benefit from domain expertise in corresponding dis- machines from samples of their behavior. IEEE Trans. Comput.,
ciplines such as Information Extraction (for text manipulating pro- 21(6):592–597, 1972.
grams), Graphics (for image manipulating programs), and Pro- [9] M. Blum, A. K. Chandra, and M. N. Wegman. Equivalence of free
gramming Languages (for data-structure manipulating programs). boolean graphs can be decided probabilistically in polynomial time.
For the third dimension of searching a program over a given space, Inf. Process. Lett., 10(2):80–82, 1980.
Logical Reasoning and Machine Learning communities can play a [10] B. Bollig, P. Habermehl, C. Kern, and M. Leucker. Angluin-style
big role. learning of NFA. In IJCAI, pages 1004–1009, 2009.
Research Questions Following are some important research [11] S. Chaudhuri, S. Gulwani, and R. Lublinerman. Continuity analysis
questions for developing next generation of program synthesis tools of programs. In POPL, pages 57–70, 2010.
and techniques. [12] K. Claessen, N. Smallbone, and J. Hughes. QuickSpec: Guessing
formal specifications using testing. In Tests and Proofs, Fourth
∙ How do we combine various forms of user intent (such as
International Conference, TAP 2010. To appear.
logical specifications, natural language, input-output examples,
traces, and higher-order, inefficient, or partial programs) in a [13] A. Cypher, D. C. Halbert, D. Kurlander, H. Lieberman, D. Maulsby,
unified programming interface? B. A. Myers, and A. Turransky, editors. Watch what I do:
programming by demonstration. MIT Press, Cambridge, MA, USA,
∙ How do we ensure a modular architecture that would al- 1993.
low reuse of various synthesis components including domain [14] C. de la Higuera. A bibliographical study of grammatical inference.
knowledge (both facts and rules for manipulating objects such Pattern Recognition, 38(9):1332 – 1348, 2005. Grammatical
as text, images, data-structures) and search techniques across Inference.
different synthesis tools and applications? [15] C. de la Higuera, P. W. Adriaans, M. van Zaanen, and J. Oncina,
∙ How do we combine the power of various search techniques editors. Proceedings of the Workshop and Tutorial on Learning
such as version space algebras, logical reasoning techniques Context-Free Grammars. Ruder Boskovic Institute, Zagreb, Croatia,
(based on abstraction, mathematical induction, theorem prov- 2003.
ing, SAT/SMT/QBF solving) and machine learning techniques [16] E. W. Dijkstra. Program inversion. In Program Construction,
(probabilistic inference, bayesian learning, genetic program- http://www.cs.utexas.edu/∼EWD/ ewd06xx/EWD671.PDF,
ming) for automated program synthesis from user intent? How pages 54–57. Springer-Verlag, 1979.
do we further incorporate quantitative program analysis tech- [17] E. W. Dijkstra. Solutions of a problem in concurrent programming
niques [28, 11] to produce programs that are not just function- control (reprint). Commun. ACM, 26(1):21–22, 1983.
ally correct, but also meet performance and resource availability [18] R. Glück and M. Kawabe. A method for automatic program inversion
constraints? based on LR(0) parsing. Fundam. Inf., 66(4):367–395, 2005.
[19] E. M. Gold. Language identification in the limit. Information and
Acknowledgments Control, 10(5):447–474, 1967.
I am grateful to my family members who let me work on this paper [20] S. A. Goldman and M. J. Kearns. On the complexity of teaching.
during my vacation that was long over-due. I would like to thank Journal of Computer and System Sciences, 50:20–31, 1995.
my collaborators on various program synthesis projects, some of [21] T. Granlund and R. Kenner. Eliminating branches using a superopti-
which are referenced in this paper: Vijay Anand, Ras Bodik, Swarat mizer and the gnu c compiler. In PLDI, 1992.
[22] M. Gualtieri. Deputize end-user developers to deliver business agility Learning and Adaptive Systems, AAAI, 2004.
and reduce costs. In Forrester Report for Application Development [46] T. A. Lau, P. Domingos, and D. S. Weld. Version space algebra and
and Program Management Professionals, April 2009. its application to programming by demonstration. In ICML, pages
[23] S. Gulwani, S. Jha, A. Tiwari, and R. Venkatesan. Component based 527–534, 2000.
synthesis applied to bitvector circuits. Technical Report MSR-TR- [47] T. A. Lau, P. Domingos, and D. S. Weld. Learning programs from
2010-12, Microsoft Research, 2010. traces using version space algebra. In K-CAP, pages 36–43, 2003.
[24] S. Gulwani and N. Jojic. Program verification as probabilistic [48] M. Li and P. M. B. Vitanyi. An Introduction to Kolmogorov
inference. In POPL, pages 277–289, 2007. Complexity and Its Applications. Springer-Verlag, Berlin, 1993.
[25] S. Gulwani, S. Srivastava, and R. Venkatesan. Program analysis as [49] Y. Li, H. Yang, and H. V. Jagadish. Nalix: an interactive natural
constraint solving. In PLDI, pages 281–292, 2008. language interface for querying xml. In SIGMOD, pages 900–902,
[26] S. Gulwani, S. Srivastava, and R. Venkatesan. Constraint-based 2005.
invariant inference over predicate abstraction. In VMCAI, pages [50] D. Mandelin, L. Xu, R. Bodı́k, and D. Kimelman. Jungloid mining:
120–135, 2009. Helping to navigate the API jungle. In PLDI, pages 48–61, 2005.
[27] S. Gulwani and A. Tiwari. Constraint-based approach for analysis of [51] Z. Manna and R. Waldinger. A deductive approach to program
hybrid systems. In CAV, pages 190–203, 2008. synthesis. ACM TOPLAS, 2(1):90–121, 1980.
[28] S. Gulwani and F. Zuleger. The reachability-bound problem. In PLDI, [52] H. Massalin. Superoptimizer - a look at the smallest program. In
2010. To appear. ASPLOS, pages 122–126, 1987.
[29] B. A. Hockey and M. Rayner. Using paraphrases of deep semantic [53] T. M. Mitchell. Generalization as search. Artif. Intell., 18(2):203–226,
representions to support regression testing in spoken dialogue 1982.
systems. In Proceedings of the Workshop on Software Engineering,
Testing, and Quality Assurance for Natural Language Processing [54] MyDoom Wikipedia Article, URL accessed Sep. 2009.
(SETQA-NLP 2009), pages 14–21, Boulder, Colorado, June 2009. [55] R. P. Nix. Editing by example. ACM Trans. Program. Lang. Syst.,
Association for Computational Linguistics. 7(4):600–621, 1985.
[30] N. Immerman. Upper and lower bounds for first order expressibility. [56] J. Oncina, P. Garcı́a, and E. Vidal. Learning subsequential transducers
J. Comput. Syst. Sci., 25(1):76–98, 1982. for pattern recognition interpretation tasks. IEEE Trans. Pattern Anal.
[31] S. Itzhaky, S. Gulwani, N. Immerman, and M. Sagiv. A simple Mach. Intell., 15(5):448–458, 1993.
inductive synthesis methodology and its applications. Technical [57] M. Pardowitz, B. Glaser, and R. Dillmann. Learning repetitive robot
Report MSR-TR-2010-35, Microsoft Research, April 2010. programs from demonstrations using version space algebra. In RA
[32] R. N. Jackiw and W. F. Finzer. The geometer’s sketchpad: ’07: Proceedings of the 13th IASTED International Conference on
programming by geometry. In Watch what I do: programming by Robotics and Applications, pages 394–399. ACTA Press, 2007.
demonstration, pages 293–307. MIT Press, Cambridge, MA, USA, [58] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of
1993. Plausible Inference. Morgan Kaufmann, 1988.
[33] S. Jha, S. Gulwani, S. Seshia, and A. Tiwari. Oracle-guided [59] P. Porras, H. Saidi, and V. Yegneswaran. An analysis of conficker’s
component-based program synthesis. In ICSE, 2010. logic and rendezvous points. Technical report, SRI International,
[34] S. Jha, S. Gulwani, S. Seshia, and A. Tiwari. Synthesizing switching March 2009.
logic for safety and dwell-time requirement. In ICCPS, 2010. [60] S. Schrödl and S. Edelkamp. Inferring flow of control in program
[35] C. G. Johnson. Genetic programming with fitness based on model synthesis by example. In KI ’99: Proceedings of the 23rd Annual
checking. In EuroGP, pages 114–124, 2007. German Conference on Artificial Intelligence, pages 171–182.
Springer-Verlag, 1999.
[36] V. Jojic, S. Gulwani, and N. Jojic. Probabilistic inference of programs
from input/output examples. Technical Report MSR-TR-2006-103, [61] J. T. Schwartz. Fast probabilistic algorithms for verification of
Microsoft Research, 2006. polynomial identities. J. ACM, 27(4):701–717, 1980.
[37] S. Katayama. Systematic search for lambda expressions. In Trends in [62] E. Y. Shapiro. Algorithmic Program DeBugging. MIT Press,
Functional Programming, 2005. Cambridge, MA, USA, 1983.
[38] G. Katz and D. Peled. Genetic programming and model checking: [63] M. Smellie. String generalisation for editing by example. http:
Synthesizing new mutual exclusion algorithms. In ATVA, pages //www.mcs.vuw.ac.nz/comp/graduates/archives/honours/
33–47, 2008. 1997/michael-smellie.ps.gz, 1997.
[39] D. E. Knuth. The Art of Computer Programming, Volume [64] D. R. Smith. KIDS – a semi-automatic program development system.
IV. http://www-cs-faculty.stanford.edu/∼knuth/ IEEE TSE, 16(9):1024–1043, 1990.
taocp.html. [65] A. Solar-Lezama. Program Synthesis by Sketching. PhD thesis,
[40] D. E. Knuth. The Art of Computer Programming, Volume I: University of California, Berkeley, 2008.
Fundamental Algorithms, 2nd Edition. Addison-Wesley, 1973. [66] S. Srivastava and S. Gulwani. Program verification using templates
[41] D. E. Knuth. The Art of Computer Programming, Volume III: Sorting over predicate abstraction. In PLDI, pages 223–234, 2009.
and Searching. Addison-Wesley, 1973. [67] S. Srivastava, S. Gulwani, S. Chaudhuri, and J. Foster. Program
[42] J. R. Koza. Genetic programming: on the programming of computers inversion revisited. Technical Report MSR-TR-2010-34, Microsoft
by means of natural selection. MIT Press, Cambridge, MA, USA, Research, 2010.
1992. [68] S. Srivastava, S. Gulwani, and J. S. Foster. VS3: SMT solvers for
[43] V. Kuncak, M. Mayer, R. Piskac, and P. Suter. Complete functional program verification. In CAV, pages 702–708, 2009.
synthesis. In PLDI, 2010. To appear. [69] S. Srivastava, S. Gulwani, and J. S. Foster. From program verification
[44] T. Lau. Programming by Demonstration: a Machine Learning to program synthesis. In POPL, pages 313–326, 2010.
Approach. PhD thesis, University of Washington, Seattle, 2001. [70] Symantec Corporation. Internet security threat report volume XIV.
[45] T. Lau, L. Bergman, V. Castelli, and D. Oblinger. Programming shell http://www.symantec.com/business/theme.jsp?themeid=
scripts by demonstration. In Workshop on Supervisory Control of threatreport, April 2009.
[71] A. Taly, S. Gulwani, and A. Tiwari. Synthesizing switching logic [75] W. Weimer, T. Nguyen, C. L. Goues, and S. Forrest. Automatically
using constraint solving. In VMCAI, pages 305–319, 2009. finding patches using genetic programming. In ICSE, pages 364–374,
[72] M. T. Vechev and E. Yahav. Deriving linearizable fine-grained 2009.
concurrent objects. In PLDI, pages 125–135, 2008. [76] I. H. Witten and D. Mo. TELS: learning text editing tasks from
[73] J. M. Vilar. Query learning of subsequential transducers. In ICG ’96: examples. In Watch what I do: programming by demonstration, pages
Proceedings of the 3rd International Colloquium on Grammatical 293–307. MIT Press, Cambridge, MA, USA, 1993.
Inference, pages 72–83. Springer-Verlag, 1996. [77] L. S. Zettlemoyer and M. Collins. Learning context-dependent
[74] H. S. Warren. Hacker’s Delight. Addison-Wesley, ’02. mappings from sentences to logical form. In ACL-IJCNLP, pages
976–984, 2009.