Artificial Intelligence 112 (1999) 57–104

Enhancing model checking in verification

by AI techniques

Francesco Buccafurri a,1 , Thomas Eiter b,2 , Georg Gottlob c,∗ ,

Nicola Leone c,3
a DIMET, Universitá di Reggio Calabria, loc. Feo di Vito, I-89100 Reggio Calabria, Italy
b Institut and Ludwig Wittgenstein Labor für Informationssysteme, Technische Universität Wien, Treitlstraße 3,
A-1040 Wien, Austria
c Institut and Ludwig Wittgenstein Labor für Informationssysteme, Technische Universität Wien, Paniglgasse 16,
A-1040 Wien, Austria

Received 30 September 1997

Abstract
Model checking is a fruitful application of computational logic with high relevance to the
verification of concurrent systems. While model checking is capable of automatically testing that
a concurrent system satisfies its formal specification, it can not precisely locate an error and suggest
a repair, i.e., a suitable correction, to the system. In this paper, we tackle this problem by using
principles from AI. In particular, we introduce the abstract concept of a system repair problem,
and exemplify this concept on repair of concurrent programs and protocols. For the development
of our framework, we formally extend the concept of counterexample, which has been proposed in
model checking previously, and provide examples which demonstrate the need for such an extension.
Moreover, we investigate into optimization issues for the problem of finding a repair, and present
techniques which gain in some cases a considerable reduction of the search space for a repair.  1999
Elsevier Science B.V. All rights reserved.

Keywords: Abductive theory revision; Model checking; Diagnosis; Repair

∗ Corresponding author. Email: [email protected].

1 Email: [email protected].
2 Email: [email protected].
3 Email: [email protected].

0004-3702/99/$ – see front matter  1999 Elsevier Science B.V. All rights reserved.
PII: S 0 0 0 4 - 3 7 0 2 ( 9 9 ) 0 0 0 3 9 - 9
58 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

1. Introduction

Model checking, which has been first proposed by Clarke and Emerson [9,10], is
an approach to automated verification of finite-state concurrent systems such as circuit
designs and communication protocols. In this approach, specifications are expressed in a
propositional temporal logic, and the concurrent system is modeled as a state transition
graph, which amounts to a Kripke structure for this logic. Checking whether the system
satisfies its specification, given by a logical formula, reduces then to test whether the Kripke
structure is a model of the formula.
Model checking has several important advantages over other methods for verification
of circuits and protocols, like mechanical theorem provers or proof checkers. The most
relevant one is that it is efficient and highly automatic. Recent advances are made in
model checking by using special data structures and algorithms, known as symbolic model
checking, some of which have more than 10120 states [8]. A number of major companies
including Intel, Motorola, Fujitsu, and AT&T have started using symbolic model checkers
to verify actual circuits and protocols. Thus, (symbolic) model checking is nowadays
considered to be one of the most fruitful and promising applications of computational
logic.
On the other hand, various techniques for diagnostic reasoning on systems have been
developed in the field of AI, including logic-based approaches like model-based diagnosis
and repair [27]. These approaches utilize general AI principles and are successfully used
in different application domains. Our work approaches a new and promising field for
application of AI techniques, which is in particular attractive for knowledge representation
and reasoning methods, and thus adds to the application perspective of this field [1].
In this paper, we study the enhancement of model checking by abductive reasoning,
which is a major technique in AI and knowledge representation, cf. [3,4,16,19,29,31,
40,43]. The work presented does not exhaustively treat this issue, and further work is
necessary; however, it is a first step towards an integration of model checking with AI
techniques, and may stimulate other work in this direction.
The main contributions of the present paper can be summarized as follows.
• We study the integration of model checking and AI principles. In particular, we
introduce the system repair problem in the context of Computational Tree Logic
(CTL) (a temporal logic used to express the specifications of concurrent systems to be
checked), which is a formal framework for repairing a concurrent system, described
by a Kripke model, at the semantical level. Notice that in a different context, repair
was introduced in [23,24,41].
The system repair problem amounts to an interesting abductive model revision
problem: Determine by abductive reasoning a suitable change of the system (i.e., of its
Kripke model) such that the specification is satisfied upon this change. Interestingly,
this problem is an intermingled abductive reasoning and theory revision problem,
which is best understood as an abductive theory revision problem. In fact, the
system repair problem can be modeled as an abductive theory revision problem in
the frameworks of [29,33]. Note that the close relationship between abduction and
revision is well-recognized, and its investigation received increasing interest more
recently, e.g., [5,29,33,34].
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 59

• We show that the proposed framework for system repair can be profitably used,
by providing an application to the repair of concurrent programs and protocols. In
particular, we describe a program repair problem, in which repair of a concurrent
program in terms of changes at the syntactical level (i.e., modifications of the program
code) is mapped to changes at the semantical level. As dealing with all possible
modifications is clearly infeasible, we restrict in our approach to some types of
modifications which seem to be relevant in practice. A repair is then a sequence
α = α1 · · · αq of basic corrections αi on the program, such that the modified program
satisfies the specification ϕ. Applying AI principles and, in particular, Occam’s
principle of parsimony, we provide also a notion of minimal solution, that prunes
solutions which are not free of redundancy.
• We face the problem of searching for a program repair. In general, the search space
for this problem is large, and might still contain a number of candidates, even if only
elementary corrections such as inverting the value of an expression or exchanging the
name of a variable in a Boolean assignment statement are adopted as repairs.
In order to alleviate this problem, we develop optimization techniques which sensibly
reduce the search space for a repair by exploiting structural information about the
failure of the system provided by a counterexample. In particular, we formulate two
pruning criteria referred to as correction execution and correction exploitation, which
can be evaluated efficiently. In fact, given the program, a collection of candidate
repairs, and a counterexample, the candidates violating these principles can be
discarded in linear time. As we demonstrate, this may yield considerable savings, and
thus applying correction execution and exploitation is an effective pruning method
which comes at low computational cost.
• We formally extend the notion of counterexample from [11], which is a heuristically
selected computation path from a conceptual counterexample tree (i.e., an evolving
branching computation) that gives a hint at the failure of the system. As shown by
examples, there are cases in which no single path is a counterexample. We therefore
introduce the concept of a multi-path, which enables representation of nested paths.
Multi-paths turn out to be a suitable tool for expressing full counterexample trees,
which is needed for our purposes.
To give the flavor of theory and application developed in this paper, we discuss a
motivating example.
Consider the concurrent program P in Fig. 1. It consists of processes PA and PB ,
which share two common Boolean variables x and y. To ensure mutual exclusion of the
assignments to x and y, some control variables, flags and turns, are introduced, following
the classical Peterson scheme [38], in which each critical section is executed obeying an
entry and exit protocol. There are then two critical sections in each process, one for the
assignments to x (statements 5 in PA and statements 5–11 in PB ), and another one for the
assignments to y (statements 12 in PA and 10 in PB , respectively); notice that in PB , the
critical section for y is nested into the critical section for x. Each variable flagiV indicates
the request of process V to enter critical section i, and turniB tells whether such a request
by process B in case of simultaneous requests should be granted. The six control variables
flagiA and flagiB, and turniB, for i = 1, 2 are shared among the two processes.
60 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Process PA Process PB

1: flag1A := true; 1: flag1B := true;

2: turn1B := false; 2: turn1B := false;
3: if flag1B and turn1B then 3: if flag1A and not turn1B then
4: goto 3; 4: goto 3;
5: x := x and y; 5: x := x and y;
6: flag1A := false; 6: flag2B := true;
7: if turn1B then 7: turn2B := false;
8: begin flag2A := true; 8: if flag2A and not turn2B then
9: turn2B := true; 9: goto 8;
10: if flag2B and turn2B then 10: y := not y;
11: goto 10; 11: x := x or y;
12: y := false; 12: flag2B := false;
13: flag2A := false; 13: flag1B := false;
end ;
14: goto 1; 14: goto 1;

Fig. 1. A concurrent program P.

The critical sections have been set up for the purpose of fulfilling some part of the system
specification. The complete specification prescribes that P satisfies mutual exclusion
for assignments to x and y, respectively, and absence of starvation. For example, PA
must not execute instruction 5, if PB executes instruction 5 or 11 at the same time.
Absence of starvation requires that a request of a process for a resource (by setting a
flag) must eventually be granted. Clearly, this makes sense only under the hypothesis that
the underlying scheduler is fair; absence of starvation cannot be ensured if the scheduler
always dispatches the same process.
Careful inspection of P shows that the program is not correct, even under fair schedules;
instruction 2 of PA should be turn1B := true. Even in this small example, however,
detecting the error is not immediate for the non-expert. Model checking allows for
checking the correctness of P (and of much larger programs) in a fully automatic way. The
specification of the system, mutual exclusion and absence of starvation, can be expressed
in the temporal logic ACTL [26]; fair schedules are specified my means of fairness
constraints [11]. Then, an automatic procedure verifies whether the program meets the
specifications or not. If the program is incorrect, however, model checkers usually can not
single out the error precisely, and are far from fixing a bug.
By using abductive reasoning, our method goes beyond error checking: it tries to locate
a bug and proposes a repair for the program, such that the modified program meets the
specification. Our approach considers possible errors both in the left and right hand side of
an assignment as well as the interchange of two successive assignments.
Like abduction, program repair comes at computational cost. Even if we assume the
case of a single error in the program and we plausibly restrict in Fig. 1 attention to the
assignments of control variables, we must consider 12 assignments (1, 2, 6, 8, 9, 13 in PA
and 1, 2, 6, 7, 12, 13 in PB ) and 6 control variables. Thus, (5 + 1) · 12 + 5 = 77 attempts of
repair should be done, namely 5 corrections of the control variable on the left hand side and
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 61

one correction of the right hand side of each statement, and 5 interchanges of neighbored
assignments to control variables (1,2 and 8,9 in PA and 1,2; 6,7; 12,13 in PB ). Each of
these corrections requires a call of the model checker to see if it works.
Towards more efficient program repair, we have designed optimization techniques, based
on counterexamples [11], for the case of a single error in the program, which is often
considered in practice. By applying these techniques, our procedure makes only 17 (instead
of 77) attempts in the worst case.
The remainder of this paper is structured as follows. In Section 2, we recall the syntax
and the semantics of the logic CTL. In Section 3, we address the problem of modifying
a system, given in terms of a Kripke structure, such that a given formula holds on it. In
the course of this, we introduce the notion of system repair problem, which provides a
general framework for the problem of properly changing a system at the semantical level
in order to meet a formal specification, given in CTL. A system repair problem constitutes
a kind of abductive model revision problem, and we outline how such a problem can be
represented in the frameworks for abductive theory change proposed in [29,33]. After that,
we consider in Section 4 the corresponding problem at the syntactical level, in particular the
one of correcting concurrent programs and protocols. The program repair problem, which
is addressed there, resorts at the semantical level to a system repair problem. In Section 5,
we then address the problem of finding repairs. For this purpose, we suitably generalize the
notion of counterexample described in [11], and show that counterexamples characterize
errors. In the subsequent Section 6, we address optimization techniques which, by use of
counterexamples, may allow to considerably reduce the number of possible repairs that
have to be considered. In particular, we formulate correction execution and correction
exploitation, investigate their computational feasibility and show the effectiveness of the
techniques on an example. The final Section 7 concludes the paper and states some issues
for further work.
In order to increase readability, proofs of technical results except a few short ones have
been moved into the appendix.

2. Computational tree logic

Computational Tree Logic (CTL) is a propositional branching-time temporal logic [9];

see [12,20] for a rich background on this and further such logics. The semantics of CTL
is given by Kripke Structures which model finite-state systems. CTL is used to represent
specification in reactive systems. Linear-time features of CTL are useful to capture ordering
of events in time. Branching-time operators allow to take into account the existence of
multiple possible future scenarios, starting from a given point of a computation. Indeed, in
a branching frame the temporal order defines a tree which branches toward the future. Thus,
every point of time has a unique past, but, in general, more than one future. Branching-
time operators allow us to deal with this form of non-determinism. Using these operators,
we will express the truth or falsehood of a certain property as being relative to a given
branch of the computation tree, such that we can express both possible properties (true
in a possible evolution of time in the future) and necessary ones (true in all possible
computation branches).
62 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

CTL is a fragment of the more general logic CTL∗ [21], which combines both
branching-time and linear-time operators. The branching-time operators are A and E,
which intuitively say “for every respectively some computation path”, and the basic linear-
time operators are X (next time), U (until), and V (unless, releases); further operators are
derived from them.

Definition 2.1. Let A be a set of atomic propositions. CTL is the set of state formulas on
A inductively defined as follows:
(1) any atomic proposition a ∈ A is a state formula;
(2) if ϕ and ψ are state formulas, then ¬ϕ, ϕ ∨ ψ, and ϕ ∧ ψ are state formulas;
(3) if ϕ and ψ are state formulas, then Xϕ, ϕUψ and ϕVψ are path formulas;
(4) if ϕ is a path formula, then E(ϕ) and A(ϕ) are state formulas.
Any formula ϕ which is formed only by (1)–(2) is called a pure state formula.

Intuitively, path formulas describe properties of paths because they use temporal
operators next time, until and unless, looking forward in a computation path.
For a formal definition of the semantics of CTL, special Kripke structures are used.
Informally, a Kripke structure consists of a labeled finite transition graph.

Definition 2.2. A Kripke structure is a quintuple M = (A, S0 , S, R, L) such that:

• A is a finite set of atomic propositions;
• S is a finite set of states;
• S0 ⊆ S is a finite set of initial states;
• R ⊆ S × S is a transition relation;
• L : S → 2A is a mapping assigning each state of S the set of atomic proposition true
in that state; L is called label function.
Given a Kripke structure M, we denote by A(M) its set of atomic propositions, by S0 (M)
the set of initial states, by S(M) the set of states, by R(M) the transition relation, and,
finally, by L(M) the label function.

Starting from initial states, R generates the (infinite) computation paths.

Definition 2.3. A path π of a Kripke structure M is an infinite sequence [s0 , s1 , . . . , si , . . .]

such that for each i > 0 (si , si+1 ) ∈ R. Given an integer i > 0 and a path π , we denote by
π(i) the ith state of π . Given an integer j > 0 and a path π , the j -suffix π j of π is the path
[π(j ), π(j + 1), . . .] (clearly, π = π 0 and π(i) = π i (0)).

The semantics of CTL is defined through an entailment relation |=, which can be applied
on states s and paths π for evaluating state or path formulas, respectively.

Definition 2.4. The entailment relation |= for state and path formulas on a Kripke structure
M is as follows (s and π are a generic state and path in M, respectively):
(1) M, s |= p, if p ∈ L(M)(s), for any atomic proposition p ∈ A(M);
(2) M, s |= ¬ϕ, if M, s 6|= ϕ (ϕ is a state formula);
(3) M, s |= ϕ1 ∨ ϕ2 , if M, s |= ϕ1 or M, s |= ϕ2 (ϕ1 , ϕ2 are state formulas);
(4) M, s |= ϕ1 ∧ ϕ2 , if M, s |= ϕ1 and M, s |= ϕ2 (ϕ1 , ϕ2 are state formulas);
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 63

(5)M, s |= E(ψ), if there exists a path π with π(0) = s such that M, π |= ψ;

(6)M, s |= A(ψ), if M, π |= ψ for all paths π with π(0) = s;
(7)M, π |= ϕ, if M, π(0) |= ϕ where ϕ is a state formula;
(8)M, π |= Xϕ, if M, π 1 |= ϕ;
(9)M, π |= ϕ1 Uϕ2 , if there exists an integer k > 0 such that M, π k |= ϕ2 and M, π j |=
ϕ1 , for all 0 6 j < k;
(10) M, π |= ϕ1 Vϕ2 , if for every k > 0, M, π j 6|= ϕ1 for all 0 6 j < k implies M, π k |=
ϕ2 .
We write M |= ϕ if M, s0 |= ϕ, for every initial state s0 ∈ S0 (M).

Intuitively, a state formula holds along a path, if it is true at its first state; ϕ1 Uϕ2 is true,
if ϕ1 is true along the path until some stage is reached at which ϕ2 is true; and ϕ1 Vϕ2 is
true, if there is no stage such that ϕ2 is false and ϕ1 is false at all previous stages. Note that
U and V are dual operators: ϕ1 Uϕ2 is true precisely if ¬ϕ1 V¬ϕ2 is false.
Two important additional operators, F (finally) and G (globally) are expressed through
U and V.

Definition 2.5. Given a state formula ϕ, the operators F and G are defined as follows:
• Fϕ = true Uϕ,
• Gϕ = false Vϕ (= ¬F¬ϕ),
where true is a Boolean tautology and false is a Boolean contradiction.

Thus, coherent with the intuition, M, π |= Fϕ means that there exists an integer k > 0
such that M, π k |= ϕ, while M, π |= Gϕ means that for every k > 0, M, π k |= ϕ.
EX, EG and EU (or, dually, AX, AF, and AV) can be seen as basic time operators of
CTL. The following equivalences are well known (see, e.g., [12]).

Proposition 2.1.

AXϕ = ¬EX(¬ϕ), A(ϕUψ) = ¬E(¬ψU(¬ϕ ∧ ¬ψ)) ∧ ¬EG¬ψ,

AGϕ = ¬EF(¬ϕ), A(ϕVψ) = ¬E(¬ϕU¬ψ),
AFϕ = ¬EG(¬ϕ), ¬A(¬ϕU¬ψ) = E(ϕVψ).

For modeling fair computations, Kripke structure with fairness constraints (FC-Kripke
structure, for short) have been proposed.

Definition 2.6. An FC-Kripke structure M is an expansion of a Kripke structure K =

(A, S0 , S, R, L) by a finite set F of CTL formulas, called fairness constraints, i.e., M =
(A, S0 , S, R, L, F ).

For any FC-Kripke structure M, we denote by F (M) its set of fairness constraints; the
other components of M are denoted as for ordinary Kripke structures.
The semantics of CTL formulas is adapted to fairness constraints by restricting the path
quantifiers to those paths along which every fairness constraint holds infinitely often, which
are called fair paths. More formally,
64 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Definition 2.7. A path π in an FC-Kripke structure M is fair, if for every ϕ ∈ F (M) and
i > 0 there exists an integer j > i such that M, π(j ) |= ϕ.

Entailment of state and path formulas from an FC-Kripke structure M is defined

analogous to entailment from a Kripke structure, with the only difference that path
quantifiers A and E evaluate to “for all fair paths in M” and “there exists a fair path in
M”, respectively. Since the notion of entailment will be clear from the context, we will use
for both entailment from a Kripke and an FC-Kripke structure the same symbol “|=”.

Notation. Throughout this paper, we use kOk to denote the size of an object O represent-
ed as a string in the standard way, i.e., the numbers of symbols in this string.

3. Abductive model revision

In this section, we present an approach for changing a Kripke structure such that it
satisfies a given formula ϕ. Our approach is in spirit of methods in the field of theory
revision and abductive reasoning, and can be viewed as a semantical approach to changing
a system which is represented by some Kripke model; typical such systems are concurrent
programs and protocols. This semantical approach can be utilized as the underlying basis
of an approach for change at the syntactical level of a system, i.e., its description in some
formal specification language. This will be exemplified with the problem of repairing
concurrent programs and protocols in the next section.

3.1. System repair problem

Given a CTL formula ϕ and an FC-Kripke structure M, a model checking technique can
be applied for verifying whether ϕ is satisfied by M or not. Actually, symbolic model
checking [8,36] gives as the result a description of the set of states where ϕ holds or
provides a counterexample, which outlines a case in which ϕ does not hold.
This is, in general, very useful to the protocol or circuit designer, because it aids him or
her in understanding which part of the system fails, and is a clue for finding a modification
of the system such that the specification, given by formula ϕ, holds. However, model
checking does not provide any methods for repairing the system. That is, there is no
component which suggests, given that the system does not satisfy the specification ϕ, a
possible modification to the system upon which it satisfies ϕ. Clearly, such a component
would be desirable in practice. Notice that the notion of repair (or therapy) in the context
of model-based diagnosis was introduced in [23,24] and independently in [41].
For this purpose, we formalize the system repair problem as a problem, given by an FC-
Kripke structure M and a formula ϕ, whose solution consists of a set of modifications to
the transition relation R (additions or deletions of tuples in R), such that ϕ is true in the
modified system. As shown by examples later on, a solution of a system repair problem
may give a useful clue of how to properly modify the system.
In what follows, we assume that M = (A, S0 , S, R, L, F ) is an FC-Kripke structure and
ϕ is a CTL formula.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 65

We start with the elementary concept of a simple modification of the transition relation,
which is an addition or a deletion of a state transition.

Definition 3.1. Let R ⊆ S × S. Every pair δ = h(s1 , s2 ), ⊕i, where s1 ∈ S, s2 ∈ S, and

⊕ ∈ {−, +}, is a simple modification. The application of δ on R, denoted δ(R), is
R ∪ {(s1 , s2 )}, if ⊕ = +, and R \ {(s1 , s2 )} otherwise.

A modification of the system M is a consistent set of simple modifications of its

transition relation R, where consistent means that no simultaneous addition and deletion
of a given pair of states is allowed. More precisely,

Definition 3.2. A modification for M is a set Γ of simple modifications for R such that
Γ contains no tuples h(s10 , s20 ), +i, h(s10 , s20 ), −i. Let Γ + = {(s1 , s2 ) | hs1 , s2 , +i ∈ Γ } and
Γ − = {(s1 , s2 ) | hs1 , s2 , −i ∈ Γ }. The set of all modifications for M is denoted by mod(M).
The result Γ (M) of Γ is the FC-Kripke structure (A, S0 , S, R Γ , L, F ) where
\  [ 

 δ(R) ∪ δ(R) \ R , if Γ 6= ∅,
R Γ = δ∈Γ − δ∈Γ +

R, otherwise.

Now we introduce the system repair problem. Intuitively, it represents the problem of
finding a system modification Γ for M, such that Γ (M) satisfies ϕ. In general, Γ must be
implemented on a formal description of the system (i.e., the code of concurrent programs
and protocols), and not every Γ might be actually feasible; therefore, we add a function Y,
which tells whether a particular modification Γ is admissible.

Definition 3.3. A system repair problem (SRP) is a triple S = hM, ϕ, Yi where M is

an FC-structure, ϕ is a formula on A(M), and Y is a computable Boolean function on
mod(M). Any modification Γ such that Y(Γ ) = true is called admissible.

Observe that Definition 3.3 does not request that M 6|= ϕ holds. In fact, if M |= ϕ,
there will be a trivial solution to the SRP given by the empty modification Γ = ∅,
which is always assumed to be admissible. This is similar to, e.g., Reiter’s definition of
diagnosis [42], where the diagnosis in case of a correct system is an empty set of faulty
components.
An SRP hM, ϕ, Yi is also called the repair problem of M with respect to ϕ under Y.
The admissibility function Y is domain-dependent; for example, in case of a concurrent
program, Y is derived from possible changes to the code of the processes.
A solution of an SRP states how the original system, which does presumably not satisfy
ϕ, has to be modified by means of a set Γ of admissible modifications of its transition
relation.

Definition 3.4. Given an SRP S = hM, ϕ, Yi, a solution for S is an admissible modifi-
cation Γ for M such that Γ (M) |= ϕ. A solution Γ for S is minimal, if there exists no
solution Γ 0 for S such that Γ 0 is properly contained in Γ .
66 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Fig. 2. Labeled transition graph.

The restriction of arbitrary solutions to minimal ones is natural and implements Occam’s
principle of parsimony. In general, a solution preference could be used to select preferred
solutions as customary, e.g., in abductive reasoning, cf. [18]. Observe that if M |= ϕ, then
Γ = ∅ is the unique minimal solution to S.

Example 3.1. Consider the SRP S = hM, ϕ, Yi, where M is the FC-Kripke structure
as obvious from Fig. 2 with unique initial state s0 and no fairness constraints, ϕ =
AGAFa, and Y ≡ true, i.e., each modification Γ is admissible. (Note that R(M) =
{(s0 , s0 ), (s0 , s1 ), (s1 , s1 )}.) It holds that M 6|= ϕ: For the path π = [s0 , s1 , s1 , . . .], we
have M, π 6|= GAFa, as it can be easily seen that M, π 1 |= EG¬a. A solution of S is
the modification Γ1 = {hs0 , s1 , −i}, i.e., delete the transition from s0 to s1 . Indeed, then
π = [s0 , s0 , . . .] is the unique infinite path starting at an initial state, and a is true at each
stage of π . Further solutions are, e.g.,
 
Γ2 = hs1 , s1 , −i, hs1 , s0 , +i and Γ3 = hs0 , s1 , −i, hs0 , s0 , −i .
Notice that Γ1 is minimal, while Γ2 and Γ3 are not. In fact, Γ1 and Γ4 = {hs1 , s1 , −i} are
all minimal solutions of S.

Comparing system repair to problems in AI, it appears that an SRP is an interesting kind
of abductive model revision problem, which involves both theory revision and abductive
reasoning. From the definition, an SRP can neither be viewed as a pure theory revision
problem (cf. [25,30]), nor as a pure abductive reasoning task as, e.g., in [3,4,16,19,31,
40,43]; rather, it is a combined problem and can be best understood (and modeled) as an
abductive theory revision problem [29,33].
On the one hand, an SRP is similar to a theory revision problem. Indeed, the FC-Kripke
structure M can be viewed as the set Th(M) of all formulas true on it, and we have to
revise this knowledge base by incorporating the formula ϕ (the specification) into it. Here,
the revised knowledge base M ◦ ϕ must amount to a modified FC-Kripke structure M 0 such
that M 0 entails ϕ.
On the other hand, an SRP involves abductive reasoning: given M and ϕ, we want
to find a particular modification Γ ∈ mod(M) such that applying Γ to M yields a
structure M 0 = Γ (M) in which ϕ is true; thus, we abduce a solution Γ for the SRP
in terms of changes to the transition relation. This can be formalized in a proper
logical language. However, this does not mean an SRP is a genuine abductive reasoning
task; in fact, abduction is applied in case of incomplete knowledge, which is in this
view about the suitable changes for transforming M into M 0 . This is a somewhat
unnatural state-oriented view, though, since like in planning, we proceed from one state
(M) to another (M 0 ), and the transition is specified in the domain theory using frame
axioms.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 67

3.2. System repair and abductive theory revision

More naturally, an SRP can be viewed as an abductive theory revision problem. We

outline in the following how this is possible in the frameworks of [33] and [29].

SRP in Lobo and Uzcátegui’s framework

In [33], the following scenario is considered. Given an abductive domain theory Σ,
a knowledge base K, a formula ω (all in a finite language), and a revision operator ◦, a
suitable formula γ is an explanation for ω (with respect to K, Σ and ◦), if the knowledge
base K ◦ (Σ ∧ γ ) entails ω. Suitability of γ means that γ is formed over a specified set Ab
of abducible atoms. An explanation γ can be seen as a proper abductively inferred revision
for incorporating ω into the knowledge base.
An SRP S = hM, ϕ, Yi can be modeled in this framework as follows. It is possible to
express the entailment problem M |= ϕ as an inference problem KM |= f (ϕ) of a formula
f (ϕ) from a knowledge base KM in a suitable logic L, where KM describes M and
f (ϕ) is a translation of ϕ into L; for example, L could be transitive closure logic if M
is serial (i.e., each state has a successor) and no fairness constraints are present [28], or
a similar extension of first-order logic with generalized quantifiers in the general case.
Following [28] and similar translations of propositional modal logic into first-order logic,
KM can be constructed as a set of literals describing the components of M; the states
are constants, ground literals ±R(s, s 0 ) describe the transition relation, and ground literals
±Pa (s) represent the label function, where Pa (s) means that a ∈ L(s), for each atom
a ∈ A(M).
Now choose Ab to be the set of all ground atoms on R, and let the domain theory Σ be
empty. Furthermore, let us restrict acceptable abductive explanations to formulas γ which
are conjunctions of literals.
Then, an acceptable abductive explanation γ for the formula f (ϕ) according to KM
and any reasonable revision operator ◦ (see [25,30]), is a conjunction of ground literals
on R. Any such γ corresponds to a set Γ ∈ mod(M) as follows: For each positive literal
R(s, s 0 ) in γ , the tuple hs, s 0 , +i is in Γ , and for each negative literal ¬R(s, s 0 ) in γ , the
tuple hs, s 0 , −i is in Γ . If Γ is admissible, then it is a solution of S; call γ admissible
in this case. Vice versa, each solution Γ of the SRP S corresponds to an admissible
explanation γ in the same way. Thus, there is a one-to-one logical correspondence between
admissible explanations γ of f (ϕ) and solutions Γ of S. In particular, modulo Y the
minimal solutions of S correspond to the most general admissible explanations γ , i.e., any
admissible explanation γ 0 with γ |= γ 0 satisfies γ 0 |= γ . The disjunction of all these γ
is an abductive explanation, provided ◦ satisfies some property [33], and amounts to the
collection of all minimal solutions of S.
Thus, an SRP can be modeled as (slightly constrained) abductive revision problem as
described in [33]. 4 We remark that in the above modeling, the domain theory Σ is empty.
Of course, we could have set Σ to the part of KM not involving R, but this would not make

4 In fact, Lobo and Uzcátegui work in a finite propositional language; their framework can be extended for the
slightly more general setting here.
68 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

a difference. Moreover, in some cases the admissibility function Y can be easily expressed
in the domain theory.

SRP in Inoue and Sakama’s framework

In [29], an extended form of abduction is proposed, which is employed for an abductive
framework of nonmonotonic theory change. The framework is detailed for autoepistemic
logic, but it can be analogously based on other logics as well. In this approach, an abductive
framework is a pair hT , H i of theories T and H , where T is the background theory
(containing domain and factual knowledge) and H is a set of generic hypotheses; an
explanation for a formula γ given hT , H i is a pair (I, O) such that I and O are instances
of formulas in H , (T ∪ I ) \ O logically entails γ , and (T ∪ I ) \ O is consistent. An expla-
nation (I, O) is minimal, if every explanation (I 0 , O 0 ) with I 0 ⊆ I and O 0 ⊆ O is identical
to (I, O).
The salient point in Inoue and Sakama’s concept of abductive explanation is that
formulas may also be removed from the background theory T , rather than only added.
This is motivated by their observation that in a nonmonotonic context, it may be necessary
to remove formulas from T in order to find an explanation for a formula γ . Observe that
removal of formulas from T is accomplished in the framework of [33] implicitly through
the revision operator ◦.
An SRP S = hM, ϕ, Yi can be modeled in Inoue and Sakama’s framework as follows.
As described above, the entailment problem M |= ϕ can be expressed as an inference
problem KM |= f (ϕ) in a suitable logic L (e.g., transitive closure logic). If we take KM as
background theory T and the set {R(x, y), ¬R(x, y)} as generic hypotheses H , then the
solutions of S naturally correspond to the explanations of γ = f (ϕ) obtained from hT , H i
as follows.
If Γ is a solution of S, then the pair (IS , OS ) is an explanation of γ , where
 
IS = R(s, s 0 ) | (s, s 0 ) ∈ Γ + ∪ ¬R(s, s 0 ) | (s, s 0 ) ∈ Γ − ,
 
OS = ¬R(s, s 0 ) | (s, s 0 ) ∈ Γ + ∪ R(s, s 0 ) | (s, s 0 ) ∈ Γ − .

On the other hand, if (I, O) is an explanation of γ , then the set

 
Γ(I,O) = hs, s 0 , +i | R(s, s 0 ) ∈ I \ O ∪ hs, s 0 , −i | ¬R(s, s 0 ) ∈ I \ O

is a solution of S, provided Γ(I,O) is admissible in terms of Y. Notice that this establishes

a natural one-to-one logical correspondence between minimal solutions and minimal
admissible explanations, where the admissible explanations (I, O) are those such that
Γ(I,O) is admissible. As in the previous case, the admissibility function Y may be expressed
in some cases in the background theory T .
Some further remarks are in order. Firstly, the above translation includes negative literals
¬R(s, s); for each pair of states s, s 0 such that there is no transition from s to s 0 . Of course,
one could use the closed world assumption (CWA) on the set of positive literals R(s, s 0 )
and obtain the same effect. However, this would blur the fact that the transition relation R
in a Kripke structure M is completely specified, i.e., it is known whether a transition from
s to s 0 exists or not. Secondly, for minimal explanations (I, O), it holds that I \ O = I , and
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 69

I = {∼L | L ∈ O}, where ∼L denotes the opposite of the literal L. In general, solutions
can also be expressed in terms of O \ I as follows. For an explanation (I, O) of γ , define
 
Γ(O,I ) = hs, s 0 , +i | ¬R(s, s 0 ) ∈ O \ I ∪ hs, s 0 , −i | R(s, s 0 ) ∈ O \ I .
Then Γ(O,I ) is a solution, provided it is admissible in terms of Y. In particular, we get again
a one-to-one logical correspondence between minimal solutions and minimal admissible
explanations as above.

4. Repair of concurrent programs

In this section, we apply the model from above to concurrent programs and protocols.
A concurrent program consists of a number of processes running in parallel. In [39], propo-
sitional temporal logic is used for representing specifications for concurrent programs and
for defining a technique to verify such a specification. We adopt Pnueli’s model [39] for
representing concurrent programs, in order to have a clear and synthetic way of repre-
senting the semantics of concurrent programs. We then define the FC-Kripke structure of
a concurrent program under the assumption that shared variables have a Boolean domain.
The transition relation is defined referring to an asynchronous model with interleaving [36].
Specifications for the program are described by using both CTL formulas and fairness con-
straints. Fairness constraints allow to express properties that must be verified infinitely of-
ten along paths (which are not expressible in CTL). Such properties are frequently required
in the context of concurrency, e.g., for the fair scheduling of processes.
Consider then the FC-Kripke structure of a concurrent program P; denote this structure
by MF (P). Let ϕ be a CTL formula comprising all specifications required for P. Suppose
that ϕ is not satisfied, that is MF (P) 6|= ϕ. The problem of modifying the original program
in order to make ϕ true is nontrivial.
In this section, we deal with this issue under particular assumptions about error types.
In particular, we address the cases in which errors are faulty assignments and disordered
successive assignments. We show that this problem has a direct mapping into the system
repair problem from the previous section. Determining a program repair and optimization
issues will be dealt with in subsequent sections.

4.1. Concurrent programs and FC-Kripke structures

A concurrent program P is composed of a finite collection P1 , . . . , Pn of processes

running in parallel. We refer to a model with shared memory; hence, all variables
x1 , x2 , . . . , xl in P, which we denote by x, are accessible to all processes.
According to Pnueli’s model [39], each process Pi can be represented by a directed
labeled graph G(Pi ) = hNi , Ei i, the graph of Pi , where Ni is the set of nodes and Ei is
the set of labeled directed edges.
The nodes Ni are an initial segment 1, 2, . . . , mi of the integers. They represent the
break points in the code of Pi , which are the points before the code and between successive
statements in ascending order. We denote by stmntP (i, b) the statement immediately after
break point b in the code of Pi ; if clear from the context, P and i are omitted. The code
between two subsequent break points is considered atomic in the parallel execution.
70 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

We suppose that the following types of statements are available:

(1) empty statement, denoted by ε;
(2) assignment statement: xi := g(x), where g(x) is an expression over variables in x
compatible with the type of xi ;
(3) jump statement: goto break_point;
(4) conditional statement: if c(x) then stmnt, where c(x) is a Boolean expression over
variables in x; and,
(5) compound statement: begin stmnt-1; . . . stmnt-n end.
In order to assure that each process Pi is nonterminating (that is usual in this framework),
we assume that the last statement of Pi is an unconditional jump. An infinite loop b : goto b
can easily be added at the end of a process.
The arcs Ei correspond to the possible execution flow of the program. From every
break point b, there is at least one arc leading to another break point, depending on the
type of stmnt(b). Moreover, each arc a is labeled with a command l(a), which is a pair
(c(x), stmnt) of a Boolean condition c(x) (the guard) and a statement stmnt, which is
either ε or an assignment.
• If stmnt(b) is ε or an assignment, an arc b → b + 1 is present, labeled with
(true, stmnt(b)).
• If stmnt(b) is a jump statement goto p, then an arc b → p is present, labeled with
(true, ε).
• If stmnt(b) is a conditional statement if c(x) then p: stmnt, where p is a break
point, then arcs b → p and b → q are present, where q is the first break point
after stmnt (note that p = b + 1). The labels of the arcs are l(b, p) = (c(x), ε) and
l(b, q) = (¬c(x)), ε), respectively.
• If stmnt(b) is a compound statement begin stmnt-1; b2 : stmnt-2; . . . bn : stmnt-n end,
then consider b1 : stmnt-1; . . . bn : stmnt-n, where stmnt-1 inherits its break point
b1 = b from stmnt(b).
Thus, more than one arc may leave from a node in the graph, reflecting the different
execution paths of a process. For convenience, we sometimes omit true and ε in commands;
in particular, () is (true, ε). An example of a process graph is shown in Fig. 3.

Remark. The above language for programs is elementary, but can be easily enriched by
further types of statements. For example, conditional statements with else-branches of
the form “if c(x) then stmnt-1 else stmnt-2” or while-loops “while c(x) do stmnt” may
be added. Such statements are straightforwardly translated into equivalent sequences of
elementary statements using jump-statements as usual. The graph of a process in such a
syntactically enriched language is then given by the graph of the transformed program.
Alternatively, the process graph could be defined directly, without resorting to a low
level transformation; however, the definition becomes more involved. In order to keep the
treatment simple, we refrain from explicitly considering an enriched syntax.
If the program contains also synchronization primitives such as semaphores, the
synchronization statement can be easily represented by using guards. For instance, if s
is a semaphore, the wait primitive P (s) corresponds to the command (s > 0, s := s − 1),
and the signal primitive V (s) corresponds to (true, s := s + 1).
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 71

Fig. 3. Graph G(PA ) for the process PA from Fig. 1.

Note that for this kind of statement, the disjunction between guards of all arcs leaving
a node, i.e., the exit condition from this node, is not necessarily true (different from the
previous statements); because of synchronization statements, a process can remain trapped
in a node until some condition is verified.

Now we provide a formal representation of the execution of a concurrent program.

Intuitively, the execution of a single process corresponds to a traversal of the process
graph driven by the result of the evaluation of the guards and involving all the actions
(assignments) defined in the commands labeling arcs. The concurrent execution can be
represented as the interleaved execution of all processes under the assumption that any
single command is atomic. If we want to faithfully model a possible interference between
the fetching and storing of operands, we may have to replace an assignment statement by a
chain of assignments [39]. For example, an assignment x := x1 or x2 , where the evaluation
of x1 or x2 should be not atomic, can be modeled by z1 := x1 , z2 := x2 ; x := z1 or z2 ,
where z1 and z2 are temporary variables and evaluating z1 or z2 is atomic (see [39] for a
detailed discussion of this topic).
Formally, a concurrent program P = P1 , . . . , Pn can be modeled using FC-Kripke
structures as follows. Let x = x1 , . . . , xp be the variables of P, which range over the
domains D1 , . . . , Dp , respectively. For each process Pi , let Ni = {1, 2, . . . , mi } be the set
of its break points.
Note that specifications for the program are expressed both by CTL formulas and by
fairness constraints.
In what follows, we assume that all variables xi are Boolean. Clearly, the general case
of finite domains Di can be modeled with Boolean variables. For each xi , an atomic
proposition xid for each value d ∈ Di can be used, corresponding to “xi = d”; alternatively,
dlog |Di |e many Boolean variables xi,j allow for a binary representation of the value of xi .
As observed in [11], automatic techniques of model checking can be applied only in the
case of small finite domains, due to the resulting size of the state space. For example,
in programs which encode concurrent protocols, variables usually represent control flags,
72 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

switches, and similar objects whose values are from small discrete domains. Hence, such
programs are suitable candidates for verification and repair as developed in the sequel.

Definition 4.1. Let P be a program. The FC-Kripke structure MF (P) = (A, S, S0 , R, L,

F ), (the Kripke structure of P, if F is understood), is as follows:
– Set of atomic propositions A. A = Vars ∪ B ∪ E, where:
• Vars = {x1 , . . . , xp } is the set of all variables in P.
• B = {bik | 1 6 i 6 n, 1 6 k 6 mi }. For each process Pi and possible break point
k ∈ {1, . . . , mi } of Pi , an atomic proposition bik exists, which is intuitively true if
Pi is currently at break point k.
• E = {e1 , . . . , en }. The atomic proposition ei tells whether process i was executed
in the latest execution step in the system.
– Set of states S. S = N1 × · · · × Nn × D1 × · · · × Dp × {1, . . . , n}. Hence, a state
is a tuple of break points (one for each process, i.e., a state of its execution flow),
an assignment to the program variables, and process number. As defined later, the
process number tells which process was executed in the last step before reaching this
state.
Given a state s ∈ S, we denote by b(s) the n-tuple of break points appearing in s, by
v(s) the p-tuple of values of variables appearing in s, and by ex(s) the last component
of s. Given a tuple t, we denote by ti the ith element of t. Hence, b(s)i is the break
point of Pi at s, and v(s)i is the value of variable xi at s.
– Set of initial states S0 . The set S0 contains only states s such that v(s)i = 1, for all i,
i.e., all processes are at their entry break points, and ex(s) = 1. (We arbitrarily choose
process number 1, since it is, by the meaning of ex(s), not relevant.) We assume that
the states in S0 are chosen according to a fixed policy which assigns the variables
initial values (e.g., value false, or a random value).
– Transition relation R. Intuitively, at state s1 , one of the processes Pk is enabled for
traversing the next arc in G(Pk ) in the computation.
A pair (s1 , s2 ) ∈ S × S belongs to R iff:
• there exists an arc a in G(Pex(s2 ) ) directed from break point b(s1 )ex(s2 ) to break
point b(s2 )ex(s2 ) ;
• b(s1 )i = b(s2 )i , for each 1 6 i 6 n, i 6= ex(s2 );
• for the label l(a) = (c(x), stmnt) of arc a, c(v(s1 )) is true and
(i) if stmnt = ε, then v(s1 ) = v(s2 );
(ii) if stmnt is an assignment xi := g(x), then v(s2 )j = v(s1 )j , for all j 6= i, and
v(s2 )i = g(v(s1 )); denote this by v(s2 ) = g(v(s1 )).
We say that (s1 , s2 ) executes the command (c(x), stmnt), and that s2 is reached from
s1 by executing stmnt(ex(s2 ), b(s1 )ex(s2 ) ); notice that for each Pk , a unique s2 is
reached from s1 by executing stmnt(k, b(s1 )k ). 5
– Label function L. The label function associates with every state s the set of atomic
propositions
L(s) = Lx (s) ∪ Lb (s) ∪ Lex (s),

5 If semaphores would be allowed, no such s might exist for P , which means that stmnt(m, k) is not
2 k
executable at s1 .
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 73

where Lx (s) = {xi ∈ x, | v(s)i = true}, Lb (s) = {biv(s)i ∈ B | 1 6 n}, and Lex (s) =
{eex(s2 ) } ⊆ E.
– Fairness constraints F . F is a set of fairness constraints.

Example 4.1. Let us reconsider program P in Fig. 1 and see how an FC-Kripke structure
MF (P) for it looks like.
The set of states is S = {1, . . . , 14}× {1, . . . , 14} ×{true, false}8 ×{1, 2}, since both PA
and PB have 14 break points, there are eight variables xi in x (x, y, flagiV and turniB,
where i = 1, 2 and V = A, B), and two process numbers (A = 1 and B = 2, say). Thus,
|S| = 142 · 28 · 2 = 100,352, i.e., there are roughly 105 states; among these, there are
28 = 256 potential initial states. If we assume that all program variables are false at the
beginning of a computation, there is a unique initial state.
The set A contains 8 + 2 · 14 + 2 = 38 atomic propositions: the eight variables of x,
the 28 variables bik , and two variables eA and eB .
The transition relation R and the label function L can be readily defined from this and
the process graphs for PA (cf. Fig. 3) and PB . For example, the pair (s1 , s2 ) is in R, where
s1 is such that v(s1 ) = (7, 8), all variables are false, and ex(s1 ) = B, and s2 is such that
v(s2 ) = (8, 8), all variables except turn2B are false, and ex(s2 ) = A. The labels of the states
s1 and s2 are L(s1 ) = {b17, b28 , eB } and L(s2 ) = {b18, b28 , turn2B, eA }, respectively.
Plausible fairness constraints for P are F = {eA , eB }, which guarantee fair scheduling.
(Recall that a path π satisfies a fairness constraint ϕ iff ϕ is true infinitely often on that
path. For instance, because of eA process A will be scheduled infinitely often.)
In the rest of our running example, we adopt this set of constraints for P.

4.2. The repair problem

In this section, we consider the problem of repairing a concurrent program P =

P1 , . . . , Pn . We assume that P with its variables x, a set of fairness constraints F (pure
state formulas), the Kripke structure MF (P), and a CTL formula ϕ on A(MF (P)) are
given.
The correctness of program P refers to MF (P), where the formulas in F encode
assumptions on the program execution; “unfair” computation paths are excluded.
Suppose a formula ϕ is a formal specification for P, and we have fairness constraints
F for P. Then, P fulfills ϕ iff MF (P) |= ϕ. If P does not fulfill ϕ, we are interested in a
change to the code of P such that the modified program P 0 fulfills ϕ. This amounts to a
(mostly nontrivial) SRP.

Example 4.2 (Continued). For P in Fig. 1, define

ϕ = ϕflags ∧ ϕcrit ,
where
^
ϕflags = AG(flagiV → AF¬flagiV),
i=1,2, V =A,B




ϕcrit = AG ¬(b112 ∧ b210) ∧ AG ¬ b15 ∧ (b25 ∨ b211) .
74 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Informally, ϕ says that in every computation, PV must eventually exit the critical section i
after entering it, and that the two processes cannot be simultaneously in a critical section.
(For example, AG(¬(b112 ∧ b210)) in ϕcrit requires that the processes A and B do not execute
the assignments to the variable y, namely instruction 12 of A and instruction 10 of B, at
the same time.)
For F = {eA , eB } (fair scheduling), MF (P) 6|= ϕ; this is not immediate.

Since we assumed that errors are present in terms of incorrect assignments or

assignments in wrong order, a solution of the program repair problem will be a sequence of
assignment modifications and assignment interchanges. An assignment modification may
affect each side of an assignment. In particular, it will either
• replace the right hand side of an assignment by a constant (true or false), or
• replace the variable on the left hand side of an assignment by another one.
Next, we will formally define the notion of program correction. To this end, we introduce
some preliminary notation.

Definition 4.2. For any assignment statement χ , we denote by var(χ) the variable of the
left hand side of χ and by expr(χ) the expression of the right hand side of χ .
An assignment χ is called simple if either expr(χ) = true or expr(χ) = false.

The next definition formalizes the notion of single program correction, which is an
atomic change to the program. A (general) program correction will be then obtained as
a sequence of single corrections (which are applied one by one in the specified order).

Definition 4.3. A single (program) correction for P is a 3-tuple hk, b, γ i, where k ∈

{1, . . . , n} is a process number, b ∈ Nk is a break point of the process Pk , and γ ∈
{swap, r-change} ∪ {l-change(xj ) | xj ∈ x}, is a modification such that the following holds:
• if γ = r-change, i.e., it is a right side modification, then stmnt(k, b) is a simple
assignment;
• if γ = l-change(xj ), for some variable xj ∈ x, i.e., a left side modification, then
stmnt(k, b) is an assignment with var(stmnt(k, b)) 6= xj ; and
• if γ = swap, i.e., an assignment interchange, then both stmnt(k, b) and stmnt(k, b + 1)
are assignments.
The modification of P by a single correction α = hk, b, γ i, denoted by P α , is the
concurrent program obtained from P by changing the code of process Pk in the following
way:
• if γ = r-change (i.e., it is a right side modification), the assignment stmntP (k, b) is
replaced by the assignment var(stmntP (k, b)) := not (expr(stmntP (k, b)));
• if γ = l-change(xj ) (i.e., it is a left side modification with variable xj ), the assignment
stmntP (k, b) is replaced by the assignment xj := expr(stmntP (k, b));
• if γ = swap (i.e., it is an interchange modification), the assignment stmntP (k, b) is
replaced by the assignment stmntP (k, b + 1) and the assignment stmntP (k, b + 1) is
replaced by the assignment stmntP (k, b).
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 75

Some remarks about the kinds of corrections we consider here are in order. In general,
a change on the right hand side of an assignment may involve a new variable or any
expression; in lack of any information about which of those changes are meaningful
in a particular context, and considering the in general tremendously large number of
functionally different expressions, we do not consider such changes here. However, our
framework could be extended to handle such modifications as well.
Furthermore, a disordering of two statements which are not assignments seems to be
a programming error which is less frequent in practice; moreover, considering respective
corrections is more involved and restricts the use of optimization techniques we develop
later. Therefore, we do not consider the interchange of arbitrary statements.

Example 4.3. The modification of the program P of Fig. 1, with the single program
correction α1 = h2, 12, l-change(flag1A)i is the program obtained from P by replacing
statement 12 of Process B by the assignment flag1A := false. The 3-tuple h1, 2, swapi is
not a correction for P, since statement 3 of PA is not an assignment.

A complex correction for P is a sequence of single corrections. The modification of P

by a complex correction is obtained by applying the single corrections in order.

Definition 4.4. Let α = α1 · · · αq be a sequence of single corrections for P. Then, the

modification P α of P by the complex correction α is recursively defined as the program

α
P α1 ···αq = P α1 ···αq−1 q , if q > 1,
and as P ∅ = P if q = 0. The length of the correction α, denoted length(α), is the number
q of single corrections in it.

Note that a modification of P by a single correction hk, b, γ i affects the code of process
Pk . It induces a change of the graph G(Pk ) and, as a consequence, of the FC-Kripke
structure associated with P. In case of an assignment correction, the only change in G(Pk )
is the label corresponding to the assignment stmnt(b). In case of an assignment interchange,
both labels corresponding to stmnt(b) and stmnt(b + 1) are changed. This merely affects
the transition relation R of MF (P). The following proposition is therefore easily derived
by an inductive argument.

Proposition 4.1. For any correction α, the FC-Kripke structures MF (P) and MF (P α )
coincide on A, S0 , S, L, and F .

(Recall that fairness constraints are fixed.) Since a correction α on P only affects R,
α can be viewed, according to Definition 3.2, as a modification of the system MF (P);
there exists a modification Γ for MF (P) such that R α = R Γ . On the other hand, given
a modification Γ of MF (P), in some cases R Γ can be obtained by a correction α and
considering the new transition relation R α .

Definition 4.5. A correction α for P induces a modification Γ ∈ mod(MF (P)), if R α =

RΓ .
76 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Let YP : mod(MF (P)) → {true, false} be the Boolean function such that for every
Γ ∈ mod(MF (P)), we have YP (Γ ) = true if there exists a correction α for P such that α
induces Γ , and YP (Γ ) = false otherwise.

Note that, by our assumptions, the function YP is clearly computable.

Next we define a repair problem P and show how it can be solved in terms of the
abductive solution of an SRP from the previous section.

Definition 4.6. A program repair problem (PRP) is a triple R = hP, F, ϕi, where P is a
concurrent program, F is a set of fairness constraints on A(M(P)), and ϕ is a formula on
A(M(P)).

A PRP R = hP, F, ϕi is also called the program repair problem for P with respect to ϕ
under F . The solution of a PRP can be given in terms of a solution to an SRP as follows.

Definition 4.7. Given a PRP R = hP, F, ϕi, a solution for R is a solution Γ for the SRP
S = hMF (P), ϕ, YP i. A repair for R is any correction α for P that induces a solution Γ
for R.

The following proposition is immediate from Definitions 3.4 and 4.7.

Proposition 4.2. For any PRP problem R = hP, F, ϕi, a program correction α is a repair
for R if and only if MF (P α ) |= ϕ.

Observe that in case P meets the specification, i.e., MF (P) |= ϕ holds, the empty
correction (q = 0 in Definition 4.4), which leaves P untouched, is a repair for R.
The process of finding an abductive solution is commonly guided by some rationality
principle which aims at pruning solutions that are less plausible. In particular, following
Occam’s principle of parsimony, solutions are pruned which are not free of redundancy,
i.e., if it is possible to remove hypotheses while preserving the solution property.
This criterion is often strengthened by further restricting the preferred solutions to those
which have a minimal cost (or, dually, a highest probability); if no cost information is
available, solutions containing a smallest number of hypotheses are often selected.
Following this lead, we adopt as rationality criterion the length of a complex correction.

Definition 4.8. Given a PRP R = hP, F, ϕi, a repair α for R is minimal, if there is no
repair β for R such that length(β) < length(α).

In particular, if P satisfies the specification ϕ, then the empty correction is the unique
minimal repair for R.
Of course, alternative notions of minimal repair could be acceptable. For example, if
we view the PRP as the underlying system repair problem, we could have accepted those
corrections α as minimal repairs such that the system modification Γ induced by α is
minimal in the sense of Definition 3.4. This would require, however, that the user is aware
of the formal representation of programs by means of FC-Kripke structures, in order to
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 77

interpret minimality of repairs in the right terms; moreover, statements in a program are the
atomic units of the user language, which suggests that repairs should be understood on the
basis of this language. And, finally, notice that at the semantical level, some complex repair
might be minimal, while it is not at the syntactical level (e.g., if it contains a sequence of
operations which cancel in their effects). Therefore, adopting the notion of minimal repair
as in Definition 4.8 is preferable in this particular application.

Example 4.4. Consider the PRP R = hP, F, ϕi, where P is the program of the running
example (Fig. 1) and ϕ, F are as in Example 4.2. As pointed out in Example 4.2,
MF (P) 6|= ϕ. It is possible to verify that, in this case, there is an error occurring in the
second statement of process PA . Indeed, it should read turn1B := true. Hence, the single
correction α = h1, 2, r-changei is a repair for D. Clearly, since length(α) = 1, it is also
minimal.

Regarding complex corrections, we note that the case of a single error in a program
is very relevant in practice. It is frequent and is usually examined first by human trouble
shooters as well as many diagnostic systems. And, many such systems are not capable of
handling complex errors at all. In the case of a single error, a minimal repair will be a
sequence α = α1 of length 1, since a single program correction will be sufficient to fix
the single error in the program. Hence, we focus with respect to practice on such minimal
repairs. Our results for optimization in finding a repair are tailored for this case; they hold,
suitably generalized, in the context of complex repairs, but are naturally weaker in effect.

5. Repairs and counterexamples

After having defined the concepts of repair problem and solution, the upcoming issue is
how to find a solution for a given problem; a suitable algorithm for this task is desired.
There is a simple brute force algorithm for solving the repair problem: check for each
possible correction α whether it is a repair, i.e., whether MF (P α ) |= ϕ, in a systematic
enumeration of all possible corrections until such a α is found; by respecting cancellation
effects of single corrections in complex corrections, the search space is finite.
Clearly, this algorithm is inefficient in general. Even if we restrict to single correction
solution candidates α = α1 , quite a number of different tests may have to be made until the
desired answer is obtained.
In order to reduce the number of cases that have to be considered, we develop
techniques which restrict the search space by exploiting counterexamples. Informally, a
counterexample for a PRP R = hP, F, ϕi is a portion of a branching computation tree
which witnesses that ϕ fails. Given a counterexample, our technique identifies corrections
α under which the counterexample is invariant, i.e., still apply if α is implemented. Such
α’s are useless as corrections and can be discarded. It happens that this way, the space of
candidate repairs may be drastically reduced.
Our concept of counterexample extends the one presented in [11] for the purpose of
(symbolic) model checking in the logic ACTL, which is a fragment of CTL. There, a
procedure for counterexample construction is outlined (see also [14]), which returns as
78 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

a result a single path in M. This path is in general not a counterexample per se, but
rather a heuristically selected path from a counterexample tree which gives some intuition
why the formula fails. The main reason for restricting the return value to a single path is
that understanding counterexample trees (and to represent them graphically) seems to be
difficult. 6
As shown below, there are simple formulas in ACTL for which no single path is
a counterexample. This is due to the possible presence of nested path quantifiers and
disjunction in a formula ϕ. A single path from the counterexample tree, as returned by
a model checking or symbolic model checking procedure, or even the full counterexample
tree represented in the customary way might not be much instructive why a formula
fails. The reason is that important structural information about the paths in a simple tree
representation is missing, namely how they are nested.
For our purposes, a full counterexample tree is needed, because otherwise a repair
of the program may not be found. In order to overcome the representational problem
with counterexample trees, we introduce the concept of a multi-path, which allows for
a structured representation of paths. Counterexamples are then particular multi-paths.
Our formalization of counterexamples seems to be an appropriate extension of path
counterexamples as described in [11]. In fact, while an ACTL formula may lack a path
counterexample, there always exists a multi-path counterexample if it fails on a Kripke
structure.
In the remainder of this paper, we restrict our attention to the fragment ACTL of CTL.
In this fragment, only universal path quantifiers are allowed, and negation is restricted
to pure state formulas. 7 For instance, the specification ϕ in our running example (see
Example 4.2) is an ACTL formula; however, the formula ψ = E F(turn1B ∧ ¬turn2B),
which states that there is some computation such that turn1B is true and turn2B false at
some point, is not an ACTL formula. Notice that ACTL is considered to be an important and
highly relevant fragment of CTL, as it allows for abstraction and compositional reasoning
[13,26]. Moreover, we assume in the following that fairness constraints in FC-Kripke
structures are pure state formulas.

5.1. Counterexamples

Informally, a multi-path represents an infinite tree T , by using a vertical axis rather than
the usual recursion from a node to its descendants. The branches of T are infinite paths; the
axis is a distinguished main path of the tree, from which other paths spring off. These paths
are main paths of subtrees of T . This view gives rise to an inductive definition of multi-
paths. The main advantage of this concept is preservation of the nesting of paths, which is
lost in the usual tree definition. Moreover, for a class of multi-paths which is sufficient for
our purposes, effective finite representations exist.

6 K. McMillan, personal communication.

7 Some authors restrict negation to atoms, which yields formulas in negation normal form (NNF) [26]. While
semantically equivalent, the syntactically larger class is more convenient. Moreover, conversion into NNF is
simple.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 79

Preliminary to the formal definition of multi-paths, we introduce multi-sequences.

Definition 5.1. Let S be the set of states. Then,

• for every state s ∈ S, Π = s is a finite multi-sequence in S;
• if Π0 , Π1 , . . . are countably infinite many multi-sequences in S, then Π = [Π0 , Π1 ,
. . .] is a multi-sequence in S.
For any multi-sequence Π , its ith element is denoted by Π(i), for all i > 0; moreover,
its depth is

0, if Π = s,
d(Π) =
sup d(Π(i)) + 1, otherwise,
i>0

and its origin is


Π(0), if Π = s,
or(Π) =
or(Π(0)), otherwise.

Next we introduce the notion of main sequence of a multi-sequence. Informally, it is the

sequence formed by the origins of all elements in a multi-sequence.

Definition 5.2. Given a multi-sequence Π , the main sequence of Π , denoted by µ(Π), is

• s, if Π = s is finite;
• the sequence [or(Π(0)), or(Π(1)), or(Π(2)), . . .], otherwise.

Multi-paths are multi-sequences which model nested paths in M.

Definition 5.3. A multi-sequence Π is a multi-path in M, if either Π is finite, or µ(Π) is

a path in M and for every i > 0, Π(i) is a multi-path in M. A multi-path Π is fair, if Π is
finite or if Π is infinite and µ(Π) is a fair path and every Π(i) i > 0, is a fair multi-path.
The main sequence of a multi-path Π is called the main path of Π .

Here, a single state is considered as a fair multi-path, which turns out to be technically
convenient later.
Note that multi-paths generalize paths. Indeed, a path can be seen as an infinite multi-
path Π such that each element Π(i) is a state. Fairness of paths is generalized accordingly.
An infinite multi-path Π represents intuitively an evolving computing tree, whose
branches are the main path µ(Π) and all paths of form π0 π1 where π0 = µ(Π)(0), . . . ,
µ(Π)(i − 1) is a finite prefix of µ(Π) and π1 is a branch of the multi-path Π(i), where
Π(i) must be infinite.

Example 5.1. Assuming proper M, the multi-sequence Π = [[s0 , s1 , s1 , . . .], s2 , s2 , . . .]

is a multi-path, which represents two paths π1 = [s0 , s1 , s1 , . . .] and π2 = [s0 , s2 , s2 , . . .]
starting at s0 (Fig. 4). π2 is the main path µ(Π) of Π . The multi-path
 
Π = [s0 , s1 , s1 , . . .], s2 , [s0 , s1 , s1 , . . .], s2 , [s0 , s1 , s1 , . . .], . . .
80 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Fig. 4. Branching paths.

has main path µ(Π) = [s0 , s2 , s0 , s2 , . . .] and represents the computation tree in which
from µ(Π) at every even state number a path [s0 , s1 , s1 , . . .] branches off; hence, Π
contains besides µ(Π) all paths of form [(s0 , s2 )i , s0 , s1 , s1 , . . .].

An important note is that in general, a multi-path Π may not directly reflect in its
structure a truly branching computation tree. In fact, the definition allows fake branching,
in the sense that two nested branching paths may amount to the same path in the structure.
For example, in the multi-path
 
Π = s0 , s1 , [s2 , s3 , s4 , . . .], s3 , s4 , . . . ,

the branch s2 , s3 , s4 , . . . is identical to the remainder of the main path s2 , s3 , s4 , . . . . This

is not a shortcoming of our definition, but an important feature; it allows to express that a
particular path is a subpath of another one. In an extended vocabulary for multi-paths, this
could be expressed more elegantly; however, we disregard such an extension here.
We are now prepared to formalize the notion of counterexample. Intuitively, a
counterexample for a formula ϕ is a multi-path Π originating at an initial state such that
ϕ is not true along Π . Since counterexamples are defined inductively, we need the concept
of a local counterexample, which may origin at an arbitrary state rather than only at an
initial state. For the technical definition of local counterexamples, we use an operation for
merging two multi-paths into a single one.

Definition 5.4. Let Π1 and Π2 be two multi-paths such that or(Π1 ) = or(Π2 ). The merge
of Π1 and Π2 , denoted by Π1 ∗ Π2 , is the multi-path recursively defined as follows:


 Π1 , if Π2 is finite;



[Π1 , Π2 (1), Π2 (2), . . .], if Π2 is infinite
Π1 ∗ Π2 =

 and Π2 (0) is finite;


[Π ∗ Π (0), Π (1), Π (2), . . .], otherwise.
1 2 2 2
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 81

Intuitively, the trees represented by Π1 and Π2 are merged at their common root.

Example 5.2. Merging Π = [[s0 , s11 , s12 , . . .], s21 , s23 , . . .] and Π 0 = [s0 , s31 , s32 , . . .]
yields

Π ∗ Π 0 = [Π, s31 , s32 , . . .]

 
= [[s0 , s11 , s12 , . . .], s21 , s22 , . . .], s31 , s32 , . . . ,
while
 
Π 0 ∗ Π = Π 0 ∗ [s0 , s11 , s12 , . . .], s21 , s22 , . . .
 
= [Π 0 , s11 , s12 , . . .], s21 , s22 , . . .
 
= [[s0 , s31 , s32 , . . .], s11 , s12 , . . .], s21 , s22 , . . . .
These merges essentially represent the same branching of three paths πi = [s0 , si1 , si2 , . . .]
for i = 1, 2, 3, starting from s0 .

Note that merging Π1 and Π2 by adding Π1 as first element to Π2 does not work, since
in general, this leads to a set of paths different from those in Π1 and Π2 ; the result may
even not be a multi-path.

Definition 5.5. Let M be an FC-Kripke structure and ϕ be an ACTL formula on A(M).

A multi-path Π in M is a local (l-) counterexample for ϕ if, depending on the structure of
ϕ, the following holds:
• if ϕ is a pure state formula: Π = s is a state and M, s 6|= ϕ;
• otherwise, if
(1) ϕ = A(ϕ1 Uϕ2 ): Π is an infinite fair multi-path and either
(1.1) there exists k > 0 such that Π(k) is an l-counterexample for ϕ1 ∨ ϕ2 , Π(i)
is an l-counterexample for ϕ2 , for each 0 6 i 6 k, and Π(j ) is a state, for
j > k; or
(1.2) Π(i) is an l-counterexample for ϕ2 , for each i > 0;
(2) ϕ = A(ϕ1 Vϕ2 ): Π is an infinite fair multi-path and there exists a k such that every
Π(j ), 0 6 j < k, is an l-counterexample for ϕ1 , Π(k) is an l-counterexample for
ϕ2 , and every Π(`) is a state, for ` > k;
(3) ϕ = AXϕ1 : Π is an infinite fair multi-path, Π(1) is an l-counterexample for ϕ1 ,
and Π(i) is a state, for each i 6= 1;
(4) ϕ = ϕ1 ∨ ϕ2 : Π = Π1 ∗ Π2 , where Πi , i = 1, 2, is an l-counterexample for ϕi ;
(5) ϕ = ϕ1 ∧ ϕ2 : Π is an l-counterexample for either ϕ1 or ϕ2 .

Recall that M 6|= ϕ if there exists an initial state s0 at which ϕ is false. Hence, we
introduce a notion of “global” counterexample.

Definition 5.6. Let M be an FC-Kripke structure and ϕ be a formula on A(M). Any

l-counterexample Π for ϕ in M such that or(Π) ∈ S0 (M) is called a counterexample for
ϕ in M.
82 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Fig. 5. Labeled transition graph.

Let us consider some examples. (A more involved example appears in Section 6.4.)

Example 5.3. Let M be the FC-Kripke structure that amounts to the labeled transition
graph in Fig. 5, where s0 is the unique initial state, A = {a1 , a2 }, and F = ∅, and consider
the formula ϕ = AFa1 .
It holds that M 6|= ϕ: Along the path π = [s0 , s1 , s1 , . . .], the atom a1 is false at each
state, which means M, π i |= ¬a1 , for every i > 0, i.e., M, π |= G¬a1 . Thus, π witnesses
the failure of ϕ in M. As easily checked, π is a counterexample of ϕ.
Consider next the formula ψ = AGAFa1 . Also this formula is false on M. Intuitively,
this is witnessed by path π again. However, from the formal definition, π is not a
counterexample of ψ, as it does not respect witness paths for the subformula AFa1 of ψ.
The multi-path Π = [[s0 , s1 , . . .], s1 , s1 , . . .] is a proper counterexample for ψ according
to the definition, as well as any multi-path [s0 , (s1 , )i , . . . , [s1 , s1 , . . .], s1 , s1 , . . .], where
i > 0.
Finally, also the formula ρ = AFAGa1 is false in M, and again the path π = [s0 , s1 , s1 ,
. . .] shows this. Formally, the multi-path [[s0 , s1 , s1 , . . .], [s1 , s1 , . . .], [s1 , s1 , . . .], . . .] is a
counterexample for ρ; in fact, it is the unique counterexample.

As mentioned above, in many cases a counterexample for a formula is (essentially) a

single path. However, there are cases in which a true computation tree is required.

Example 5.4. Consider the FC-structure M from Example 5.3, but now the formula
ϕ = AFa1 ∨ AFa2 .
Clearly, M 6|= ϕ: For every ai , i = 1, 2, there is an infinite path πi = s0 , si , si , . . .
which never reaches a state at which ai is true; hence, every disjunct AFai in ϕ is false.
A counterexample for ϕ is the multi-path Π = [[s0 , s1 , s1 , . . .], s2 , s2 , . . .], which results
by merging the πi ’s into Π = (π1 ∗ π2 ). Notice that no counterexample for ϕ exists that is
an ordinary path, and that π1 ∗ π2 , π2 ∗ π1 are the only (isomorphic) counterexamples.
In this spirit, examples of concurrent programs P can be given which do not satisfy
ϕ = AFa1 ∨ AFa2 and such that no single path is a counterexample.

Example 5.5. Let M be the FC-Kripke structure that amounts to the labeled transition
graph in Fig. 6, where s0 is the unique initial state and F = ∅. Consider the formula
ϕ = AFAGa. It is easy to verify that M 6|= ϕ. Indeed, there is a path π = [s0 , s0 , . . .]
starting from the initial state where always the nested formula AGa does not hold, as, for
each i > 0, there exists a path starting at π(i) where sometimes a is not true (e.g., on the
path [s0 , s1 , s2 , s2 , . . .] a is never true).
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 83

Fig. 6. Another labeled transition graph.

The multi-path
 
Π = [s0 , s1 , s2 , s2 , . . .], [s0 , s1 , s2 , s2 , . . .], . . .
is a counterexample for ϕ.
Note that no single path is a counterexample for ϕ. Indeed, for each i > 0, the suffix π i
of a counterexample path π for ϕ must be a witness of ¬AGa = EF¬a. Thus, s1 must
occur in π i . Now, every (single) path in M has a suffix cycle which involves either s0 or
s2 . Therefore, for each path π there exists k > 0 such that either π(i) = s0 for each i > k,
or π(i) = s2 for each i > k. Because a is true in both s0 and s2 , no path can witness the
falsity of AGa.

Besides the above very simple examples, many other cases can be found in which any
counterexample is a truly branching computation tree. They include formulas of form
AF(AGϕ ∨ AG¬ϕ), which informally state that any computation must commit at some
point about a condition ϕ, and AF(ϕ ∨ AGψ), which state that at some stage ϕ is true or,
regardless of how the computation proceeds, ψ holds.
Our next result states that the existence of an l-counterexample for a formula ϕ at a state
s in an FC-Kripke structure M implies that ϕ is not true at s in M and vice versa. As a
consequence, ϕ is not true in M if and only if there exists a counterexample for ϕ. In fact,
a counterexample has as its origin an initial state of the FC-Kripke structure.

Theorem 5.1. Let M be an FC-Kripke structure, ϕ a formula on A(M), and s ∈ S(M).

Then, M, s 6|= ϕ if and only if there exists an l-counterexample Π for ϕ such that or(Π) =
s.

An immediate consequence of the previous proposition is the following.

Corollary 5.2. For any FC-Kripke structure M and formula ϕ on A(M), M 6|= ϕ if and
only if there exists at least one counterexample Π for ϕ in M.

5.2. Representation and computation of counterexamples

Counterexamples are infinite objects, and therefore it is not a priori clear that there is
a finite representation for them. Fortunately, it is possible to finitely represent a relevant
class of multi-path counterexamples that is sufficient for our purposes, in a similar way as
a relevant class of path counterexamples.
If a path counterexample exists for an ACTL formula ϕ, then a path counterexample π
exists for ϕ which is finitely representable [14]. In particular, it is easy to see that if a path
counterexample exists for an ACTL formula ϕ, then a path counterexample π exists for ϕ
84 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

which consists of a finite prefix π(0), . . . , π(k − 1) and a finite cycle π(k), . . . , π(k + m),
such that π(`) = π(k + fk,m (`)) for all ` > k, where fk,m (`) = (` − k) mod m [14].
Each fairness constraint must be satisfied at least once on this cycle. Thus, such a path
counterexample can be represented as a simple expression of the form Prefix(Cycle)∞
over the alphabet of all states.
For example, the path π = [s0 , s2 , s2 , s2 , . . .], which loops at state s2 , can be represented
by the expression [s0 , (s2 , )∞ ], and the path
σ = [s0 , s1 , s0 , s1 , s3 , s2 , s0 , s1 , s2 , s0 , s1 , . . .],
which loops between s2 , s0 , and s1 , can be represented by [s0 , s1 , s0 , s1 , s3 , (s2 , s0 , s1 , )∞ ]
(if we omit the redundant square brackets and commas, simply by s0 s1 s0 s1 s3 (s2 s0 s1 )∞ ).
Similarly, if a multi-path counterexample exists for an ACTL formula ϕ, then a multi-
path counterexample π exists for ϕ which is finitely representable. To see this, first
note that each counterexample is, from its definition, a multi-path whose depth is finite.
Using an inductive argument, it can be seen that if a counterexample of certain depth
d exists, then a counterexample Π of depth less or equal d exists which consists of a
finite prefix Π(0), . . . , Π(k − 1) and a (finite) cycle Π(k), . . . , Π(k + m) = Π(k) such
that Π(`) = Π(k + fk,m (`)) for all ` > k, where again fk,m (`) = (` − k) mod m. This
counterexample can be represented as an expression of form Prefix(Cycle)∞ over the
alphabet of states and counterexamples of depth smaller d; thus, if we bottom out the
representation, it can be represented as a nested expression built over an alphabet of states
and paths (or even states alone, if we wish).
For example, the counterexample Π = [[s0 , s1 , s1 , . . .], s2 , s2 , . . .] in Example 5.4 can
be represented by [s0 (s1 )∞ , (s2 , )∞ ], and the counterexample
 
Π = [s0 , s1 , s2 , s2 , . . .], [s0 , s1 , s2 , s2 , . . .], . . .
in Example 5.5 by [(s0 s1 (s2 )∞ , )∞ ] (here s0 (s1 )∞ and s0 s1 (s2 )∞ are the simplified path
expressions [s0 , (s1 , )∞ ] and [s0 , s1 , (s2 , )∞ ], respectively).
We face now the problem of computing counterexamples and provide a method, based
on a procedure of [14], for constructing a counterexample. Briefly, given an ACTL formula
ϕ and a state s (which is usually an initial state of the FC-Kripke structure), the procedure
described in [14] produces a single path, starting from s, which demonstrates or hints at
the failure of ϕ. In our framework, for most ACTL formulas this single path corresponds
to the main path of a counterexample; the structure of a counterexample itself depends on
nestings of A in ϕ. For instance, if ϕ is a pure state formula, then a counterexample for
ϕ in M is simply an initial state; a counterexample for a formula AGϕ is a path; and, a
counterexample for a formula AG(ϕ1 ∨ AGϕ2 ) consists of two nested paths: the first starts
from an initial state, and the second branches off at a certain point of the first.
Therefore, if the ACTL formula ϕ has only one (universal) quantifier, then it admits path
counterexamples that can be correctly (and efficiently) computed by using the procedure
designed in [14]. 8

8 Note that this procedure is applicable here, as the Kripke structure of a program has always a total transition
relation by construction.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 85

Formulas with more than one universal quantifier do not admit, in general, path
counterexamples (see Examples 5.4 and 5.5) and the procedure of [14] cannot be
employed to compute counterexamples of such formulas (as it computes simple paths;
while branching multi-paths are required). A suitable extension of the algorithm of [14] is
needed for computing multi-path counterexamples in the general case.
Examples 5.4 and 5.5 show the two basic sources that imply the need of multi-paths
in counterexamples: disjunction and nested quantifiers. Multi-path counterexamples of a
formula consisting of the disjunction of atomic (i.e., with one quantifier) ACTL formulas,
say ϕ = AFa1 ∨ AFa2 , can be computed very easily:
(i) compute an initial state, say s0 , on which ϕ = AFa1 ∨ AFa2 is false (e.g., by
computing the suitable fixpoint for the negation of ϕ, i.e., EG¬a1 ∧ EG¬a2 );
(ii) call the procedure of [14] on AFa1 and on AFa2 with s0 as the initial state;
(iii) create the multi-path counterexample for ϕ by merging the two path counterexam-
ples for AFa1 and AFa2 , respectively, returned by the procedure.
The computation of multi-path counterexamples of ACTL formulas with nested quantifiers
is harder and requires a recursive extension of the procedure of [14]. A naive way to make
this extension is to recursively call (top-down on the structure of the formula) the procedure
of [14] to build the counterexamples of the nested ACTL subformulas. For instance, a
(multi-path) counterexample of the formula ϕ = AFAGa of Example 5.5 can be computed
as follows. Call first the procedure of [14] on ϕ; this procedure returns a fair path π such
that, for each integer i, π(i) is a state where EF¬a is true (i.e., AGa is false). Then,
for each integer i, call the procedure again to compute a path counterexample, for AGa
starting from π(i) (i.e., with π(i) as the initial state), say πi . The multi-path Π such that
Π(i) = πi is a multi-path counterexample for ϕ = AFAGa. 9 Other nested ACTL formulas
can be treated in a similar way.
It is worth noting that several optimizations are possible in the computation of
counterexamples. Relevant optimizations can be designed by singling out cases of nested
ACTL formulas where paths are sufficient to witness the falsity. For instance, AFAFψ,
AGAFψ, AXAGψ, AGAGψ (which is equivalent to AGψ), where ψ is a propositional
formula, admit path counterexamples. A detailed analysis of the (nested) formulas that
admit path counterexamples leads beyond this paper, and is carried out elsewhere [7].

6. Optimization techniques for repair

In the previous sections, we have introduced the problem of program repair and the
concept of counterexample. In this section, we present some optimization techniques which
allow to cut the search space for a repair. The techniques utilize counterexamples from
above, and are most effective in the case of single correction repairs. An important aspect
is that they are efficiently applicable; namely, given a PRP, a counterexample γ , and a
single correction α, a sound test whether whether α can be discarded as a repair can done
in linear time in the size of the input.

9 Note that the path π returned by [14] is finitely represented as a prefix and cycle (see above). Consequently,
a finite number of calls to the procedure is sufficient, and the multi-path Π is finitely represented.
86 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

The basic observation underlying our techniques is that a counterexample must contain
certain transitions which prove the failure of a formula ϕ. Any repair must remove these
transitions, i.e., avoid that such transitions take place. Thus, if a correction leaves these
transitions unchanged, then it can not amount to a repair. By determining such corrections
from a counterexample at hand, a number of useless corrections might be excluded.
Before we present the particular techniques that we have developed in this spirit, we
have to introduce a suitable notion of equivalence between multi-paths. In the rest of this
section, P = P1 , . . . , Pn is a concurrent program on variables x having a set F of pure
state formulas as fairness constraints, and ϕ is an ACTL formula.

6.1. Equivalent multi-paths and weak corrections

Definition 6.1. For any formula ϕ, we denote by S

ap(ϕ) the set of atomic propositions in
ϕ. For any set F of formulas, we denote ap(F ) = ϕ∈F ap(ϕ).

Definition 6.2. Let M and M 0 be two FC-Kripke structures such that F (M) = F (M 0 ), and
let A0 ⊆ A(M) ∩ A(M 0 ). Then,
(i) states s ∈ S(M) and s 0 ∈ s(M 0 ) are equivalent on A0 , denoted s ≈A0 s 0 , if

L(M)(s) ∩ A0 = L(M 0 )(s 0 ) ∩ A0 ;

(ii) multi-paths Π in M and Π 0 in M 0 , are equivalent on A0 , denoted Π ≈A0 Π 0 , if

either
(ii.1) both Π = s and Π 0 = s 0 are states and s ≈A0 s 0 , or
(ii.2) both Π and Π 0 are infinite and Π(i) ≈A0 Π 0 (i), for every i > 0.

Intuitively, if Π and Π 0 are equivalent on A0 then, for every formula ϕ with ap(ϕ) ⊆ A0 ,
we have that ϕ is true on Π if and only if ϕ is true on Π 0 .
The next proposition is the basis of later results. It states a transfer results for
counterexamples between structures: Given a counterexample Π for a formula ϕ in M,
any multi-path in a structure M 0 that is equivalent to Π on the variables of ϕ is a
counterexample for ϕ in M 0 . As a consequence, if M 0 is the result of a repair to the FC-
Kripke structure of a program M with respect to ϕ, it is impossible to find in M 0 a “pattern
of behavior” equivalent on ap(ϕ) to any counterexample in M. This is captured formally
by the next propositions.

Proposition 6.1. Let M and M 0 be two FC-Kripke structures which coincide on S, S0 , L,

and F . Let Π be an l-counterexample in M for a formula ϕ on ap(ϕ) ⊆ A(M). If Π 0 is a
fair multi-path in M 0 such that Π ≈ap(ϕ) Π 0 , then Π 0 is an l-counterexample for ϕ in M 0 .

We next introduce the concept of a weak correction for a counterexample, that singles
out corrections that are certainly useless to fix the error (i.e., modifying the concurrent
program by any set of weak corrections would not allow to entail the CTL formula at hand).
Prior to this, we fix a notation for the variables which are touched by a single correction.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 87

Definition 6.3. For any single correction α = hk, b, γ i for P, V (α) is the following set of
variables:



{xj , var(stmnt(k, b))}, if γ = l-change(xj );
V (α) = {var(stmnt(k, b))}, if γ = r-change;


{var(stmnt(k, b)), var(stmnt(k, b + 1))}, if γ = swap.
Sq
For any complex correction α = α1 · · · αq , let V (α) = i=1 V (αi ).

Definition 6.4. Let Π be a fair multi-path in MF (P) such that or(Π) ∈ S0 . A correction α
for P is called weak with respect to Π , if there exists a fair multi-path Π 0 in M 0 = MF (P α )
such that or(Π 0 ) ∈ S0 and Π ≈A0 Π 0 for A0 = A(M) − V (α).

Intuitively, a weak correction for Π does not modify the set of formulas holding on Π
(if we see Π 0 as the “image” of Π under the correction α), apart from formulas involving
propositions in V (α) (i.e., that are explicitly modified by α).
The next proposition states that a program repair not involving variables from ϕ
cannot be a weak correction with respect to any counterexample of ϕ. In fact, if a
repair is a weak correction for some counterexample, the modified program will produce
the same counterexample for the formula in the modified system. Roughly speaking,
counterexamples have to be modified by a correction if we hope to repair the system
through it. This result will be utilized later.

Proposition 6.2. Let α be a repair for the PRP R = hP, F, ϕi, such that
V (α) ∩ ap(ϕ) = ∅.
Then, for every counterexample Π for ϕ in MF (P), α is not weak with respect to Π .

Proof. Towards a contradiction, suppose α is a weak correction with respect to Π .

Hence, there exists a fair multi-path Π 0 in M 0 = MF (P α ) such that Π ≈A0 Π 0 where
A0 = A \ V (α). By hypothesis on ϕ, ap(ϕ) ⊆ A0 , and thus Proposition 6.1 implies that Π 0
is a counterexample for ϕ in M 0 . Hence, by Corollary 5.2, α is not a repair for R, which is
a contradiction. 2

The consequences of this proposition will be exploited in Section 6.3 for optimization
issues (where the result is used to prove the main theorem). Intuitively, if we know that
a correction is weak with respect to a counterexample, then we can discard it from the
“candidates” repair, unless it touches explicitly the atomic propositions in ap(ϕ).

6.2. Correction execution

The first optimization method respects whether the counterexample passes through a
statement that has been modified; if not, then it is still valid, and the respective correction
is ruled out as a repair.
Informally, given a counterexample Π and a single correction α = hk, m, γ i, we say that
Π executes α if there exists some path in Π along which process Pk passes through the
88 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

break points affected by α. Here, we must take into account that a computation path may
pass only through part of them, due to jumps; it could pass break point m but not m + 1.
We next prove formally that if a correction is a repair, then it must be executed by any
counterexample. For that, we must formalize correction execution.

Definition 6.5. Let α = hk, m, γ i be a correction. A path π executes α, if there exists i > 0
such that ex(π(i + 1)) = k (i.e., the transition from i to i + 1 executes Pk ) and, further
• γ ∈ {r-change, l-change(xj )} and b(π(i))k = m, or
• γ = swap and b(π(i))k ∈ {m, m + 1}.
A multi-path Π executes α, if Π is infinite and, further, either the main path µ(Π) of
Π executes α, or there exists an integer i > 0 such that Π(i) executes α.
A path π (respectively, multi-path Π ) executes a correction α = α1 · · · αq , if it executes
αi , for some i = 1, . . . , q.

We note a couple of simple lemmas, which are useful in the proof of the next result.

Lemma 6.3. Let α = hk, m, γ i be a single correction, and let (s1 , s2 ) ∈ R. If

(i) γ ∈ {r-change, l-change(xj )} and b(s1 )k 6= m, or
(ii) γ = swap and b(s1 )k ∈
/ {m, m + 1}, then (s1 , s2 ) ∈ R α .

Proof. Observe that the effect of changes to G(Pk ) by α is restricted to labels of arcs
which are leaving m in case (i) and leaving m or m + 1 in case (ii). 2

Lemma 6.4. Let α be a correction for P, and let π be a path in MF (P) which does not
execute α. Then π is also a path in MF (P α ).

Proof. If π does not execute α = α1 · · · αq , then π does not execute any of α1 , . . . , αq .

Thus, by Lemma 6.3, we obtain that π is a path in P α1 . By a repeated argument, we obtain
that π is a path in P α1 ···αi , for every i = 1, . . . , q. 2

Lemma 6.5. Let R = hP, F, ϕi be a PRP for which some repair exists. Then, every
counterexample Π for ϕ in MF (P) is infinite.

The next theorem states that, given a PRP R and a repair α for it, every counterexample
executes α.

Theorem 6.6. Let α be a repair for the PRP R = hP, F, ϕi. Then, every counterexample
Π for ϕ in MF (P) is infinite and executes α.

This result is intuitive; if a counterexample does not touch the statements which have
been corrected, the same multi-path will be present in the Kripke structure of the corrected
program. As a consequence of Theorem 6.6, while looking for possible repairs, we can rule
out a priori any correction that is not executed on a counterexample.
An important observation is that correction execution can be tested efficiently. In fact,
this is possible in linear time if counterexamples are properly represented, e.g., in the
scheme of Section 5.2.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 89

Proposition 6.7. Given a PRP R = hP, F, ϕi, a collection C of complex corrections

α = α1 · · · αq , and a counterexample Π for ϕ, the corrections in C which are not executed
by Π can be discarded in time O(kPk + kCk + kΠk), i.e., in linear time.

Thus, checking for correction execution is a low cost pruning principle which can benefit
in reduction of the search space.

6.3. Correction exploitation

We can identify another important property that repairs must verify, and exploit it for
further optimization.
Recall that a correction α involves a set of variables, denoted by V (α). Then, only
computations that evaluate some variable in V (α) can be influenced by the correction α.
Indeed, the values of variables not in V (α) and guards of labels in the process graphs
can change only if variables whose values are affected by the correction α are referenced.
A path that does not evaluate variables of V (α) will be transformed by the correction into
an equivalent path on A \V (α); the same happens to a multi-path. Hence, a further property
that a repair α must satisfy is that all counterexamples have to evaluate some variable in
V (α). We formalize this intuition next.

Definition 6.6. A path π exploits a correction α = hk, m, γ i, if there exists an integer i > 0
such that the command executed in (π(i), π(i + 1)) evaluates 10 some variable in V (α).
A multi-path Π exploits a correction α = hk, m, γ i, if Π is infinite and, furthermore,
either µ(Π) exploits α or there exists an integer i > 0 such that Π(i) exploits α.
A path π (respectively, multi-path Π ) exploits a correction α = α1 · · · αq , if it exploits
αi for some i = 1, . . . , q.

The next result states that under certain conditions, any counterexample must exploit a
given repair. It is based on the fact that if a path π does not exploit a single correction α,
then we can find a path π 0 in the modified structure which is modulo V (α) equivalent to
π and starts from any given state equivalent to the initial state of π (see Lemma A.1 in
Appendix A).

Theorem 6.8. Let α be a repair for the PRP R = hP, F, ϕi such that V (α) ∩ (ap(ϕ) ∪
ap(F )) = ∅. Then, every counterexample Π for ϕ in MF (P) exploits α.

Notice that the condition V (α) ∩ ap(F ) = ∅ is not much restrictive, since F should
be concerned about fair executions (and speak about processes and break points) rather
than the value of program variables, which should be done in the specification ϕ. On the
other hand, the condition V (α) ∩ ap(ϕ) = ∅ is more restrictive, but cannot be removed in
general. For example, if ϕ says that some variable ok must always be committed to true

10 Note that an assignment statement evaluates the variables appearing in its right hand side; a conditional
statement evaluates the variables appearing in its if condition.
90 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

in a computation (AFAGok), a correction of ok := false to ok := true might eliminate all

counterexamples, even if no computation references the value of ok.
An important point is that like correction execution, correction exploitation is a pruning
principle which can be applied efficiently. In fact, in the setting of Section 6.2, it is possible
to implement correction exploitation in linear time.

Proposition 6.9. Given a PRP R = hP, F, ϕi, a collection C of complex corrections

α = α1 · · · αq such that V (α) ∩ (ap(ϕ) ∪ ap(F )) = ∅, and a counterexample Π for ϕ, the
corrections in C which are not exploited by Π can be discarded in time O(kPk + kCk +
kΠk), i.e., in linear time.

6.4. Example for optimization

Let us consider an example which demonstrates that by the optimization techniques in

Sections 6.2 and 6.3, quite some savings can be gained in finding a repair.

Example 6.1. Reconsider the concurrent program P in Fig. 1, and let ϕ and F be as in
Example 4.2, viz. ϕ = ϕflags ∧ ϕcrit , where
^
ϕflags = AG(flagiV → AF¬flagiV)
i=1,2, V =A,B

and




ϕcrit = AG ¬(b112 ∧ b210) ∧ AG ¬ b15 ∧ (b25 ∨ b211) ,
and F = {eA , eB }. As already mentioned, the program is not correct. Therefore, MF (P) 6|=
ϕ must hold and a counterexample for ϕ must exist.
It can be verified that indeed MF (P) 6|= ϕ: there is a path π from an initial state, leading
to a state at which flag1B = true, and where another path π 0 starts along which flag1B
is always true. Thus, the formula EF(flag1B ∧ EGflag1B) is true, which means that the
formula AG(flag1B → AF¬flag1B) is false; since the latter is a conjunct of ϕ, also ϕ is
false. This should give rise to a counterexample.
Indeed, consider the following path π (we show of each state si , left to right, the break
points for PA and PB , the program variables that are true, and the process lastly executed;
in the initial state s0 , all variables are false):

π(0) = s0 = 1 1 PA
π(1) = s1 = 2 1 flag1A PA
π(2) = s2 = 2 2 flag1A, flag1B PB
π(3) = s3 = 3 2 flag1A, flag1B PA
π(4) = s4 = 3 3 flag1A, flag1B PB
···

At s4 , where flag1B is true, the path π 0 starts:

 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 91

π 0 (0) = s4
π 0 (1) = s5 = 5 3 flag1A, flag1B PA
π 0 (2) = s6 = 5 4 flag1A, flag1B PB
π 0 (3) = s7 = 5 3 flag1A, flag1B PB
π 0 (4) = s8 = 6 3 flag1A, flag1B PA
π 0 (5) = s9 = 6 4 flag1A, flag1B PB
π 0 (6) = s10 = 6 3 flag1A, flag1B PB
π 0 (7) = s11 = 7 3 flag1B PA
π 0 (8) = s12 = 14 3 flag1B PA
π 0 (9) = s13 = 1 3 flag1B PA
π 0 (10) = s14 = 2 3 flag1A, flag1B PA
π 0 (11) = s15 = 3 3 flag1A, flag1B PA
π 0 (12) = s5 = 5 3 flag1A, flag1B PA
π 0 (13) = s6 = 5 4 flag1A, flag1B PB
π 0 (14) = s7 = 5 3 flag1A, flag1B PB
π 0 (15) = s8
···
π 0 (i) = s15
···
where flag1B is always true. The computation goes on such that PB loops between break
points 3 and 4, and PA loops between break points 1–3, 5–7, and 14.
Notice that starting from s1 , not all computation paths are wrong. Indeed, for each state
in which flag1A is false, PB could go beyond break point 3 into the critical section.
Suppose then an oracle for a counterexample (e.g., a call to a procedure) returns the
following multi-path Π , which formally represents the intuitive computation from above:
 
s0 , s1 , s2 , s3 , [s4 , s5 . . . , s14 , s15 , s5 , . . . , s14 , . . .], s5 , . . . , s14 , s15 , s5 , . . . , s14 , . . .

here, the branching path and the remaining main path are identical; i.e., the branching path
is a subpath of the main path, and thus Π intuitively amounts to a single path. Note that Π
can be represented, using the scheme in Section 5.2, by the expression
 
s0 , s1 , s2 , s3 , s4 (s5 · · · s15 )∞ , (s5 , . . . , s15 , )∞ .

Now let us look for a repair for this program, where we assume that a single error
is present and thus focus on single correction repairs, with possible further restriction to
particular correction types.
Then, the naive repair approach considers in PA the assignments after break point
i ∈ {1, 2, 6, 8, 9, 13} and in PB after break point j ∈ {1, 2, 6, 7, 12, 13}.
92 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

For simplicity, let us first consider single statement repairs which change right hand sides
of assignments. Then, our optimization technique allows us to restrict by Theorem 6.6
attention in PA to i ∈ {1, 2, 6} and in PB to j ∈ {1, 2}; the other assignments (8, 9, 13
in PA and 6, 7, 12, 13 in PB ) are not executed by Π . Thus, only 5 out of 12 candidate
repairs remain to be considered. In case of arbitrary single assignment repairs, also the left
hand side of some of these five candidate statements may be changed; in principle, there
are five possibilities for each change since there are six control variables flagiA, flagiB
and turniB, i = 1, 2. However, only flag1A, flag1B, and turn1B are referenced along Π .
Applying Theorem 6.8, we thus can exclude any correction which changes a left hand side
to turn2B, leaving four possibilities for each change. However, knowing that the conjunct
AG(flag1B → AF¬flag1B) of ϕ fails, applying Theorem 6.8 again this number can be
cut down to two possibilities for each change. Thus, taking also right hand changes into
account, in total (2 + 1) · 5 = 15 out of 6 · 12 = 72 candidate repairs remain. Finally,
if also assignment interchange is considered, then applying Theorem 6.6 we obtain that
15 + 2 = 17 out of 72 + 5 = 77 candidate repairs remain (only the interchange of 1, 2 in
PA and 1, 2 in PB , respectively, is executed by Π ).
Apparently, a single correction repair for P is α = hA, 2, γ i where γ = r-change, i.e.,
statement 2 in PA is changed to turn1B := true. Indeed, the modified program P α does not
enable PB to loop forever between 3 and 4.

6.5. Further optimization

The utilization of correction execution and correction exploitation in an algorithm for

computing a program repair is shown in Fig. 7. The call counterexample(MF (P), ϕ) in
line 4 returns a counterexample Π for ϕ and assigns ψ the conjunct of ϕ which is falsified
by Π (in case ϕ is not a conjunction, ψ is assigned ϕ.)
Of course, further and stronger optimization techniques are imaginable, which may give
rise to other else if clauses before line 10, but they may come at a higher computational
price which must be paid for analyzing the structure of P, ϕ, and the counterexample Π .
For example, the exploitation technique from Section 6.3 can be sharpened by
incorporating that the counterexample must actually “see” the effect of a single correction
α, rather than only referencing a variable xi in V (α); it might well happen that the value
of xi is overwritten before it is referenced, and thus the correction is not useful, provided
that it does not directly affect the program specification or fairness constraints.
For an illustrating, simple example, consider the following (part of a) process:

1: x := true;
2: y := false;
3: x := y;
4: z := not x;
..
.

Here, a correction of stmnt(1) to x := false is not “seen” by the program; likewise, a

correction of stmnt(1) to y := true is not seen by the program. Thus, these corrections
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 93

Algorithm S-REPAIR
Input: PRP R = hP, F, ϕi and FC-Kripke structure MF (P).
Output: Repair α of length 6 1 for R, if one exists; “no”, otherwise.

1: if MF (P) |= ϕ then
2: return () (* empty correction *)
3: end if ;
4: (Π, ψ) := counterexample(ϕ, MF (P));
5: for each single correction α do
6: if Π does not execute α then
7: discard α
8: else if (V (α) ∩ (ap(ψ) ∪ ap(F )) = ∅) and (Π does not exploit α) then
9: discard α
10: else if MF (P α ) 6|= ψ then
11: discard α
12: else
13: return α
14: end if
15: end for ;
16: return “no”

Fig. 7. Procedure for finding a single correction repair.

of the program are useless and can be discarded (provided that they do not directly affect
the specification or fairness constraints).
For a single right side modification, we can say that a computation path sees this
correction, if
(1) it evaluates at some point k the variable, say x, of the left hand side of the corrected
statement,
(2) at some point j 6 k the corrected statement was executed, and
(3) between j and k no assignment to x was made.
Thus, it is guaranteed that at point k after the execution of the corrected statement,
the computation path experiences the effect of the correction (either by evaluating x in
an if condition or by using it in the right hand side of an assignment). If no path of a
counterexample sees the correction, than it is useless (if it does not interfere with the
specification or the fairness constraints).
Importantly, the test whether a counterexample sees a correction is efficiently possible,
given the counterexample representation that we have outlined in Section 5.2. For
assignment swaps and complex corrections, the formalization of strong exploitation is
similar, but it is more involved and also may have higher evaluation cost.
Another important optimization issue is the investigation of more sophisticated tech-
niques for pruning the search space by using counterexamples. Connected with this is an
optimized procedure for counterexample construction. So far, we were satisfied by having
any counterexample which proves that a formula fails on a structure. However, in order
to locate an error and to find a suitable repair, some counterexamples are obviously more
94 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

useful than others. For example, if a counterexample Π references a single variable during
its execution and another counterexample Π 0 references all variables, then Π might be
preferable since it allows to attribute the error to fewer statements than Π 0 . Thus, a formal,
comparative notion of quality of counterexamples has to be developed, and an optimized
counterexample procedure which finds as good counterexamples as possible. This may turn
out to be difficult and computationally complex, which calls for appropriate heuristics.

6.6. Complexity issues

We close this section with some brief comments on the computational complexity of the
problems that we have considered, namely deciding whether a program is correct, finding
a counterexample, and finding a minimal correction.
As for checking the correctness of a program P with respect to a specification ϕ under
fairness constraints F (i.e., deciding whether the empty correction () is a repair of the PRP
R = hP, F, ϕi), as shown below this problem is intractable if the input is R but is solvable
in polynomial time if R and MF (P) are given for input, since model checking MF (P) |= ϕ
for any CTL formula ϕ is polynomial in the size of MF (P) and ϕ [9,10]. Note that since
the FC-Kripke structure MF (P) associated with P and F can have exponentially many
states in the size of P and F , a simple reduction of deciding program correctness given
hP, F, ϕi to model checking MF (P) |= ϕ does not yield a polynomial time algorithm in
general. However, in special cases, e.g., if the number of processes in P and the number
of variables in P are bounded by constants, MF (P) is constructible from P and F in
polynomial time (recall that all variables are Boolean and thus have a fixed domain size),
which means that deciding program correctness given hP, F, ϕi is feasible in polynomial
time.
Constructing a counterexample, represented as in Section 5.2, for a given formula ϕ
in a given FC-Kripke Structure may take exponential time in general, but is possible
in polynomial time if the nesting depth of the path quantifier A in ϕ is bounded by a
constant; this is a restriction which is met in practice. On the other hand, constructing a
counterexample for a formula ϕ in MF (P) given hP, F, ϕi is clearly exponential, even if
the nesting depth of A is one: imagine a single process P which increases a counter having
n bits from 0 to 2n − 1 and then loops forever. The specification ϕ = AFa, where a is
assigned false in the first step and is not modified subsequently by P is violated by the
process; any counterexample for ϕ must involve at least 2n different states.
Finally, the problem of finding a repair, given a PRP R = hP, F, ϕi for input, is also
intractable. This is no surprise, and can be shown by the following simple reduction from
the well-known NP-complete satisfiability problem. Suppose ψ is a pure state formula on
atoms a1 , . . . , an . Let P be the following process:

Process P
1: a1 := true;
2: a2 := true;
...
n: an := true;
n + 1: goto n + 1
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 95

Let the specification be ϕ = AFψ; informally, it says that at some point, the formula
ψ is satisfied. To achieve this, a correction α may change the right hand sides of some
assignments from true to false; left side modifications and assignment interchanges are
clearly not needed. It holds that the PRP R = hP, F, ϕi where P = P and F = ∅ has a
repair—regardless of the underlying initialization policy for program variables—just if the
formula ψ is satisfiable. As a consequence, deciding the existence of a repair is NP-hard.
Using a second process P 0 which is similar to P but assigns false to all variables a1 , . . . , an ,
we obtain that the program P = P , P 0 is correct with respect to ϕ = AGψ under fairness
constraints F = ∅ (i.e., the empty correction () is a repair for hP, F, ϕi), just if ψ is a
tautology; this proves intractability of deciding program correctness given hP, F, ϕi for
input.
Observe that the FC-Kripke structure MF (P) associated with the program P = P and
F from above has exponentially many states in n. Hence, the previous intractability result
does not immediately carry over to deciding the existence of a repair if R = hP, F, ϕi
and MF (P) are given for input. However, it can be shown that deciding the existence of
a repair for a given PRP R = hP, F, ϕi is NP-hard even if P contains a single process
and the number of variables in P is bounded by a constant. Since in this case, MF (P) is
constructible from R in polynomial time, it follows that deciding the existence of a repair
is also intractable if the input is R and MF (P). Clearly, this implies that also computing a
minimal repair is intractable in both settings.
We leave a detailed exploration of all these complexity issues for future work.

7. Conclusion

Model checking, which is successfully used in verifying concurrent systems, appears to

be an area which has high potential for applying AI principles. In this direction, we have
presented an approach to combine model checking with repair, which has been considered
in the area of automated diagnosis. In the course of the formalization of this approach,
theory revision and abductive reasoning play an important role.
To our knowledge, our approach to integrate repair into model checking is novel. Notice
that automated diagnosis and repair of programs was investigated, e.g., in [15,45,46]. The
framework and setting in [15] (logic programs) is quite different; the approach of [45,46] is
developed for functional programming languages and generates out of a (faulty) program
an instance of a model-based diagnosis problem. This is quite different from our approach,
which aims at using techniques and concepts from model checking and combining them
with AI principles.
For a successful integration of repair into the model checking paradigm, we had to
extend the notion of a counterexample as described in [11] formally such that it is
technically available on the whole language of ACTL formulas. For this purpose, we have
introduced the concept of multi-paths as a suitable formalization of counterexample trees.
We then have presented optimization techniques which, as demonstrated on an example,
may allow for a considerable reduction of the search space for a program repair.
These optimization techniques, correction execution and correction exploitation, can be
implemented to run efficiently.
96 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Naturally, not all interesting and relevant issues can be addressed in this paper which
introduces our approach, and a number of them must be left for further work.
We have already discussed some interesting optimization issues in Section 6.5. Another
important direction of research concerns the extension of the framework by further types
of corrections. The current framework allows for right modifications and left modifications
of assignments, as well as the interchange of assignments. A prototype implementation of
this and further optimization techniques is planned for the future. Further corrections, e.g.,
more complex right hand side modifications, could be desirable. Connected with this, an
analysis of common errors in concurrent programming or protocol specification would be
acknowledged in order to identify relevant errors which our framework should be able to
handle.
Furthermore, it remains to be analyzed how abstract principles for abductive reason-
ing [34] can be exploited in this specific application domain, as well as whether abductive
algorithms and computational devices developed in AI (e.g., truth maintenance systems)
are fruitfully applicable.

Acknowledgement

We would like to thank O. Grumberg, K. McMillan, and S. Campos for their clarifying
comments on counterexamples, in particular on how counterexample paths are computed
and used in the SMV procedure. Moreover, we are grateful to K. Inoue for his comments
on connections between the the system repair problem and the framework presented
in [29], and to M. Stumptner and F. Wotawa for bibliographic hints on diagnosis of
programs. Finally, the authors appreciate the review comments which helped to improve
the presentation of this paper. This work was partially supported by the Austrian Science
Fund Project N Z29-INF.

Appendix A. Proofs

Theorem 5.1. Let M be an FC-Kripke structure, ϕ a formula on A(M), and s ∈ S(M).

Then, M, s 6|= ϕ if and only if there exists an l-counterexample Π for ϕ such that or(Π) =
s.

Proof. The statement clearly holds for pure state formulas, as Π = s is the desired
counterexample. We prove that it holds also for nonpure state formulas ϕ by structural
induction, starting from a pure state formula.
(⇐H) Suppose then ϕ is a nonpure state formula, and that Π is an l-counterexample for
ϕ such that or(Π) = s. We have to show that M, s 6|= ϕ. Consider the following possible
cases:
(1) ϕ = A(ϕ1 Uϕ2 ). It holds that M, s 6|= A(ϕ1 Uϕ2 ) if and only if either M, s |=
EG(¬ϕ2 ) or M, s |= E(¬ϕ2 U(¬ϕ1 ∧ ¬ϕ2 )) (cf. Proposition 2.1). Π is an infinite
multi-path and either
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 97

(1.1) there exists an integer k > 0 such that Π(k) is an l-counterexample for ϕ1 ∨ϕ2 ,
Π(i) is an l-counterexample for ϕ2 , for each 0 6 i 6 k, and Π(j ) is a state,
for j > k, or
(1.2) Π(i) is an l-counterexample for ϕ2 , for each i > 0.
Let π = µ(Π) be the main path of Π ; π is a fair path. In the case (1.1), M, π |=
¬ϕ2 U(¬ϕ1 ∧ ¬ϕ2 ). Indeed, for the state π(k) = or(Π(k)) in π , we have M, π(k) 6|=
ϕ1 ∨ ϕ2 : either ϕ1 ∨ ϕ2 is a pure state formula, which means Π(k) = π(k), or ϕ1 ∨ ϕ2
is not a pure state formula and Π(k) = Π 0 ∗Π 00 is the merge of two counterexamples
for ϕ1 and ϕ2 , respectively. By the induction hypothesis, M, π(k) 6|= ϕi , for i = 1, 2;
it follows M, π(k) |= ¬ϕ1 ∧ ¬ϕ2 . Moreover, for every i such that 0 6 i 6 k,
the induction hypothesis implies M, π(i) |= ¬ϕ2 . This implies the claim M, π |=
¬ϕ2 U(¬ϕ1 ∧ ¬ϕ2 ). As a consequence, M |= E(¬ϕ2 U(¬ϕ1 ∧ ¬ϕ2 ), which means
M, s 6|= A(ϕ1 Uϕ2 ).
In the case (1.2), we have M, π |= G¬ϕ2 . Indeed, the induction hypothesis implies
M, π(i) |= ¬ϕ2 , for every i > 0, since π(i) is the origin of an l-counterexample for
ϕ2 .
(2) ϕ = A(ϕ1 Vϕ2 ). Then, Π is infinite and there exists a k > 0 such that Π(j ) is
a counterexample for ϕ1 , for every 0 6 j < k, and Π(k) is a counterexample for
ϕ2 . Hence, by the induction hypothesis, M, or(Π(j )) 6|= ϕ1 , for 0 6 j < k and
M, or(Π(k)) 6|= ϕ2 . Let π = µ(Π) be the main path of Π . Since or(Π(i)) = π(i),
for every 0 6 i 6 k, and ¬ϕ1 , ¬ϕ2 are state formulas, it follows that M, µj |= ¬ϕ1 ,
for 0 6 j < k and M, µk |= ¬ϕ2 . Consequently, M, µ |= ¬ϕ1 U¬ϕ2 , which implies
M, π(0) |= E(¬ϕ1 U¬ϕ2 ). Since π(0) = s and E(¬ϕ1 U¬ϕ2 ) is equivalent to ¬ϕ, it
follows M, s 6|= ϕ.
(3) ϕ = AXϕ1 . Π is an infinite multi-path, such that Π(1) is an l-counterexample
for ϕ1 . Let π = µ(Π) be the main path of Π . π is a fair path. By the induction
hypothesis, M, π(1) 6|= ϕ1 . Hence, M, s 6|= ϕ.
(4) ϕ = ϕ1 ∨ ϕ2 . There exist two multi-paths Π1 and Π2 such that Π = Π1 ∗ Π2 , Π1
is an l-counterexample for ϕ1 , Π2 is an l-counterexample for ϕ2 , and or(Π1 ) =
or(Π2 ) = s. Hence, by the induction hypothesis, M, s 6|= ϕ1 and M, s 6|= ϕ2 .
Therefore, M, s 6|= ϕ.
(5) ϕ = ϕ1 ∧ ϕ2 . Without loss of generality, Π is a counterexample for ϕ1 with
or(Π) = s. By the induction hypothesis, M, s 6|= ϕ1 , hence M, s 6|= ϕ.
(H⇒) Consider a nonpure state formula ϕ, and suppose that M, s 6|= ϕ. We have to show
that an l-counterexample Π for ϕ exists such that or(Π) = s.
(1) ϕ = A(ϕ1 Uϕ2 ). It holds that M, s 6|= ϕ if and only if either
(i) M, s |= E(¬ϕ2 U(¬ϕ1 ∧ ¬ϕ2 )) or
(ii) M, s |= EG(¬ϕ2 ) (cf. Proposition 2.1).
In the case (i), there exist a fair path π and an integer k > 0, such that π(0) = s,
M, π(i) 6|= ϕ2 (i.e., M, π(i) |= ¬ϕ2 ), for each 0 6 i < k, and M, π(k) 6|= ϕ1 ∨
ϕ2 (i.e., M, π(k) |= ¬ϕ1 ∧ ¬ϕ2 ). Thus, by the induction hypothesis, for each
0 6 i 6 k, there exists an l-counterexample Πi for ϕ2 such that or(Πi ) = π(i),
and by the induction hypothesis and an argument as in case (4), there exists an
l-counterexample Πk for ϕ1 ∨ ϕ2 such that or(Πk ) = π(k). Consider now the multi-
sequence Π = [Π0 , Π1 , . . . , Πk , π(k + 1), π(k + 2), . . .]. Clearly, Π1 is a fair multi-
98 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

path and or(Π1 ) = s. Indeed, µ(Π) = π , and each element Π(i) is a multi-path. It
is easily seen that Π is a l-counterexample for ϕ such that or(Π) = s.
In the case (ii), there exists a path π such that π(0) = s and M, π |= G(¬ϕ2 );
since ϕ2 is a state formula, this means M, π(i) 6|= ϕ2 , for every i > 0. Hence,
by the induction hypothesis, there exists an l-counterexample Πi for ϕ2 such that
or(Πi ) = π(i), for every i > 0. Consider the multi-sequence Π = [Π0 , Π1 , Π2 , . . .].
Clearly, Π is a fair multi-path with or(Π) = s, and is a l-counterexample for ϕ.
(2) ϕ = A(ϕ1 Vϕ2 ). We have M, s |= E(¬ϕ1 U¬ϕ2 ) by duality of V and U. Hence,
there exists a fair infinite path π such that π(0) = s and M, π |= ¬ϕ1 U¬ϕ2 . The
latter means that there exists a k such that M, π j |= ¬ϕ1 , for every 0 6 j < k and
M, π k |= ¬ϕ2 . Since ϕ1 and ϕ2 are state formulas, it follows that M, π(j ) 6|= ϕ1
for every 0 6 j < k and M, π(k) 6|= ϕ2 . By the induction hypothesis, there exist an
l-counterexample Πj for ϕ1 at π(j ), 0 6 j < k and a l-counterexample Πk for ϕ2
at π(k). Let then Π be the multi-sequence
 
Π = Π0 , Π1 , . . . , Πk , π(k + 1), π(k + 2), . . . .
Clearly, Π is an infinite fair multi-path and or(Π) = s. Hence, Π is an l-counter-
example for ϕ at s.
(3) ϕ = AXϕ1 . If M, s 6|= AXϕ, then there exists a fair path π such that π(0) = s and
M, π(1) 6|= ϕ1 . By the induction hypothesis, there exists an l-counterexample Π 0 for
ϕ1 such that or(Π 0 ) = π(1). Hence, the multi-path Π = [π(0), Π 0 , π(2), π(3), . . .]
is an l-counterexample for ϕ such that or(Π) = s.
(4) ϕ = ϕ1 ∨ ϕ2 . M, s 6|= ϕ implies that M, s 6|= ϕi , for i = 1, 2; therefore, by
the induction hypothesis, there exists an l-counterexample Πi for ϕi such that
or(Πi ) = s, i = 1, 2.
Consequently, Π = Π1 ∗ Π2 is an l-counterexample for ϕ such that or(Π) = s.
(5) ϕ = ϕ1 ∧ ϕ2 . Without loss of generality, M, s 6|= ϕ1 ; by the induction hypothesis,
ϕ1 has a l-counterexample Π such that or(Π) = s, which is a l-counterexample for
ϕ such that or(Π) = s. 2

Proposition 6.1. Let M and M 0 be two FC-Kripke structures which coincide on S, S0 , L,

and F . Let Π be an l-counterexample in M for a formula ϕ on ap(ϕ) ⊆ A(M). If Π 0 is a
fair multi-path in M 0 such that Π ≈ap(ϕ) Π 0 , then Π 0 is an l-counterexample for ϕ in M 0 .

Proof. We proceed by structural induction on ϕ, where at the basis are pure state formulas.
Basis. ϕ is a pure state formula. Then Π = s for some state s and M, s 6|= ϕ.
Since Π 0 ≈A0 Π , by Definition 6.2, Π 0 = s 0 such that s ≈A0 s 0 . Hence, L(s) ∩ A0 =
L0 (s 0 ) ∩ A0 . Thus, M, s 6|= ϕ implies M 0 , s 0 6|= ϕ since ϕ is on A. Consequently, Π 0 is
an l-counterexample for ϕ in M 0 .
Induction. Let ϕ be a nonpure state formula, let Π be a l-counterexample for ϕ, and let
Π 0 be a fair multi-path such that Π ≈A0 Π 0 . We consider all possible cases for ϕ:
(1) ϕ = A(ϕ1 Uϕ2 ). Either (1) or (2) must hold:
(1) there exists an integer k > 0 such that:
(a) Π 0 (k) ≈A0 Π(k), where Π(k) is a l-counterexample for ϕ1 ∨ ϕ2 in M.
Hence, by Definition 5.5 Π(k) is both an l-counterexample for ϕ1 and for ϕ2
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 99

in M. Thus, by the induction hypothesis, Π 0 (k) is both an l-counterexample

for ϕ1 and for ϕ2 in M 0 , and hence also for ϕ1 ∨ ϕ2 in M 0 .
(b) Π 0 (i) is fair and Π 0 (i) ≈A0 Π(i), where Π(i) is an l-counterexample for
ϕ2 in M, for each 0 6 i 6 k. By the induction hypothesis, Π 0 (i) is an
l-counterexample for ϕ2 in M 0 .
(c) Π 0 (j ), for each j > k, is a state in M 0 as Π 0 (j ) ≈A0 Π(j ).
(2) Π 0 (i) ≈A0 Π(i), where Π(i) is an l-counterexample for ϕ2 , for each i > 0. By
induction hypothesis, Π 0 (i) is an l-counterexample for ϕ2 in M 0 .
Hence, by Definition 5.5, Π 0 is an l-counterexample for ϕ in M 0 .
(2) ϕ = A(ϕ1 Vϕ2 ). Π 0 is infinite, and there exists a k > 0 such that every Π(j ),
0 6 j < k, is an l-counterexample for ϕ1 , Π(k) is an l-counterexample for ϕ2 , and
every Π(`) is a state, for ` > k. Since Π ≈A0 Π 0 and ϕ is on A, by the induction
hypothesis Π 0 (j ) is a l-counterexample for ϕ1 in M 0 , for every 0 6 j < k, and Π 0 (k)
is a l-counterexample for ϕ2 in M 0 . Moreover, Π 0 (`) must be a state in M 0 for every
` > k. Hence, Π 0 is a l-counterexample for ϕ in M 0 .
(3) ϕ = AXϕ1 . Hence, Π 0 (1) is an l-counterexample for ϕ1 . By the induction
hypothesis, Π 0 (1) is an l-counterexample for ϕ1 in M 0 . Moreover, due to Π(i) ≈A0
Π 0 (i), Π 0 is infinite and Π 0 (i) is a state, for i 6= 1. Therefore, by Definition 5.5, Π 0
is an l-counterexample for ϕ in M 0 .
(4) ϕ = ϕ1 ∨ ϕ2 . Suppose without loss of generality that ϕ1 is a nonpure state formula.
Since Π is an l-counterexample for ϕ in M, Π = Π1 ∗ Π2 where Πi is an
l-counterexample for ϕi in M, i = 1, 2. Π ≈A0 Π 0 implies that Π 0 = Π10 ∗ Π20 such
that Πi0 ≈A0 Πi ; hence, by the induction hypothesis, Πi0 is an l-counterexample for
ϕi in M 0 , for i = 1, 2. Consequently, Π 0 is an l-counterexample for ϕ in M 0 .
(5) ϕ = ϕ1 ∧ ϕ2 . Π is an l-counterexample in M for either ϕ1 or ϕ2 . Hence, by the
induction hypothesis, Π 0 is an l-counterexample in M 0 for either ϕ1 or ϕ2 . Thus, by
Definition 5.5, Π 0 is an l-counterexample for ϕ in M 0 . 2

Lemma 6.5. Let R = hP, F, ϕi be a PRP for which some repair exists. Then, every
counterexample Π for ϕ in MF (P) is infinite.

Proof. Towards a contradiction, suppose there exists a single state counterexample Π = s

for ϕ in MF (P). By Definition 5.6, or(Π) = s, and or(Π) ∈ S0 (MF (P)). Since MF (P)
and MF (P α ) for a repair α coincide on S0 and L, clearly Π is also a counterexample for
ϕ in MF (P α ). Corollary 5.2 and Proposition 4.2 imply that α is not a repair for R, which
is a contradiction. 2

Theorem 6.6. Let α be a repair for the PRP R = hP, F, ϕi. Then, every counterexample
Π for ϕ in MF (P) is infinite and executes α.

Proof. Let Π be a counterexample for ϕ in M = MF (P). By Lemma 6.5, Π is an infinite

fair multi-path. Towards a contradiction, suppose Π does not execute α. We prove that Π
is also a fair infinite multi-path in M 0 . Hence, Π is a counterexample for ϕ in M 0 , which
contradicts that α is a repair.
100 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

In order to establish this, it suffices to show by induction on k > 0 that every fair multi-
path Π of M of depth d(Π) 6 k which does not execute α is a fair multi-path in M. (Notice
that from the definition, counterexample multi-paths have finite depth.)
Basis. k = 0. In this case, Π = s is a single state, and Π is by Definition 5.3 a fair
multi-path in M 0 .
Induction. k > 0. Let Π be such that d(Π) = k. We have to show that
(i) the main path µ(Π) is also a fair path in M 0 , and
(ii) Π(i) is a fair multi-path in M 0 , for every i > 0.
For (i), we note that µ(Π) is a fair path in M which does not execute α. Therefore, by
Lemma 6.4, µ(Π) is also a path in M 0 . As F (M) = F (M 0 ), clearly µ(Π) is also fair in
M 0 . For (ii), we note that each Π(i) is a fair multi-path in M which does not execute α
such that d(Π(i)) < k, and apply the induction hypothesis. 2

Lemma A.1. Let α = hk, m, γ i be a correction for P such that V (α) ∩ ap(F ) = ∅. Let π
be a fair path in MF (P) which does not exploit α. Let s0 be a state such that s0 ≈A0 π(0),
where A0 = A \ V (α). Let the sequence π 0 in MF (P α ) be defined by π 0 (0) = s0 , and
π 0 (i + 1) = si+1 , for all i > 0, where si+1 is the state reached from π 0 (i) by executing
stmnt(ki , `i ) in P α , ki = ex(π(i + 1)), `i = b(π(i + 1)ki ). Then, π 0 is a fair path and
π 0 ≈A0 π .

Proof. By induction on i, we establish that π(i) ≈A0 π 0 (i), for all i > 0, and (π 0 (i − 1),
π 0 (i)) ∈ R α , i > 1; this and the fact that ap(F ) ⊆ A0 (as ap(F ) ∩ V (α) = ∅) implies
π 0 (i) |= F iff π(i) |= F , i > 0. This proves the lemma.
Basis. For i = 0, this holds by hypothesis on s0 = π 0 (0).
Induction. Assume the statement holds for i, and consider i + 1. Since by the induction
hypothesis π(i) ≈A0 π 0 (i), we have b(π(i)) = b(π 0 (i)), i.e., the executions corresponding
to π 0 and π up to π 0 (i), respectively, π(i) have reached the same break points in all
processes. By definition of R, π(i + 1) is reached from π(i) by executing stmnt(ki , `i )
(for short, stmnt) in process Pk , where ki = ex(π(i + 1)) and `i = b(π(i))ki . Consider the
corresponding statement stmnt(ki , `i ) in P α , denoted by stmnt0 , and let si+1 be the state
reached from π 0 (i) by its execution. 11 There are two cases.
(I) stmnt0 = stmnt. If stmnt is an assignment, then, since π 0 (i) ≈A0 π(i) and stmnt may
only evaluate variables in A0 , the transitions (π(i), π(i + 1)) and (π 0 (i), π 0 (i + 1)) amount
to assigning the same value to the same variable xi ; hence, clearly si+1 ≈A0 π(i + 1);
otherwise, if stmnt is a conditional statement, for the same reason the expression c(x)
evaluates to the same at π(i) and π 0 (i), and hence (π(i), π(i + 1)) and (π 0 (i), π 0 (i + 1))
amount to executing the branching to the same break point; again, si+1 ≈A0 π(i + 1).
(II) stmnt 6= stmnt0 . Then, stmnt0 is a changed statement, which is either due to an
assignment modification or an assignment swap. In any case, stmnt and stmnt0 are
assignments xi := g 0 (x) and xj := g(x), respectively, where xi , xj ∈ V (α), and their
executions in Pki and Pkα , respectively, lead to the same break points. Hence, clearly
si+1 ≈A0 π(i + 1). 2

11 Note that also if semaphores would be allowed, stmnt0 would be executable at π 0 (i) as π 0 (i) ≈ 0 π(i).
A
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 101

Proposition 6.7. Given a PRP R = hP, F, ϕi, a collection C of complex corrections

α = α1 · · · αq , and a counterexample Π for ϕ, the corrections in C which are not executed
by Π can be discarded in time O(kPk + kCk + kΠk), i.e., in linear time.

Proof. We sketch a possible algorithm with this property. In the first step, all distinct pairs
π(i), π(i + 1) are determined such that π(i) and π(i + 1) are successive states in the
computation tree represented by Π . For each such pair, the statement stmnt(k, b) which is
executed by the transition from π(i) to π(i + 1) can be determined easily. In step two, all
these statements stmnt(k, b) from step one are marked in P. From the marked program, it
is for a single correction α = hk, m, γ i easy to tell by looking up at most two statements
in P whether α is executed by Π or not; for a complex correction α = α1 · · · αq , this is
possible by a sequence of lookups for the single αi ’s. This way, in step three all corrections
α in C are examined.
It is not hard to see that computing all pairs π(i), π(i + 1) is possible by a recursive
procedure in linear time. Moreover, the lookup and marking of a statement stmnt(b, k) in
P can be done in constant time, provided the representation of P allows for random access
to its statements. (A random access structure can be built in linear time; alternatively,
a two-dimensional Boolean array T can be used where T (k, b) tells whether statement
stmnt(k, b) was executed by some transition in Π .) Thus, also step two can be done in
linear time. Since determining whether α = α1 · · · αq is executed by Π takes at most 2·q
lookups in P, it follows that all complex corrections not executed by Π can be discarded
from C in linear time.
Note that in the case where C has few and small corrections, i.e., the total number
of simple component corrections αi occurring in C is bounded by a constant, a natural
variant of the algorithm is the following. Steps two and three are replaced by a test for
each stmnt(k, b) from step one and correction α in CTL whether α is executed by Π by
virtue of stmnt(k, b). Each such test is possible in constant time. Observe that the resulting
algorithm does not access P at all.
We finally remark that for inputs where P is not available, O(n log n) algorithms for
correction execution are possible by using sorting techniques. 2

Theorem 6.8. Let α be a repair for the PRP R = hP, F, ϕi such that


V (α) ∩ ap(ϕ) ∪ ap(F ) = ∅.
Then, every counterexample Π for ϕ in MF (P) exploits α.

Proof. Let Π be a counterexample for ϕ in M = MF (P). By way of contradiction,

suppose that Π does not exploit α. We prove that α is a weak correction with respect
to Π , i.e., there exists a fair multi-path Π 0 in M 0 = MF (P α ) such that Π and Π 0 are
equivalent on A0 = A \ V (α). Thus, a contradiction arises by Proposition 6.2.
We show by induction on the depth d(Π) of Π that if Π is a fair multi-path in M which
does not exploit α and s0 is a state such that s0 ≈A0 or(Π), then there exists a fair multi-
path Π 0 in M 0 such that Π ≈A0 Π 0 and or(Π 0 ) = s0 . (Notice that, from the definition, every
counterexample has finite depth.) In case of the counterexample, choose s0 = or(Π).
Basis. d(Π) = 0. Then, Π = s is a state. Clearly, Π 0 = s0 is the desired fair multi-path
in M 0 .
102 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

Induction. Consider d(Π) > 0. Then, Π = [Π0 , Π1 , . . .] is an infinite multi-path such

that µ(Π) is a fair path which does not exploit α and every Πi is a fair multi-path in M
which does not exploit α. By Lemma A.1, there exists a path π (1) for α = α1 such that
π (1) ≈A01 π (0) = µ(Π), where A01 = A \ V (α1 ), π (1) (0) = s0 , and π (1) is fair in MF (P α1 ).
By repeated application of Lemma A.1, we obtain that for every i = 1, . . . , q there exists
a fair path π (i) in MF (P α1 ···αi ) such that π (i) ≈A0i π (i−1) , where A0i = A \ V (αi ) and
S
π (i) (0) = s0 . It follows that π (q) ≈B π (0) for B = A \ i V (αi ). Since B = A0 and π (0) =
µ(Π), it follows that the states π (q) (i) and µ(Π)(i) satisfy π (q) (i) ≈A0 µ(Π)(i), for every
i > 0. Hence, by the induction hypothesis, for every i > 0 there exists a fair multi-path Πi0
in M 0 such that or(Πi0 ) = π (q) (i) and Πi ≈A0 Πi0 . Let Π 0 = [Π00 , Π10 , . . .]. It is easily
verified that Π 0 is a fair multi-path in M 0 such that Π ≈A0 Π 0 and or(Π 0 ) = π (q) (0) = s0 .
This concludes the induction and the proof of the theorem. 2

Proposition 6.9. Given a PRP R = hP, F, ϕi, a collection C of complex corrections

α = α1 · · · αq such that


V (α) ∩ ap(ϕ) ∪ ap(F ) = ∅,
and a counterexample Π for ϕ, the corrections in C which are not exploited by Π can be
discarded in time O(kPk + kCk + kΠk), i.e., in linear time.

Proof. The algorithm is similar to the one sketched in the proof of Proposition 6.7.
As there, in the first step all distinct pairs π(i), π(i + 1) are determined such that π(i)
and π(i + 1) are successive states in the computation tree represented by Π . In the second
step, by a scan through these pairs, the set of variables EV(Π) which are evaluated in at
least one transition π(i), π(i + 1) can be easily determined by referring to the program P.
From EV(Π), it is easy to verify whether Π exploits a single correction α by checking
whether some xi ∈ V (α) occurs in EV(Π). For a complex correction α = α1 · · · αq , this
can be done analogously by taking V (α) in place of V (α).
Step one is possible in linear time, and also step two can be done in linear time if P is
suitably represented, such that random access to its statements stmnt(k, b) is possible; here,
multiple examination of a stmnt(k, b) in constructing EV(Π) can be avoided by marking
stmnt(k, b) in P after the first access to it. Step three is also possible in linear time, by
considering the corrections α in C one by one. The result follows. 2

References

[1] L.C. Aiello, D. Nardi, Perspectives in knowledge representation, Appl. Artificial Intelligence 5 (1) (1991)
29–44.
[2] M. Ben-Ari, Z. Manna, A. Pnueli, The temporal logic of branching time, Acta Informatica 20 (1983) 207–
226.
[3] G. Brewka, K. Konolige, An abductive framework for generalized logic programs and other nonmonotonic
systems, in: Proc. IJCAI-93, Chambéry, France, 1993, pp. 9–17.
[4] G. Brewka, J. Dix, K. Konolige, Nonmonotonic Reasoning—An Overview, Chapter 5: Abduction, CSLI
Lecture Notes 73, CSLI Publications, Stanford University, 1997, pp. 65–86.
[5] C. Boutilier, V. Becher, Abduction as belief revision, Artificial Intelligence 77 (1997) 43–94.
 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104 103

[6] M.C. Browne, E.M. Clarke, D. Dill, Checking the correctness of sequential circuits, in: Proc. 1985
International Conference on Computer Design, Port Chester, New York, 1985.
[7] F. Buccafurri, T. Eiter, G. Gottlob, N. Leone, On ACTL formulas having deterministic counterexamples,
Technical Report INFSYS RR-1843-99-01, Institut für Informationssysteme, TU Wien, 1999.
[8] J.R. Burch, E.M. Clarke, K.L. McMillan, D.L. Dill, J. Hwang, Symbolic model checking: 10120 states and
beyond, Inform. and Comput. 98 (2) (1992) 142–170.
[9] E.M. Clarke, E.A. Emerson, Synthesis of synchronization skeletons for branching time temporal logic,
in: Logic of Programs: Workshop, Yorktown Heights, NY, Lecture Notes in Computer Science, Vol. 131,
Springer, Berlin, 1981.
[10] E.M. Clarke, E.A. Emerson, A.P. Sistla, Automatic verification of finite-state concurrent systems using
temporal logic specifications, ACM Trans. Programming Languages and Systems 8 (2) (1986) 244–263.
[11] E.M. Clarke, O. Grumberg, D.E. Long, Verification tools for finite-state concurrent systems, in: J.W. de
Bakker, W.P. de Roever, G. Rozenberg (Eds.), A Decade of Concurrency—Reflections and Perspectives,
Lecture Notes in Computer Science, Vol. 803, Springer, Berlin, 1994, pp. 124–175.
[12] E.M. Clarke, O. Grumberg, D.E. Long, Model checking, in: M. Broy (Ed.), Deductive Program Design,
Proc. NATO ASI Series F, Vol. 152, Springer, Berlin, 1996.
[13] E.M. Clarke, O. Grumberg, D.E. Long, Model checking and abstraction, ACM Trans. Programming
Languages and Systems 16 (5) (1994) 1512–1542.
[14] E.M. Clarke, O. Grumberg, K. McMillan, X. Zhao, Efficient generation of counterexamples and witnesses
in symbolic model checking, in: Proc. 32nd ACM/SIGDA Design Automation Conference (DAC-95), ACM
Press, New York, 1995. Also: Technical Report CMU-CS-94-204, Carnegie Mellon University, Pittsburgh,
PA, 1994.
[15] L. Console, D. Theseider Dupré, G. Friedrich, Model-based diagnosis meets error diagnosis in logic
programs, in: Proc. IJCAI-93, Chambéry, France, 1993, pp. 1494–1499.
[16] L. Console, D. Theseider Dupré, P. Torasso, On the relationship between abduction and deduction, J. Logic
Comput. 1 (5) (1991) 661–690.
[17] O. Coudert, C. Berthet, J.C. Madre, Verification of synchronous sequential machines, based on symbolic
execution, in: Proc. 1989 International Workshop on Automatic Verification Methods for Finite State
Systems, Grenoble, France, Lecture Notes in Computer Science, Vol. 407, Springer, Berlin, 1989.
[18] T. Eiter, G. Gottlob, The complexity of logic-based abduction, J. ACM 42 (1) (1995) 3–42.
[19] T. Eiter, G. Gottlob, N. Leone, Semantics and complexity for abduction from default theories, Artificial
Intelligence 90 (1997) 177–222.
[20] E.A. Emerson, Temporal and modal logics, in: J. van Leeuwen (Ed.), Handbook of Theoretical Computer
Science, Chapter 16, Elsevier, Amsterdam, 1990, pp. 995–1072.
[21] E.A. Emerson, J.Y. Halpern, “Sometimes” and “not never” revisited: On branching versus linear time
temporal logic, J. ACM 33 (1) (1986) 151–178.
[22] E.A. Emerson, E.M. Clarke, Using branching-time logic to synthesize synchronization skeletons, Sci.
Comput. Programming 2 (1982) 241–266.
[23] G. Friedrich, G. Gottlob, W. Nejdl, Hypothesis classification, abductive diagnosis, and therapy, in:
G. Gottlob, W. Nejdl (Eds.), Proc. International Workshop on Expert Systems in Engineering, Lecture Notes
in Artificial Intelligence, Vol. 462, Springer, Berlin, 1990, pp. 69–78.
[24] G. Friedrich, G. Gottlob, W. Nejdl, Formalizing the repair process—Extended report, Ann. Math. Artificial
Intelligence 11 (1994) 187–201.
[25] P. Gärdenfors, Knowledge in Flux, Bradford Books/MIT Press, Cambridge, MA, 1988.
[26] O. Grumberg, D.E. Long, Model checking and modular verification, ACM Trans. Programming Languages
and Systems 16 (3) (1994) 843–871.
[27] W. Hamscher, (Ed.), Readings in Model-Based Diagnosis, Morgan Kaufmann, San Mateo, CA, 1992.
[28] N. Immerman, M. Vardi, Model checking and transitive-closure logic, in: Proc. Computer Aided Verification
(CAV-97), 1997, pp. 291–302.
[29] K. Inoue, Ch. Sakama, Abductive framework for nonmonotonic theory change, in: Proc. IJCAI-95, Montreal,
Quebec, 1995, pp. 204–210.
[30] H. Katsuno, A.O. Mendelzon, Propositional knowledge base revision and minimal change, Artificial
Intelligence 52 (1991) 253–294.
[31] K. Konolige, Abduction versus closure in causal theories, Artificial Intelligence 53 (1992) 255–272.
104 F. Buccafurri et al. / Artificial Intelligence 112 (1999) 57–104

[32] O. Kupferman, O. Grumberg, Buy one, get one free!!, J. Logic Comput. 6 (4) (1996) 523–539.
[33] J. Lobo, C. Uzcátegui, Abductive change operators, Fundamenta Informaticae 27 (1996) 385–412.
[34] J. Lobo, C. Uzcátegui, Abductive consequence relations, Artificial Intelligence 89 (1–2) (1997) 149–171.
[35] D.E. Long, Model checking, abstraction and compositional reasoning, Ph.D. Thesis, Carnegie Mellon
University, Pittsburgh, PA, 1993.
[36] K.L. McMillan, Symbolic model checking: An approach to the state explosion problem, Ph.D. Thesis,
Carnegie Mellon University, Pittsburgh, PA, 1992.
[37] B. Misra, E.M. Clarke, Hierarchical verification of asynchronous circuits using temporal logic, Theoret.
Comput. Sci. 38 (1985) 269–291.
[38] G. Peterson, Myths about the mutual exclusion problem, Inform. Process. Lett. 12 (3) (1981) 115–116.
[39] A. Pnueli, The temporal semantics of concurrent programs, Theoret. Comput. Sci. 13 (1981) 45–60.
[40] D. Poole, A logical framework for default reasoning, Artificial Intelligence 36 (1988) 27–47.
[41] G.M. Provan, D. Poole, The utility of consistency-based diagnostic techniques, in: Proc. Internat. Conference
on the Principles of Knowledge Representation and Reasoning (KR-91), Morgan Kaufmann, San Mateo,
CA, 1991, pp. 461–472.
[42] R. Reiter, A theory of diagnosis from first principles, Artificial Intelligence 32 (1987) 57–95.
[43] B. Selman, H.J. Levesque, Support set selection for abductive and default reasoning, Artificial Intelligence
82 (1996) 259–272.
[44] A.P. Sistla, E.M. Clarke, The complexity of propositional linear temporal logics, J. ACM 32 (3) (1985)
733–749.
[45] M. Stumptner, F. Wotawa, Model-based program debugging and repair, in: Proc. 9th International
Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems
(IEA/AIE-96), Fukuoka, Japan, 1996, pp. 155–160.
[46] M. Stumptner, F. Wotawa, A model-based approach to software debugging, in: S. Abu-Hakima (Ed.), Proc.
7th International Workshop on the Principles of Diagnosis (DX-96), Val Morin, Quebec, 1996, pp. 214–223.
[47] M.Y. Vardi, P. Wolper, An automata-theoretic approach to automatic program verification, in: Proc. 1st
Annual Symposium on Logic in Computer Science (LICS-86), IEEE Computer Society Press, 1986,
pp. 332–344.