2018 ACM/IEEE 13th International Workshop on Automation of Software Test

S oftware Testing as A Problem of Machine Learning:

Towards a Foundation on Computational Learning Theory
(Extended Abstract of Keynote Speech)
Hong Zhu
School of Engineering, Computing and Mathematics
Oxford Brookes University, Oxford OX33 1HX, UK. Tel. ++44 01865 484580
[email protected]
In recent years, the application of machine learning techniques to The main conclusions that we can draw are two folds. First, a
software testing has been an active research area. Among the most function is learnable implies that it is testable. Thus, learning is a
notable work reported in the literature are those experiments on more difficult computational problem than testing. Second, when
the uses of supervised and semi-supervised learning techniques to a machine learning technique is used for test automation, its
develop test oracles so that the correctness of software outputs and inductive inference power (i.e. the set of functions that is
behaviours on new test cases can be predicated [1]. Experiment learnable for the inference device) determines the set of functions
data show that it seems a promising approach to the test oracle that are testable.
automation problem. In general, software testing is an inductive Similarly, given a Probably Approximately Correct (PAC)
inference in the course of which the tester attempts to deduce inference machine M, we can define an test adequacy
general properties of a software system by observing the measurement KM, (t, p), which is a function from test sets t and
δ

behaviours of the system on a finite number of test cases [2]. programs p to real number adequacy scores in the range [0,1]. The
Thus, there is a great potential for the application of machine following theorem links test adequacy to software reliability.
learning to software testing.
Theorem 3. For a finite random test set t, the program p is correct
Since 1980s, researchers have studied the relationships between on t w.r.t a specification s which is in a set P of functions PAC
software testing and inductive inferences. In this talk, I will brief learnable by M, then the δ-probable reliability of p is KM, (t, p).

δ

review the main results in this area from a theoretical perspective.

The existing work can be roughly classified into two categories. A practical implication of Theorem 3 is that the complexity of the
The first is to define test adequacy criteria based on inductive software under test should be taken into consideration in
inference techniques. For example, Weyuker proposed an reliability estimation since the complexity of the function
adequacy criterion explicitly involving inductive inference [4]. In determines learnability, and thus testability.
particular, a test is defined to be adequate if the program under The second category is to analyze existing software testing
test can be derived from test cases. More recently, Fraser and techniques and methods from an inductive inference point of
Walkingshaw further developed Weyuker’s work by employing view. Considering testing as a process of inductive inference, the
Probably Approximately Correct inductive inference protocol to question is whether the induction converges to a right conclusion
define behavioral adequacy criterion, which requires an accurate when testing stops. Because test adequacy criteria are used as stop
model of the software to be derivable from adequate test cases [5]. rules, the analysis of testing methods can be performed via
In general, an inductive inference device M is a function. It takes a examining test adequacy criteria using various inductive inference
finite subset X of input/output pairs of a function f on a domain D, protocols. In [3], Weyuker’s axioms of test adequacy criteria were
as input and produces a function M(f) such that it is correct on the studied with identification in the limit. It was proved that the
set X of input/output pairs. An adequacy criteria CM(t, p) can then adequacy criterion CM(t, p) satisfies Weyuker’s axioms if the
be defined as











, where t is a finite test inference machine M satisfies certain properties, such as
set, p is a function on D under test,


is the subset of conservative. Zhu and Hall’s axioms of test adequacy
input/output pairs of p with input from t. Employing the measurement were also examined, but using Valiant’s PAC
identification in the limit protocol, the following were proved [3]. inference protocol. It was proved that the adequacy criterion
KM, (t, p) satisfies Zhu and Hall’s axioms. In other words, the
Theorem 1. A program p is correct w.r.t. specification s after
δ

axiom systems of test adequacy do catch the key properties of

successfully tested on a finite test set t, if t is adequate according
inductive inference nature underlying software testing techniques.
to criterion CM(t, p), p is explanatorily learnable by M, s is
behaviourally learnable by M, and M converges to a function that REFERENCES
is consistent with s on t.
[1] Almaghairbe, R. 2017. Formulating Test Oracles via Anomaly
Theorem 2. If both program p and specification s belong to a set Detection Techniques. PhD Thesis, University of Strathclyde, UK.
of functions that are learnable by identification in the limit, the [2] Zhu, H., Hall, P., and May, J. 1992. Inductive inference and software
correctness of a program p w.r.t. a specification s can be testing. Journal of Software Testing, Verification, and Reliability,
determined by testing on a finite number of test cases.
2:69-81.
[3] Zhu, H. 1996. A formal interpretation of software testing as inductive
Permission to make digital or hard copies of part or all of this work for inference. Journal of Software Testing, Verification and Reliability,
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that 6:3-31.
copies bear this notice and the full citation on the first page. Copyrights [4] Weyuker, E. J. 1983. Assessing test data adequacy through program
for third-party components of this work must be honored. For all other inference. ACM Transactions on Programming Languages and
uses, contact the Owner/Author.
Systems, 5(4), 641-655.
A S T ' 1 8, May 28–29, 2018, Gothenburg, Sweden [5] Fraser, G and Neil Walkinshaw, N. 2015. Assessing and generating
© 2018 Copyright is held by the owner/author(s). test sets in terms of behavioural adequacy. Software Testing,
ACM ISBN 978-1-4503-5743-2/18/05.
https://doi.org/10.1145/3194733.3194745 Verification And Reliability, 25:749–780

Authorized licensed use limited to: FUDAN UNIVERSITY. Downloaded on July 24,2020 at 01:54:39 UTC from IEEE Xplore. Restrictions apply.