Checking Security Properties of

Cloud Service REST APIs

Abstract—Most modern cloud and web services are program- in order to thoroughly exercise the cloud service deployed
matically accessed through REST APIs. This paper discusses behind that API, with the goal of finding unhandled exceptions
how an attacker might compromise a service by exploiting (service crashes) that can be detected by a test client as “500
vulnerabilities in its REST API. We introduce four security rules
that capture desirable properties of REST APIs and services. Internal Server Errors”. While that work looks promising and
We then show how a stateful REST API fuzzer can be extended reports many new bugs found, its scope is restricted to the
with active property checkers that automatically test and detect detection of unhandled exceptions.
violations of these rules. We discuss how to implement such In this paper, we introduce four security rules that capture
checkers in a modular and efficient way. Using these checkers, we
desirable properties of REST APIs and services.
found new bugs in several deployed production Azure and Office-
365 cloud services, and we discuss their security implications. All • Use-after-free rule. A resource that has been deleted
these bugs have been fixed. must no longer be accessible.
Keywords-Test generation; Security; Cloud and Web services; • Resource-leak rule. A resource that was not created
REST APIs successfully must not be accessible and must not “leak”
any side-effect in the backend service state.
I. I NTRODUCTION • Resource-hierarchy rule. A child resource of a parent
Cloud computing is exploding. Over the last few years, resource must not be accessible from another parent
thousands of new cloud services have been deployed by cloud resource.
platform providers, like Amazon Web Services [2] and Mi- • User-namespace rule. A resource created in a user
crosoft Azure [13], and by their customers who are “digitally namespace must not be accessible from another user
transforming” their businesses by modernizing their processes namespace.
while collecting and analyzing all kinds of new data. Violations of such rules might allow an attacker to hijack cloud
Today, most cloud services are programmatically accessed resources or bypass quotas (Elevation-of-Privilege attack), or
through REST APIs [9]. REST APIs are implemented on to steal information from other users (Information-Disclosure
top of the ubiquitous HTTP/S protocol, and offer a uni- attack), or to corrupt the backend service state so that it no
form way to create (PUT/POST), monitor (GET), manage longer operates properly (Denial-of-Service attack), as will be
(PUT/POST/PATCH) and delete (DELETE) cloud resources. discussed later.
Cloud service developers can document their REST APIs and We show how a stateful REST API fuzzer can be extended
generate sample client code by describing their APIs using to test and detect violations of such rules. For each rule, we
an interface-description language such as Swagger (recently define an active property checker which (1) generates new API
renamed OpenAPI) [25]. A Swagger specification describes requests to test specific rule violations and (2) detects any such
how to access a cloud service through its REST API, including rule violation. In other words, each checker actively tries to
what requests the service can handle, what responses may be break its rule in addition to monitoring for any rule violation.
received, and the response format. We discuss how to implement such checkers in a modular way,
How secure are all those APIs? Today, this question is still so that checkers do not interfere with each other. Since each
largely open. Tools for automatically testing cloud services checker generates new tests, in addition to an already-large
via their REST APIs and checking whether these services state space exploration, we also discuss how to implement each
are reliable and secure are still in their infancy. Some tools individual checker efficiently, by eliminating likely-redundant
available for testing REST APIs capture live API traffic, tests whenever possible.
and then parse, fuzz, and replay the traffic with the hope By construction, these checkers can find security rule vio-
of finding bugs [4], [21], [6], [26], [3]. Recently, stateful lations beyond the “500 Internal Server Errors” that can be
REST API fuzzing [5] was proposed to specifically test more detected by baseline stateful REST API fuzzing. Using these
deeply services deployed behind REST APIs. Given a Swagger checkers, we found new bugs in several production Azure
specification of a REST API, this approach automatically and Office-365 cloud services. The use of security checkers
generates sequences of requests, instead of single requests, increases the value of REST API fuzzing by detecting more
∗ The work of this author was mostly done while visiting Microsoft types of bugs at a modest incremental testing cost.
Research. This paper makes the following contributions:
 • We introduce rules that describe security properties of delete. The request body b may include additional parameters
REST APIs. and their values that may be required or optional for the
• We design and implement active checkers to test and request to be executed successfully.
detect violations of these rules. For instance, here is a request to get the properties of a
• We present detailed experimental results evaluating the specific Azure DNS zone [14] (shown on multiple lines):
performance and effectiveness of these active checkers h User-auth-token i GET
on three production cloud services. https://management.azure.com/
• With these checkers, we found new bugs in several subscriptions/{subscriptionId}/
production Azure and Office-365 cloud services, and we resourceGroups/{resourceGroupName}/
discuss their security implications. providers/Microsoft.Network/
The rest of the paper is organized as follows. In Sec- dnsZones/{zoneName}
tion II, we recall background information on stateful REST ?api-version=2018-03-01 { }
API fuzzing. In Section III, we introduce rules that capture This request is of type GET, its path requires three
desirable properties of secure REST APIs and present active resource names, namely a subscriptionID, a
checkers to test and detect violations of these rules. In Sec- resourceGroupName, and a zoneName, and its body (at
tion IV, we present experimental results with active checkers the end) denoted by { } is empty.
on production cloud services. In Section V, we discuss new REST API requests of type PUT or POST typically cre-
bugs found by these checkers and their security implications. ate new resources, while DELETE requests destroy existing
In Section VI, we discuss related work, and we conclude the resources. A request whose execution creates a new resource
paper in Section VII. of type T is called a producer for the resource type T . A
newly created resource is represented by its identifier, or
II. S TATEFUL REST API F UZZING id for short. Because resources are dynamically created, we
In this section, we recall the definition of stateful REST API will sometimes call them dynamic objects. A request which
fuzzing [5], before introducing in Section III security property requires a resource name of type T in its path or in its body is
checkers that can be implemented as extensions of this basic called a consumer for the resource type T . We will sometimes
scheme. refer to the resource name of type T as the dynamic object
We consider cloud services accessible through REST APIs. type. In the Azure DNS zone example above, the GET request
A client program sends messages, called requests, to a service shown consumes three resources of type subscriptions,
and receives messages back, called responses. Such messages resourceGroups, and dnsZones respectively, but does
are sent over the HTTP/S protocol. Each response is associated not produce any new resource.
with a single HTTP status code which is either in the 2xx, Inside resource paths or request bodies of individual re-
3xx, 4xx or 5xx ranges. quests, the user is allowed to specify that some specific values,
Swagger [25], also known as OpenAPI, is an example called fuzzable values, are to be chosen randomly among a
of specification language to define REST APIs. A Swagger (small finite) set of specific values. For instance, a user might
specification describes how to access a service through its specify that a given integer value in the body of a request
REST API, including what requests the service can handle, may be, say, either 0, 10, 1000000, or -10. Such a set
what responses may be received, and the respective response of values is called a fuzzing dictionary. Given a request with
format. fuzzable values, a rendering of that request denotes a mapping
We define a REST API as a finite set of requests. Each of each fuzzable value to a single concrete value selected in
request r is a tuple of the form ha, t, p, bi where its fuzzing dictionary. Thus, a request with n fuzzable values
• a is an authentication token, which can each take k possible values results in nk possible
• t is the request type,
renderings. A rendering is called valid if the execution of the
corresponding request returns a valid response (defined in the
• p is a resource path, and
next paragraph). Users are responsible for identifying values
• b is the request body.
they want to fuzz and their associated fuzzing dictionaries.
A request type t is any of the following five REST-allowed We define the state space of a service as a directed graph
values: PUT (create or update), POST (create or update), GET where nodes represent service states and edges are transitions
(read, list or query), DELETE (delete), PATCH (update). The between these. Given a state s of the service, executing a
resource path p is a string identifying a cloud resource and single request r leads to a successor state s0 : this execution is
its parent hierarchy. Typically, p is a (non-empty) sequence r
denoted by s → s0 . The execution of a request r in a state s is
matching the regular expression either valid if it triggers a 2xx response, invalid if it triggers a
(/hresourceTypei/hresourceNamei/)+ 3xx or 4xx response, or a bug if it triggers a 5xx response.
where resourceType denotes the type of a cloud resource Given an initial state where no resources exist, the state
and resourceName is the specific name of the resource of space of the service reachable from that initial state can
that type. The last resource named in the path is typically be explored by executing sequences of requests. Such an
the specific resource that the request tries to create, access, or exploration is stateful when it attempts to explore service states
that are reachable only using sequences of multiple requests: must fail and thus return a “404 Not Found” HTTP status
earlier requests in a sequence may produce resources that are code in their response.
consumed in subsequent requests in that sequence in order to A use-after-free violation occurs when a resource that has
exercise more requests and reach deeper service states. been deleted still remains accessible through the API. This
State-space exploration can be performed using various must never happen. It is a clear bug that may lead to bypassing
search strategies, e.g., a systematic breadth-first search or a resource quotas and corrupting the service backend state.
random search [5]. State spaces can be large, even infinite, be- Resource-leak rule. A resource that was not successfully
cause the length of request sequences is not bounded, because created must not be accessible, and must not “leak” any
the sets of possible renderings can be very large, and because associated resources in the backend service state. In other
the service under test is viewed as a blackbox. Fortunately, words, if the execution of a PUT or POST request to create a
a partial state-space exploration may be sufficient to reveal new resource fails (for any reason), any subsequent operation
interesting bugs. In our context, a bug is defined as a 500 on that resource must also fail with a 4xx response. Fur-
HTTP status code being received as a response after executing thermore, no side-effects associated with successful creation
a request sequence. Such “500 Internal Server Errors” are of that resource type must occur in the backend service state
unhandled exceptions triggered by unexpected input request and be visible to the user. For instance, a failed-to-be-created
sequences, which may corrupt the service state and severely resource must not be counted in the user’s resource counter
damage the service health: it is safer to fix such bugs rather towards service quotas, and the name of the failed-to-be-
than risk a live incident with unknown consequences. created resource must be reusable by the user.
In what follows, we will sometimes use the term test cases As an example, after issuing a malformed PUT request to
to refer to executions of request sequences, while tests refer create URI /users/user-id1, a 4xx response must be
to executions of single requests. We will also call the general received. Any subsequent request to access (read, update, or
state-space exploration algorithm of this section the main delete) this URI must also fail.
driver of stateful REST API fuzzing. A resource-leak violation occurs when a resource that was
not successfully created nevertheless “leaks” some side-effect
III. S ECURITY C HECKERS FOR REST API S in the backend service state. For instance, the resource may be
In this section, we define and describe active checkers listed by a subsequent GET request, yet it cannot be deleted
for security rules of REST APIs. First, in Section III-A, we with a DELETE request, or subsequent attempts to re-create
introduce four REST API security rules. In Section III-B, we this resource return “409 Conflict” responses. Such violations
describe how to implement active checkers for testing and must never happen, as they may have unintended consequences
detecting security rule violations. Each active checker focuses on the capacity for that resource type (e.g., if resource quota
on a single type of security rule violation. In Section III-C, we limits are reached and no new resources can be created) and
discuss how each checker can be combined in a modular way on the performance of the service (e.g., due to unnecessarily
with the other checkers and with the main driver of stateful large database tables).
REST API fuzzing. In Section III-D, we propose a new search Resource-hierarchy rule. A child resource of a parent
strategy for scalable test generation with property checkers. resource must not be accessible from another parent
In Section III-E, we describe how to group together checker resource. In other words, if a resource child is
violations in order to avoid reporting the same bug multiple successfully created from a resource parent and
times to the user. identified as such in service resource paths of the form
hparentTypei/parent/hchildTypei/child/, the
A. Security Rules child resource must not be accessible (i.e., must not be
We introduce four security rules that capture desirable successfully read, updated or deleted) when substituting the
properties of REST APIs and services. We illustrate each rule parent resource by any other parent resource.
with an example and discuss its security implications. All four For example, after issuing POST requests to URIs
rules are inspired by past real bugs in deployed cloud services, /users/user-id1, /users/user-id2, and
which were found either by manual penetration testing or by /users/user-id1/reports/report-id1 to create
root cause analysis of customer-visible incidents. Examples of users user-id1, user-id2, and then add report
new, previously-unknown bugs we found as rule violations report-id1 to user user-id1, subsequent requests
in deployed production Azure and Office-365 services are to URI /users/user-id2/reports/report-id1
presented later in Section V. must fail since, according to the resource-hierarchy rule,
Use-after-free rule. A resource that has been deleted must report report-id1 belongs to user user-id1 but not to
no longer be accessible. In other words, after a successful user user-id2.
DELETE operation on any resource, any subsequent operation A resource-hierarchy violation occurs when a sub-resource
– like read, update, or delete – on that resource must fail. originally created from a parent resource is accessible from
For example, after issuing a DELETE request to URI a different parent resource with no parent-child relation-
/users/user-id1 in order to delete the account with iden- ship. When such violations are possible, an attacker might
tifier user-id1, all subsequent attempts to use user-id1 be able to provide an unauthorized parent object identifier
 1 Inputs: seq, global cache, reqCollection 1 Inputs: seq, global cache, reqCollection
2 # Retrieve the object types consumed by the last request and 2 # Retrieve the object types produced by the whole sequence and by
3 # locally store the most recent object id of the last object type. 3 # the last request separately to perform type checking later on.
4 n = seq.length 4 seq obj types = PRODUCES(seq)
5 req obj types = CONSUMES(seq[n]) 5 target obj types = PRODUCES(seq[−1])
6 # Only the id of the last object is kept, since this is the 6 for target obj type in target obj types:
7 # object actually deleted. 7 for guessed value in GUESS(target obj type):
8 target obj type = req obj types[−1] 8 global cache[target obj type] = guessed value
9 target obj id = global cache[target obj type] 9 for req in reqCollection:
10 # Use the latest value of the deleted object and execute 10 # Skip consumers that don’t consume the target type.
11 # any request that type−checks. 11 if CONSUMES(req) != target obj type:
12 for req in reqCollection: 12 continue
13 # Only consider requests that typecheck. 13 # Skip requests that don’t typecheck.
14 if target obj type not in CONSUMES(req) 14 if CONSUMES(req) − seq obj types:
15 continue 15 continue
16 # Restore id of deleted object. 16 # Execute the request accessing the ’’guessed’’ object id.
17 global cache[target obj type] = target obj id 17 EXECUTE(req)
18 # Execute request on deleted object. 18 assert ’’HTTP status code in 4xx class’’
19 EXECUTE(req) 19 if mode != ’exhaustive’:
20 assert ’’HTTP status code is 4xx’’ 20 break
21 if mode != ’exhaustive’: Fig. 2: Resource-leak checker.
22 break
Fig. 1: Use-after-free checker. We enforce the first principle by running all the checkers
(e.g., user-id3), and then steal (read) or hijack (write) whenever the main driver has finished executing a new test
an unauthorized child object (e.g., report-id1). Resource- case. We enforce the second principle by prioritizing the order
hierarchy violations are clear bugs, are potentially dangerous, of applying checkers based on their semantics, so that they
and must never happen. operate on different test cases and do not interfere with each
User-namespace rule. A resource created in a user namespace other (more on this later in this section). In what follows,
must not be accessible from another user namespace. In the we present implementation details of each checker as well as
context of REST APIs, we consider user namespaces defined optimizations to limit state-space explosion.
by the user token used to interact with the API (e.g., OAUTH Use-after-free checker. The implementation of the use-after-
token-based authentication [18]). free rule checker is described in Figure 1 in python-like
For example, after issuing a POST request to create URI notation. The algorithm is called after the main driver executes
/users/user-id1 using token token-of-user-id1, a DELETE request (see Figure 4) and takes three inputs:
resource user-id1 must not be accessible using another a sequence seq of requests, which is the latest test case
token token-of-user-id2 of another user. executed by the main driver; the global cache of dynamic
A user namespace violation occurs when a resource created objects, denoted global_cache, which contains the most
within the namespace of one user is accessible from within the recent object types and ids for the dynamic objects created so
namespace of another user. If such a violation ever occurs, an far; and the request collection, denoted reqCollection,
attacker might be able to execute REST API requests using an which is the set of all available API requests.
unauthorized authentication token, and perform unauthorized First, the types of the dynamic objects consumed by the
operations on resources belonging to another (victim) user. last request are retrieved (line 5) and the id of the last
object type, denoted target_obj_type, is stored in a
B. Active Checkers temporary variable, denoted target_obj_id. Although the
We implement active checkers for the rules defined in last request may be consuming more than one object type, we
Section III-A. An active checker monitors the state space consider the last type in req_object_types as the actual
exploration performed by the main driver of stateful REST type of the deleted object. (For example, a DELETE request on
API fuzzing and suggests new tests to assert that specific the URI /users/userId1/reports/reportId1 con-
rules are not violated. Thus, an active checker augments the sumes two object types (users and reports) but only deletes
search space by executing new tests targeted at violating report objects.) After this initial setup, the for-loop (line 12)
specific rules. In contrast, a passive checker monitors the iterates over all requests available in reqCollection and
search performed by the main driver without executing new skips those that do not consume the target object type (line 14).
tests. Once a request, req, that consumes the target object type is
We design active checkers following a modular design based found, the target object id is restored in the global cache of
on two principles: dynamic objects (line 17) and is therefore used by the function
1) Checkers are independent from the main driver of state- EXECUTE (line 19) which executes request req. Note that
ful REST API fuzzing and do not affect its state space the target object id is repeatedly restored in the global cache
exploration. because the function EXECUTE uses object ids available in
2) Checkers are independent from each other and generate global_cache when executing a request. If any of these
tests by analyzing the requests executed by the main requests succeeds, line 20 will trigger a use-after-free violation
driver, excluding those executed by other checkers. (see Section III-A).
 1 Inputs: seq, global cache 1 Inputs: seq, global cache, reqCollection
2 # Record the object types consumed by the last request 2 # Execute the checkers after the main driver.
3 # as well as those of all predecessor requests. 3 n = seq.length
4 n = seq.length 4 if seq[n].http type == ’’DELETE’’:
5 last request = seq[n] 5 UseAfterFreeChecker(seq, global cache, reqCollection)
6 target obj types = CONSUMES(seq[n]) 6 else:
7 predecessor obj types = CONSUMES(seq[:n]) 7 if seq[n].http response == ’’4xx’’:
8 # Retrieve the most recent id of each child object consumed 8 ResourceLeakChecker(seq, global cache, reqCollection)
9 # only by the last request. These are the objects whose 9 else:
10 # hierarchy we will try to violate. 10 ResourceHierarchyChecker(seq, global cache)
11 local cache = {} 11 UserNamespaceChecker(seq, global cache)
12 for obj type in target obj types − predecessor obj types: Fig. 4: Checkers dispatcher.
13 local cache[obj type] = global cache[obj type]
14 # Render sequence up to before the last request trigger a resource-leak violation (see Section III-A) or asserts
15 EXECUTE(seq, n−1)
16 # Restore old children object ids that do NOT belong to that no such violation occurs for the given request sequence
17 # the current parent ids and must NOT be accessible from those. (line 18).
18 for obj type in local cache: Finally, in order to limit the number of additional tests
19 global cache[obj type] = local cache[obj type]
20 EXECUTE(last request) generated for each input sequence, the inner loop (optionally)
21 assert ’’HTTP status code is 4xx’’ terminates when one request for each guessed object is found
Fig. 3: Resource-hierarchy checker. (line 19). We evaluate this optimization in Section IV.
Resource-hierarchy checker. The implementation of the
Finally, in order to limit the number of additional tests gen- resource-hierarchy rule checker is described in Figure 3. The
erated for each request sequence, the inner loop (optionally) algorithm takes two inputs: a sequence of requests, denoted
terminates when one request for each target object type is seq, which is the latest test case executed by the main driver
found (line 21). This option is used if the variable mode is not and the current global cache of dynamic objects, denoted
set to value exhaustive. We present detailed experimental global_cache. First, the algorithm records the object types
results regarding the impact of this optimization in Section IV. consumed by the last request of the current sequence, de-
Resource-leak checker. The resource-leak rule checker is noted target_obj_types (line 6), and the object types
described in Figure 2. The algorithm takes the same three consumed by all other requests of the sequence before the
inputs as the use-after-free checker. This checker operates on last request, denoted predecessor_obj_types (line 7).
request sequences executed by the main driver whose last Afterwards, the ids of the objects consumed only by the last
request led to an invalid HTTP status code in the response request are stored locally (lines 12 and 13). These are the
(see Figure 4). Initially, the algorithm identifies the dy- child objects whose hierarchy the checker will try to violate
namic object types produced by the whole sequence, denoted by executing requests that try to access them using invalid
seq_obj_types, and produced by the last request, denoted parent objects. To this end, in line 15, the current sequence
target_obj_types (lines 4 and 5). The main logic of is executed up to (and not including) the last request. Finally,
the algorithm is implemented in three nested for loops. The the old child object ids are restored (lines 18 and 19) and the
first loop (line 6) iterates over all object types produced by last request is executed using the old child object ids on top
the last request. The second loop (line 7) iterates over object of new parent object ids (line 20). These parent object ids
ids “guessed” for the current object type for which an invalid are not proper parent objects of the restored child object ids.
HTTP status code was received. The function GUESS takes as This way, the algorithm tries to trigger a resource-hierarchy
argument an object type and returns a set of possible object ids violation (see Section III-A) or asserts that no such violation
matching this type and which were not created successfully. occurs for the given request sequence (line 21).
For instance, if the creation of a dynamic object with object User-namespace checker. Due to space constraints, we omit a
type “x” and object id “objx1” fails through the API (according detailed presentation of this checker. In a nutshell, this checker
to the response received), the checker will attempt to execute attempts to re-execute the valid last request of any test case
any request that consumes the object type “x” and assert it fails executed by the main driver using a different authentication
when using the object id “objx1”. Note that the total number token. If this succeeds, an attacker with a different authenti-
of guessed values per object id is limited to a user-provided cation token could hijack the objects used in the last request,
parameter value in order to avoid an explosion in the number and a user namespace violation (see Section III-A) is reported.
of additional tests.
In line 8, a guessed object-id value is temporarily added C. Combining All Checkers
to the global cache of properly-created dynamic objects. The four checkers defined in the previous section are
Then the inner loop (line 9) iterates over all requests in executed as follows. Whenever the stateful REST API fuzzer
reqCollection to find requests that are executable (given reaches a new state (as defined in Section II), its main driver
the object types produced by the current sequence) and that calls the code shown in Figure 4. Depending on the last request
consume the given target object type. These requests are executed, this code activates the checkers that are applicable
executed (line 17) using the “guessed” object ids previously to the current state. We now discuss important properties of
registered in the global cache. This way, the algorithm tries to these checkers and of their combination.
Contribution beyond stateful REST API fuzzing. The of length n, instead of to all of them as in BFS [5]. BFS-
checkers extend the main driver of baseline stateful REST Fast provides full grammar coverage only with respect to all
API fuzzing in two ways: (1) they extend the state space by possible renderings of individual requests but does not explore
executing additional tests and (2) they check for responses all request sequences of a given sequence length.
other than 5xx and can flag unexpected 2xx responses as Although BFS-Fast scales better compared to BFS, it does
rule-violation bugs. Thus, they clearly increase the bug-finding so by exploring only a subset of all possible request sequences.
capabilities of the main driver: they can find bugs that the main Unfortunately, this limits the number of violations the security
driver alone would not find. checkers can actively check. To alleviate this limitation, we
Active property checking versus passive monitoring. As introduce a new search strategy, called BFS-Cheap.
discussed earlier, the checkers we define extend the search BFS-Cheap follows the inverse trade-off of BFS-Fast: it
space explored by the main driver with additional test cases sacrifices full coverage of all possible request renderings at
aimed at triggering and detecting specific rule violations. In every state but explores all possible request sequences for a
contrast, passive runtime monitoring of these rules in conjunc- given sequence length, albeit not with all possible renderings.
tion with the main driver, i.e., without executing those new Specifically, given a set of sequences of length n, called
tests, would likely be unable to detect rule violations. Specif- seqSet, and a set of requests, called reqCollection,
ically, use-after-free and resource-leak rule violations would BFS-Cheap operates as follows:
likely not be detected with passive monitoring alone because For each sequence seq ∈ seqSet, append each
the default state space exploration, performed by the main req ∈ reqCollection to the end of seq, execute
driver, would likely not attempt to re-use deleted resources the new sequence while considering the possible
or resources after a failure, respectively. Similarly, resource- renderings of req, and add to seqSet at most
hierarchy and user-namespace rule violations would not be one valid (if any) and one invalid (if any) sequence
detected by passive monitoring either because the baseline rendering.
main driver does not attempt to substitute object identifiers Valid renderings are used by the use-after-free, resource-
or authentication tokens, respectively. In other words, the hierarchy, and user-namespace checkers, while invalid render-
additional test cases generated by the checkers are necessary ings are used by the resource-leak checker.
to find rule violations and are not redundant with respect to BFS-Cheap thus provides a middle-ground between BFS
non-checker tests. and BFS-Fast (see Section IV-B for an experimental eval-
Complementarity among the checkers. The four checkers uation). It explores all possible request sequences up to a
we define complement each other: no two checkers will ever given sequence length (like BFS) and adds at most two new
generate the same new tests, by construction, because their renderings for each sequence in order to avoid an enormous
preconditions are all mutually exclusive. First, the use-after- seqSet (like BFS-Fast). Two new renderings per sequence
free checker is the only checker activated by request sequences explored allow for active checking of all the security rules
that end in a DELETE request. Second, the resource-leak defined in Section III-A while maintaining a tractable number
checker is the only checker activated when the last request of sequences in seqSet as the sequence length increases.
executed returns an invalid HTTP status code. Third, the Note that the suffix “cheap” comes from the fact BFS-Cheap
resource-ownership checker is the only checker activated on is a cheaper version of BFS where at most one valid rendering
request sequences with valid renderings that do not end in a is added to the BFS “frontier” setSeq for each new sequence.
DELETE request. Fourth and last, the user-namespace checker This leads to the creation of fewer resources than those
executed tests using an attacker token different from the created when all valid renderings of each request sequence are
authentication token used by the main driver and all other explored, as in BFS. For instance, imagine a request definition
checkers, so it clearly extends the state space in another, with an enum type describing ten different flavours of the
orthogonal dimension. same resource type. BFS-Cheap will stop creating resources
once one resource of one flavour is successfully created. In
D. Search Strategies for Checkers
contrast, BFS and BFS-Fast, will create ten resources of the
The main search strategy used for test generation in stateful same type with ten different flavours.
REST API fuzzing [5] is a breadth-first search (BFS) in the
search space defined by all possible request sequences. This E. Bug Bucketization
search strategy provides full grammar coverage both with Before discussing examples of real violations found with
respect to all possible renderings of each individual request active checkers, we define the bucketization scheme used to
and with respect to all possible request sequences up to group together similar violations. In the context of active
a given sequence length. However, since the search space checkers, we define “bugs” as rule violations. Each bug is
explored by BFS is typically enormous, the search does not associated with the request sequence that was executed to
scale well as the sequence length increases. Therefore, an trigger it. Given this property, we use the following procedure
optimization called BFS-Fast was introduced. With BFS-Fast, to create per-checker bug buckets:
whenever the search depth increases to a new value n + 1, Whenever a new bug is found, compute all non-
each request is appended to at most one request sequence empty suffixes of the request sequence that triggers
 Total Search Max Checker Stats
API Tests Main Checkers
Req. Strategy Len. Use-Aft-Free Leak Hierarchy NameSpace
Azure A 13 BFS 3 3255 48.1% 51.9% 11.5% 1.5% 0.1% 38.8%
BFS-Cheap 4 4050 55.0% 45.0% 10.0% 0.8% 2.4% 31.8%
BFS-Fast 9 4347 59.2% 40.8% 15.5% 0.2% 0.1% 25.1%
Azure B 19 BFS 5 7721 46.4% 53.6% 3.6% 0.4% 0.2% 49.4%
BFS-Cheap 5 7979 46.2% 53.8% 3.5% 0.4% 0.2% 49.7%
BFS-Fast 40 17416 65.3% 34.7% 0.3% 0.0% 0.1% 34.3%
O-365 C 18 BFS 3 11693 89.4% 10.6% 0.0% 1.0% 0.1% 9.5%
BFS-Cheap 4 10982 95.9% 4.1% 0.0% 0.0% 0.1% 4.0%
BFS-Fast 33 18120 66.9% 33.1% 0.0% 0.0% 0.1% 33.0%
TABLE I: Comparison of BFS, BFS-Fast and BFS-Cheap. Shows the maximum sequence length (Max Len.), the number of requests sent
(Tests), the percentage of tests generated by the main driver (Main) and by all four checkers combined (Checkers) and individually, with
each search strategy after 1 hour of search. The second column shows the total number of requests in each API.

the bug, starting with the smallest one. If a suffix in this section. There is no randomness in the renderings
exists in a previously-recorded bug bucket, add the generated. We ran our fuzzing experiments using a single-
new sequence to that existing bug bucket. Otherwise, threaded fuzzer running on a PC connected to the internet and
create a new bug bucket for the new sequence. a valid service subscription that allows access to each service
This bug bucketization scheme is the same as the one in API. No other special test setup or service knowledge was
stateful REST API fuzzing [5], but we maintain separate, per- required. As in [5], our fuzzer includes a garbage-collector that
checker bug buckets because the failure conditions are defined deletes no-longer-used resources (dynamic objects) in order to
differently for each rule. Each bug will always be triggered by avoid exceeding service quota limits.
one checker for a specific sequence length (because of checker We fuzz production services already deployed and acces-
complementarity), except for “500 Internal Server Error” bugs sible to anyone with a valid subscription, but we have no
which may be triggered by both the main driver and checkers. visibility as to what happens inside the backend of the services
For 500 bugs, the new sequence will be added only once to we test. Our fuzzer only observes the HTTP status codes of the
the bug bucket of the main driver or checker that triggered it responses it receives. All client-side requests are sent over the
first. internet to the target services, and responses are parsed when
they are received. Because we do not control the deployment
IV. E XPERIMENTAL E VALUATION of these services, the experiments reported in this section are
In this section, we report results of experiments with three not fully controlled. However, we performed these experiments
production cloud services. These services and our experimen- several times and the results did not vary significantly.
tal setup are described in Section IV-A. Then, we compare
in Section IV-B the three search strategies described in Sec- B. Comparing Search Strategies
tion III-D. Next, we present results showing the number of rule We now compare our new search strategy, BFS-Cheap, with
violations reported by each checker on the three cloud services BFS and BFS-Fast when using security checkers to fuzz real
as well as the impact of various optimizations (Section IV-C). services. We present results of experiments with two Azure
and one Office-365 services, denoted by Azure A, Azure B,
A. Experimental Setup and O-365 C respectively.
We report results of experiments performed with three cloud Table I shows individual experiments with the three search
services, whose names are anonymized (to avoid targeting strategies on each service, over a fixed time budget of one
them): Azure A and Azure B are two Azure [13] management hour per experiment. For each experiment, we report the total
services, and O-365 C is an Office365 [16] messaging service. number of requests in the API (Total Req.), the maximum
The number of requests in the REST API of each of these sequence length generated (Max Len.), the total number of
three services ranges from 13 to 19 requests. We selected requests sent (Tests), the percentage of the requests sent by
those three services because their size and complexity are the main driver (Main) and the active checkers (Checkers) as
representative among the cloud services we analyzed. So far, well as the individual contribution of each checker.
we have performed similar experiments with about a dozen Table I clearly shows that, for all services, BFS reaches
production services, and our general experience with these the smallest depth, BFS-Fast reaches the largest depth, and
other services is summarized in Section V. BFS-Cheap provides a trade-off between these two extremes,
Every service we consider has a publicly-available Swagger while being closer to BFS than BFS-Fast. The total number of
specification [15]. For each service, we compile its specifica- tests generated varies across services, depending on the speed
tion to produce a test-generation grammar, similarly to prior of the responses received from each service. For any given
work [5]. Each grammar is encoded as executable python code. service, this number remains roughly similar except for BFS-
For a given service and API, the same grammar and fuzzing FAST with Azure B and O-365 C where the total number of
dictionaries were used across all the experiments reported tests increases significantly. For O-365 C, this increase seems
 Total Statistics Bug Buckets
API Mode
Req.
Tests Checkers Main Use-Aft-Free Leak Hierarchy NameSpace
Azure A 13 optimized 4050 45.0% 4 3 0 0 0
exhaustive 2174 54.5% 4 3 0 0 0
Azure B 19 optimized 7979 46.2% 0 0 1 0 0
exhaustive 9031 63.9% 0 0 1 0 0
O-365 C 18 optimized 10982 4.1% 1 0 0 1 0
exhaustive 11724 11.4% 0 0 0 1 0

TABLE II: Comparison of modes optimized and exhaustive for two Azure and one Office-365 services. Shows the number of requests
sent in 1 hour (Tests) with BFS-Cheap, the percentage of tests generated by all four checkers combined (Checkers), and the number of bug
buckets found by the main driver and each of the four checkers. Optimized finds all the bugs found by exhaustive but its main driver explores
more states faster given a fixed test budget (1 hour).

to be due to a significantly lower number of failed requests We observe that the number of tests varies for different
generated by BFS-FAST for these two services compared to services and checker modes. However, the percentage of tests
BFS and BFS-Cheap. Such failed requests are sent back to the generated by the checkers is always higher with the exhaustive
client (our fuzzer) with larger time delays. Delaying responses mode, as expected. Since in the optimized mode the checkers
to failed requests is a well-known mechanism used by services produce fewer tests per visited state, the main driver is allowed
to throttle future requests, i.e., to try to slow them down. For to explore more states faster. Yet, despite the lower number of
Azure B, BFS-Fast executes more tests because its request checker tests per visited state, for all three services considered,
sequences are deeper but include many DELETE requests the optimized mode finds all the unique bugs (bug buckets)
which are faster to execute (their responses are received almost found by the exhaustive mode. Also, for the O-365 C service,
instantly): BFS-Fast executes about 9 times more DELETE the main driver finds one more bug with the optimized mode
requests than BFS or BFS-Cheap. compared to the exhaustive mode within one hour of search.
The total percentage of checker tests (Checkers) is the Table II reveals an interesting inversion that further demon-
highest for BFS and the lowest for BFS-FAST, while BFS- strates the value of the optimized checkers mode. In Azure
Cheap is again in between. Indeed, while BFS-Fast generates A, we observe that the optimized mode produces almost
the largest number of tests, its search space is pruned and twice as many tests than than the exhaustive mode (4050
activates checkers less often, as discussed in Section III-D versus 2174). At first sight, this is counter-intuitive. After
– this is the precise motivation for introducing BFS-Cheap a deeper investigation, we discovered that some of the tests
in that section. An exception is the 33% spike in checker- produced by the exhaustive mode of the user-namespace
generated tests by BFS-FAST for O-365 C. This spike seems checker have significantly larger response times for service
to be due to a larger number of successful requests (see the Azure A. Indeed, this specific checker in exhaustive mode
previous paragraph), which in turn led to more checker tests. executes additional tests compared to the optimized mode, but
From the individual checker statistics in Table I, we observe containing expensive operations (i.e., high latency) that slow
that the number of tests they each generate varies from service down the overall test throughput.
to service. This number depends on the number of DELETE During the course of all experiments with these three
requests executed for the use-after-free checker, the number of services, we found and reported a total of 7 unique bugs to
failed resource-creation requests for the resource-leak checker, the developers of those services, including 4 500 bugs found
and the depth of the object hierarchy for the resource-hierarchy by the main driver and 3 bugs found by each of the checkers
checker. In contrast, the user-namespace checker is triggered except the user-namespace checker. In the next section, we
more consistently more often and contributes the largest per- discuss several interesting bugs found thanks to the checkers
centage of checker-generated tests. introduced in this paper.
For all three services, the number of bugs found is nearly
the same for all three search strategies and is discussed next. V. E XAMPLES OF REST API S ECURITY
V ULNERABILITIES
C. Comparing Checker Optimizations At the time of this writing, we have fuzzed nearly a dozen
We now compare the performance of the two modes opti- production Azure and Office-365 cloud services of size and
mized and exhaustive discussed in Section III. complexity similar to the three services used in the previous
Table II shows how many requests were sent in one hour section. In almost all cases, our fuzzing was able to find about
of fuzzing with BFS-Cheap in the Tests column, and what a handful of new bugs in each of these services. About two
percentage of those requests were generated by either the main thirds of those bugs are “500 Internal Server Errors”, and
driver of Section II or by any of the four checkers. The table about one third are rule violations reported by our new security
also shows how many unique bugs (bug buckets) were found checkers. We reported these bugs to the service owners, and
in one hour of search by the main driver and by each of the all have been fixed.
checkers. Results are presented for both the optimized and the We emphasize that, even when the security checkers do
exhaustive modes previously discussed. not find any bugs, they increase confidence that the rules
they check cannot be violated and therefore they increase shows that the user view is correct: the CM resource named
confidence in the overall service reliability and security. X attempted to be created in Step 1 has not been created.
This section presents examples of real bugs found in However, the second PUT request in Step 3 proves that the
deployed Azure and Office-365 services and discuss their service still remembers the failed creation of the CM resource
security relevance. We anonymize the name of those services named X attempted in the first PUT request of Step 1. This
and key details not to target any specific service. bug is potentially dangerous: an attacker could create an
Use-after-free violation in Azure. In an Azure service, we unbounded number of such “zombie” resources by repeating
found the following use-after-free violation. Step 1 using many different names, and exceed his/her official
1) Create a new resource R (with a PUT request). quota since such failed resource creations are (correctly) not
2) Delete resource R (with a DELETE request). counted towards the user’ resource quota. Yet, they are clearly
3) Create a new child resource of the deleted resource R remembered (incorrectly) somewhere in the backend service.
and of a specific type (with another PUT request). Other Example: Eager Resource-Accounting DoS Attack.
This sequence of requests results in a “500 Internal Server Er- After fuzzing another Azure service for about five hours,
ror”. The Use-after-free checker catches this as (1) it attempts we accidentally triggered a severe health degradation for that
to re-use in Step 3 the deleted resource in Step 2 and (2) the service. We summarize here the findings on its root cause.
response of Step 3 is different from the expected “404 Not Our fuzzing tool uses a garbage collector not to exceed
Found” response. quotas for the cloud resources created during fuzzing. For
Resource-hierarchy violation in Office365. In an Office365 instance, if a default quota for a resource type Y is 100, at most
messaging service where users can post messages and then 100 resources of that type can be created at any time, and our
reply and edit these, the resource-hierarchy checker detected garbage collector makes sure that the number of live resources
the following bug. never exceeds quotas by deleting (using a DELETE request)
resources that are no longer used. Without garbage collection,
1) Create a first message msg-1 (with a request POST
our fuzzing tool would typically reach quota limits in minutes
/api/posts/msg-1).
and would not be able to continue state-space exploration.
2) Create a second message msg-2 (with a request POST
In this specific Azure service, any PUT request to create a
/api/posts/msg-2).
resource of a specific type, let us call it IM, returns a response
3) Create a reply reply-1 to the first message (with a
quickly but actually triggers other tasks that take minutes to
request POST /api/posts/msg-1/replies/reply-1).
complete in the service backend. Similarly, a DELETE request
4) Edit the reply reply-1 with a PUT request using
for an IM resource also returns quickly but also triggers
msg-2 as message identifier (with a request PUT
delete tasks that also take minutes to complete. However,
/api/posts/msg-2/replies/reply-1).
such PUT and DELETE requests for IM resources update
Surprisingly, the last request in Step 4 returns a “200 Al- counters towards quotas eagerly, too quickly, without waiting
lowed” response while it must have returned a “404 Not for the several minutes actually needed to fully complete these
Found” response. This rule violation reveals that the imple- tasks. As a result, an attacker could create-then-delete quickly
mentation of the API that posts a reply does not analyze many IM resources without exceeding his/her quota, while
the full hierarchy when checking permissions for a reply. triggering a huge number of backend tasks, hence literally
Missing hierarchy validation checks are potential security flooding the backend service. Such a Denial-of-Service attack
vulnerabilities: an attacker might be able to exploit them to was accidentally triggered by our fuzzing tool.
access child objects by bypassing the parent hierarchy. A fix to this vulnerability is to update usage counters
Resource-leak violation in Azure. In another Azure service, towards quotas for DELETE requests only when all delete
the resource-leak checker triggered the following bug. backend operations have been completed, i.e., minutes later
1) Create a new resource of type CM and of name X with in the case of IM resources. This way, the amount of backend
a specific malformed body (with a PUT request). This tasks is still linearly bounded by the official quota, since
returns a “500 Internal Server Error”, which is already subsequent IM resource-creation PUT requests will be blocked
a bug. until preceding DELETE requests have been fully completed.
2) Get a list of all resources of type CM: the returned list
is empty. VI. R ELATED W ORK
3) Create a new resource of type CM with the same name X Our work extends stateful REST API fuzzing [5]. Given
as in Step 1 with a well-formed body but in a different a Swagger specification of a REST API, this specification
region (e.g., US-West versus US-Central) with a PUT is compiled into a fuzzing grammar, which is then used to
request. automatically generate sequences of requests that satisfy the
Unexpectedly, the last request in Step 3 returns a response specification. Stateful REST API fuzzing automates the gener-
“409 Conflict” instead of an expected “200 Created”. This ation of a fuzzing grammar compared to traditional grammar-
behavior means that the service has reached an inconsistent based fuzzing [20], [22], [24] where the user manually writes a
state: the failed request in Step 1 has left unintended side- grammar. The BFS and BFS-Fast search strategies are inspired
effects on the service state. Indeed, the GET request in Step 2 by test generation algorithms used in model-based testing [27],
[12], [28] for generating minimal test suites that cover an entire of API requests and their responses as in traditional runtime
finite-state-machine model of a system under test. This paper verification [8], [11], but also generate new tests specifically
extends stateful REST API fuzzing (i) by introducing a set of aimed at triggering rule violations. Similarly to [10], we use
security rules for REST APIs and corresponding checkers for multiple independent security checkers simultaneously. But
efficiently testing and detecting violations of these rules; and unlike [10], we do not use symbolic execution, constraint
(ii) by introducing a new search strategy, BFS-Cheap, which generation and solving in order to generate new tests. Indeed,
offers a middle-ground between BFS and BFS-Fast when using the inner workings of the services we test are invisible to
active checkers. our fuzzing tool and its checkers, which only see REST
Since REST API requests and responses are transmitted over API requests and responses. Since cloud services are usually
the HTTP protocol, HTTP-fuzzers can be used to fuzz REST complex distributed systems whose components are written
APIs. Fuzzers like Burp [7], Sulley [23], BooFuzz [6], or in different languages, general symbolic-execution-based ap-
the commercial AppSpider [4] and Qualys’s WAS [21], can proaches seem problematic, but it would be worth exploring
capture/replay HTTP traffic, parse HTTP requests/responses this option further in future work.
and their contents (like embedded JSON data), and then fuzz In practice, the main technique used today to ensure the
those using either pre-defined heuristics [4], [21] or user- security of cloud services is penetration testing, or pen testing
defined rules [23], [6]. Some tools to capture, parse, fuzz, and for short, which means security experts review the architecture,
replay HTTP traffic have recently been extended to leverage design, and code of cloud services from a security perspective.
Swagger specifications in order to parse HTTP requests over Since pen testing is labor intensive, it is expensive and limited
REST APIs and guide their fuzzing [4], [21], [26], [3]. in scope and depth. Fuzzing tools and security checkers,
However, these tools do not perform any global analysis of like those discussed in this paper, can partly automate the
Swagger specifications and therefore cannot generate new discovery of specific classes of security vulnerabilities, and
sequences of requests: their fuzzing is stateless, i.e., restricted are complementary to pen testing.
to fuzzing parameter values of individual requests. Therefore,
adding active checkers to stateless fuzzers is problematic. In
contrast, our work extends stateful REST API fuzzing with VII. C ONCLUSION
active checkers targeting specific REST API rule violations.
Because most HTTP-fuzzers were born as extensions of We introduced four security rules that capture desirable
traditional web-page crawlers and scanners, they often support properties of REST APIs and services. We then showed how a
a long list of HTTP-focused properties they can check, such stateful REST API fuzzer can be extended with active property
as checking for proper HTTP-usage in responses and even checkers that automatically test and detect violations of these
checking for cross-site-scripting attacks or SQL-injections rules. So far, we have fuzzed nearly a dozen production Azure
if whole web-pages (with HTML and Javascript code) are and Office-365 cloud services using the fuzzer and checkers
returned as part of the responses. However, for most REST described in this paper. In almost all cases, our fuzzing was
APIs, responses do not include web-pages, and most of the able to find about a handful of new bugs in each of these
aforementioned checking capabilities are irrelevant. services. About two thirds of those bugs are “500 Internal
Compared to HTTP-fuzzers and web scanners, our paper Server Errors”, and about one third are rule violations reported
introduces new security rules that are targeted specifically at by our new security checkers. We reported all these bugs to
REST API usage. These rules are security-related because their the service owners, and all have been fixed.
violations might be exploited by a malicious attacker to harm Indeed, violations of the four security rules introduced in
the health of a service or steal unauthorized information or this paper are clearly potential security vulnerabilities. The
resources. In contrast, we do not discuss in this paper how bugs we found have all been taken seriously by the respective
to check other REST API usage rules [9], such as request service owners: our current bug “fixed/found” ratio is nearly
idempotence (i.e., repeating identical requests like GET or 100%. Moreover, it is safer to fix these bugs rather than risk
PATCH have no further effect on the outcome), which are a live incident – provoked intentionally by an attacker or
not “exploitable” when violated. triggered by accident – with unknown consequences. Finally,
Given the widespread use of REST APIs, there is sur- it helps that these bugs are easily reproducible and that our
prisingly little guidance provided on secure REST API us- fuzzing approach reports no false alarms.
age. Most of the security guidance from organisations like How general are these results? To find out, we need to
OWASP [19] (Open Web Application Security Project) or fuzz more services through their REST APIs and check
from books on REST APIs [1] or micro-services [17] is about more properties to detect different kinds of bugs and security
managing authentication tokens and API keys. No detailed vulnerabilities. Given the recent explosion of REST APIs for
guidance is provided regarding REST API input validation cloud and web services, there is surprisingly little guidance
and resource management. To the best of our knowledge, the about REST API usage from a security point of view. Our
four security rules introduced in this paper are new. paper makes a step in that direction by contributing four rules
In Section III, we used the term active checker from [10] whose violations are security-relevant and which are non-
to denote that our checkers do not simply monitor sequences trivial to check and satisfy.
 R EFERENCES
[1] S. Allamaraju. RESTful Web Services Cookbook. O’Reilly, 2010.
[2] Amazon. AWS. https://aws.amazon.com/.
[3] APIFuzzer. https://github.com/KissPeter/APIFuzzer.
[4] AppSpider. https://www.rapid7.com/products/appspider.
[5] V. Atlidakis, P. Godefroid, and M. Polishchuk. RESTler: Stateful REST
API Fuzzing. In 41st ACM/IEEE International Conference on Software
Engineering (ICSE’2019), May 2019.
[6] BooFuzz. https://github.com/jtpereyda/boofuzz.
[7] Burp Suite. https://portswigger.net/burp.
[8] D. Drusinsky. The Temporal Rover and the ATG Rover. In Proceedings
of the 2000 SPIN Workshop, volume 1885 of Lecture Notes in Computer
Science, pages 323–330. Springer-Verlag, 2000.
[9] R. T. Fielding. Architectural Styles and the Design of Network-based
Software Architectures. PhD Thesis, UC Irvine, 2000.
[10] P. Godefroid, M. Levin, and D. Molnar. Active Property Checking. In
Proceedings of EMSOFT’2008 (8th Annual ACM & IEEE Conference
on Embedded Software), pages 207–216, Atlanta, October 2008. ACM
Press.
[11] K. Havelund and G. Rosu. Monitoring Java Programs with Java
PathExplorer. In Proceedings of RV’2001 (First Workshop on Runtime
Verification), volume 55 of Electronic Notes in Theoretical Computer
Science, Paris, July 2001.
[12] R. Lämmel and W. Schulte. Controllable Combinatorial Coverage in
Grammar-Based Testing. In Proceedings of TestCom’2006, 2006.
[13] Microsoft. Azure. https://azure.microsoft.com/en-us/.
[14] Microsoft. Azure DNS Zone REST API. https://docs.microsoft.com/en-
us/rest/api/dns/zones/get.
[15] Microsoft. Microsoft Azure Swagger Specifications. https://github.com/
Azure/azure-rest-api-specs.
[16] Microsoft. Office. https://www.office.com/.
[17] S. Newman. Building Microservices. O’Reilly, 2015.
[18] OAuth. OAuth 2.0. https://oauth.net/.
[19] OWASP (Open Web Application Security Project). https://www.owasp.
org.
[20] Peach Fuzzer. http://www.peachfuzzer.com/.
[21] Qualys Web Application Scanning (WAS). https://www.qualys.com/
apps/web-app-scanning/.
[22] SPIKE Fuzzer. http://resources.infosecinstitute.com/fuzzer-automation-
with-spike/.
[23] Sulley. https://github.com/OpenRCE/sulley.
[24] M. Sutton, A. Greene, and P. Amini. Fuzzing: Brute Force Vulnerability
Discovery. Addison-Wesley, 2007.
[25] Swagger. https://swagger.io/.
[26] TnT-Fuzzer. https://github.com/Teebytes/TnT-Fuzzer.
[27] M. Utting, A. Pretschner, and B. Legeard. A Taxonomy of Model-Based
Testing Approaches. Intl. Journal on Software Testing, Verification and
Reliability, 22(5), 2012.
[28] M. Yannakakis and D. Lee. Testing Finite-State Machines. In Proceed-
ings of the 23rd Annual ACM Symposium on the Theory of Computing,
pages 476–485, 1991.