The Inevitability of Failure: The Flawed Assumption of

Security in Modern Computing Environments

Peter A. Loscocco, Stephen D. Smalley, Patrick A. Muckelbauer, Ruth C. Taylor,

S. Jeff Turner, John F. Farrell

[email protected]

National Security Agency

Abstract reality, operating system security mechanisms play a criti­
cal role in supporting security at higher levels. This has
Although public awareness of the need for secu­
been well understood for at least twenty five years
rity in computing systems is growing rapidly, current
[2][54][39], and continues to be reaffirmed in the literature
efforts to provide security are unlikely to succeed.
[1][35]. Yet today, debate in the research community as to
Current security efforts suffer from the flawed
what role operating systems should play in secure systems
assumption that adequate security can be provided in
persists [11]. The computer industry has not accepted the
applications with the existing security mechanisms of
critical role of the operating system to security, as evi­
mainstream operating systems. In reality, the need for
denced by the inadequacies of the basic protection mecha­
secure operating systems is growing in today’s com­
nisms provided by current mainstream operating systems.
puting environment due to substantial increases in
connectivity and data sharing. The goal of this paper The necessity of operating system security to overall
is to motivate a renewed interest in secure operating system security is undeniable; the underlying operating
systems so that future security efforts may build on a system is responsible for protecting application-space
solid foundation. This paper identifies several secure mechanisms against tampering, bypassing, and spoofing
operating system features which are lacking in main- attacks. If it fails to meet this responsibility, system-wide
stream operating systems, argues that these features vulnerabilities will result.
are necessary to adequately protect general applica­ The need for secure operating systems is especially
tion-space security mechanisms, and provides con­ crucial in today’s computing environment. Substantial
crete examples of how current security solutions are increases in connectivity and data sharing have increased
critically dependent on these features. the risk to systems such that even a careful and knowl­
Keywords: secure operating systems, mandatory se­ edgeable user running on a single-user system is no longer
curity, trusted path, Java, Kerberos, IPSEC, SSL, fire- safe from the threat of malicious code. Because the dis­
walls. tinction between data and code is vanishing, malicious
code may be introduced, without a conscious decision on
1 Introduction the part of a user to install executable code, whenever data
Public awareness of the need for security in com­ is imported into the system. For example, malicious code
puting systems is growing as critical services are could be introduced with a Java applet or by viewing
becoming increasingly dependent on interconnected apparently benign data that, in actuality, contains execut­
computing systems. National infrastructure compo­ able code [32][62]. More so than ever, secure operating
nents such as the electric power, telecommunication systems are needed to protect against this threat.
and transportation systems can no longer function
The goal of this paper is to motivate a renewed inter­
without networks of computers [50]. The advent of
est in secure operating systems. By consolidating a num­
the World Wide Web has especially increased public
ber of well-documented examples from the literature, it
concern for security. Security is the primary concern
argues that the threats posed by the modern computing
of businesses which want to use the Internet for com­
environment cannot be addressed without support from
merce and maintaining business relationships [24].
secure operating systems and, as was stated in [8], that any
The increased awareness of the need for security security effort which ignores this fact can only result in a
has resulted in an increase of efforts to add security to “fortress built upon sand.” Section 2 describes a set of
computing environments. However, these efforts suf­ secure operating system features which are typically lack­
fer from the flawed assumption that security can ade­ ing in mainstream operating systems but are crucial to
quately be provided in application space without information security. The need for these features is high-
certain security features in the operating system. In
lighted in section 3, which examines how application- role-based access control [22] and type enforcement
space access control and cryptography cannot provide [39][7][13].1
meaningful security without a secure operating system. Likewise, as defined in [59], this paper uses a more
Section 4 provides concrete examples of how security general notion of discretionary security in which a dis­
efforts rely on these operating system security features. cretionary security policy is considered to be any secu­
Section 5 discusses the role of operating system security rity policy where ordinary users may be involved in the
with respect to overall system security. definition of the policy functions and/or the assignment
2 The Missing Link of security attributes. Here discretionary security is not
synonymous with identity based access control; IBAC,
This section identifies some features of secure oper­
like any other security policy, may be either mandatory
ating systems which are necessary to protect applica­
or discretionary[58].
tion-space security mechanisms yet are lacking in
mainstream operating systems. They form the “missing An operating system’s mandatory security policy
link” of security. Although this section only deals with may be divided into several kinds of policies, such as an
features, it is important to note that features alone are access control policy, an authentication usage policy,
inadequate. Assurance evidence must be provided to and a cryptographic usage policy. A mandatory access
demonstrate that the features meet the desired system control policy specifies how subjects may access objects
security properties and to demonstrate that the features under the control of the operating system. A mandatory
are implemented correctly. Assurance is the ultimate authentication usage policy specifies what authentica­
missing link; although approaches to providing assur­ tion mechanisms must be used to authenticate a princi­
ance may be controversial, the importance of assurance pal to the system. A mandatory cryptographic usage
is undeniable. policy specifies what cryptographic mechanisms must
be used to protect data. Additionally, various sub-
The list of features in this section is not intended to
systems of the operating system may have their own
be exhaustive; instead it is merely a small set of critical
mechanism usage policies. These subsystem-specific
features that demonstrate the value of secure operating
usage policies may be dependent on the cryptographic
systems. A more complete discussion on secure operat­
usage policy. For example, a network usage policy for a
ing systems, including discussions of assurance, can be
router might specify that sensitive network traffic should
found in [25], [59] or [20]. Subsequent sections argue
be protected using IPSEC ESP [4] in tunneling mode
the necessity of these features by describing how appli­
prior to being sent to an external network. The selection
cation-space security mechanisms and current security
of a cryptographic algorithm for IPSEC ESP may be
efforts employing them are vulnerable in their absence.
deferred to the cryptographic usage policy.
Mandatory security A secure system must provide a framework for
The TCSEC [20] provides a narrow definition of defining the operating system’s mandatory security pol-
mandatory security which is tightly coupled to the icy and translating it to a form interpretable by the
multi-level security policy of the Department of underlying mandatory security mechanisms of the oper­
Defense. This has become the commonly understood ating system. Without such a framework, there can be
definition for mandatory security. However, this defini­ no real confidence that the mandatory security mecha­
tion is insufficient to meet the needs of either the nisms will provide the desired security properties.
Department of Defense or private industry as it ignores
An operating system which provides mandatory
critical properties such as intransitivity and dynamic
security may nonetheless suffer from the presence of
separation of duty [12][22]. This paper instead uses the
high bandwidth covert channels. This is an issue when-
more general notion of mandatory security defined in
ever the mandatory security policy is concerned with
[59], in which a mandatory security policy is considered
confidentiality. This should not, however, be a reason to
to be any security policy where the definition of the pol-
ignore mandatory security. Even with covert channels,
icy logic and the assignment of security attributes is
an operating system with basic mandatory controls
tightly controlled by a system security policy adminis­
improves security by increasing the required sophistica-
trator. Mandatory security can implement organization-
wide security policies. Others have referred to this same
concept as non-discretionary security in the context of 1. Actually, long ago, the term non-discretionary controls
was used for multi-level security as well [39].
tion of the adversary. Once systems with basic manda­ ment environment [48]. For example both the
tory controls become mainstream, covert channel Sidewinder firewall and the DTE firewall use type
exploitation will become more common and public enforcement for confinement [6][12].
awareness of the need to address covert channels in Although one could attempt to enforce a mandatory
computing systems will increase[57]. security policy through discretionary security mecha­
In any system which supports mandatory security, nisms, such mechanisms can not defend against careless
some applications require special privileges in the man­ or malicious users. Since discretionary security mecha­
datory policy in order to perform some security-relevant nisms place the burden for security on the individual
function. Such applications are frequently called trusted users, carelessness by any one user at any point in time
applications because they are trusted to correctly per- may lead to a violation of the mandatory policy. In con­
form some security-related function and because they trast, mandatory security mechanisms limit the burden
are trusted to not misuse privileges required in order to to the system security policy administrator. With only
perform that function. If the mandatory security mecha­ discretionary mechanisms, a malicious user with access
nisms of a secure operating system only support coarse- to sensitive data and applications may directly release
grained privileges, then the security of the overall sys­ sensitive information in violation of the mandatory pol-
tem may devolve to the security of the trusted applica­ icy. Although that same user may also be able to leak
tions on the system. To reduce the dependency on sensitive information in ways that do not involve the
trusted applications, the mandatory security mecha­ computing system, the ability to leak the information
nisms of an operating system should be designed to sup- through the computing system may increase the band-
port the principle of least privilege. Type enforcement is width of the leak and may decrease its traceability. In
an example of a mandatory security mechanism which contrast, with mandatory security mechanisms, he may
may be used both to limit trusted applications to the only leak sensitive information through covert channels,
minimal set of privileges required for their function and which limits the bandwidth and increases accountability,
to confine the damage caused by any misuse of these if covert channels are audited.
privileges [48][28]. Furthermore, even with users who are benign and
The mandatory security mechanisms of an operat­ careful, the mandatory security policy may still be sub­
ing system may be used to support security-related func­ verted by flawed or malicious applications when only
tionality in applications by rigorously ensuring that discretionary mechanisms are used to enforce it.2 The
subsystems are unbypassable and tamperproof. For distinction between flawed and malicious software is not
example, type enforcement may be used to implement particularly important in this paper. In either case, an
assured pipelines to provide these properties. An application may fail to apply security mechanisms
assured pipeline ensures that data flowing from a desig­ required by the mandatory policy or may use security
nated source to a designated destination must pass mechanisms in a way that is inconsistent with the user’s
through a security-related subsystem and ensures the intent. Mandatory security mechanisms may be used to
integrity of the subsystem. Many of the security require­ ensure that security mechanisms are applied as required
ments of these applications may be ensured by the and can protect the user against inadvertent execution of
underlying mandatory security mechanisms of the oper­ untrustworthy applications. Although the user may have
ating system. [48] carefully defined the discretionary policy to properly
Operating system mandatory security mechanisms implement the mandatory policy, an application may
may also be used to rigorously confine an application to change the discretionary policy without the user’s
a unique security domain that is strongly separated from approval or knowledge. In contrast, the mandatory pol-
other domains in the system. Applications may still mis­ icy may only be changed by the system security policy
behave, but the resulting damage can now be restricted administrator.
to within a single security domain. This confinement In the case of personal computing systems, where
property is critical to controlling data flows in support of the user may be the system security policy administra­
a system security policy [33]. In addition to supporting tor, mandatory security mechanisms are still helpful in
the safe execution of untrustworthy software, confine­
ment may support functional requirements, such as an
2. A discussion of the formal limitations of discretionary
isolated testing environment or an insulated develop­ security mechanisms appears in [29].
protecting against flawed or malicious software. In the Mach microkernel; both systems provide mechanisms
simplest case, where there is only a distinction between for mandatory access control and a mandatory policy
the user’s ordinary role and the user’s role as system framework.
security policy administrator, the mandatory security
Trusted path
mechanisms can protect the user against unintentional
A trusted path is a mechanism by which a user may
execution of untrustworthy software. With a further sub-
directly interact with trusted software, which can only
division of the user’s ordinary role into various roles
be activated by either the user or the trusted software
based on function, mandatory security mechanisms can
and may not be imitated by other software [20]. In the
confine the damage that may be caused by flawed or
absence of a trusted path mechanism, malicious soft-
malicious software.
ware may impersonate trusted software to the user or
Although there are a number of commercial operat­ may impersonate the user to trusted software. Such
ing systems with support for mandatory security, none malicious software could potentially obtain sensitive
of these systems have become mainstream. These sys­ information, perform functions on behalf of the user in
tems have suffered from a fixed notion of mandatory violation of the user’s intent, or trick the user into
security, thereby limiting their market appeal. Further- believing that a function has been invoked without actu­
more, these systems typically lack adequate support for ally invoking it. In addition to supporting trusted soft-
constraining trusted applications. In order to reach a ware in the base system, the trusted path mechanism
wider market, operating systems must support a more should be extensible to support the subsequent addition
general notion of mandatory security and must support of trusted applications by a system security policy
flexible configuration of mandatory policies. administrator [28].
Mainstream commercial operating systems rarely The concept of a trusted path can be generalized to
support the principle of least privilege even in their dis­ include interactions beyond just those between trusted
cretionary access control architecture. Many operating software and users. The TNI introduces the concept of a
systems only provide a distinction between a completely trusted channel for communication between trusted soft-
privileged security domain and a completely unprivi­ ware on different network components [44]. More gen­
leged security domain. Even in Microsoft Windows NT, erally, a mechanism that guarantees a mutually
the privilege mechanism fails to adequately protect authenticated channel, or protected path, is necessary to
against malicious programs because it does not limit the ensure that critical system functions are not being
privileges that a program inherits from the invoking pro­ spoofed. Although a protected path mechanism for local
cess based on the trustworthiness of the program [65]. communications could be constructed in application
Current microkernel-based research operating sys­ space without direct authentication support in the oper­
tems have tended to focus on providing primitive protec­ ating system, it is preferable for an operating system to
tion mechanisms which may be used to flexibly provide its own protected path mechanism since such a
construct a higher-level security architecture. Many of mechanism will be simpler to assure [59] and is likely to
these systems, such as the Fluke microkernel [23] and be more efficient.
the Exokernel [41], use kernel-managed capabilities as Most mainstream commercial operating systems
the underlying protection mechanism. However, as dis­ are utterly lacking in their support for either a trusted
cussed in [59], typical capability architectures are inade­ path mechanism or a protected path mechanism.
quate for supporting mandatory access controls with a Microsoft Windows NT does provide a trusted path for a
high degree of flexibility and assurance. L4 [38] pro­ small set of functions such as login authentication and
vides some support for mandatory controls through its password changing but lacks support for extending the
clans and chiefs mechanism and its IPC mechanism for trusted path mechanism to other trusted applications
identifying senders and receivers but still lacks a coher­ [65]. For local communications, NT does provide serv­
ent framework for using these mechanisms to meet the ers with the identity of their clients; however, it does not
requirements of a mandatory policy. Furthermore, L4 provide the server identity to the client.
assumes that there will only be a small number of dis­
tinct security domains [38]. Flask [56], a variant of the 3 General Examples
Fluke microkernel, provides a mandatory security This section argues that without operating system
framework similar to that of DTOS [43], a variant of the support for mandatory security and trusted path, appli-
cation-space mechanisms for access control and cryp­ operating system may be used to ensure that all accesses
tography cannot be implemented securely. These to the protected objects are mediated by the enforcer
arguments will then be used to reinforce the discussion component.
in section 4, which analyzes concrete examples.
3.2 Cryptography
3.1 Access Control An analysis of application-space cryptography may
An application-space access control mechanism be decomposed into an analysis of the invocation of the
may be decomposed into an enforcer component and a cryptographic mechanism and an analysis of the crypto­
decider component. When a subject attempts to access graphic mechanism itself. The analysis of this section
an object protected by the mechanism, the enforcer draws from the discussions in [51][15] [60][61][55][52].
component must invoke the decider component, supply­ As an initial basis for discussion, suppose that the
ing it with the proper input parameters for the policy cryptographic mechanism is a hardware token that
decision, and must enforce the returned decision. A implements the necessary cryptographic functions cor­
common example of the required input parameters is the rectly and that there is a secure means by which the
security attributes of the subject and the object. The cryptographic keys are established in the token. Even in
decider component may also consult other external this simplified case, where the confidentiality and integ­
sources in order to make the policy decision. For exam­ rity of algorithms and keys is achieved without operat­
ple, it may use an external policy database and system ing system support, this section will demonstrate that
information such as the current time. there are still vulnerabilities which may only be effec­
If a malicious agent can tamper with any of the tively addressed with the features of a secure operating
components in the access control mechanism or with system.
any inputs to the decision, then the malicious agent can One vulnerability in this simplified case is that
subvert the access control mechanism. Even if the com­ invocation of the token cannot be guaranteed. Any legit­
ponents and all of the inputs are collocated within a sin­ imate attempt to use the token might not result in a call
gle file, the operating system security mechanisms are to the token. The application that performs the crypto­
still relied upon to protect the integrity of that file. As graphic invocation might be bypassed or modified by
discussed in the prior section, only mandatory security malicious applications or malicious users. Malicious
mechanisms can rigorously provide such integrity guar­ applications might impersonate the cryptographic token
antees. to the invoking application.
Even with strong integrity guarantees for the policy Mandatory security and protected path features in
decision inputs, if an authorized user invokes malicious the operating system address this vulnerability. Manda­
software, the malicious software could change an tory security mechanisms may be used to ensure that the
object’s security attributes or the policy database’s rules application that invokes the cryptographic token is
without the user’s knowledge or consent. The access unbypassable and tamperproof against both malicious
control mechanism requires a trusted path mechanism in software and malicious users. Unbypassability could
the operating system in order to ensure that arbitrary also be achieved by using an inline cryptographic token,
propagation of access cannot occur without explicit which is physically interposed between the sender of the
authorization by a user. data to be protected and the receiver of the protected
If a malicious agent can impersonate the decider data; however, this would be less flexible. A protected
component to the enforcer component, or if a malicious path mechanism may be used to ensure that malicious
agent can impersonate any source of inputs to the deci­ software cannot impersonate the cryptographic token to
sion, then the malicious agent can subvert the mecha­ the invoking application.
nism. If any of the components or external decision Misuse of the cryptographic token is a second vul­
input sources are not collocated within a single applica­ nerability in the simplified case. Misuse may involve the
tion, then the access control mechanism requires a pro­ use of a service, algorithm, session or key by an unau­
tected path mechanism. thorized application. Without operating system support
If a malicious agent can bypass the enforcer compo­ for identifying callers, a cryptographic token can do lit­
nent, then it may trivially subvert the access control tle more than require that a user activate it, after which,
mechanism. Mandatory security mechanisms in the any service, algorithm, session or key authorized for that
user may be used by any application on the system. In If the assumption that the cryptographic mechanism
this case, the cryptographic token may be misused by is confined to a single hardware token is removed and
applications operating on behalf of other users or may implemented in software instead, the confidentiality and
be misused by malicious software operating on behalf of integrity of the cryptographic mechanism’s code and
the authorized user. Furthermore, unless the crypto­ data becomes dependent on the operating system,
graphic token has a direct physical interface for user including both memory protection and file protection.
activation, malicious software can spoof the token to the Mandatory security is needed to rigorously ensure the
user, obtain authentication information, and subse­ mechanism’s integrity and confidentiality. If any exter­
quently activate the cryptographic token without the nal inputs, such as input parameters to a random number
user’s knowledge or consent. Even with a direct physi­ generator, are used by the cryptographic mechanism, the
cal interface to the user, it is impractical for the crypto­ input sources and the path between the input sources
graphic token to require user confirmation for every and the cryptographic mechanism must be protected
cryptographic operation. with mandatory security mechanisms.
This second vulnerability may be addressed 4 Concrete Examples
through mandatory security, trusted path and protected
This section further demonstrates that secure oper­
path features in the operating system. A trusted path
ating systems are necessary by showing that some
mechanism obviates the need for a separate physical
widely accepted security solutions critically rely on the
interface for activation. A protected path mechanism
features of secure operating systems. In particular, this
permits the cryptographic token to identify its callers
section examines mobile code security efforts, the Ker­
and enforce fine-grained controls over the use of ser­
beros network authentication system, firewalls and net-
vices, algorithms, sessions and keys. As an alternative to
work security protocols.
having the token deal with fine-grained controls over its
usage, mandatory security mechanisms may also be 4.1 Mobile Code
used to provide such controls. For example, mandatory
A number of independently-developed security
security mechanisms may be used to isolate the token
solutions for the World Wide Web, each with its own
for use only by applications executed by the user who
protection model, have been developed to protect
activated the token. Furthermore, the mandatory secu­
against the threats from malicious mobile code. How-
rity mechanisms can reduce the risk of malicious soft-
ever, systems relying on these security solutions are vul­
ware being able to use the cryptographic token and may
nerable because of a lack of operating system support
consequently limit the use of the trusted path mecha­
for security. Primarily, this section will emphasize this
nism to highly sensitive actions.
point by focusing on efforts to secure Java [27], but
Hence, even in the simplest case, the features of a other efforts will also be used to highlight issues.
secure operating system are crucial to addressing the
The primary threat that these solutions attempt to
vulnerabilities of application-space cryptography. In the
address is the threat of hostile mobile code gaining
remainder of this section, the assumptions of the simpli­
unauthorized access to a user’s files and resources in
fied case are removed, and the additional vulnerabilities
order to compromise confidentiality or integrity. The
are examined.
threat is not limited to interpreted applets loaded from
If the assumption that initial keys are securely the network by a web browser; both [26] and [30]
established within the token is removed, then there is the extend this threat model to include helper applications
additional vulnerability that the initial keys may be which may have been actively installed by a user. There
observed or modified by an unauthorized entity. Unless is little distinction between mobile code and what is tra­
the initial keys are provided via a dedicated physical ditionally considered data. For example, consider that
interface to the cryptographic token, the operating sys­ Postscript documents are actually programs with poten­
tem must protect the path between the initial key source tial access to the local filesystem. Consequently, helper
and the cryptographic token and may need to protect the applications which operate on untrustworthy data, such
initial key source itself. Mandatory security mechanisms as Postscript viewers, must either be executed in a less
may be used to rigorously protect the path and the key flexible mode of operation, or must be carefully con-
source. A trusted path may be required for initial keying. fined by the operating system.
 The basic Java Security Model is based on the can reduce the risk of malicious code entering the sys­
notion of “sandboxing.” The system relies on the type- tem, provide some measure of trust that an applet will
safety of the language in conjunction with the Java behave properly, and provide another piece of informa­
Security Manager to prevent unauthorized actions [27]. tion to use in making an access control decision. How-
Efforts are currently underway to add additional security ever, as with the general application-space cryptography
features to Java, such as capabilities, an expanded described in section 3.2, the digital signature verifica­
access control model, or additional controls over access tion mechanism depends on secure operating system
to certain class libraries [70]. features to guarantee invocation, to protect the integrity
The fundamental limitation of these approaches is of the mechanism, and to protect the integrity of the
that none can be guaranteed to be tamperproof or unby­ locally cached public keys.
passable. For example, although the Java language is The need for an operating system trusted path
claimed to be secure, the Java Virtual Machine (JVM) mechanism was highlighted by [67] which demonstrates
will accept byte code which violates the language the ease with which a trojan horse applet can capture
semantics and which can lead to security violations [32]. credit card numbers, PIN numbers or passwords by per­
JVM implementation errors have led to violations of the fectly emulating a window system dialog box. The pro-
language’s semantics [19]. A significant portion of the posed solution was an ad hoc user-level trusted path
Java system is currently in the form of native methods mechanism which required a user to customize his dia­
which are implemented as object code and are not sub­ log box with a complicated graphical pattern. This solu­
ject to the JVM’s type-safety checks. The JVM is not tion is not adequate as it only increases the
able to protect itself from tampering by other applica­ sophistication required in the trojan horse.
tions. Finally, the Java security model can offer no pro­ Other systems attempt to provide alternative secu­
tection from the many other forms of malicious mobile rity solutions to the mobile code threat. The Janus sys­
code. In [30], the authors call for trusted systems to sup- tem [26] interposes on Solaris system calls to constrain
port a system-wide solution to address the threats pre­ untrusted native applications, and Safe-Tcl [49] pro­
sented by non-Java code. vides a “safe interpreter” which attempts to limit the
Even if such problems with the JVM did not exist, command set available to untrusted code. However, like
these security solutions would still suffer from the fun­ the Java security solutions, these systems are subject to
damental limitation that they rely on application-space the same vulnerabilities as any other application-space
access control for security. They all depend on the local access control mechanism; consequently, they require
file system to preserve the integrity of the system code, secure operating system support.
including class files. All of the systems which store pol- Beyond enabling all of the mobile code systems
icy locally depend on file system access control to pre- mentioned above to function securely, a secure system
serve the integrity of the policy files. Section 3.1 could also simplify them. Rather than implementing
demonstrated the importance of secure operating system their security primitives in application space where they
features for supporting application-space access control. are vulnerable, they could utilize the system security
Another popular approach to “securing” mobile services to provide a better overall system. A properly
code is to require digitally signed applets and limit exe­ designed secure system would provide a flexible, eco­
cution to those originating from trusted sources [27]. In nomic foundation with one consistent security model for
fact, native ActiveX security is based entirely on digital all of the different virtual machine efforts to use.
signatures, as it has no form of access control [24][27].
The basic flaw with this approach is that it is an all-or- 4.2 Kerberos
nothing proposition; the user cannot constrain a native Kerberos [31][47] is a network authentication ser­
ActiveX control to a limited security domain. Manda­ vice originally developed for Project Athena at MIT. In
tory security mechanisms in the operating system may addition to providing an authentication service, Ker­
be used for this purpose, by confining the browser to a beros supports the establishment of session keys to sup-
distinct security domain. port network confidentiality and integrity services.
Note that, although not sufficient by themselves, Derivatives of Kerberos have been used to provide
digital signatures will play an important part in mobile authentication and key establishment services for AFS
code security, even on secure operating systems. They [64], DCE [53], and ONC RPC [21]. Kerberos and sys-
tems that rely on Kerberos have been suggested as a more, suppose that the client workstation is a single-user
means of providing security for the World Wide Web system which does not export any services to other sys­
[18][36][37]. tems. In spite of these assumptions, a user is still vulner­
Kerberos is based on symmetric cryptography with able to attacks by malicious software, such as mobile
a trusted key distribution center (KDC) for each realm. code downloaded by the user.
The Kerberos KDC has access to the secret key of every If the malicious software could spoof the client-side
principal in its realm. Consequently, a compromise of authentication program to the user, then it may be able
the KDC can be catastrophic. This is generally to obtain a user’s password. Even with one-time pass-
addressed by requiring that the KDC be both physically words, this attack would permit the malicious software
secure and dedicated solely to running the Kerberos to act on behalf of the user during the login session. A
authentication server [46].3 A typical environment also trusted path mechanism in the client workstation’s oper­
uses physically-secure dedicated systems for the servers ating system can be used to prevent such an attack.
using Kerberos. Without these environmental assump­ Additionally, such a trusted path mechanism in combi­
tions, the Kerberos authentication service and the Ker­ nation with support for a network protected path can be
berized server applications would require secure used to provide a trusted path between users and server
operating system features to rigorously ensure that they applications.
are tamperproof and unbypassable. For the sake of argu­ If the malicious software can read the files used by
ment, the remainder of this section will consider these the Kerberos client software to store tickets and session
environmental assumptions to be true and focus only on keys, then the malicious software may directly imper­
the security of the client workstations. sonate the user to the corresponding Kerberized server
Kerberos was designed for an environment where applications. Even if the session keys are encapsulated
the client workstations and the network are assumed to within a hardware cryptographic token, the malicious
be completely untrustworthy [10][45]. However, since software can invoke the cryptographic token on behalf
the software on the client workstation mediates all inter- of the user, exploiting the misuse vulnerability discussed
actions between its user and the Kerberized server appli­ in section 3.2. Mandatory security mechanisms can be
cations, this assumption implies that the Kerberized used to rigorously protect either the file or the crypto­
server applications must view all client applications as graphic token against access by malicious software.
potentially malicious software. Furthermore, a Kerber­
ized server application has no means of establishing a 4.3 Network Security Protocols
trusted path to a user on a client workstation, since that The IPSEC network security protocols [5][3][4] are
would require trusted code on the client workstation. used to provide authentication, integrity, and confidenti­
Thus, in a system that uses Kerberos, malicious software ality services at the IP layer. Typical implementations of
executed by a user is free to arbitrarily modify or leak a the IPSEC protocols rely on application-space key man­
user’s information, with no means of confinement; no agement servers to perform key exchanges and supply
distinctions between a user’s legitimate requests and the keys for security associations. The IPSEC module in the
requests of malicious software are possible. Given the network stack communicates with the local key manage­
increasing ease with which malicious software may be ment server via upcalls to retrieve the necessary infor­
introduced into a system, the Kerberos environmental mation.
model seems untenable. As noted in [14], secure end-to-
SSL [69] is another network security protocol that
end transactions require trusted code at both end points.
provides authentication, integrity, and confidentiality
As a basis of further discussion, suppose that there services and a negotiation service for keys and crypto­
is a base set of trustworthy software on the client work- graphic algorithms. SSL, however, is implemented
stations which is protected against tampering, but that entirely in application space and requires no kernel
the client workstation operating system still lacks mech­ modifications. SSL has been implemented as a library
anisms for mandatory security and trusted path. Further- that interposes on socket calls to incorporate the SSL
protocol between the underlying transport protocol of
3. Variants of Kerberos have been proposed that use asym­ the socket (e.g., TCP) and the application protocol (e.g.,
metric cryptography either to reduce the cost incurred by a
penetration of the KDC or to completely eliminate the HTTP).
need for the KDC [63] [66][42][18].
 Since it relies on application-space cryptography, through the firewall. Malicious insiders can construct
the key management server used by IPSEC is subject to tunnels to permit outsiders to perform inbound calls
the vulnerabilities described in section 3.2 and requires through the firewall or may provide ways of bypassing a
mandatory security mechanisms in the operating system firewall entirely. Additionally, malicious insiders can
for adequate protection. In turn, since the protection exploit data leaked between users within the firewall.
provided by IPSEC depends on the protection of the Although internal firewalls may be used to partition
keys, mandatory security mechanisms in the operating insiders into multiple trust classes, the granularity of
system are also crucial to meeting the security require­ protection is quite limited in comparison to what can be
ments of IPSEC. Since the complete SSL implementa­ provided by a secure operating system.
tion operates in application space, it is directly subject to The ability of malicious insiders to leak data
the vulnerabilities described in section 3.2 and requires through the firewall can be confined by mandatory secu­
mandatory security mechanisms in the operating system rity mechanisms in the operating systems of the internal
for adequate protection. hosts. Likewise, mandatory security mechanisms in the
Both IPSEC and SSL are intended to provide secure operating systems of the internal hosts can confine out­
channels. However, as noted in [14], an end-to-end siders who perform inbound calls through tunnels con­
secure transaction requires a secure channel and secure structed by a malicious insider to the security domains
end points. If an attacker can penetrate one of the end in which the malicious insider is allowed to operate.
points and directly access the unprotected data, then the In addition to the threat of malicious insiders, a fire-
protection provided by IPSEC and SSL is only illusory. wall is at risk from the threat of malicious software exe­
cuted by benign insiders. Typically, firewalls do not
4.4 Firewalls
require that insiders strongly authenticate themselves to
A network firewall is a mechanism for enforcing a the firewall in order to access external services through
trust boundary between two networks. The analysis of the firewall [40]. Hence, if a benign insider executes
this section is based on the discussions in [17][9][11][6]. malicious software on an internal host, the malicious
Commonly, firewalls are used to maintain a separation software may seek to subvert the protection of the fire-
between insiders and outsiders for an organization’s wall in the same fashion as a malicious insider. An
computing resources. Internal firewalls may also be example of using a malicious Java applet to enable out­
used to provide separation between different groups of siders to penetrate a firewall is given in [40]. Even if
insiders or to provide defense-in-depth against outsid­ insiders are required to strongly authenticate themselves
ers. to the firewall, a benign insider may still execute a trojan
Modern firewall architectures typically involve the horse whose overt purpose requires external access; in
use of bastion hosts; in a screened subnet architecture, this case, the malicious software may still subvert the
there may be an external bastion host on a perimeter net- protection of the firewall.
work, which is highly exposed to outsiders, and an inter­ Mandatory security mechanisms in the operating
nal bastion host on the internal network, which is systems of the internal hosts may be used to protect
exposed to the external bastion host. The security of the users against execution of malicious software or to con-
bastion hosts is crucial to the security provided by the fine such software when it is executed. If strong authen­
firewall. To reduce risk, bastion hosts are typically dedi­ tication is required prior to accessing external services,
cated systems, only providing the minimal services mandatory security mechanisms could be used to ensure
required. Even with such minimal configuration, flaws that only trustworthy software on the internal hosts can
in the proxy servers on the bastion host may permit pen­ communicate with the strong authentication mechanism
etration. However, mandatory security mechanisms in on the firewall. In any case, the mandatory security
the operating systems of the bastion hosts may be used mechanisms would limit the ability of malicious soft-
to confine proxy servers so that penetrations are nar­ ware to leak information or support inbound calls.
rowly limited. Similarly, the bastion host’s mandatory
Firewalls are also susceptible to malicious data
security mechanisms may be used to protect proxy serv­
attacks [62]. Some example malicious data attacks rele­
ers against tampering.
vant to firewalls are described in [68][40][16]. As with
Firewalls provide no protection against malicious malicious insiders and malicious software, mandatory
insiders. Typically, insiders can easily leak information security mechanisms in the operating systems of the
bastion hosts and the internal hosts may be used to con- assumptions will result in residual vulnerabilities. As an
fine malicious data attacks. example, covert channels remain a serious technical
When inbound services are supported by a firewall, challenge for secure operating system designers. These
the firewall itself cannot protect the remote system limitations must be understood, and suitable measures
against compromise. The remote system’s operating must be taken to deploy complementary mechanisms
system must protect against misuse of the allowed designed to compensate for such problems. In the covert
inbound services and must protect any information channel example, auditing and detection mechanisms
acquired through the inbound service against leakage. should be utilized to minimize the chances that known
Mandatory security mechanisms in the remote system’s channels are exploited. In turn, these should depend on
operating system may be used to provide such protec­ secure operating systems to protect their critical compo­
tion. Additionally, mandatory security mechanisms in nents, such as audit logs and intrusion sensors, because
the internal host’s operating system are needed to con- they are subject to the same types of vulnerabilities as
fine any attack from a penetrated remote system. those discussed throughout this paper.
When a benign insider wishes secure access to a 6 Summary
remote service, the firewall itself cannot provide com­ This paper has argued that the threats posed by the
plete protection for the use of the remote service. The modern computing environment cannot be addressed
internal host’s operating system must protect against any without secure operating systems. The critical operating
attempts by the server to trick the client into misusing its system security features of mandatory security and
privileges, as in the case where a browser executes a trusted path have been explained and contrasted with the
malicious applet provided by a server; mandatory secu­ inadequate protection mechanisms of mainstream oper­
rity mechanisms in the internal host’s operating system ating systems. This paper has identified the vulnerabili­
may be used to confine these client applications. ties that arise in application-space mechanisms for
5 System Security access control and cryptography and has demonstrated
how mandatory security and trusted path mechanisms
No single technical security solution can provide
address these vulnerabilities. To provide a clear sense of
total system security; a proper balance of security mech­
the need for these operating system features, this paper
anisms must be achieved. Each security mechanism pro­
has analyzed concrete examples of current approaches
vides specific security functions and should be designed
to security and has shown that the security provided by
to only provide those functions. It should rely on other
these approaches is inadequate in the absence of such
mechanisms for support and for required security ser­
features. Finally, the reader was given a perspective of
vices. In a secure system, the entire set of mechanisms
system security where both secure operating systems
complement each other so that they collectively provide
and application-space security mechanisms must com­
a complete security package. Systems that fail to
plement each other in order to provide the correct level
achieve this balance will be vulnerable.
of protection.
As has been shown throughout this paper, a secure
By arguing that secure operating systems are indis­
operating system is an important and necessary piece to
pensable to system security, the authors hope to spawn a
the total system security puzzle, but it is not the only
renewed interest in operating system security. If security
piece. A highly secure operating system would be insuf­
practitioners were to more openly acknowledge their
ficient without application-specific security built upon
security solution’s operating system dependencies and
it. Certain problems are actually better addressed by
state these dependencies as requirements for future
security implemented above the operating system. One
operating systems, then the increased demand for secure
such example is an electronic commerce system that
operating systems would lead to new research and
requires a digital signature on each transaction. A appli­
development in the area and ultimately to commercially
cation-space cryptographic mechanism in the transac­
viable secure systems. In turn, the availability of secure
tion system protected by secure operating system
operating systems would enable security practitioners to
features might offer the best system security solution.
concentrate on security services that belong in their par­
No single security mechanism is likely to provide ticular components rather than dooming them to try to
complete protection. Unsolved technical problems, address the total security problem with no hope of suc­
implementation errors and flawed environmental cess.
7 References [21] M. Eisler et al. Security Mechanism Independence
in ONC RPC. Proceedings of the 6th USENIX
[1] M. Abrams et al, Information Security: An UNIX Security Symposium, July 1996.
Integrated Collection of Essays, IEEE Comp. 1995.
[22] D. Ferraiolo and R. Kuhn. Role-Based Access
[2] J. Anderson, Computer Security Technology Control. Proceedings of the 15th National
Planning Study, Air Force Elect. Systems Div., Computer Security Conference, 1992.
ESD-TR-73-51, October 1972.
[23] B. Ford et al. Microkernels Meet Recursive Virtual
[3] R. Atkinson. IP Authentication Header (AH). IETF Machines. Proceedings of 2nd USENIX Symposium
RFC 1826, August 1995. on Operating Systems Design and Implementation,
[4] R. Atkinson. IP Encapsulating Security Payload October 1996.
(ESP). IETF RFC 1827, August 1995. [24] S. Garfinkel. Web Security and Commerce. O’Reilly
[5] R. Atkinson. Security Architecture for the Internet & Associates, Cambridge, 1997.
Protocol. IETF RFC 1825, August 1995. [25] M. Gasser. Building a Secure Computer System.
[6] Badger et al. DTE Firewalls, Initial Measurement Van Nostrand Reinhold Company, New York, 1988.
and Evaluation Report. Trusted Information [26] I. Goldberg et al. A Secure Environment for
Systems Technical Report #0632R, March 1997. Untrusted Helper Applications. Proceedings of 6th
[7] L. Badger et al. Practical Domain and Type USENIX Unix Security Symposium, July 1996.
Enforcement for UNIX. Proceedings of IEEE [27] L. Gong. Java Security: Present and Near Future.
Symposium on Security and Privacy, May 1995. IEEE Micro, May/June 1997.
[8] D. Baker. Fortresses Built Upon Sand. Proceedings [28] R. Graubart. Operating System Support for Trusted
of the New Security Paradigms Workshop, 1996. Applications. Proceedings of the 15th National
[9] S. Bellovin and W. Cheswick. Network Firewalls. Computer Security Conference, 1992.
IEEE Communications, September 1994. [29] M. Harrison et al. Protection in Operating Systems.
[10] S. Bellovin and M. Merritt. Limitations of the Communications of the ACM 19(8), August 1976.
Kerberos Authentication System. Computer [30] T. Jaeger et al. Building Systems that Flexibly
Communications Review 20(5), October 1990. Control Downloaded Executable Content.
[11] B. Blakley. The Emperor’s Old Armor. Proceedings Proceedings of the 6th USENIX Security
of the New Security Paradigms Workshop, 1996. Symposium, July 1996.
[12] W. Boebert and R. Kain, A Further Note on the [31] J. Kohl and C. Neuman. The Kerberos Network
Confinement Problem. Proceedings of the 30th Authentication Service V5. IETF RFC 1510,
IEEE International Carnahan Conference on September 1993.
Security Technology, 1996. [32] M. Ladue. When Java Was One: Threats from
[13] W. Boebert and R. Kain. A Practical Alternative to Hostile Byte Code. Proceedings of the 20th
Hierarchical Integrity Policies. Proceedings of the National Information Systems Security Conference,
8th National Computer Security Conference, 1985. 1997.
[14] E. Brewer at al. Basic Flaws in Internet Security [33] B. Lampson. A Note on the Confinement Problem.
and Commerce. http://http.cs.berkeley.edu/ Communications of the ACM 16(10), 1973.
~gauthier/endpoint-security.html, 1995. [34] B. Lampson et al. Authentication in Distributed
[15] W. Brierley. Integrating Cryptography into Trusted Systems: Theory and Practice. Proceedings of the
Systems: A Criteria Approach. Proceedings of the 13th ACM Symposium on Operating Systems
8th IEEE Conference on Computer Security Principles, 1992.
Applications, 1992. [35] J. Lepreau et al. The Persistent Relevance of the
[16] Computer Emergency Response Team. Advisory Local Operating System to Global Applications.
93:16. Proceedings of the 7th ACM SIGOPS European
Workshop, September 1996.
[17] D. Chapman and E. Zwicky. Building Internet
Firewalls. O’Reilly, 1995. [36] S. Lewontin. The DCE-Web Toolkit. Proceedings
of the 3rd International World Wide Web
[18] D. Davis. Kerberos Plus RSA for World Wide Web Conference, 1995.
Security. Proceedings of the 1st USENIX Workshop
on Electronic Commerce, July 1995. [37] S. Lewontin and M. Zurko. The DCE Web Project:
Providing Authorization and Other Distributed
[19] D. Dean et al. Java Security: From HotJava to Services to the World Wide Web. Proceedings of
Netscape and Beyond. Proceedings of the IEEE the 2nd International World Wide Web Conference,
Symposium on Security and Privacy, 1996. 1994.
[20] DOD 5200.28-STD. Department of Defense [38] J. Liedtke. L4 Reference Manual. Research Report
Trusted Computer System Evaluation Criteria, RC 20549, IBM T. J. Watson Research Center,
December 1985. September 1996.
[39] T. Linden. Operating System Structures to Support [57] Secure Computing Corporation. DTOS Covert
Security and Reliable Software. ACM Computing Channel Analysis Plan, Technical report MD A904-
Surveys 8(4), Dec. 1976. 93-C-4209 CDRL A017, May 1997.
[40] D. Martin et al. Blocking Java Applets at the [58] Secure Computing Corporation. DTOS Generalized
Firewall. Proceedings of the Internet Society Security Policy Specification, Technical report MD
Symposium on Network and Distributed Systems A904-93-C-4209 CDRL A019 June 1997. (http://
Security, 1997. www.securecomputing.com/randt/HTML/
[41] D. Mazieres and M. Kaashoek. Secure Applications dtos.html)
Need Flexible Operating Systems. Proceedings of [59] Secure Computing Corporation. DTOS General
the 6th Workshop on Hot Topics in Operating System Security and Assurability Assessment
Systems, May 1997. Report, Technical report MD A904-93-C-4209
[42] A. Medvinsky et al. Public Key Utilizing Tickets CDRL A011 June 1997. (http://www.
for Application Servers. IETF Draft Jan 1997 securecomputing.com/randt/HTML/dtos.html)
expires July 1997. [60] Secure Computing Corporation. LOCKed
[43] S. Minear. Providing Policy Control Over Object Workstation Cryptographic Services Study,
Operations in a Mach Based System. Proceedings Technical Report MD A904-94-C-6045 CDRL
of the 5th USENIX Security Symposium, April A009, September 1995. .
1995. [61] Secure Computing Corporation. Security
[44] NCSC-TG-005. Version 1. NCSC Trusted Network Requirements Specification and Requirements
Interpretation, July 1987. Rationale Report for the Technical Study
Demonstrating the Feasibility of Software-Based
[45] C. Neuman and J. Steiner. Authentication of Cryptography on INFOSEC Systems, Technical
Unknown Entities on an Insecure Network of report MDA904-91-C-7103 CDRL A011 and
Untrusted Workstations. Proceedings of the Usenix A012, May 1994.
Workshop on Workstation Security, August 1988.
[62] W. Sibert. Malicious Data and Computer Security.
[46] C. Neuman and T. Ts’o. Kerberos: An Proceedings of the 19th National Information
Authentication Service for Computer Networks. Systems Security Conference, 1996.
IEEE Communications Magazine, September 1994.
[63] M. Sirbu and J. Chuang. Distributed Authentication
[47] C. Neuman et al. The Kerberos Network in Kerberos using Public Key Cryptography.
Authentication Service V5 R6. IETF Draft July Proceedings of the Symposium on Network and
1997, expires Jan 1998. Distributed System Security, 1997.
[48] R. O’Brien and C. Rogers. Developing [64] M. Spasojevic and M. Satyanarayanan. An
Applications on LOCK. Proceedings of the 14th Empirical Study of a Wide-Area Distributed
National Computer Security Conference, 1991. System. ACM Transactions on Computer Systems
[49] J. Ousterhout et al. The Safe-Tcl Security Model. 14(2), May 1996.
Sun Labs Technical Report TR-97-60, March 1997. [65] S. Sutton and S. Hinrichs. MISSI B-level Windows
[50] President’s Commission On Critical Infrastructure NT Feasibility Study Final Report. Technical
Protection. Research and Development Report, NSA MISSI Contract MDA904-95-C-4088,
Recommendations for Protecting and Assuring December 1996.
Critical National Infrastructures, September 1997 [66] B. Tung et al. Public Key Cryptography for Initial
[51] M. Roe and T. Casey. Integrating Cryptography in Authentication in Kerberos. IETF Draft expires Jan
the Trusted Computing Base. Proceedings of the 1998.
6th IEEE Conference on Computer Security [67] J. Tyger and A. Whitten. WWW Electronic
Applications, 1990. Commerce and Java Trojan Horses. Proceedings of
[52] RSA Laboratories. Public Key Cryptography the 2nd Usenix Workshop on Electronic Commerce,
Standard No. 11 - Cryptoki Version 2.0. RSA November 1996.
Laboratories, pp. 24-25, April 1997. [68] W. Venema. Murphy’s Law and Computer Security.
[53] R. Salz. DCE 1.2 Contents Overview. Open Group Proceedings of the 6th USENIX Unix Security
RFC 63.3, October 1996. Symposium, 1996.
[54] J. Saltzer and M. Schroeder. The Protection of [69] D. Wagner and B. Schneier. Analysis of the SSL 3.0
Information in Computer Systems. Proceedings of Protocol. Proceedings of the 2nd USENIX
the IEEE, 63(9), September 1975. Workshop on Electronic Commerce, November,
1996.
[55] B. Schneier. Applied Cryptography, 2nd Edition.
John Wiley & Sons, New York, 1996. p. 169-187, [70] D. Wallach et al. Extensible Security Architectures
216-225. for Java. Technical Report 546-97, Dept. of
Computer Science, Princeton University, April
[56] Secure Computing Corporation. Assurance in the 1997.
Fluke Microkernel:: Formal Security Policy Model,
Technical report MD A904-97-C-3047 CDRL
A003, March 1998.