1

CSIT 561 – COMPUTER SECURITY

MODULE 7 : OPERATING SYSTEM
SECURITY

Bharath K. Samanthula
Department of Computer Science
Montclair State University

Slides are adopted from Chapter 12, Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043)..
 2

Objectives
• Basic security functions provided by operating systems
• System resources that require operating system
protection
• Operating system design principles
• How operating systems control access to resources
• The history of trusted computing
• Characteristics of operating system rootkits

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 3

Security in OSs
• OS is the fundamental controller of all system resources –
which makes it a primary target of attack, as well
• OS initializes at system boot time
• First, Primitive functions (e.g., I/O), device drivers (act as early
defense line)
• Second, process controllers
• Third, file and memory management routines
• Finally, user Interface
• What if malware embeds itself in OS ?
OR what if malware circumvents OS?

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 4

OS – Security Features
Security-relevant features:
• Enforced sharing
• Interprocess communication and synchronization
• Protection of critical data
• Guaranteed fair service
• Interface to hardware
• User authentication
• Memory protection
• File and I/O device access control
• Allocation and access control to general objects

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 5

Operating System Functions

Users

User Interface
Synchronization,
Concurrency
Control, Deadlock
Operating Services Management,
System
Communication,
Accounting

Resource Allocation

Data

CPU
Memory

Program
Libraries
I/O Devices

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 6

History of Operating Systems

• Single-user systems, no OS
• Users manually enter their programs using
switches, keyboard, or punch cards
• Not good in terms of economy and productivity
• Multiprogrammed OS, aka monitors
• Multiple users
• Multiple programs
• Scheduling, sharing, concurrent use
• Personal computers
• Multitasking returned to the mainstream in the
1990s, and with it came all the lessons of the
early shared computers
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 7

Protected Objects
• Memory
• Sharable I/O devices, such as disks
• Serially reusable I/O devices, such as printers
• Sharable programs and subprocedures
• Networks
• Sharable data

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 8

OS Layered Design
Subprocesses of User Processes

User Processes

Compilers, Database Managers

Operating system
visualized in layers, from Utility Functions

most critical (bottom) to File Systems, Device Allocation

least critical. Operating
System Scheduling, Sharing,
Memory Management

Synchronization, Allocation
Operating
System Security Functions
Kernel
Security
Kernel Hardware

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 9

Functions Spanning Layers

Authentication is a good example of a function that needs to span the layers in
the layered model.

Least
Trusted
Code

User Authentication Module

User
Interface

User ID
Lookup

Most Authentication
Trusted Data Comparison
Code
Authentication
Data Updates
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 10

Modular OS Design
Users Users Users Users Users
User Mode
Privileged Mode
User Interface

Sec File Object A/V Net Backup Shell

System Services Interface

I/O Time Synch Memory Comm Sec

Primitive Services

Microkernel Kernel Mode Drivers

Hardware Interface and Abstraction

Hardware

Modern OSs are built from discrete modules. These modules generally come
from a variety of sources and are subject to updating/overwriting, so they
cannot trust one another.
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 11

Virtualization
• With virtualization, the OS presents each user with just
the resources that user should see
• The user has access to a virtual machine (VM), which
contains those resources
• The user cannot access resources that are available to
the OS but exist outside the VM
• A hypervisor, or VM monitor, is the software that
implements a VM
• Translates access requests between the VM and the OS
• Can support multiple OSs in VMs simultaneously

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 12

Virtualization
• Sandbox: a protected environment from which a process
can have only limited, controlled impact on outside
resources
• Initial design of Java is designed to run code, called applets,
download from untrusted sources with limited access to resources
• Honeypot: A VM meant to lure an attacker into an
environment that can be both controlled and monitored
• Cliff Stoll (sys admin at Lawrence Berkeley National
Laboratory)simulated the effect of a slow speed, unreliable
connection and created fake documents to lure a tenacious hacker,
identified as a KGB spy

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 13

Separation and Sharing

• Methods of separation:
• Physical
• Temporal
• Logical
• Cryptographic
• Methods of supporting separation/sharing:
• Do not protect – when procedures are run at different times
• Isolate
• Share all or share nothing – public or private objects
• Share but limit access – OS guards objects using access controls
• Limit use of an object
• Limit operations allowed on an object (e.g., users may be allowed to
derive statistical summaries from a database, but not to access specific
data values)

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 14

Hardware Protection of Memory

Several ways of protecting a memory space
• Fence
• Fence Registers
• Base/Bound Registers
• Segmentation
• Paging
• Paging + Segmentation

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 15

Hardware Protection of Memory

Addresses Memory
A fence is a method to 0
confine users to one
side of a boundary Hardware
Address Operating System
Very restrictive as a
Limitation
predefined amount of n
space is reserved for
OS
n+ 1

Addressing User Program Space

Range

High
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 16

Fence Registers
Like fences, but fence registers allow for the boundary to change.

Address Address
Limit Limit
Register Register
n +1 p +1

Addresses Memory Addresses Memory

0 0

Operating Operating
System Version 1 System Version 2

n p

n +1 p +1

User Program User Program

Space Space
Addressing Addressing
Range Range

High High
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 17

Base/Bounds Registers
Addresses Memory
0

Operating
Base Register System
n
n +1
n +1
User A
Program Space
Bounds Register p
p
p +1 User Program
User B Space
Program Space
q
q +1 User C
Program Space

High

With base and bounds registers, memory space can be broken into more
than two sections, allowing for multiple users.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 18

Two Pairs of Base/Bounds Registers

Operating
System

User A
Data Base Program Space

User B
Data Bounds Data Space

User A
Data Space User Program
and Data
Space
User C
Program Space

Program Base User C

Data Space

Program Bounds User B

Program Space

This separates executable memory from data memory for each user,
making it harder for bugs/attacks to overwrite code.
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 19

Segmentation
• A program is divided into separate, logical pieces (e.g., an
array, a procedure). Each segment has its own set of access
rights. The operating system maintains a table of each
segment and its true memory address, and it translates calls to
each segment using that table (shown on next slide).
Advantages:
• The operating system can move segments around as
necessary, which is very helpful as segments grow and shrink.
• Segments can be removed from memory if they aren’t being
used currently.
• Every legitimate address reference must pass through the OS,
providing an opportunity for access control.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 20

Segmentation
Physical Placement of
Program’s Segments
Logical Arrangement of
Program
Operating
MAIN System
Segments

SEG_A SUB

SUB MAIN

DATA_SEG
Segments for
Other Users

SEG_A

DATA_SEG

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 21

Segment Address Translation

Segment Translation Table
Address
MAIN c 0
Logical Program
SEG_A g
MAIN
SUB a a
SEG_A DATA_SEG h
b
FETCH<DATA_SEG,20>
c
SUB
d
DATA_SEG e
+ f
g
h

Location 20 within Segment DATA_SEG i

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 22

Paging
• Problem with segmentation: segment size which can vary
among segments
• A program can generate a valid reference <X, 9999>, but in reality
segment X may only be 200 bytes long
• Similar to segmentation, but programs are broken into
fixed-size fragments (pages) rather than being broken
down by logical unit. Because programs aren’t broken into
logical units, paging doesn’t allow different parts of a
program to have different access rights.
• Paging allows the security advantages of segmentation
with more efficient memory management

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 23

Paging
Page Translation Table Memory
Logical Program Page Address Address
0
Page 0 0 b a
Page 1 b
FETCH<4,37> 1 f Page 0
c Location
Page 2 2 i Page 4
d 37, Page 4
Page 3 3 l e
Page 7
Page 4 4 c f
Page 1
Page 5 5 g g
Page 5
h
Page 6 6 n
i
Page 2
Page 7 7 e j
k
l
Page 3
m
+
n
Page 6
o

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 24

Paged Segmentation
Segment Translation Table
Page Memory
Segment Table Address
0
MAIN
Page Translation Tables a
Logical Program SEG_A For Segment MAIN b
Page Address DATA_SEG Page 1
MAIN c
SUB
0 c MAIN Page 0
DATA_SEG 1 f d
SEG_A
e
FETCH<DATA_SEG,20> SEG_A Page 1
For Segment SEG_A
Page Address f
MAIN Page 1
SUB 0 n g
SEG_A Page 2
1 e h
DATA_SEG
2 g i
SUB Page 0
For Segment SUB j
Page Address k
0 i
l
DATA_SEG Page 0
For Segment DATA_SEG m
Page Address
20 = Page 0 n
0 l SEG_A Page 0
o
1 b
+
Segment DATA_SEG Word 20

Programs can be broken into segments, and the segments are then combined to fill pages. This
approach creates an extra layer of translation but allows for the benefits of both paging and
segmentation.
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 25

Principles of Secure OS Design

• Simplicity of design
• OSs are inherently complex, and any unnecessary complexity only
makes them harder to understand and secure
• Layered design
• Enables layered trust

• Layered trust
• Layering is both a way to keep a design logical and understandable
and a way to limit risk
• Example: very tight access controls on critical OS functions, fewer
access controls on important noncritical functions, and few if any
access controls on functions that aren’t important to the OS

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 26

Kernelized Design
• A kernel is the part of the OS that performs the lowest-
level functions
• Synchronization
• Interprocess communication
• Message passing
• Interrupt handling

• A security kernel is responsible for enforcing the security

mechanisms of the entire OS
• Typically contained within the kernel

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 27

Reference Monitor
S
S
S

Opera
ting S
ystem
Truste o r
d Soft
ware Opera
ting S
ystem
Truste ro
O d Soft
O ware
O
O Reference
Monitor

The reference monitor is the most important part of the security kernel,
controlling access to objects. A reference monitor must be tamperproof,
unbypassable, and analyzable.

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 28

Trusted Systems
• A trusted system is one that has been shown to warrant
some degree of trust that it will perform certain activities
faithfully
• Characteristics of a trusted system:
• A defined policy that details what security qualities it enforces
• Appropriate measures and mechanisms by which it can enforce
security adequately
• Independent scrutiny or evaluation to ensure that the mechanisms
have been selected and implemented properly

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 29

History of Trusted Systems

Security Controls E.C. Information
for Computer Technology
Systems Security
Trusted Computer Evaluation Common
System Evaluation Criteria Criteria
Criteria

1970 1983 1991 1994

1972 1988 1992

Security
Technology Combined
British,
Planning Federal
German,
Study Criteria
French
Criteria
Attempts to declare computers trustworthy go back almost 50 years. Over the years, changes
in technology have resulted in new requirements, and the explosion of new devices and
software have made it impossible to keep up.
https://www.us-cert.gov/bsi/articles/best-practices/requirements-engineering/the-common-
criteria
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 30

Trusted Computing Base (TCB)

User applications

Utilities

User request interpreter

User process coordination, synchronization
Non-TCB
User environment: objects, names (e.g., files)
User I/O
Procedures, user processes
Creation and deletion of user objects
Directories
Extended types
Segmentation, paging, memory management

Primitive I/O
Basic operations
TCB Clocks, timing
Interrupt handling
Hardware: registers, memory
Capabilities

The TCB portion of the OS is the part we depend on for enforcement of

security policy. The TCB monitors and protects the secrecy and integrity of four
basic interactions: process activation, execution domain switching, memory
protection, and I/O operation.
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 31

Other Trusted System Characteristics

• Secure startup
• System startup is a tricky time for security, as most systems load basic
I/O functionality before being able to load security functions
• Intel’s trusted boot technology uses TPMs to achieve secure startup.
• Trusted path
• An unforgeable connection by which the user can be confident of
communicating directly with the OS
• Object reuse control
• OS clears memory before reassigning it to ensure that leftover data
doesn’t become compromised
• Audit
• Trusted systems track security-relevant changes, such as installation
of new programs or OS modification
• Audit logs must be protected against tampering and deletion
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 32

Rootkits
• A rootkit is a malicious software package that
attains and takes advantage of root status or
effectively becomes part of the OS
• Rootkits often go to great length to avoid being
discovered or, if discovered and partially
removed, to reestablish themselves
• This can include intercepting or modifying basic OS
functions

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 33

Rootkit Evading Detection

Will call’s
Intercepted Yes
result reveal
function call
rootkit?

No

Pass call to Execute call but

operating system monitor result and
function adjust as necessary

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 34

Summary
• OSs have evolved from supporting single users and
single programs to many users and programs at once
• Resources that require OS protection: memory, I/O
devices, programs, and networks
• OSs use layered and modular designs for simplification
and to separate critical functions from noncritical ones
• Resource access control can be enforced in a number of
ways, including virtualization, segmentation, hardware
memory protection, and reference monitors
• Rootkits are malicious software packages that attain root
status or effectively become part of the OS

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.