Chapter 4

Operating System Security

Yirga Y. (PhD)
[email protected]
Department of Information Technology
06/04/2024 Adanced information Security 1
Outline
• What is Operating System Security?
• Goal of Security System
• Threats to Operating System
• OS policies

06/04/2024 Adanced information Security 2

What is OS Security?

• It refers to the processes or measures taken

to protect the operating system from
dangers, such as viruses, worms, malware, and
remote hacker intrusions.
• The process of ensuring OS availability,
confidentiality, integrity.
• It comprises all preventive-control
procedures that protect any system assets that
could be stolen, modified, or deleted if OS
security is breached.
06/04/2024 Adanced information Security 3
Operating System Security

• Providing a protection system to computer system

resources:
• CPU, memory, disk, software programs, and
most importantly data/information stored in the
computer system.
• A computer system must be protected against
unauthorized access, malicious access to system
memory, viruses, worms, etc.
• Authentication
• One Time passwords
• Program Threats
• System Threats

06/04/2024 Adanced information Security 4

…cont.
• It refers to practices and measures that can
ensure the confidentiality, integrity, and
availability (CIA) of operating systems.
• Goal: To protect the OS from various
threats, including malicious software such as
worms, trojans, and other viruses,
misconfigurations, and remote intrusions.
• The implementation of control techniques
that can protect your assets from
unauthorized modification and deletion or
theft.

06/04/2024 Adanced information Security 5

…con’t

06/04/2024 Adanced information Security 6

Protected Objects
• The rise of multiprogramming meant that several aspects of a computing system
required protection:
• memory
• sharable I/O devices, such as disks
• serially reusable I/O devices, such as printers and tape drives
• sharable programs and sub procedures
• networks
• sharable data
• a directory of files
• a data structure, such as a stack
• a table of the operating system

06/04/2024 Adanced information Security 7

Protection in General-Purpose OS

• Protection features provided by general-purpose operating systems:

• Protecting memory, files, and the execution environment
• Controlled access to objects
• User authentication
• OS goals: controlling shared access and implementing an interface to allow that access.
• Operating system functions can be:
• Access control
• Identity and credential management
• Information flow
• Audit and integrity protection

06/04/2024 Adanced information Security 8

Threats to Operating System
• Malware
• It contains viruses, worms, Trojan horses, and other dangerous software.
• May corrupt files, delete the data, replicate to propagate further, and even crash a system.
• It frequently goes unnoticed by the victim user while criminals silently extract important data.
• Network Intrusion
• Such as masqueraders, misfeasors, and unauthorized users.
• Buffer Overflow
• It is also known as buffer overrun.
• It is the most common and dangerous security issue of the OS.
• A condition at an interface under which more input may be placed into a buffer and a data
holding area than the allotted capacity, and it may overwrite other information.
• Attackers use such a situation to crash a system or insert specially created malware that allows
them to take control of the system.
06/04/2024 Adanced information Security 9
What are Common OS Security Threats?
• Malware
• It encompasses a range of attack vectors such as
viruses, worms, trojans, and rootkits.
• Injected into a system without the owner’s consent,
or by masquerading/hidden as legitimate software,
with the objective of stealing, destroying, or
corrupting data, or compromising the device.
• It can also replicate, allowing it to spread further in
a corporate network and beyond.
• Malware attacks often go undetected by the target
user, allowing for the quiet extraction of sensitive
data.

06/04/2024 Adanced information Security 10

…cont.
• Network Intrusion
• It occurs when an individual gains access to a system for improper use.
• Types:
• Careless insiders: authorized users who neglect to follow security
policies or best practices, causing exposure of sensitive assets.
• Malicious insiders: authorized users who misuse their privileges for
malicious indigence.
• Masqueraders: external individuals who pose as legitimate users,
exploiting the account or credentials of an authorized user to gain
access to the system.
• An unauthorized person who gains access to a system and uses an
authorized person's account.
• Clandestine users: attackers who penetrate the system by gaining
supervisory control and going around access controls.
• Misfeasor: is a legitimate user who gains unauthorized access to and
misuses programs, data, or resources.

06/04/2024 Adanced information Security 11

…cont.

• Buffer Overflow
• The main function of a buffer is to temporarily store data.
• Each buffer has a capacity of data it can hold.
• During a buffer overflow attack, the buffer or other
temporary data stores are overflowing with data.
• When the buffer overflows, the program attempting to
write the data may overwrite other memory locations
containing important information.
• Threat actors look for buffer overflow vulnerabilities,
which they can exploit to inject scripts that help them
hijack the system or crash it.

06/04/2024 Adanced information Security 12

 …cont.
• Program Threats
• If a user program made process do malicious tasks.
• E.g. a program installed in a computer which can store and send user credentials via network to
some hacker.
• Some well-known program threats.
• Trojan Horse: a program traps user login credentials and stores them to send to malicious user
who can later on login to computer and can access system resources.
• Trap Door: If a program which is designed to work as required, have a security hole in its code
and perform illegal action without knowledge of user.
• Logic Bomb: a program misbehaves only when certain conditions met otherwise it works as a
genuine program. It is harder to detect.
• Virus: can replicate themselves on computer system. They are highly dangerous and can
modify/delete user files, crash systems.
• A virus is generally a small code embedded in a program. As user accesses the program, the
virus starts getting embedded in other files/ programs and can make system unusable for user.
06/04/2024 Adanced information Security 13
 …cont.
• System Threats
• The misuse of system services and network connections to put the user in trouble.
• It can be used to launch program threats on a complete network called a program attack.
• System threats create such an environment that operating system resources/ user files are misused.
• Following is a list of some well-known system threats.
• Worm:
• A process that can choke down a system’s performance by using system resources to extreme levels.
• The process generates multiple copies where each copy uses system resources, preventing all other processes to
get the required resources.Worms processes can even shut down an entire network.
• Port Scanning:
• A mechanism or means by which a hacker can detect system vulnerabilities to make an attack on the system.
• It is a common technique hackers use to discover open doors or weak points in a network.
• Denial of Service:
• Attacks normally prevent the user to make legitimate use of the system.
• For example, a user may not be able to use the internet if denial of service attacks the browser's content settings.

06/04/2024 Adanced information Security 14

 Ways to ensure OS Security?
1. Authentication
• The process of identifying every system user and associating the programs executing with those
users.
• Match an identified user with the programs or data they are allowed to access.
• All OS have controls that can be used to verify that users who run a particular program are
authorized to do so.
• In general, operating systems identify and authenticate users in three ways.
1. Username/Password
• Every user contains a unique username and password that should be input correctly before
accessing a system.
2. User Attribution
• Usually include biometric verification, such as fingerprints, retina scans, etc.
• This authentication is based on user uniqueness and is compared to database samples already
in the system. Users can only allow access if there is a match.
3. User card and Key
• To login into the system, the user must punch a card into a card slot or enter a key produced
by a key generator into an option provided by the operating system.
06/04/2024 Adanced information Security 15
…cont.
2. One Time passwords
• Along with standard authentication, one-time passwords give an extra layer of security.
• Every time a user attempts to log into the One-Time Password system, a unique password is needed.
• Once a one-time password has been used, it cannot be reused. One-time passwords may be
implemented in several ways.
1. Secret Key
• The user is given a hardware device that can generate a secret id that is linked to the user's id.
• The system prompts for such a secret id, which must be generated each time you log in.
2. Random numbers
• Users are given cards that have alphabets and numbers printed on them.
• The system requests numbers that correspond to a few alphabets chosen at random.
3. Network password
• Some commercial applications issue one-time passwords to registered mobile/email addresses,
which must be input before logging in.
06/04/2024 Adanced information Security 17
…cont.
• To offer an additional layer of security when
combined with standard authentication
measures.
• Users must enter a unique password generated
each time they log in to the system.
• A one-time password cannot be reused.

06/04/2024 Adanced information Security 18

…cont.
3. Firewalls
• Are essential for monitoring all incoming and outgoing traffic.
• It imposes local security, defining the traffic that may travel through it.
• An efficient way of protecting network systems or local systems from any network-
based security threat.
4. Physical Security
• The most important method of maintaining operating system security is physical
security.
• An attacker with physical access to a system may edit, remove, or steal important
files since operating system code and configuration files are stored on the hard
drive.

06/04/2024 Adanced information Security 19

Protection in File System
• Types of Access :
• The files which have direct access of any user have the need of
protection. The mechanism of the protection provides the facility
of the controlled access by just limiting the types of access to the
file.
• Several types of operations can be controlled
• Read – Reading from a file.
• Write – Writing or rewriting the file.
• Execute – Loading the file and after loading the execution process
starts.
• Append – Writing the new information to the already existing
file, editing must end at the end of the existing file.
• Delete – Deleting the file which is of no use and using its space
for another data.
• List – List the name and attributes of the file.

06/04/2024 Adanced information Security 20

Virtualization
• It enables you to abstract software from hardware,
effectively separating the two.
• The advantage: it introduces a high level of efficiency
and flexibility while providing greater security coverage.
• There are many types of virtualization: desktop, application,
network, server, network, storage, and OS virtualization.
• What is OS virtualization?
• OS virtualization enables you to multiple isolated
user environments using the same OS kernel.
• The technology that creates and enables this type of
isolation is called a “hypervisor”, which serves as a
layer located between the device and the virtualized
resources.

Adanced information Security 21

06/04/2024
Categories: OS virtualization

• Fully locked-down VM
• Should be used to provide access to sensitive data and
corporate systems, such as IT environments,
payment systems, and sensitive customer data.
• Unlocked, open VM
• Should be used to provide unrestricted access to
non-corporate resources. For example, full web
browsing sessions, installation of applications, and
use of external devices.
• Semi-locked-down VM
• Should be used to provide access to standard
corporate applications and resources, such as office
documents, company email, and internal services.

06/04/2024 Adanced information Security 22

Testing and Validating OS Security
• An ongoing process that requires constant testing.
• Depending on the risk and priority of a system,
security posture tests may take place on a monthly,
weekly, or daily basis.
• Vulnerability Assessment
• Involves testing for weaknesses that may be lying
undetected in an operating system.
• Identifying vulnerabilities allows you to identify
possible vectors for an attack so you can better
understand the risk to your system.
• It attempts to stay on top of newly exposed
vulnerabilities by locating, classifying, and
prioritizing them according to severity and
impact.
06/04/2024 Adanced information Security 23
Typical methods used for OS vulnerability assessment

Scanning the software and applications

on an OS.
Scanning for malware.
Scanning for missing patches and
updates.

06/04/2024 Adanced information Security 24

Penetration Testing
• It is a security assessment strategy that uses vulnerability assessment to identify how an
attacker may successfully exploit vulnerabilities in the system.
• This method involves simulating an exploit to evaluate system security.
• It helps discover vulnerabilities and seeks to identify the methods an attacker may use
to exploit them.
• Security teams can leverage the insights provided by penetrating to put in place effective security
measures.
• Types:
• White Box: The penetration tester has full technical knowledge of the system being tested.
• Grey Box: The penetester has limited technical knowledge of the system being tested.
• Black Box: The penetester doesn’t have any prior technical knowledge of the system
being tested.

06/04/2024 Adanced information Security 25

…con’t

06/04/2024 Adanced information Security 26

OS: Security Policies and Procedures
• Various OS security policies may be implemented based on the organization that you are working in.
• As OS security policies and procedures cover a large area, there are various techniques to address them.
• Security policies handle all preventative activities and procedures to ensure an OS protection, including
steal, edited, and deleted data.
• Some of them are as follows:
• Installing and updating anti-virus software.
• Ensure the systems are patched or updated regularly.
• Implementing user management policies to protect user accounts and privileges.
• Installing a firewall and ensuring that it is properly set to monitor all incoming and outgoing traffic.
• OS security policies and procedures are developed and implemented to ensure that you must first
determine which assets, systems, hardware, and date are the most vital to your organization.

06/04/2024 Adanced information Security 27

Research points

• Analysis of memory management issues

• Managing services
• File protection

06/04/2024 Adanced information Security 28

Reading Assignment

• Methods of memory protection

 Memory Protection using Keys
 Memory Protection using Rings
 Capability-based addressing
 Memory Protection using masks
 Memory Protection using Segmentation
 Memory Protection using Simulated segmentation
 Memory Protection using Dynamic tainting
• File protection methods

06/04/2024 Adanced information Security 29

 Thankyou!

06/04/2024 Adanced information Security 30