Systems Programming (SENG360)

Lecturer: Maksat Atagoziev

Office Hours: Monday 13:00 – 15:00,
Tuesday 10:00 – 12:00
 Course Info
Course Objectives This course provides an introductory
overview of operating systems and system programming,
mainly focusing on system-level programming based on OS
services and other APIs.
Course Content Topics include system calls, file I/O, files
and directories, memory management, process control,
inter-process communication (IPC), socket-based network
programming, remote procedure call (RPC) programming,
and basic security mechanisms. Course work includes
programming homework.

2
 Course Info
Course Learning Outcomes
• Overview of operating systems, functionalities and charateristics of OS
• System errors and error handling
• Variations on File I/O
• Resource management techniques
• Spawning and managing child processes
• Interprocess communications
• Messaging systems
• Client/server development
• Performance measurement
• Distributed processing infrastructures
• Parallel processing infrastructures
• Introduction to Server-Client Sockets
• Introduction to Threads
3
 Course Info

Textbook:
1. Systems Programming and Designing and Developing Distributed Applications, 1st
edition , Richard Anthony.
2. The Linux Programming Interface: A Linux and UNIX System Programming, 1st
Edition, Michael Kerrisk
3. Linux System Programming: Talking Directly to the Kernel and C Library, Second
Edition, Robert Love

4
Course Info
Weekly Topics
1 Overview of systems programming
2 Users, files and manuals
3 Directories, file properties and file systems
4 Terminal control and signals
5 Event driven programming
6 Processes and programs
7 i/o redirection and pipes
8 Mid-term Exam
9 Servers and sockets
10 Threads
11 Memory management system and processes carried out under the operating systems
12 Concurrency control
13 Linkers and Loaders
14 Compilers
15 Enterpreters and Debuggers
16 Final Exam
5
Overview of systems programming - 1
You developed various programs and applications. The
goal was to write programs that “accomplished”
something interesting in that it provided a service for
the user.
Usually resulting in an “app” that interacted with the
user. Examples of common applications are internet
browsers, word processors, games, database access
programs, spreadsheets, etc.. So, what you have been
doing in your courses has been:

6
Overview of systems programming - 2
Applications Programming is the programming of
software to provide services for the user directly.
Systems Programming, on the other hand is different. It
has a different focus … and can be defined as follows:
Systems Programming is the programming
of software that provides services for other software
… or for the underlying computer system.

7
Overview of systems programming - 3
So, when you are doing systems programming, you are
writing software that does not typically have a front-
end GUI that interacts with the user.
It is often the case where the software runs “behind-
the-scenes”… sometimes as a process/thread running
in the background.

8
Overview of systems programming - 4
Some examples of systems programs are:
1. Firmware (e.g., PC BIOS and UEFI).
2. Operating systems (e.g., Win, Mac OSX, GNU/Linux, BSD, etc.)
3. Game Engines (e.g., Unreal Engine 4, Unity 3D, Torque3D)
4. Assemblers (e.g., GNU AS, NASM, FASM, etc...)
5. Macro Processors (e.g., GNU M4)
6. Linkers and Loaders (e.g., GNU ld which is part of GNU binutils)
7. Compilers and Interpreters (e.g., gcc, python, Java VM)
8. Debuggers (e.g., gdb)
9. Text editors (e.g., vim)
10. Operating system shell (e.g., bash)
11. Device Drivers (e.g., for Bluetooth, network cards, etc..)

9
Overview of systems programming - 5
• Much more lower level than typical application
software
• Tied to the actual hardware of the machine that it is
running on
• Uses the operating system directly through system
calls
• It is important to have a good understanding of the
machine that it will be running on

10
Overview of systems programming - 6
• Applications programming is at a higher level than
systems programming
• Closer to the way we think as humans
• It is more natural and can make use of high level
programming languages and specialized libraries
• System software is the layer between the hardware
and application software
• It can deal directly with the hardware and usually
controls it
• It is therefore, more naturally written in a lower level
programming language
• Provides “efficient” services to applications.
11
Overview of systems programming - 7
• Computer organization

12
Overview of systems programming - 8
• Hardware components: PC

13
Overview of systems programming - 9
• Hardware components: DRAM vs. Disk
• Speed
• Capacity
o Memory Hierarchy

• Volatility: Volatile vs. Non-volatile

o Need to write data into disk explicitly for persistency (file I/O)

• Interface: Byte-unit interface vs. Sector-unit interface

o Need to load a program from disk to RAM before execution (loading)

14
Overview of systems programming - 10
• Memory hierarchy:

• 15 https://static.tvtropes.org/pmwiki/pub/images/memory_hierarchy.jpg
Overview of systems programming - 10
• Memory hierarchy:

• 16 https://editor.analyticsvidhya.com/uploads/77254Computer%20Memory%20Hierarchy%20Pyramid.png
 Overview of systems programming - 11
System program: address the following issues
• How to run this application program on CPU?
• What is the role of printf()?
• How the string is displayed on Monitor?
• How this program can be executed with other programs
concurrently?
• What are the differences between local and global
variables?
• What kinds of techniques can be applied to enhance the
performance of this program?

17
 Overview of systems programming - 12
System program: address the following issues
• How to run this application program on CPU? (object, binary,
compiler, assembler, loader, …)

• What is the role of printf()? (library, linker, …)

• How the string is displayed on Monitor? (device driver, file system,
…)

• How this program can be executed with other programs

concurrently? (process, scheduler, context switch, IPC (Inter process
communication), ... )

• What are the differences between local and global

variables? (data, stack, heap, virtual memory, buddy system, …)
• What kinds of techniques can be applied to enhance the
performance of this program? (compiler optimization (loop unrolling,
reordering), CPU optimization (pipeline, superscalar, out-of-order execution), …)
18
Overview of systems programming - 13
System program: Classification

19
Overview of systems programming - 14
Concept: Language Hierarchy

20
Overview of systems programming - 15
Overall structure and 6 key components

21
Overview of systems programming - 16
System software is the layer between the hardware and
application software.
It can deal directly with the hardware and usually
controls it.
It is therefore, more naturally written in a lower level
programming language.
In general, it provides “efficient” services to applications.

22
Overview of systems programming - 17
The goal of writing systems software is to make efficient
use of resources (e.g., computer memory, disk space,
CPU time, etc..).
In some cases, performance may be critical (e.g., like a
fast game engine).
In fact, it is often the case that small improvements in
efficiency can save a company a lot of money.
Therefore, in this course, we will be concerned about
writing efficient code...
We will also be trying to get a better grasp of the
computer’s operating system

23
Overview of systems programming - 18

24
Overview of systems programming - 19
An Operating System is:
• System software
• Manages computer hardware and software resources
• Provides common services for computer programs
• There are various operating systems out there of which
you may have heard of:
o Windows
o Mac OSX
o Unix
o Linux
o Android
o Chrome OS

25
Overview of systems programming - 20
Operating systems are the layer of software that sits
between the computer’s hardware and the user
applications
It is considered the “boss” of the computer
It manages everything that is going on visibly, as well as
behind the scenes.
Some operating systems provide time-sharing features
that schedule tasks at various times depending on how
the computer’s resources (e.g., memory, disk storage,
printers and devices) are allocated at any different time.

26
Overview of systems programming - 21
Applications perform system calls to gain access to the
resources:

27
Overview of systems programming - 22
Some of the functionality that is provided by the
operating system:
File I/O,
• provides file system organization and structure
• allows access to read and write files
• provides a measure of security by allowing access
permissions to be set

28
Overview of systems programming - 23
Some of the functionality that is provided by the
operating system:
Device I/O,
• allows communication with devices through their
drivers
• e.g., mouse, keyboard, printer, bluetooth, network card,
game controllers…
• manages shared access (i.e., printer queue)

29
Overview of systems programming - 24
Some of the functionality that is provided by the
operating system:
Process Management,
• allows the starting/stopping/alteration of running
executables
• can support multitasking
o (i.e., multiple processes running concurrently)

• allocates memory for each process

o needed for program instructions, variables & keeping track of function
calls/returns

30
Overview of systems programming - 25
Some of the functionality that is provided by the
operating system:
Virtual memory,
• provides memory to every process
o dedicated address space, allocated when process starts up

• appears (to the process) as a large amount of

contiguous memory but in reality, some is fragmented in
main memory & some may even be on disk
• programs use virtual memory addresses as opposed to
physical memory addresses

31
Overview of systems programming - 26
Behaviors: 1) initial state

32
Overview of systems programming - 27
Behaviors: 2) generate a file (user’s viewpoint)
vi test.c

33
Overview of systems programming - 28
Behaviors: 2) generate a file (system’s viewpoint)

34
Overview of systems programming - 28
Behaviors: 2) generate a file (system’s viewpoint)

35
Overview of systems programming - 29
Behaviors: 3) compile the file (user’s viewpoint)

36
Overview of systems programming - 30
Behaviors: 3) compile the file (system’s viewpoint)

37
Overview of systems programming - 31
Behaviors: 4) execute the a.out (user’s viewpoint)

38
Overview of systems programming - 32
Behaviors: 4) execute the a.out (system’s viewpoint)
o To run a.out, OS first loads it into memory

39
Overview of systems programming - 33
Behaviors: 4) execute the a.out (system’s viewpoint)
Then, OS makes a new process (active object)

40
Overview of systems programming - 34
Behaviors: 4) execute the a.out (system’s viewpoint)
Then, OS schedule the process

41
Overview of systems programming - 35
Behaviors: 4) execute the a.out (system’s viewpoint)
Actually there are multiple processes managed by scheduler

42
 Overview of systems programming - 36
Operating system: summary
• Process manager (Task manager): CPU
o process manipulation, schedule, IPC, signal, context switch
o fork, exec, wait, getpid, (pthread_create) , …

• Virtual Memory: Main memory

o page, segment, address translation, buddy, LRU
o brk, (malloc, free), …

• File system: Storage

o file, directory, disk scheduling, FAT
o open, read, write, mknod, pipe, (fopen, fwrite, printf), …

• Device driver: Device

o IO port management, interrupt, DMA
o open, read, write, ioctl, module, … ü Network protocol: Network
o connection, routing, fragmentation
o socket, bind, listen, send, receive, …
43
 Overview of systems programming - 37
The language of choice for programming Unix-based
operating systems is C.
• closer to hardware
• used to write Unix OS, device drivers
• very fast, with negligible runtime footprint

44
 Overview of systems programming - 38
Tools for Systems Programming
We will now discus 4 tools that are essential for systems
programming:
1. Shells
2. Text Editors
3. Compilers
4. Debuggers

45
 Overview of systems programming - 39
Tools for Systems Programming - Shell
A Shell is a command line user interface that allows access to
an operating system’s services.
A shell allows the user to type in various commands to run
other programs.
It also serves as a command line interpreter.
Multiple shells can be run at the same time (in their own
separate windows).
In Unix, there are three major shells:
• sh - Bourne shell
• bash - Bourne-again shell (default shell for Linux)
• csh - C shell
46
 Overview of systems programming - 40
Tools for Systems Programming - Shell
The shells differ with respect to their command line
shortcuts as well as in setting environment variables
A shell allows you to run programs with command line
arguments (i.e., parameters that you can provide when you
start the program in the shell).
The parameters (a.k.a. arguments or options) are usually
preceded by a dash – character.

47
 Overview of systems programming - 41
Tools for Systems Programming - Shell
There are some common shell commands that you can make
use of within a shell:
o e.g., alias, cd, pwd, set, which

There are some common system programs that you can

make use of within a shell:
o e.g., grep, ls, more, time, sort

You can even get some help by accessing a kind of “user

manual” through access of what are called man pages with
the man command.
We will make use of various shell commands and programs
throughout the course.

48
 Overview of systems programming - 42
Tools for Systems Programming - Text Editors
A Text Editor is a program that allows you to produce a text-
based file
There are some common ones such as: vi/vim, emacs and
gedit.
You will need to use one for writing your programs in this
course and for building make files (discussed later)
They all require you to use various “hot keys” and
commands in order to be quick and efficient at
editing(without the use of a mouse)

49
 Overview of systems programming - 43
Tools for Systems Programming - Compilers
A Compiler is computer software that transforms source
code written in one programming language into another target
programming language.
In this course, we will make use of the GNU compiler.
• GNU is a recursive acronym for "GNU's Not Unix!". It was chosen because GNU's
design is Unix-like, but differs from Unix by being free software and containing no Unix
code.

The command for using the compiler is gcc.

• options that you can provide (-o allows you to specify the output file)

Compilers produce code that is meant to be run on a

specific machine. Linux-compiled code, for example, will not
run on Windows, Unix nor MacOS machines.

50
 Overview of systems programming - 44
Tools for Systems Programming - Debuggers
A Debugger is a program that is used to test and debug
other programs
There are two main advantages of using a debugger:
• It allows you to control the running (i.e., execution) of your
code.
o can start/stop/pause your program
o good to slow things down in time-critical and resourcesharing scenarios

• It allows you to investigate what is happening in your

program
o can view your variables in the midst of your program
o can observe the control flow of your program to find out whether certain methods are
being called and in what order

51
 Overview of systems programming - 45
Computer organization

52
 Overview of systems programming - 46
Computer organization When a program load

53
 Overview of systems programming - 47
Computer organization When printf(“Hello World\n”) is invoked

54
 Overview of systems programming - 48
Memory matters - array programming example

55
 Overview of systems programming - 49
Memory matters - array programming example
Memory layout of the array programming example (Note that, in limited memory, some data
are swapped out and in )

56
 Overview of systems programming - 50
CPU also matters - Loop unrolling example
Two programs show different resource utilization in CPU

57
 Overview of systems programming - 51
Abstraction
Key of System Program: Abstraction
• Abstraction is the process of generalization by reducing
the information content of a concept or an observable
phenomenon, typically in order to retain only information
which is relevant for a particular purpose.
• In computer science, abstraction tries to reduce and factor
out details so that the programmer can focus on a few
concepts at a time. A system can have several abstraction
layers whereby different meanings and amounts of detail
are exposed to the programmer.

58
 Overview of systems programming - 52
Abstraction
CPU:

59
 Overview of systems programming - 53
Abstraction
Multitasking:

60
 Overview of systems programming - 54
Abstraction
Memory management:

61
 Overview of systems programming - 55
Abstraction
File system:

62
 Overview of systems programming - 56
Abstraction
Device driver:

63
 Overview of systems programming - 57
Abstraction
Data representation:

64
 Overview of systems programming - 58
Abstraction
Security and reliability:

65
 Overview of systems programming - 59
Abstraction
Software layers (Layered architecture):

66
 Overview of systems programming - 60
Summary
• Definition of System Program
o Supporting computing environments
o Managing hardware directly

o 3 Types of System Program

o Compilation system, operating system, runtime system
o Hardware consideration

• Concept of Abstraction
o Information hiding
o Layered architecture

67
References

• Dr. Christine Laurendeau

• Dr. Jongmoo Choi

68