Learning Progress Review

LINUX, DOCKER & DOCKER COMPOSE

Composed by DigiRUNS – Batch 10:

1. Anissa Noverika P
2. Fadhil Rausyanfikr
3. Muhammad Irham Dinan
4. Muhammad Eko Purwanto
 Linux & Unix Administration System
What is Unix?
• The Unix operating system is a set of programs that act as a link between the computer and the user.

• It allocates the system resources and coordinate all the details of the computer's internals is called
the operating system or the kernel.

• Users communicate with the kernel through a program known as the shell. The shell is a command
line interpreter; it translates commands entered by the user and converts them into a language that is
understood by the kernel.

• The Unix operating system comprises three parts: The kernel, the standard utility programs, and the
system configuration files.
 LINUX VS UNIX
UNIX LINUX
A family of multitasking, multi-user computer
A family of free and open-source software operating
operating systems that derived from the original AT &
system built around the Linux kernel
T Unix
Source code is available to the general public
Source code is not available to the general public

Contains the Command Line Interface and Graphical

Contains the Command Line Interface User Interface

Used for servers, workstations, mainframes and high- Used for personal computers, desktops; also used for
end computers game development, embedded system etc
Not portable Portable and can be executed on various hard drives
Variants: Solaris, HP UNIX, BSD, AIS etc Variants: Ubuntu, Fedora, Red Hat, CentOS, Debian etc
Supports xfs, ramfs, nfs, vfat, cramfsm, ext3, ext4,
Supports zfs, js, hfx, gps, xfs, vfxs file systems
ext2, ext1, ufs, autofs, devpts, nts file systems
Installation requires more sophisticated high-end
Does not require more specific hardware components
hardware
Expensive Free
 What is Linux?
• Linux is a free and open-source
operating system which is made up of
several software components. They
collectively manage hardware
resources and allow users to access
the web or edit a text file in a text
editor.

• Since Linux is an open-source

operating system, individuals can use,
copy, study, and modify files in any
way they like.
Linux Distro

o A Linux distribution, often shortened to Linux distro, is an operating system compiled from
components developed by various open source projects and programmers.

o Each distribution includes the Linux kernel (the foundation of the operating system), the
GNU shell utilities (the terminal interface and commands), the X server (for a graphical
desktop), the desktop environment, a package management system, an installer and other
services

o A single Linux distribution may contain thousands of software packages, utilities

and applications.
Linux Distro
 Linux Command
I. COMMAND LINE INTERFACE (CLI)
• A Linux command is a program or
utility that runs on the CLI.

• Command Line Interface (CLI) is

an interface that accepts lines of
text and processes them into
instructions in a computer.
 II. Ownership & Permission type

Since Linux is a clone of UNIX, the multi-

user operating system which can be
accessed by many users simultaneously, it
raises security concerns as an unsolicited
or malign user can corrupt, change or
remove crucial data. Thus, for effective
security, Linux divides authorization into 2
levels:
1. Ownership
2. Permission
 A. OWNERSHIP
Every file and directory on your Unix/Linux system is assigned 3 types of
owners:
• User: The person who created a file becomes its owner. Hence, a user is also
sometimes called an owner.

• Group: A user- group can contain multiple users. All users belonging to a group
will have the same Linux group permissions access to the file

• Other: Any other user who has access to a file. This person has neither created
the file, nor he belongs to a usergroup who could own the file. Practically, it
means everybody else
 B. PERMISSION [1/5]

The permission system in Linux defines user’s behaviour. Every file and directory in
your UNIX/Linux system has following 3 permissions defined for all the 3 owners:

• Read: This permission gives individual the authority to open and read a file. Read
permission on a directory gives one the ability to lists its content.

• Write: The write permission gives individual the authority to modify the contents of
a file stored in the directory.

• Execute: In Windows, an executable program usually has an extension “.exe” and

which individuals can easily run. In Unix/Linux, individual cannot run a
program unless the execute permission is set. If the execute permission is
not set, one might still be able to see/modify the program code (provided
read & write permissions are set), but not run it.
 [2/5]
• In Linux, we can set permissions (read, write, execute) on a file/directory for the
owner, group and the world by using the ‘chmod’ command which stands for
‘change mode’

• There are 2 ways to use the command:

1. Absolute mode
2. Symbolic mode
 Absolute (Numeric) Mode in Linux [3/5]

• In this mode, file permissions are not represented as characters but a three-digit octal number.

Number Permission Type Symbol

0 No Permission —
1 Execute -x
2 Write -w-
3 Execute + Write -wx
4 Read r–
5 Read + Execute r-x
6 Read +Write rw-
7 Read + Write +Execute rwx
 Symbolic Mode in Linux [4/5]

• In the symbolic mode, you can modify permissions of a specific owner. It makes use of
mathematical symbols to modify the Unix file permissions.

Operator Description
+ Adds a permission to a file or directory
– Removes the permission

Sets the permission and overrides the

=
permissions set earlier.
 [5/5]

• The various owners are represented as –

User Denotations
u user/owner
g group
o other
a all
 [1/2]
III. CRONTAB
• Cron is a system process that will automatically perform tasks as per the specific
schedule.

• Crontab, stands for “cron table”, is also the name of the program, which is used to
edit that schedule.

• Crontab is popular because it can be scheduled to run an automated process as

root. Therefore, having an automated process running as root makes system
changes easier.
 Crontab Format [2/2]
• Crontab of Linux has six fields. The first five fields define the time and date of execution, and
the 6’th field is used for command execution. The syntax is :
 IV. Linux Basic Command [1/4]
No. Command Function

To find the path of your

1 pwd command
current working directory

To navigate through the

2 cd command
Linux files and directories

Lists files and directories

3 ls command
within a system

Compares two contents of

a file line by line. After
4 diff command analyzing them, it will
display the parts that do
not match.
It lists, combines, and
5 cat command writes file content to the
standard output.
No. Command Function
[2/4]
To copy files or
6 cp command directories and their
content
To move and rename files
and directories.
7 mv command Additionally, it doesn’t
produce an output upon
execution.

To create one or multiple

directories at once and
8 mkdir command
set permissions for each
of them.

To delete files within a

9 rm command
directory.
No. Command Function
[3/4]
To find a word by searching
10 grep command through all the texts in a
specific file
Allow users perform tasks
sudo (SuperUser Do)
11 that require administrative
command
or root permissions.
To create an empty file or
generate and modify a
12 touch command
timestamp in the Linux
command line.
Modifies a file or directory’s
13 chmod command read, write, and execute
permissions
Allows user to change the
ownership of a file,
14 chown command
directory, or symbolic link to
a specified username.
 [4/4]
No. Command Function
The system will list up to 500
previously executed commands,
15 history command
allowing user to reuse them
without re-entering
Displays a line of text or string
16 echo command
using the standard output
To terminate an unresponsive
program manually. It will signal
17 kill command misbehaving applications and
instruct them to close their
processes.
To search for files within a
18 find command specific directory and perform
subsequent operations.
 [1/4]
Linux Directory Hierarchy

“/” – root directory: Inside root, there are various directories

(folders) and each of these directories is used to store a specific type of data.
 [2/4]

/bin directory: The bin is short for binaries. It contains all the binary files for the applications.

/sbin directory: It contains all the system binaries that a system administrator will use.

/boot directory: It contains all of the files which an operating system needs to boot.

/dev directory: All the devices of the computer will be listed here.

/etc directory: This folder is where all of your configurations which are system-wide like apt,
source.list, etc will be stored.

/lib32 and /lib64 directory: These are where all of the libraries are stored.
 [3/4]

/mnt and /media directory: “mnt” stands for the mount. All of the devices which are mounted
manually will be displayed here. Media directory contains devices like
the USB, cdrom, etc.

/opt directory: “opt” stands for optionals. It is where all of the proprietary programs like google
chrome, team viewer, etc are stored.

/root directory: the “/” and “/root” both are two different things. “/root” is the root directory for
the user who has root privileges. User need to have root permission to access the
“/root” directory.

/srv directory: This directory is used when user have a server running on its system. So, all the files
of services data can be accessed here.
 [4/4]

/sys directory: This directory is used to interact with the kernel. The data available here is not
physically written to the disk. It is created every time when a system boots up.

/tmp directory: It contains all the temporary files. All the applications which are running on the
system use this directory to store temporary data.

/usr directory: All the applications which are installed by the user will reside here. It is also known as
“Unix System Resource”.

/var directory: var is a variable directory. It stores information related to the system log files, system
crash reports, etc.

/home directory: It is where all of the users and there files and folders will be displayed.
 [1/8]
Linux Disks & File Systems

• The primary role of file systems in Linux is to hold files.

• Linux, like UNIX, uses file systems to represent other objects that the system
needs to operate.

• Linux uses a two-part software implementation as a way to improve both system

and programmer efficiency.
 [2/8]

• The first part of this two-part implementation is the Linux virtual filesystem which provides a single set
of commands for the kernel, and developers, to access all types of filesystems.
• The filesystem-specific device drivers are the second part of the implementation. The device driver
interprets the standard set of filesystem commands to ones specific to the type of filesystem on the
partition or logical volume.
A. Disks [3/8]

• A disk is a piece of hardware that is capable of storing data.

• A disk can be, and often is, divided into partitions. Each partition can then be assigned a
different purpose.

• Disk partitions are displayed in the first column of output from the df command, as shown
below:

This entry shows the file representing partition 1 on the first hard drive (hda). Depending on
the hardware configuration, hard disks may also appear as sda, sdb, and such or with a higher
letter designation, i.e. hde or higher.
• Command check disk : [4/8]
o du command: It is used to check the information of disk usage of files and
directories on a system. It displays a list of all the files along with their
respective sizes. By default, size given is in kilobytes.

o df command: The df command stands for "disk-free," and shows available and used
disk space on the Linux system.
o fdisk -l command: It shows disk size along with disk partitioning information

o parted -l command: The parted is a command line tool that helps to manage the
hard disk partitions. The parted -l command is used to lists hard
disk partitions
 [5/8]
B. File Systems
• A file system is a set of methods to access
and organize storage.

• It is comprised of some amount of storage

from a disk or disks, a logical method to
name, organize and secure its contents, and
a set of system calls to allow programs,
including the operating system, to interact
with it.
1. Ext, Ext2, Ext3 and Ext4 file system [6/8]

- The file system Ext stands for Extended File System.

- Ext2 is the first Linux file system that allows managing two terabytes of data.

- Ext3 is developed through Ext2; it is an upgraded version of Ext2 and

contains backward compatibility.

- Ext4 file system is the faster file system among all the Ext file systems. It is a
very compatible option for the SSD (solid-state drive) disks, and it is the
default file system in Linux distribution.
2. JFS File System [7/8]
- JFS stands for Journaled File System
- It is an alternative to the Ext file system. It can also be used in place of Ext4,
where stability is needed with few resources. It is a handy file system
when CPU power is limited.

3. ReiserFS File System

- ReiserFS is an alternative to the Ext3 file system and have improved
performance and advanced features.

4. XFS File System

- It is considered as high-speed as JFS.
- NASA still uses this file system with its high storage server
(300+ Terabyte server).
 [8/8]
5.Btrfs File System
- Btrfs stands for the B tree file system.
- It is used for fault tolerance, repair system, fun administration, extensive
storage configuration, and more. It is not a good suit for the
production system.

6. Swap File System

- The swap file system is used for memory paging in Linux operating system
during the system hibernation.
 Linux Kernel & Boots [1/3]

- The Linux kernel is the main component of a Linux operating system and is the core interface between
a computer’s hardware and its processes.
- The kernel has 4 jobs:
a. Memory management: Keep track of how much memory is used to store what, and
where.

b.Process management: Determine which processes can use the central processing unit
(CPU), when, and for how long.

c. Device drivers: Act as mediator/interpreter between the hardware and processes.

d.System calls and security: Receive requests for service from the processes.

- It is very important to learn the booting process to understand the working of any operating system.
• The stages of Linux boot process are:
[2/3]
 [3/3]

1. The machine’s BIOS or boot microcode hundreds and runs a boot loader.

2. Boot loader finds the kernel image on the disk and loads it into memory, to start the system.

3. The kernel initializes the devices and their drivers.

4. The kernel mounts the basis filesystem.

5. The kernel starts a program referred to as init with a method ID zero

6. init sets the remainder of the system processes in motion.

7. For some purpose, init starts a method permitting you to log in, typically at the top or close
to the top of the boot sequence.
 System Logging
What so Important?

- In Linux, system logs are human-readable records of the core system activities, such as: user logins
and login failures, operating system booting, system failures, etc, that performed by services,
daemons, and system applications.

- Syslog is specifically responsible for creating logs via the System Logger. Syslog comprises of several
components such as the Syslog Message Format, Syslog Protocol, and the Syslog Daemon.

- The /var/log directory stores most of the logs on a Linux system. The /var directory mostly contains
variable files and directories i.e data that is bound to change often.
 System Time [1/3]

What so Important?

- In Linux, time is measured in seconds since the Unix epoch. The Unix epoch is January 1, 1970 UTC.
This means that the time on your Linux machine is always relative to this date.

- Checking the time on your Linux machine is important for a number of reasons. For example, if you
are running a cron job, you will want to make sure that the time is set correctly so that the job will
run at the correct time.
 System Time [2/3]

How to check time

- There are a few different ways to check the time on your Linux machine:
a) to use the date command.
This command displays the current date and time in human-readable format.

b) to use the timedatectl command.

This command allows you to view and change the system time and date settings.

c) to use the hwclock command.

This command shows or sets hardware clock information on Linux systems.
 System Time [3/3]

Date Format Options

- The following are the date format options: $date +%[format-option]

- %D: Display date as mm/dd/yy. - %m: Displays the month of year (01 to 12).
- %d: Display the day of the month (01 to 31). - %y: Displays last two digits of the year(00 to 99).
- %a: Displays the abbreviated name for weekdays (Sun to Sat). - %Y: Display four-digit year.
- %A: Displays full weekdays (Sunday to Saturday). - %T: Display the time in 24 hour format as HH:MM:SS.
- %h: Displays abbreviated month name (Jan to Dec). - %H: Display the hour.
- %b: Displays abbreviated month name (Jan to Dec). - %M: Display the minute.
- %B: Displays full month name(January to December). - %S: Display the seconds.
 Users
What so Important?

- Users command in Linux system is used to show the user names of users currently logged in to the
current host. It will display who is currently logged in according to FILE. If the FILE is not specified,
use /var/run/utmp. /var/log/wtmp as FILE is common.

User Environment
There are several commands available that allow us to list and set environment variables in Linux:
• env – This command allows us to run other programs in a specific environment without changing
the current one.
• printenv – print the environment variables you want to view.
• set – puts a value into an environment variable.
• unset – remove environment variables.
• export – put values ​into an environment variable.
 Linux Process
What so Important? Commands

- An instance of a program is called a Process. - $Top

In simple terms, any command that you give This utility tells the user about all the running
to your Linux machine starts a new process processes on the Linux machine.

Type of Processes

- Foreground Processes: They run on the

screen and need input from the user.
- Background Processes: They run in the
background and usually do not need user
input.
 Shell Script Programming
What is It?

- A computer program can run in the UNIX shell command line interpreter for file manipulation,
program execution and printing text is called Shell Scripting.
- Individual commands follow a sequence and perform the operations and hence it is known as Shell
Script. This is useful for the repetitive tasks in the system administration.

What so useful?

- We need to write shell scripts to avoid repetitive work and automation, to perform the routine
backups by system admins using shell scripting, too Adding new functionality to the shell, to
perform System monitoring and so on.
 DOCKER
What is docker?

Docker is an open source platform

that enables developers to build,
deploy, run, update and
manage containers—standardized,
executable components that combine
application source code with the
operating system (OS) libraries and
dependencies required to run that
code in any environment.
 Why Use Docker?

• Single host deployment - This means you can run everything on a single piece of
hardware

• Quick and easy configuration - Due to YAML scripts

• High productivity - Docker Compose reduces the time it takes to perform tasks

• Security - All the containers are isolated from each other, reducing the threat landscape
 Docker Architecture [1/3]
Docker follows Client-Server architecture, which includes the three main components that are Docker Client,
Docker Host, and Docker Registry.
 [2/3]
1. Docker Client
• Docker client uses commands and REST APIs to communicate with the Docker Daemon
(Server).
• When a client runs any docker command on the docker client terminal, the client terminal
sends these docker commands to the Docker daemon.
• Docker daemon receives these commands from the docker client in the form of command
and REST API's request.
• Docker Client uses Command Line Interface (CLI) to run the following commands:
• Docker build
• Docker pull
• Docker run
 [3/3]

2. Docker Host 3. Docker Registry

• Docker Host is used to provide an • Docker Registry manages and stores the
environment to execute and run Docker images.
applications. • Docker Client uses Command Line
Interface (CLI) to run the following
• It contains the docker daemon, commands:
images, containers, networks, and • Pubic Registry - Public Registry is also
storage. called as Docker hub
• Private Registry - It is used to share
images within the enterprise.
 Docker Basic Command [1/4]

Docker search—
o We can use the command to search for public images on the Docker hub.
o It will return information about the image name, description, stars, official
and automated.

Docker pull –platform linux/x86_64 mysql

o Now that we know the name of the image, we can pull that from the
Docker hub using the command pull.
o Tags are used to identify images inside a repository.
o If we don’t specify a tag Docker engine uses the :lastest tag by default.
o So, in the previous example, Docker pulled the mysql:latest image
Docker images
Now we should have some images in our local machine, and to confirm let’s run the following
[2/4]
command to list all the local images.

Docker run
–env MYSQL_ROOT_PASSWORD=my-secret-pw –detach mysql

Docker ps
We can list all the running containers by using the following command.
Docker stop [3/4]
To stop a container, use the command with either the container id or container name. We
may stop a container if we want to change our docker run command.

Docker restart
To restart our stopped contained

Docker rename
container_before container_after

Docker exec
To run a new command in a running container

Docker logs
This command is helpful to debug our Docker containers. It will fetch logs from
a specified container.
 [4/4]

Docker rm container_name
In case we want to remove a container, we can use the following command.

Docker rmi
If we want to free some disk space, we can use the command with the image id to
remove an image.
 Docker Object [1/4]

1. Images
➢ Docker images are the read-only binary templates used to create Docker
Containers.
➢ It uses a private container registry to share container images within the
enterprise and also uses public container registry to share container images
within the whole world.
➢ Metadata is also used by docket images to describe the container's abilities.
 [2/4]
2. Docker containers
➢ Containers are the structural units of Docker, which is used to hold the entire
package that is needed to run the application.

➢ The advantage of containers is that it requires very less resources.

➢ In other words, we can say that the image is a template, and the container is a
copy of that template.
3. Docker Networking [3/4]

Using Docker Networking, an isolated package can be communicated. Docker

contains the following network drivers:

❖Bridge - Bridge is a default network driver for the container. It is used when
multiple docker communicates with the same docker host.
❖Host - It is used when we don't need for network isolation between the
container and the host.
❖None - It disables all the networking.
❖Overlay - Overlay offers Swarm services to communicate with each other. It
enables containers to run on the different docker host.
❖Macvlan - Macvlan is used when we want to assign MAC addresses to the
containers.
 [4/4]
4. Docker Storage

Docker Storage is used to store data on the container. Docker

offers the following options for the Storage:

❖ Data Volume - Data Volume provides the ability to create persistence

storage. It also allows us to name volumes, list volumes,
and containers associates with the volumes.
❖ Directory Mounts - It is one of the best options for docker storage. It
mounts a host's directory into a container.
❖ Storage Plugins - It provides an ability to connect to external storage
platforms.
 Docker Deployment
Here is the basic process of developing and deploying a web app using
docker:

1. Install Docker on the machines you want to use it.

2. Set up a registry at Docker Hub.
3. Initiate Docker build to create your Docker Image.
4. Set up your ’Dockerized‘ machines.
5. Deploy your built image or application.
Docker Deployment
 Dockerfile vs Docker Compose
• Dockerfile and Docker Compose are both part of the Docker universe
but are different things with different functions.
• A Dockerfile describes how to build a Docker image,
• Docker Compose is a command for running a Docker container.
1. Dockerfile
o A Dockerfile is a text document that contains all the commands a user needs to
build a Docker image, a file used to execute code in a Docker container.
o When a user runs the Docker run command and specifies WordPress, Docker
uses this file, the Dockerfile, to build the image.
2. Docker Compose

Command:
• docker-compose up [OPTION]
• docker-compose up -d

Docker Compose is a tool that assists in defining and sharing multi-container

applications.

By using Compose, we can define the services in a YAML file, as well as spin them
up and tear them down with one single command.
Docker Compose
 Docker for Data Engineering
A brief of Problems

- A data science project often involves a whole team of data scientists, data engineers, software
architects, often working along with other software development teams to create a viable solution.
You can have a situation where different data scientists end up working with different versions of
the library only to realize after hours of debugging that there are small differences in their
environments. Docker lets you create a consistent environment for data scientists and data
engineers to deal with these kinds of situations.

Docker as Solutions

- Ease of model building - Dockerizing your data science project can help set things up faster
- Deployment - Deploying a data science solution is much easier with docker in place.
 Docker for Data Engineering
Use Case

For example, if you wanted to deploy Airflow, you could just have a VM, install Python, Postgres and Airflow. But,
what if you also needed a different version of Postgres for a different app? What if you also have a whole bunch of
other things that are running on that VM? What if some of them need Python 2 instead of Python 3?

While Airflow isn’t the heaviest piece of software, imagine something like Spark with Python. You’d need Python,
some version of the JVM, some version of Spark, as well as other dependencies. If you have a laptop, there’s a very
high likelihood you’ll run into some dependency conflicts.

Docker allows you to completely isolate processes in a way that’s reproducible. If an image works locally, it’ll work
elsewhere, like production. Docker, as a technology, isn’t technically required anywhere. However, there are loads
of benefits, specifically reproduction of an environment.