ADMIN

Network & Security

DDoS AlmaLinux
Monitoring Defense Build System

Network & Security ISSUE 79

Dashy, LibreNMS, Tier 0 Systems, Graphite

Authelia IAM
ScaleFlux Computational Storage
OSSEC
Intrusion detection and
host-based intrusion prevention
DDoS Defense
Restoring Hybrid Identities
DIY Docker Images Local Azure Arc
Build and host Docker Manage on-premises
images servers in Azure

Kubernetes Backups GitOps

Dos and don’ts of Synchronize repository
backing up K8s storage changes DVD
E
WWW.ADMIN-MAGAZINE.COM
FR E
 Welcome to ADMIN W E LCO M E

Technology Predictions
for 2024 and Beyond
I must admit that I really hate it when I read industry pundit predictions
for the next year. No, seriously, I really hate the predictions because they’re
generally based on nothing but conjecture or the hope of filling a page
or perhaps starting a flame war to drive up page views.
I enjoy writing pieces like this as much as I enjoy writing an article
in “corporate speak” or the latest marketing buzzword-laden rant
about some aspect of mundane technology. I do rather enjoy be-
ing an “in your face” antagonist of sorts. Maybe it’s age. Maybe
it’s my own personality flaw. Or maybe it’s my techno-lingo-jaded
self that wants to scream every time I hear terms such as single
pane of glass, hyper-converged architecture, or quiet cutting.
Yeah, it’s that last one, for sure.
Wait no longer. Here are my 10 technology predictions for 2024
and beyond. Be warned that these predictions are chock-full of
corporate speak, buzzwords, and my own sarcastic creations.
1. Industry pundits will predict massive artificial intelligence (AI)
adoption across all business sectors.
2. Technology companies will make more “tough decisions” and
perform more quiet cutting of their US-based workforce.
3. An industry pundit will predict “The 2024 Tech Boom.”
4. A different industry pundit will predict “The Great 2024 Tech Crash.”
5. A technology writer will post an article titled “2024: The Year of the
Linux Desktop.”
6. Industry analysts will predict that AI will replace tech workers.
7. Someone will write an article titled “The 10 Best Tech Stocks to Invest in Right Now.”
8. A major financial magazine will publish an article discussing “The Great 2024 Tech Stock Selloff.”
9. There will be a new Facebook hoax that will hit the national newsfeeds as fact.
10. Elon Musk will buy the Moon and charge us a monthly subscription fee to look at it.
Yes, these predictions are meant to be humorous, but I wish someone would give me $10,000 for each one that comes
true. I wouldn’t be able to retire, but I’d have a nice vacation and some new camera equipment to show for it.
I’ve predicted for 20 years that tech workers will someday band together and create a Tech Workers Union. I know
there have been a couple of solid attempts over the years, and there might still be one or two out there, but they
haven’t made enough of an impact to gather a big following. I attempted to start such a union in the early 2000s
but was threatened by my employer, so I scrapped the project. Organizing a union does me no good if I have no
oppressive and exploitative job to defend myself against. The company also sent a very strong message to all others
with similar aspirations. I’m still hopeful that, someday, tech workers will unite and protect themselves, but perhaps
Lead Image © rendeeplumia, 123RF.com

by then, the prediction of mass tech worker extinction at the hands of AI will have come true.
Whatever technology predictions come true for 2024, you can place your money on one sure thing – none of
them will benefit the tech worker in any capacity. We are at the bottom of the corporate food chain, supply all
the labor, and keep the business running from the trenches, and yet, we are the ones that have targets on our
backs. I think it would be far more profitable to keep the lowly tech workers employed and replace our highly
paid overlords with their much lower cost AI alternatives. Think of the millions of dollars that move would save.
But I predict that it won’t happen. Not in 2024 or my lifetime, anyway.
Ken Hess • ADMIN Senior Editor

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 3
S E RV I C E Table of Contents

ADMIN Network & Security

Features Tools Containers and Virtualization

10 Dashy 28 AlmaLinux OS 46 DIY Docker Images

Create your own command The AlmaLinux Build System lets When facing the challenge of
center in small infrastructures or you build, test, sign, and release packaging your application in a
test environments with flexible packages from a single interface. container, take into account your
dashboards that control and needs in terms of handling and
monitor relevant applications 32 Authelia security and investigate sensible
and services. Add access controls to web options for hosting your own
applications that do not have registry.
14 LibreNMS their own user administration.
Check out this monitoring 52 Kubernetes Backups
environment with auto-discovery, 34 iRedMail Stateful applications that store
alerting, the ability to scale Deploy this full-featured email their information in a container’s
even in very large environments server on a number of platforms persistent volume can be backed
with many devices, and flexible in a matter of minutes. up in a variety of ways.
dashboards and widgets for
special views. 40 ScaleFlux 56 Local Azure Arc
See performance gains of 50 Manage your on-premises servers
19 Tier 0 Monitoring percent and more with computing with Windows Admin Center in
We show you how monitoring power built directly into the Azure.
your sensitive IT systems can be network card.
a more secure experience.
Security
24 Graphite
This open source tool offers 60 OSSEC
real-time monitoring with Detect and fix security
comprehensive and fast data problems in real time at the
collection from virtually any operating system level with
system. functions such as log analysis,
file integrity checks, Windows
registry monitoring, and rootkit
detection.

Service
3 Welcome
6 News
97 Back Issues
98 Call for Papers

4 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Table of Contents S E RV I C E

10, 14, 19, 24 | Monitoring

Tools for your IT systems
This issue takes a deep dive into monitoring solutions
for your IT infrastructure, including Dashy, LibreNMS,
Tier 0 systems, and Graphite.

Highlights

40 ScaleFlux 46 DIY Docker Images 60 OSSEC

Relieve load on the CPU and Building Docker containers is This powerful free intrusion
ensure higher bandwidth not particularly complicated, detection and host-based
and lower latency by simply and CI/CD tools and a good intrusion prevention system
replacing your storage DIY image registry will help can detect and combat malware
hardware with computational make the experience more and cyberattacks and can even
storage. convenient. be run on a virtual machine.

Management Nuts and Bolts On the DVD

64 Restoring Hybrid 74 GitOps FreeBSD 14.0

Identities Applying DevOps practices The FreeBSD open source operating sys-
We look into contingency through infrastructure automation tem derives from 4.4 BSD Lite2 and can
measures for hybrid directory of version control repositories be used for everything from software
services with Entra ID, the is a popular approach to development to games to Internet ser-
Graph API, and its PowerShell synchronizing system changes, vice provision. The base distribution has
implementation. both in and outside the context full source code for the kernel and all
of Kubernetes. utilities, and the Ports Collection makes
70 Ralph Asset Management it easy to install your favorite traditional
Keep things simple, without 80 BGP Routing Protocol Unix utilities. You will also find:
compromising flexibility, when We look at the Border Gateway • PIE support enabled by default
managing data centers with this Protocol, how it routes packets
open source asset management through the Internet, its
• Unprivileged operation with chroot
system and configuration weaknesses, and some hardening • Support for NFS-over-TLS with two new
database. strategies. daemons
Before installing, check the errata docu-
86 DDos Defense ment online (https://www.freebsd.org/
Targeted attacks cannot be releases/14.0R/errata/).
prevented, but they can be
effectively mitigated.
@adminmagazine
91 Terminating OpenSSH
@adminmag Disconnect OpenSSH user sessions
after a certain period of inactivity
with the systemd-logind service.
ADMIN magazine
93 Performance Dojo
@adminmagazine Exploring low-cost parallel
computing.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 5
 NEWS ADMIN News

News for Admins

Tech News
Vim 9.1 Now Available
The Vim project has announced the release of version 9.1 of the popular text editor.
“This release is dedicated to Bram Moolenaar, Vim's lead developer for more than 30 years, who
passed away half a year ago,” the announcement states. “The Vim project wouldn't exist without
his work!”
Vim 9.1 is mainly a bug fix release, but it also offers new features and improvements, such as:

• Smooth scroll support

• New :defer command to help with cleaning up a function
• Support for adding virtual-text to a buffer
• Support for Vim9 classes and objects for the Vim9 scripting language

Download the latest release from the Vim website (https://www.vim.org/download.php).

Microsoft Introduces Copilot Key to PC Keyboards

Microsoft has announced the addition of a new Copilot key for Windows PC keyboards, marking
the first significant change to the keyboard in nearly three decades.
“The Copilot key joins the Windows key as a core part of the PC keyboard,” says Yusuf Mehdi
in the announcement (https://blogs.windows.com/windowsexperience/2024/01/04/introducing-a-new-copilot-key-to-
kick-off-the-year-of-ai-powered-windows-pcs/), “and when pressed, the new key will invoke the Copilot in
Windows experience.”
A company disclaimer notes that the timing and availability of the Copilot feature will vary by
market and device and will require a Microsoft account.

Google Announces AI Hypercomputer

In addition to its release of the Gemini AI model (https://www.fosslife.org/google-announces-gemini-ai-model),
Get the latest Google has announced an AI Hypercomputer, “a groundbreaking supercomputer architecture that
IT and HPC news employs an integrated system of performance-optimized hardware, open software, leading ML
Lead Image © vlastas, 123RF.com

in your inbox frameworks, and flexible consumption models.”

According to the announcement, “AI Hypercomputer employs systems-level codesign to boost ef-
Subscribe free to ficiency and productivity across AI training, tuning, and serving.”
ADMIN Update The AI Hypercomputer “features performance-optimized compute, storage, and networking”
and HPC Update as well as out-of-the-box support for “popular ML frameworks such as JAX, TensorFlow, and
bit.ly/HPC-ADMIN-Update PyTorch,” the announcement says.

6 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
NEWS ADMIN News

Best Practices to Prepare for Post-Quantum Security Challenges

“The rise of quantum computing introduces a unique set of challenges related to the manage-
ment of digital certificates,” says Murali Palanisamy (of AppViewX) on the RSA Conference
Library blog.
“Unfortunately, today’s encryption is widely understood to be vulnerable to quantum attacks;
researchers have already shown a quantum computer could break some of the tougher encryption
used today (considered unbreakable until now) in 104 days (https://www.fujitsu.com/global/about/resources/
news/press-releases/2023/0123-01.html). Palanisamy, he notes, as quantum computing evolves, that time
could shrink to hours, minutes, or only seconds.
In the article Palanisamy outlines best practices to help security professionals prepare for a
quantum future, including:

• Get full visibility, automation, and control of certificates.

• Establish flexibility for the transition to post-quantum algorithms
• Implement strong certificate and key management practices
• Monitor compliance

Read more at the RSA Conference Library (https://www.rsaconference.com/library/blog/

six-steps-for-mitigating-quantums-impact-on-digital-certificates).

Open Networking Foundation Projects Transferred to LF

The Open Networking Project (ONF) has announced that its open source networking proj-
ects will become independent projects under the Linux Foundation (LF), and the ONF will be
dissolved.
“Linux Foundation is merging ONF’s marquee portfolio of broadband, mobile, edge, and cloud
networking projects under the LF umbrella to help usher in the next phase of community growth,”
says Jim Zemlin, LF executive director.
“The move creates independent, community-led governance for the three major project areas:
Broadband, Aether, and P4, and sets the projects up for broader collaboration and adoption,” the
announcement states (https://opennetworking.org/news-and-events/press-releases/onf-merges-market-leading-portfo-
lio-of-open-source-networking-projects-into-the-linux-foundation/). Current ONF members will be welcomed to
the new projects with transition support.
Supporting organizations for the new projects include Cornell University, Deutsche Telekom,
Google, Intel, Netsia, Radisys, and Türk Telekom, the announcement says.

IBM Hybrid Cloud Mesh Now Generally Available

IBM’s multi-cloud networking solution, IBM Hybrid Cloud Mesh (https://www.ibm.com/products/hybrid-
cloud-mesh), which was introduced earlier this year, is now generally available.
The product is “designed to allow organizations to establish simple, scalable secured
application-centric connectivity,” the company says (https://www.ibm.com/blog/announcement/
app-centric-connectivity-a-new-paradigm-for-a-multicloud-world/).
It is “engineered for both CloudOps and DevOps teams to seamlessly manage and scale network
applications, including cloud-native ones running on Red Hat OpenShift.”
Key features include:

• Continuous infrastructure and application discovery

• Seamless connectivity
• Security
• Observability
• Traffic engineering capabilities

According to the announcement, key architecture components are the Mesh Manager and the
Edge and Waypoint Gateways.

8 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 ADMIN News NEWS

IBM Announces Quantum System Two

IBM recently announced IBM Quantum Heron, which is part of a series of quantum processors engineered
to deliver “the highest performance metrics and lowest error rates of any IBM Quantum processor to date.”
Specifically, IBM Heron offers a “five-times improvement over the previous best records set
by IBM Eagle,” the company says (https://www.ibm.com/quantum/blog/quantum-roadmap-2033).
Additionally, IBM has revealed the IBM Quantum System Two – a modular quantum computer –
which has already begun operations with three IBM Heron processors and supporting control elec-
tronics, according to the announcement.
With these pieces in place, the company has extended its Quantum Development Roadmap
to 2033, with expectations of increasing “the size of quantum circuits able to be run and help
to realize the full potential of quantum computing at scale.”
"We are firmly within the era in which quantum computers are being used as a tool to explore
new frontiers of science," said Dario Gil, IBM SVP and Director of Research.

Red Hat to Remove Xorg from RHEL 10

Red Hat has announced the decision to remove Xorg server and other X servers (except Xwayland)
from Red Hat Enterprise Linux 10.
In a blog post (https://www.redhat.com/en/blog/rhel-10-plans-wayland-and-xorg-server), Carlos Soriano San-
chez explains that the transition from the X Window System to the newer Wayland-based stack has
been happening for the past 15 years. But, now, he says, “Wayland has been recognized as the de
facto windowing and display infrastructure solution.”
During this transition period, Red Hat has been supporting both the Xorg and Wayland stacks,
and this decision will allow the development community to focus their efforts solely on a modern
stack and ecosystem, Sanchez says

European Commission Launches Large AI Grand Challenge

The European Commission (EC) has launched a new competition called the Large AI Grand Chal-
lenge (https://aiboost-project.eu/large-ai-grand-challenge/), which aims to foster innovation and excellence
in large-scale AI models.
“Participants in the challenge are invited to submit a proposal for the development of a language
foundation model, utilizing one of the EuroHPC JU targeted facilities (i.e., LUMI or Leonardo
supercomputers). The model must be trained from scratch, possess a minimum of 30 billion
parameters, and be trained following state-of-the-art optimal scaling laws for computing and
training data size,” the website says.
The competition, launched in collaboration with EuroHPC Joint Undertaking (https://eurohpc-ju.
europa.eu/), will run from November 16 to January 16, 2024.

Linux Foundation to Form High Performance Software Foundation

The Linux Foundation (LF) has announced its intent to form the High Performance Software
Foundation (HPSF).
“Through a series of technical projects, HPSF aims to build, promote, and advance a portable
software stack for high performance computing (HPC),” the announcement says.
Initial projects within the foundation include:

• Spack – A flexible HPC package manager.

• Kokkos – A performance-portable programming model for writing modern C++ applications in
a hardware-agnostic way.
• Apptainer – A container system and image format specifically designed for secure high-perfor-
mance computing.
• WarpX – A performance-portable Particle-in-Cell code with advanced algorithms that won the
2022 Gordon Bell Prize.

The formation of HPSF is expected to be completed in May 2024.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 9
 F E AT U R E Dashy

A flexible, customizable, personal dashboard

Command and Control

Create your own command center in small infrastructures or test environments with flexible dashboards
that control and monitor relevant applications and services. By Holger Reibold
Many standardized interfaces be the Docker-based installation. A yarn
can be set up for applications on search in the Docker repository, yarn build
the local network. One of the most start
popular candidates is Webmin, docker search dashy
whose strengths lie in managing To design dashboards, you need icons
legacy company servers. As a tool, reveals the package you need. In the that represent the different areas
though, Webmin is too complex and search results you will find a lissy93/ and links. Dashy comes with a set of
too powerful for many small envi- dashy entry. Download this container standard icons, but if you want to use
ronments. Administrators looking to your system and launch Dashy: the correct icons for popular devices
for a lightweight dashboard system (e.g., a Fritz!Box or a specific envi-
to manage conveniently the links docker pull lissy93/dashy ronment), enter
and entry points to various services docker run -p 8080:80 lissy93/dashy
will find Dashy [1] an interesting cd ./public/item-icons
alternative. The environment performs various git clone https://github.com/walkxcode/U
Dashy is jam packed with useful tests and outputs a success message. dashboard-icons.git
functions for creating individual dash- The local Dashy installation can be
boards. It also lets you integrate sta- accessed on http://localhost:8080. to download an icon set.
tus checks and use dynamic widgets Docker is not actually necessary for
and user-defined layouts. Dashy is deployment; instead, you can install Basic Configuration
Lead Image © peshkova, 123RF.com

open source, and the developers offer Dashy on any standard Linux system.
support on GitHub. Besides Git, you also need Node.js Now that you have Dashy running,
and Yarn: you will want to customize the
Installation environment by editing the YAML-
git clone U based /public/conf.yml configura-
Of the various ways to get Dashy up https://github.com/Lissy93/dashy.git && U tion file. The file has three root
and running, the easiest way has to cd dashy attributes:

10 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Dashy F E AT U R E

Q pageInfo is where you store the that alerts you to possible typing dashboard environment, click on the
dashboard metadata, such as the errors or misconfigurations. Press pencil icon in the Config area to en-
title, the description, the naviga- Save Changes to apply the changes. ter edit mode. Dashy now positions
tion bar links, and the footer text. Simple changes, such as changes a placeholder to let you create the
Q appConfig is where you define to the title, take effect immediately, section, which you click to open the
the dashboard settings, such as whereas for others, you have to section properties. In the Add New
themes, authentication, language, reload the environment with yarn Section dialog, define the key proper-
and customizations. build. ties, such as the name, the icon, the
Q sections is an array of sections, Further cross-system functions are default sort order, and the number
each of which in turn comprises available on the Dashy splash page. of rows and columns. When you
an array of elements, called items. Various design templates are avail- select the icons, you can use the file
The developers provide a complete able in the Layout selection menu names in the icons folder. Click Save
list of all available configuration op- top right. The design configura- to save the section configuration and
tions [2], which is very helpful for tor lets you fine-tune the template populate it with some initial links by
customizing Dashy to suit your needs. details and adjust the layout and clicking on the plus sign.
You can manage authentication, set item size. For all of these tasks, you The procedure for creating a link in
up user-defined themes, or enable always need to enable the Open Set- a section is similar. Follow the Add
reporting, for example. tings Menu at top right. New Item link and specify the prop-
The good news is that you do not erties, such as the name, description,
have to make these adjustments at Composing Dashboards and icon, in the Edit Item dialog
the console level but can access the (Figure 2). In the Service URL input
configuration settings by clicking During the initial configuration, field, specify the target URL – the
on the wrench icon in the Config Dashy comes up with a Getting IP address or hostname – that is
section. Select Update Configura- Started section (Figure 1), which linked to the item entry. In the Open-
tion | Edit Config and change the gives you a first impression of the ing Method input field, you define
settings in the configuration edi- dashboard design approach. You whether access is by a new browser
tor to suit your requirements. To need to create sections and bundle tab or a new window.
modify the title and description of access to specific services and infor- In the More Fields area, you can
your Dashy installation, open the mation in them. In a practical ap- include further information in the
pageInfo section and edit the title plication, you might want, say, Sta- configuration, such as a status
and description options by clicking tus Information, Productivity, and check and item ID. Use Hot Key to
on the values and adjusting them Network sections. Other potential assign a numeric key between 0 and
appropriately. subdivisions could be Applications, 9 to systems that you want to access
To expand the basic configuration, External Services, and Devices. particularly frequently. After saving,
click to the left of the section name Further options can be found in the you will find the initial entries in
and run the Insert command. For Dashy Showcase [3]. the section you just created.
your first steps, it makes sense to If you would like to add a section Removing items and sections is just
output notifications if unexpected of your own design to the existing as simple: In edit mode, click on
errors and
crashes occur.
To do this, use
the enableEr-
rorReporting
option, which
is disabled
by default. In
the configura-
tion editor,
add an entry
of type Auto
and assign
it an option
name and a
value of true.
The editor has
an integrated
syntax checker Figure 1: Once Dashy is installed, the Getting Started section quickly creates your first dashboards.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 11
 F E AT U R E Dashy

the slider icon of a group and run Adapting Dashy box ensures that the online or offline
the Remove command. While in edit status of each service is displayed in
mode, Dashy records this fact in the Once you have gained some initial ex- the dashboard in the form of status
footer. Press Save Locally to save perience with the typical procedure for information. It is well worth enabling
the dashboard configuration and using Dashy-based dashboard design, this option because you can then tell
then access the integrated systems. you can customize the environment at a glance whether all the services
In the footer, you will find further further. In edit mode, you can access you need are available.
backup options, such as an export further configuration options by click- Besides the language, which Dashy
function in Config Saving Options. ing on Edit App Config, which has takes from your browser configura-
Export Config tells Dashy to display the settings for the appConfig section tion, you can also specify the back-
the source code; alternatively, you that you edit in the visual user inter- ground as the Background Image,
can export the configuration file and face editor. The editor basically offers along with the Default Layout and
import it on a third-party system. maximum usability, but at the price of Default Icon Size. If you work with
reliability. Alternatively, you can work different services in parallel, you
Listing 1: Example auth Element directly in the YAML configuration will want to enable the Enable Multi-
appConfig: dialog. According to the developers, a Tasking option, which ensures that
auth: REST API is currently being developed open applications remain open in
users: that will support configuration adjust- the background. However, enabling
- user: <Holger> ments from the command line, scripts, this option comes at the expense of
hash: <Hash for password of User Holger> and third-party applications. performance.
type: admin In the application configuration, use
- user: <Klaus> Default Opening Method to define the Registration and
default method for opening the item
hash: <Hash for password of User Klaus>
settings – the default is a new browser
Authorizations
type: normal
tab. Checking the Enable Status Checks In addition to what are primarily
visual customizations, Dashy has a
login page and front-end authentica-
tion. To enable this feature, you need
to add users to the auth section below
appConfig in the configuration file.
Access protection is not automatically
enabled for a new installation.
In the auth element, you need to
create a user array, assigning each
user a name, a hash, and a user type
(admin or normal). The hash is a SHA-
256 hash of the password. Listing 1
shows an example configuration.
The easiest way to generate the hash
is with an online tool such as a SHA
generator [4]. After enabling authen-
tication, users are redirected to the
login page.
Dashy supports a guest mode that
gives all users read-only access to the
secure dashboard without having to
log in. To set this up, set the appCon-
fig.auth.enableGuestAccess option to
true. The environment also supports
the implementation of granular access
authorizations to make specific sec-
tions or elements visible to or usable
for certain users only. You have three
options from which to choose:
Q hideForUsers defines the sections
and elements that are visible to all
Figure 2: Items – in this case a Synology NAS – appear on a status page as links. users except those in this list.

12 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Dashy F E AT U R E

Q showForUsers defines items hidden under the wrench icon. It lets you dashboards child’s play, and the
from all users except those listed. save the current Dashy configura- extensive customization options,
Q hideForGuests defines items visible tion in the cloud and restore it if whether you use the integrated
for logged-in users, but not for required. Before the data is trans- tools or edit the configuration
guests. ferred, it is encrypted with the file directly, do not affect Dashy’s
Users who are not of the admin type CloudBackup.js script by the AES flexibility. Q
cannot write any changes to the lo- crypto.js method.
cal storage medium. In the appConfig The procedure is otherwise simple:
section, use preventWriteToDisk and Assign a name to the backup, press Info
preventLocalSave to restrict further Update Backup, and your data [1] Dashy: [https://dashy.to]
authorizations. To disable all UI con- moves to the cloud of your choice. [2] Configuration options:
figuration functions, including View You will want keep the ID generated [https://github.com/Lissy93/dashy/blob/
Config, set the disableConfiguration by this process safe so that you can master/docs/configuring.md]
option to true. Alternatively, you use it, together with your password, [3] Dashy Showcase:
can disable the user interface config for the Restore a Backup function, [https://github.com/Lissy93/dashy/blob/
functions for all non-admin users by if needed. In this way, you can also master/docs/showcase.md]
changing disableConfigurationForNon- easily transfer a configuration to a [4] SHA generator: [https://www.liavaag.org/
Admin to true. third-party system. English/SHA-Generator/]

Backing Up the Configuration Conclusions The Author

Holger Reibold, computer scientist, has worked
Cloud Backup & Restore is another Dashy is a small but powerful as an IT journalist since 1995. His main interests
practical function found as a tab tool that makes creating your own are open source tools and security topics.
 F E AT U R E LibreNMS

Versatile network and system monitoring

Digital Watchdog
The LibreNMS open source monitoring environment, unlike its predecessor Observium, comes through the back
door at no cost, with auto-discovery, alerting, the ability to scale even in very large environments with many
devices, and flexible dashboards and widgets for special views. By Thomas Joos

LibreNMS [1] promises flexible net- packages for Ubuntu 20.04/22.4, web GUI and command-line interface
work and system monitoring and CentOS 8, and Debian 11, along with (CLI) or use the auto-discovery mecha-
combines various functions from Docker, virtual machine (VM) im- nism (Figure 1). To set up your initial
tools such as Nagios and Cacti. An- ages, and an online demo [2]. devices in the GUI, navigate to Devices
other advantage of the software is If you opt for a local installation, | Add Device. LibreNMS requires the
that, unlike its predecessor Obser- you can access LibreNMS on http:// hostname or IP address as well as vari-
vium, it has no annual license fees for localhost: 8080. You need to define ous Simple Network Management Pro-
additional features or services such the access credentials during the tocol (SNMP)-specific details, such as
as quick updates, rule-based group- configuration process. If you use the version and port.
ing, and scalability across multiple the VMs, the credentials are already Alternatively, you can access the CLI
servers. predefined (username librenms, over SSH. To create a new host, in the
Monitoring with LibreNMS is easier password D32fwefwef). directory of your LibreNMS installa-
now thanks to the auto-discovery LibreNMS is configured by default to tion run the command:
function, and the web-based in- update the environment automati-
terface provides a wide range of cally. If you want to run a manual ./lnms device:add U
customization options for visual- update, you can use the ./daily.sh [--v1] [--v2c] U
izing information. The auto-update command as the librenms user. In [-c <yourSNMPcommunity>] <hostname>
mechanism also ensures that the principle, you can disable the update
monitoring environment is always up mechanism in the global settings For example, if you want to use
to date, but first you need to get the under Updates in the web graphical SNMP v2c to monitor a host named
software set up. user interface (GUI). However, the myhost.server.com in the My_Com-
Lead Image © erythropterus, 123RF.com

developers advise against doing so pany SNMP community, the com-

Integrating Initial Devices because the daily.sh script not only mand would be:
installs the latest system compo-
One major advantage of LibreNMS nents, but also handles other tasks ./lnms device:add --v2c U
is that the developers make it really such as cleaning up the database. -c My_Company myhost.server.com
easy for administrators to carry out To integrate new infrastructure compo-
an initial evaluation of the environ- nents into the monitoring environment, LibreNMS also supports ping-only
ment. They provide the installation you can add them manually with the devices, when only the availability

14 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 LibreNMS F E AT U R E

and response time

are relevant. You
can set these up
in the web GUI
by disabling the
use of SNMP.
LibreNMS only
visualizes the
Internet Control
Message Protocol
(ICMP) response
graph for these
devices.

Grouping
Devices
The task of man-
aging different
device types can
be simplified
with the group-
ing function. Li-
breNMS supports
static and dy-
namic groups: For
dynamic group-
ing, use the Rule
Editor (Figure 2),
which relies on Figure 1: LibreNMS records a wide range of information about monitored devices and offers various visualization
information from and interaction options.
the MySQL data-
base. For example, if you use the
dc<X>.<device type>.server.com for-
mat for the hostname, you need to
select the devices.hostname entry in
the selection menu. In the dynamic
configuration, regular expressions
such as equal, not equal, and the
like are also used to select devices.
The procedure for static grouping is
simpler. You simply need to enter the
hostnames and press Save to save the
group configuration. The advantage
of this kind of grouping is that you
can use alert mapping to assign indi-
vidual rules for sending alerts to the
various groups.
Once you have created your initial
devices, you can discover their status
and recorded metrics under Devices |
All Devices. Various actions are avail-
able in the tabular overview; for ex-
ample, you can view the host-specific
details or open a Telnet connection.
In daily use, the selection menus for
narrowing down the scope of the Figure 2: The Rule Editor offers a massive range of functions and configurations.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 15
 F E AT U R E LibreNMS

table and the search function prove to Configuring Your Dashboard Q Shared allows all users to cus-
be very useful. tomize the view.
Further important functions are avail- Once you have designated the initial To set a dashboard configuration as
able in the web-based interface. Al- hosts for monitoring, you will obvi- the default, open the user settings
though the strengths of LibreNMS lie ously want to know how they are do- (the person icon beside the cogwheel
in hardware monitoring, you can also ing. LibreNMS has extensive dashboard | My Settings) and select your pre-
keep an eye on any kind of service. To functions for this purpose. You have the ferred configuration in the Prefer-
do this, select Services | Add Service option of creating various overviews for ences section with the Dashboard
and specify the device and the check a range of tasks and populating them drop-down menu. You can use the
type in the matching configuration individually with a selection of widgets widget menu to put together the vari-
dialog. The handling of service checks (Table 1). All this activity takes place in ous modules that meet your informa-
is simplified by the template function, the Overview menu, which you can use tion requirements.
which lets you check entire device to generate new views and populate the
groups. You can access the status of existing views with the modules you Auto Discovery
the RAM, CPUs, and storage media need. Selecting Overview | Show Dash-
from the Health menu in the web GUI. board Editor switches to edit mode. To To get to know LibreNMS and its
extend the functionality of the current specifics, it certainly makes sense to
Listing 1: Define SNMP dashboard, click on the pencil icon and create hosts manually, but this ap-
// v1 or v2c then on the green plus sign to create a proach is not particularly effective in
$config['snmp']['community'][] = "user-defined_community"; new dashboard. You need to assign au- larger environments and is prone to
$config['snmp']['community'][] = "another_community"; thorizations to any newly created view; error. You can make your task easier
Use the following changes for SNMP v3: the three options are: with the auto-discovery function.
// v3 Q Private allows the dashboard to be LibreNMS supports various detection
$config['snmp']['v3'][0]['authlevel'] = 'authPriv'; displayed and edited only by the methods, applying them every six
$config['snmp']['v3'][0]['authname'] = '<my_username>';
user creating it. hours by default.
$config['snmp']['v3'][0]['authpass'] = '<my_password>';
Q Shared Read allows third parties to The first step is to make adjust-
$config['snmp']['v3'][0]['authalgo'] = 'SHA';
view the dashboard information ments to the config.php configura-
$config['snmp']['v3'][0]['cryptopass'] = 'crypto';
$config['snmp']['v3'][0]['cryptoalgo'] = 'AES';
but not make any changes to the tion file. In particular, you can use
configuration. SNMP v1, v2c, or v3. To define the
SNMP details for versions 1 and 2,
Table 1: Current Widgets you need to extend the configura-
Alerts Displays all warning messages. tion file as in Listing 1.
Alert History Lists the historical warnings. To avoid integrating systems into your
Alert History Stats Statistics of historical warnings.
monitoring setup indiscriminately,
you need to define the subnets:
Availability Map Displays all devices with colored tiles and lets you list all
services and ignored/disabled devices.
lnms config:set nets.+ '192.168.0.0/24'
Component Status Shows all components and their statuses.
lnms config:set nets.+ '172.2.4.0/22'
Device Summary Horizontal/Vertical Total number of devices.
Device Types Shows all events on devices. You can also specifically exclude
Eventlog Outputs the event logfile. devices you want the auto-discovery
Globe Map Visualizes a globe map with locations. mechanism to ignore and install an
Graph Generates diagrams of devices. agent there. An example of an exclu-
sion configuration is:
Graylog Displays all syslog entries from Graylog.
External Images Supports external images on the dashboard.
lnms config:set U
Notes Can be used for HTML tags and embedded links to external autodiscovery.nets-exclude.+ U
websites. Also acts as a digital notepad.
'192.168.0.1/32'
Server Stats Visualizes the CPU, memory, and storage utilization. Only
devices of the Server type are listed.
By default, LibreNMS does not use
Syslog Displays all syslog entries.
IP addresses for device detection but
Top Devices Lists the top devices by data volume, uptime/response searches for reverse DNS names. If
time, poll duration, processor/CPU utilization, or memory
utilization. you want to use IP-based detection,
you need to enable the matching de-
Top Errors Lists the most common error messages.
tection mechanism:
Top Interfaces Lists the interfaces in relation to the traffic load.
World Map Displays the locations of your devices. $config['discovery_by_ip'] = true;

16 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 LibreNMS F E AT U R E

LibreNMS uses various methods to Listing 2: Override SQL Query

detect network devices automati-
SELECT *,AVG(processors.processor_usage) as cpu_avg FROM devices, processors WHERE (devices. device_id =
cally – specifically, ARP, XDP, OSPF,
? AND devices.device_id=processors.device_id) AND (devices.status = 1 && (devices.disabled = 0 &&
BGP, and SNMP scans. They are all
devices.ignore = 0)) = 1 HAVING AVG(processors.processor_usage) > 30
enabled by default in the configura-
tion file.
approach to outputting alerts for criti- You can configure a user-defined SQL
Advanced Monitoring cal conditions is almost as important query on the Advanced tab by first
as the monitoring. The Rule Editor is enabling the Override SQL option and
The SNMP-based checks provide available for configuring alerts and entering the SQL query in the query
you with an initial impression of the can be accessed from the Alerts | field. If the average CPU load exceeds
status of a network environment, but Alert Rules item. In theory, LibreNMS 30 percent, the query will look like
what about business-critical servers supports highly complex rules that Listing 2.
and applications? After all, you want are based on mathematical calcula- The configuration for failed devices
to make sure that the web server on tions or MySQL queries. In principle, is particularly simple. The code is
which the company applications run however, your rules must comprise at devices.status != 1. If you want to
is working properly. LibreNMS offers least three elements: an entity, a con- output a warning in case of high CPU
three solutions: You can open a direct dition, and a value. Logical operators load per core (not the total load for
connection to the application, extend are used as possible conditions. all cores), then use:
the functionality of the snmpd daemon, To create an initial rule, follow the
or use an agent. Create new alert rule link in the rule macros.device_up = 1 AND U
The auto-discovery function simpli- manager. Alternatively, you can use processors.processor_usage >= 90
fies monitoring, but if you have a spe- Create rule from collection to use an
cific information requirement or the existing rule configuration as the Compared with many other products,
pertinent client information cannot basis for a new rule. Assign a name LibreNMS also offers useful flexibility
be read externally, the use of agents and a severity level in the Severity when it comes to choosing notifica-
is a potential remedy. The good news dropdown. The Max alerts box lets tion options. The SysContact stored
is you can integrate applications into you define the maximum number of in the SNMP configuration and the
the monitoring process even after cre- alerts used for an event. A value of respective LibreNMS user of an alarm
ating the matching host. -1 stands for an unlimited number of configuration are notified by default
Extended SNMP monitoring is en- messages. quite simply by selecting Alerts |
abled by default on all Unix-based
platforms, which means, for example,
that you can easily monitor the da-
tabase and mail and web servers
running on a Debian system. Note
that the extended check is carried out
as the snmpd user, which may be an
account with standard privileges. In
this case, you will need to use sudo.
In the device settings, which you can
access by clicking on the gear icon of
the device entry, you can enable the
applications to be monitored with the
module settings.
The LibreNMS agent for Linux [3]
collects data from remote systems
and uses the Checkmk software. The
matching check script is included in
the agent package; a Windows ver-
sion is now also available [4].

Alerting
Strictly speaking, monitoring hard-
ware and software is only one area Figure 3: LibreNMS supports numerous ways of sounding the alarm in the event of critical
of responsibility because a targeted incidents.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 17
 F E AT U R E LibreNMS

Alert Transports and using Create files and accumulates them; once Conclusions
alert transport to create a notification sufficient data has been received or
configuration. More than 40 transport a defined time has elapsed, it then LibreNMS combines impressive func-
types are available in the Transport writes the updates to the RRD file, tionality and flexibility and is easy to
type dropdown. The Mail option significantly reducing load. learn. In terms of functionality, the
(Figure 3) notifies the stored email To benefit from this ability to reduce environment leaves little to be de-
account; alternatively, notifications the server load, you first need to in- sired. Only the alerting configuration
can also be sent by Signal or as short stall the rrd-chached package, is a little tricky, because users have
messages. to struggle with the database data
For the message dispatch feature to apt -q install rrdcached and it is not always immediately clear
work, you need to store its services in which database element represents
the LibreNMS configuration file, but if and edit the LibreNMS configuration: which device function. Even so, this
you just want to send email, you can issue is not likely to trouble you after
also do so in the web GUI by mousing lnms config:set rrdtool_version '1.5.5' a brief learning curve. Q
over the cogwheel icon and choosing
Global Settings. On the Alerting tab Next, enable the tool on https://li-
under Email Options, select sendmail brenms_system/poller/rrdtool and use Info
or SMTP; if you choose the SMTP op- http://librenms_system/poller/ to ac- [1] LibreNMS: [https://www.librenms.org]
tion, enter the IP address and access cess further server query customiza- [2] LibreNMS demo environment:
data of the SMTP server. tion options that can help reduce the [https://demo.librenms.org/login]
server and network load. [3] LibreNMS agent for Linux: [https://github.
Performance Tuning Because LibreNMS collects all data com/librenms/librenms-agent]
in a MySQL database, the develop- [4] LibreNMS agent for Windows:
PHP-based environments are not ers recommend running the MySQL [https://github.com/Checkmk/checkmk/
exactly considered to be the most Tuner [5] tuning script on a daily tree/v1.2.6b5/agents/windows]
powerful or fastest. Although these basis. Another tip for MySQL is to [5] MySQL tuning script: [https://raw.
requirements are less important configure my.cnf in the [mysqld] group githubusercontent.com/major/
for smaller environments, they are as follows: MySQLTuner-perl/master/mysqltuner.pl]
all the more important for larger
enterprises. The recommended ap- innodb_flush_log_at_trx_commit = 0 The Author
proach is to leverage any optimiza- Thomas Joos is a freelance IT consultant and
tion potential you can find. More Instead of a value of 0, you can also has been working in IT for more than 20 years.
specifically, the LibreNMS develop- use 2 for a significant gain in I/O per- In addition, he writes hands-on books and
ers recommend using the rrdcached formance – but at the risk of losing papers on Windows and other Microsoft topics.
daemon, which fields updates for up to one second of MySQL data if Online you can meet him on [http://thomasjoos.
existing round-robin database (RRD) the MySQL server crashes. spaces.live.com].
Q

18 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Tier 0 Monitoring F E AT U R E

Secure status and event monitoring of tier 0 systems

Keeping a Close Watch

We show you how monitoring your sensitive IT systems can be a more secure experience. By Evgenij Smirnov

From a security perspective, tier 0 Directory (AD) domain controllers, In the case of highly privileged tier 0
systems such as domain controllers, enterprise certification authorities, systems, however, this process cre-
privileged access workstations, or iden- and sometimes systems that are heav- ates an area of tension that has barely
tity management systems provide direct ily integrated into the AD, such as been taken into consideration for years
access to digital resources, so more and Exchange servers. As hybrid IT has and is only now becoming the focus
more IT teams are making sure these progressed, new typical roles such as of those responsible for IT security
systems have additional protection, the Entra ID Connect server (formerly as the threat situation continues to
which includes monitoring to make Azure AD Connect) have been added, worsen. If you want to monitor tier 0,
sure they are working properly. and they clearly belong in tier 0. The the increasingly powerful monitoring
Regardless of whether you use a administration workstations, or privi- systems themselves become tier 0 and
tiering model with a formal descrip- leged access workstations, used to theoretically also need to be operated
tion (guidelines, firewall rules, and manage tier 0 systems must also be by separate administrative identities
access groups; e.g., the Microsoft considered tier 0. and protected in line with the rules for
tiering model [1]) in your infra- If errors occur, it is the monitoring tier 0. This perspective in turn opens
structure or simply apply common systems’ task to notify administrators up new rifts in company-wide moni-
sense and good account hygiene in by email, SMS, or other channels. toring setups, potentially leading to the
your daily administration, every IT In many organizations, the monitor- kind of operational disruptions that
landscape has systems and objects ing systems are even set up to initi- monitoring is intended to avoid. This
that can be classified as tier 0 – the ate remedial action automatically in article is not about security monitor-
Lead Image © Brian Jackson, 123RF.com

parts of the environment that enable the event of certain malfunctions, ing of tier 0 systems, but about classic
complete control over the identity ranging from a simple forced restart status and event monitoring.
and security infrastructure, which of a service or the entire server to
makes them both particularly vul- complex workflows that expand the
nerable and particularly worthy of disks virtual machines (VMs), move With or Without an Agent
protection. the VMs themselves to a different Virtually no other architectural is-
In a Windows server landscape, host or cluster, or trigger database sue has prompted so much debate in
these elements are usually the Active reorganizations. the monitoring community over the

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 19
F E AT U R E Tier 0 Monitoring

last 20 years as the use of dedicated during operation, which gives it an Manager (SCOM) [4] make extensive
agents. In fact, modern operating sys- apparent advantage at first glance. use of this design.
tems offer numerous remoting proto- Ideally, agents establish a connection
cols, such as Remote Procedure Call/ to the monitoring system; in other Drawing Clear Boundaries
Distributed Component Object Model words, the network communication
(RPC/DCOM), Windows Management originates from the monitored system, Whether with or without agents, as
Instrumentation (WMI), Windows and no separate inbound connections soon as your monitoring system is en-
Remote Management (WinRM), or are needed. Unfortunately, however, trusted with status monitoring of tier
Simple Network Management Proto- the old “comfort versus security” 0 systems, it is very likely to become a
col (SNMP), that let the monitoring dichotomy often means that the secu- part of tier 0 itself. It will either store
system query the current status of rity situation is far from perfect. credentials authorized to log on to tier
the hardware and software over the Many monitoring systems have a 0 systems or have the ability to distrib-
local network, read out event logs, push mechanism for distributing ute executable code to these systems –
and even trigger actions, if required – agents to the new computers to be or both. According to the fundamental
whether to obtain additional informa- monitored. For this strategy to work, concepts of security tiering, you will
tion about the system status or react the target systems need to accept in- therefore need a separate monitoring
to a status that has been identified as coming Server Message Block (SMB) system specifically for tier 0. This in
faulty. This approach is the one taken and RPC connections (Windows) turn means more setup and mainte-
by PRTG Network Monitor [2] by and SSH with stored authentication nance overhead, additional resource
Paessler, for example. (Unix/Linux). Access credentials that consumption, possibly additional
Proponents of agentless monitor- are sufficiently privileged to install licensing costs, and, above all, a break
ing argue that no additional active agents therefore need to be stored on in the dependency chains of the ser-
content needs to be installed and the monitoring system. vice definitions. Ultimately, the status
executed on the servers to be moni- Monitoring agents also increasingly of the AD (tier 0) cannot be taken into
tored because everything you need feature automatic update mechanisms, account in the service dependency tree
is already provided by the operating which means that anyone who has ad- of a tier 1 service if this status is not
system. If you assume that a monitor- ministrative rights for the monitoring available in tier 1 monitoring.
ing agent is subject to less stringent system can use an unattended process Almost everything that applies to the
quality control than the operating to install arbitrary executable code au- classification of monitoring systems
system itself, you can claim greater tomatically on the monitored devices. in the security tiers also applies to
operational stability. From a security Monitoring agents often can run data backup, virus protection, and
perspective, though, this approach scripts in a variety of languages as platform provisioning (virtualization,
comes at a very high price: probes. As a rule, these scripts do not storage, network management), par-
Q Network traffic on the remoting have to be explicitly registered but ticularly when it comes to configura-
protocols needs to be allowed in simply stored in a special folder to be tion management such as software
the inbound direction. Admins executed by an agent. Again, these distribution and automation systems.
sometimes fail to restrict these traf- scripts are often distributed by auto- Consistent separation of these compo-
fic relationships to the IP addresses mated update mechanisms. For ex- nents, depending on the level of pro-
of the monitoring systems. ample, almost all monitoring products tection, makes holistic mapping of IT
Q All remoting protocols require au- based on Nagios and that use either services across tier boundaries both
thentication to access the required NSClient++ or their own agents, such technically more difficult and organi-
data. SNMP is the only protocol as Checkmk [3], have this function, zationally more necessary than ever.
that uses its own authentication which makes it possible to inject arbi- Some organizations have successfully
and is independent of the moni- trary script code through the monitor- implemented the approach of com-
tored system. The use of the operat- ing agent. plete tier separation with a separate
ing system’s protocols, on the other Although the agents usually run in server platform, data backup, moni-
hand, means that highly privileged the system context on the respective toring, and even malware protection.
service accounts need to be stored servers, some applications, such as This approach is particularly useful
on the monitoring system. Very SQL, SharePoint, or Exchange, need in large companies where separate
often, the same access credentials more to retrieve application-specific teams manage the tier 0 systems any-
are used for a large number of sys- status information. In other words, way. If a malfunction detected during
tems, which in turn means that the the access credentials required for monitoring does affect the shared
passwords of these accounts are not application-specific access not only infrastructure components, such as
changed often enough. need to be stored on the monitoring the physical network, diagnostics and
An agent-based monitoring archi- system but must also be transferred troubleshooting are coordinated by a
tecture does not require you to store to the agent. Systems such as Mi- shared process on the ticket system.
highly privileged access credentials crosoft’s System Center Operations For most IT organizations, however,

20 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Tier 0 Monitoring F E AT U R E

this kind a separation is not practi- setting up a separate monitoring before the changeover, you can
cable. In the following sections, I take instance for tier 0 systems and then continue to use them, but you will
a look at the available alternatives. hardening the firewall rules. You will want to change the passwords
want to locate the new monitoring to be on the safe side before you
Bare Essentials instance fully in tier 0 and either re- store these access credentials on
register any agents or ideally even the tier 0 monitoring system. Re-
The monitoring system’s features are roll them out again and restrict ad- gardless of the password change,
often not fully utilized. In many orga- ministration of the tier 0 instance to you also need to delete the stored
nizations, administrators prefer to use authorized workstations and admin tier 0 credentials from the monitor-
the ticket system as their monitoring accounts. If privileged access creden- ing system that is now exclusively
front end. They deliberately avoid tials were stored in the general moni- dedicated to tier 1.
monitoring the current status of the toring system before this changeover, In this construct, all cross-tier com-
systems and services and focus on you will also need to separate them: munication originates from the moni-
event-centric monitoring by automati- Q If the access, Run As, or service toring system (tier 0) and is directed
cally opening a service request when accounts used previously are also to the tier 1 systems for dispatching
an error occurs by email or via an ap- used on tier 1 systems, it can be email, making API calls, or both,
plication programming interface (API) extremely time-consuming to cre- to open tickets automatically. This
call. This approach requires a very ate new accounts for tier 1. It method ensures that the tier 0 sys-
good understanding of what is normal makes more sense to grant the re- tems are protected from the influence
in the particular IT environment and quired tier 1 permissions explicitly of less privileged tiers.
a reliable process for documenting to the existing accounts, revoke the
and maintaining threshold values. tier 0 permissions, and create new
If your monitoring system works in accounts for tier 0. Distributed Monitoring
this way, achieving the required sepa- Q If the access accounts were already In both agent-based and agentless
ration requires just a little overhead: exclusively assigned to tier 0 monitoring, the risk for the highly
F E AT U R E Tier 0 Monitoring

privileged systems and accounts across the tiers, while leaving the monitoring product for tier 0 and
comes from data collection (access configuration and administration to including tier 0 monitoring in tier 1
credentials stored on the monitoring the respective Checkmk servers in the monitoring such that the status data
system) or from system administra- individual tiers. of the individual tier 0 systems are
tion (distribution of agents, updates, In Microsoft SCOM, setting up tier- displayed like operating parameters.
scripts). Displaying the current sys- specific distributed monitoring is some- The prerequisite is that you can en-
tem status in the form of dashboards what more complex and requires you able access to the tier 0 monitoring
and alerts by email, SMS, or API call, to create at least three management data without giving up control over
on the other hand, is harmless. groups (MGs) [7] (Connected MG, Tier this system’s configuration. Potential
Some monitoring systems can per- 1, and Tier 0), each with a separate solutions include:
form monitoring and alerting on sepa- SQL database, at least one dedicated Q REST API access over a separate
rate systems. In most cases, this sepa- management server, and sophisticated port and without the possibility of
ration is part of a general architecture role-based access control (RBAC) in influencing the configuration.
for distributed monitoring. Some sys- between to enforce the administrative Q Regular data export from tier 0
tems such as Checkmk [5] or Zabbix separation between the groups. monitoring and transfer to a tier 1
[6] support this type of architecture Other monitoring products offer sim- system, where the data is analyzed
by default. However, the emergence ilar approaches. When you purchase, and processed to create sensors.
of distributed monitoring strategies is install, and configure a distributed Agents such as NSClient can, for
not attributable to security consider- monitoring system, you need to example, evaluate the content of
ations, but to the need to consolidate make absolutely sure that the goal of text files or execute scripts.
and centrally process status data and tier separation is achieved and that Q An active monitoring agent on the
configurations from geographically you do not open a backdoor to your tier 0 monitoring system that sends
distributed locations. tier 0 systems. NSCA messages or SNMP traps to
The Zabbix proxy seeks to offer “conve- the tier 1 system on its own initia-
nience and standardization” and makes Heterogeneous Monitoring tive and does not transmit its own
the central server the configuration hub status, but that of the monitored
for all connected systems. Checkmk’s The various distributed monitoring tier 0 machines. Tier 1 monitoring
Livestatus technology takes the same technologies require the same product must have a corresponding accep-
approach (Figure 1). However, it offers to be used in the individual zones tor, which is becoming increasingly
an alternative, in the form of Livedump or tiers. However, you might be able rare, especially for NSCA.
and CMD dump, that lets you central- to consolidate the monitoring data You can also link two instances of
ize data storage and dashboard views in tier 1 with the use of a different the same monitoring product using

Figure 1: Depending on the technology, distributed monitoring enables tier separation. © Checkmk GmbH [5]

22 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Tier 0 Monitoring F E AT U R E

methods described above if the Risky Updates technically possible with a variety of
software you use is flexible enough monitoring systems. Knowing how
to do so. The increasingly popular automatic your IT and application teams use the
agent updates should also be treated functions of the existing monitoring
Moving Monitoring into Tier 0 with caution in tier 0. Ever since the systems is of paramount importance.
Solorigate attack on software manu- If you are planning to purchase a new
In small IT teams, where infrastruc- facturer SolarWinds [8], IT security monitoring system or change your
ture and application management is admins have been aware of the dan- monitoring system, you will definitely
essentially the responsibility of the ger that software updates can pose. want to revisit the agent vs. agentless
same group of people, it can be quite If the attackers manage to infiltrate discussion and give security consider-
practical to declare the entire moni- a manufacturer’s development en- ations top priority. Q
toring setup to be a tier 0 service. vironment, they can add their own
Onboarding a new system usually routines to the delivered software. Info
involves tier 0 activities anyway, such These routines can then be passed [1] Microsoft tiering model:
as creating groups, service accounts, unnoticed with an update to custom- [https://learn.microsoft.com/en-us/
policies, and firewall rules. If IT opti- ers’ privileged systems and compro- security/privileged-access-workstations/
mizes its processes, the administrator mise them. privileged-access-access-model]
responsible – who is already logged In tier 0, you will want to get rid of [2] PRTG Network Monitor:
in to a privileged access workstation automatic agent updates and explic- [https://www.paessler.com/prtg/
with their tier 0 access credentials at itly verify all new versions before prtg-network-monitor]
this point – can also add the new sys- deploying them to the highly privi- [3] Checkmk local checks:
tem to the monitoring setup. leged systems. If you use the same [https://docs.checkmk.com/latest/en/
Monitoring products that have the product for tier 0 and tier 1 moni- localchecks.html]
technical ability to separate the visual- toring, tier 1 monitoring can act as a [4] SCOM RunAS profiles:
ization (dashboards) from the controls staging environment for testing the [https://learn.microsoft.com/
are particularly useful. This means tier 0 agents, allowing you to check en-us/system-center/scom/
that you can hang a display terminal both operational characteristics, plan-security-runas-accounts-profiles]
for the monitoring dashboard on the such as stability and resource con- [5] Checkmk distributed monitoring:
wall in IT operations with a clear sumption, and the security of the [https://docs.checkmk.com/latest/en/
conscience, and without having to new agent versions, for example, by distributed_monitoring.html]
soften the boundaries of your network running an additional anti-malware [6] Zabbix distributed monitoring with proxies:
segmentation. Checkmk lets you use scan against the agent installers or [https://www.zabbix.com/documentation/
Livedump for this purpose. delaying the installation in tier 0 current/en/manual/distributed_monitoring]
This approach has some limitations, and first monitoring the network [7] SCOM design:
although it is quite simple to imple- traffic of the agents in tier 1 for sus- [https://learn.microsoft.com/en-us/
ment. Automatic inclusion of new sys- picious behavior. system-center/scom/plan-mgmt-group-
tems in the monitoring setup, for ex- Ultimately, of course, you need to design?view=sc-om-2022#design-
ample, should be treated with caution. compare the up- and downsides of considerations]
If your monitoring system is not smart this approach carefully. If a manu- [8] Analyzing Solorigate:
enough to use different credentials for facturer patches highly critical vul- [https://www.microsoft.com/en-us/
onboarding, depending on the target nerabilities that have already been security/blog/2020/12/18/analyzing-
system, highly privileged credentials exploited in the field, the risk of a solorigate-the-compromised-dll-file-that-
could be used far too often. successful attack by these vulnerabili- started-a-sophisticated-cyberattack-and-
In classic tiering according to the ties may be greater than that of a sup- how-microsoft-defender-helps-protect/]
legacy Microsoft model, attempts ply chain attack. However, because
to access a tier 1 system with a the agents run on systems that are Author
tier 0 account would be rejected. already highly isolated, vulnerabili- Evgenij Smirnov has been working with
If you can ensure that the tiering ties of this kind should be virtually computers since the age of 5 and delivering
guidelines take effect early on in the impossible to attack from the outside, IT solutions for almost 30 years. His Active
deployment process and the moni- which should give you sufficient time Directory and Exchange background naturally
toring system uses tier 1 credentials to evaluate the updates. led to PowerShell, of which he's been an avid
if the tier 0 login fails, automatic user and proponent since its first release.
onboarding can be practicable in Conclusions Evgenij is an active community lead at home in
principle. From a security point of Berlin, a leading contributor to German online
view, there is nevertheless some risk, A compromise between consistent communities, and an experienced user group
because tier 0 credentials could be visibility and strict separation is not and conference speaker. He is a Microsoft Cloud
compromised. easy to achieve in monitoring but is and Datacenter Management MVP since 2020.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 23
F E AT U R E Graphite

Real-time monitoring with Graphite

Single Stack
The open source Graphite tool offers real-time monitoring for IT environments, with comprehensive and
fast data collection from virtually any system. By Holger Reibold

Graphite [1], which started life in 2006 Graphite-Web provides the user inter- Relay, and the Graphite-Web interface
as a side project of a monitoring tool face and API for displaying charts and reads the data either from the cache
for an airline and travel portal, has dashboards. or directly from a storage medium.
evolved into a powerful system today The Carbon service is responsible for Which transmission method to use in
that, according to developer Chris Da- feeding the various items of raw data each individual case depends primar-
vis, helps Etsy, Booking.com, GitHub, to the stack, which in turn is handed ily on the applications reading the
Salesforce, Reddit, and many other over for long-term storage to the data or the scripts you deploy. Some
companies monitor their business pro- Whisper databases. The admin inter- applications have special tools or
cesses. The software has been under acts with the Graphite web user inter- APIs that can help you transfer data
the Apache 2.0 open source license face (UI) or application programming to Carbon. The easiest way to learn
since 2008. The ecosystem has grown interface (API), which in turn queries about the specifics of the transfer pro-
considerably in the meantime and of- Carbon and Whisper for the data. cess is to use the plaintext protocol.
fers a complete set of collection agents The key benefit of the tool is that Carbon is backed up by a number
and voice connections for all typical it bundles and consolidates a wide of daemons that form the storage
application scenarios. mix of source data, making the data back end. A simple Graphite installa-
available to third-party applications. tion typically only uses one daemon
Inner Workings Graphite supports various output (carbon-cache.py). For larger envi-
styles and formats (e.g., CSV, XML, ronments, the carbon-relay.py and
At its core, Graphite handles two JSON), which sets the stage for em- carbon-aggregator.py daemons are
tasks: It stores numerical time series bedding custom charts into external usually added to distribute the load
data and renders the data in the form websites or dashboards. of metrics processing and handle
of graphs. Feeding raw data into the Graphite has flexible options for con- custom aggregation tasks. Basically,
system is particularly easy. To bundle suming raw data. The monitoring en- all Carbon daemons expect time
Lead Image © peshkov, 123RF.com

raw data in all its diversity, Graphite vironment supports three main meth- series data from third-party sources
relies on the interaction of three soft- ods: plaintext, the Python-specific and can field the data by common
ware components: (1) Carbon que- pickle data format, and Advanced transmission protocols. However,
ries the various time series data; (2) Message Queuing Protocol (AMQP). the daemons differ in the way they
Whisper, the database library, handles The information sent to Graphite process the incoming data. Knowl-
the task of storing the data; and (3) is managed by Carbon and Carbon edge of these different processing

24 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Graphite F E AT U R E

capabilities is essential when it to Ubuntu 18.04 LTS. That said, the In the first line, you assign a label
comes to implementing sophisticated REsynthesize [3] fork is designed for to the rule, followed by the regular
storage back ends. CentOS 8.1 or higher. expression, which you specify with
pattern= and then the retention rate
Installation Configuration (retentions=).
In the example, the [garbage_collec-
Graphite is a complex Linux-based The Carbon module is the heart of tion] label is primarily for documen-
environment mainly programmed the Graphite environment. It is con- tation purposes. Graphite logs this
in Python. The environment uses trolled through various configuration and the matching metrics in the cre-
the Cairo graphics library to render files that reside in the /opt/graphite/ ates.log file. This pattern is applied
the graphs, which results in vari- conf/ directory. Because the initial to all metrics that end with garbage-
ous dependencies that typical server installation does not create a configu- Collections. Note that Graphite uses
installations do not usually cover. ration file, you will need to create one Python syntax for regular expressions.
In the source-based installation, the manually. The easiest way to do so is The retentions line states that each
check-dependencies.py script helps to glean from the various sample files datapoint is equivalent to 10 seconds
you handle the checks. The easiest (conf.example) by copying them to the and that you want to keep the data
way to install Graphite is in Docker, configuration directory and removing from the last 14 days available.
the .example suffix. Graphite has the In addition to the various configura-
docker run -d U configuration files shown in Table 1. tion files, you can manage the use of
--name graphite U For an initial installation, you will metrics with white- and blacklists.
--restart=always U primarily need to edit carbon.conf In the case of whitelist functionality,
-p 80:80 U and storage-schemas.conf. The main Graphite accepts only those metrics
-p 2003-2004:2003-2004 U carbon.conf configuration file bundles explicitly whitelisted; all blacklisted
-p 2023-2024:2023-2024 U the settings of the various daemons; metrics are rejected. The advantages
-p 8125:8125/udp U the configuration itself is broken into are obvious: You can explicitly filter
-p 8126:8126 U sections: out all metrics that are not relevant
graphiteapp/graphite-statsd Q [cache] controls the carbon-cache to your information needs. To enable
daemon, this filter function, raise the USE_
which means you could also use Q [relay] controls the carbon-relay WHITELIST flag in the carbon.conf file.
a Windows system to evaluate the daemon, and Graphite then searches the directory
setup. For a standard installation, Q [aggregator] controls the car- defined with the GRAPHITE_CONF_DIR
make sure the following conditions bon-aggregator daemon. option for the blacklist and whitelist
are met, along with having Python The developers recommend paying configurations (blacklist.conf and
version 2.7 or later in place: special attention to the [cache] sec- whitelist.conf). If you have not cre-
Q cairocffi Python module tion when you use it for the first time. ated a whitelist configuration or it is
Q Django 1.11.19 or newer The storage-schemas.conf file is where empty, the software will let all met-
Q django-tagging 0.4.6 you assign metric paths to the pat- rics pass through.
Q pytz Python module terns and store the frequency and du-
Q scandir Linux function ration of data storage for the Whisper Reading Metrics
Q fontconfig library component. The patterns are regular
Q Web server gateway interface expressions defined in three lines: The challenge when using Graphite
(WSGI) and a web server (Apache is to read in the different data, but
also requires the mod_wsgi module) [garbage_collection] the monitoring environment is very
You will also need the Graphite pattern = garbageCollections$ flexible. You can use the three meth-
web app, Carbon, and the Whisper retentions = 10s:14d ods already mentioned: plaintext,
database library, which is part of
the Graphite project. Depending on Table 1: Graphic CONF Files
whether you go for optional func- File Name Function
tions, further Python modules (e.g., carbon.conf The main configuration file used to define the settings for each
python-memcache, python-lpad, and Carbon daemon.
python-rrdtool) may be needed.
storage-schemas.conf Determines the retention rates for storing metrics.
The use of Synthesize [2], an instal-
lation script for Graphite and the as- storage-aggregation.conf Specifies how lower precision data is aggregated.
sociated services on Linux, all go to relay-rules.conf Relay rules used to communicate metrics to specific back ends.
prove that the installation is unlikely aggregation-rules.conf Aggregation rules that bundle different metrics.
to pose a massive challenge. How-
rewrite-rules.conf Lets you rename metrics with regular expressions.
ever, the use of Synthesize is limited

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 25
F E AT U R E Graphite

pickle, and AMQP. The acquired Basically, Graphite is used to collect which the developers have provided
data is sent to and managed by two numerical data and transmit the data an online overview [4]. For example,
modules: carbon and carbon-cache. to Carbon for analysis. Each data the software collaborates with col-
The method you use to feed Graphite series has a unique identifier that is lectd, the well-known system data
with data depends on the environ- based on the metric designation and acquisition daemon. You can use the
ment and the data you want to various tags. A naming system is es- collectd plugin write-graphite to sub-
process. Various tools and APIs are sential. The second step is to config- mit collectd metrics to Carbon.
available. For test data, it is easiest ure data retention, answering various Alternatively, Graphite can extract the
to use the plaintext protocol; if you questions in the process: How often is metrics from collectd RRD files if you
have large volumes of data, pickle the data generated? What kind of pre- add those files to STORAGE_ DIR/rrd.
is recommended; and AMQP is the cision do you need? Over what time For example, in practice, you can link
best choice if Carbon is listening on period do you want to acquire data? <host.name>/load/load.rrd file to rrd/
a message bus. To create a naming scheme, modify collectd/<host_name>/load/load.rrd
The simplest approach is the plain- the /opt/graphite/conf/storage-sche- to generate the collectd.<host_name>.
text protocol. The data must use the mas.conf file. Graphite requires a mes- load.load.{short,mid,long}term graph.
<metric path> <metric value> <met- sage format, such as,
ric timestamp> format. Carbon then Visualizing Data
takes care of translating the result- echo "test.bash.stats 42 `date +%s`" U
ing line of text into a metric that the | nc localhost 2003 The Graphite dashboard is respon-
web interface and Whisper database sible for visualizing the data and lets
understand. For testing purposes, I comprising the metric namespace, the users display various sources. Dash-
ran the Netcat (nc) program on Unix value you want to assign to the met- board access is by http://<my.graph-
to generate a socket and send data ric, and the timestamp. ite.host>/dashboard or the composer.
to Carbon: Unfortunately, Graphite’s visualiza-
Integrating External tion capabilities are quite limited, but
PORT=2003 you can apply functions to the data or
SERVER=graphite.system
Software pool graphs from different hosts.
echo "local.random.diceroll 4 `date +%s`" U Graphite is designed to work with As mentioned, the integrated visu-
| nc ${SERVER} ${PORT} about a hundred different tools, for alization functions turn out to be

Figure 1: The Graphite dashboard only provides a simple tool for visualizing metric data, so employing Grafana makes more sense.

26 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Graphite F E AT U R E

difficult to use in practice, which is with very little configuration overhead. monitoring tool – probably with the
what prompts most administrators As an alternative, a tool like graph- aim of addressing a larger target
to turn to more user-friendly tools ite-beacon [6] is a good choice. This group. A second look, however,
that provide meaningful evaluations. simple alerting application for Graph- shows that it is primarily an aggre-
Grafana [5] is very popular, not least ite runs asynchronously and sends gator that collects a wide variety of
because it has a native plugin to help notifications on the basis of Graphite metrics. Graphite handles this task
integrate Graphite data sources (Fig- metrics. The advantage is that it has excellently – not least because of
ure 1). Integration is easy with a little no dependencies other than the Tor- the comprehensive ecosystem that
help from the Grafana query editor nado package and is easy to imple- has grown up around the tool. How-
(Figure 2). ment. To install, use pip or apt-get: ever, Graphite’s full potential is only
revealed in combination with more
pip install graphite-beacon capable visualization tools.
Alerting Q

Monitoring critical systems and vi- apt-get update

sualizing data is one thing, but you apt-get install graphite-beacon Info
primarily need to be able to respond [1] Graphite: [https://graphiteapp.org]
to critical events, for which typical The configuration is handled by a [2] Synthesize: [https://github.com/
alerting and notification functions config.json file located in the same obfuscurity/synthesize/]
are used. By definition, a monitoring directory as graphite-beacon. When [3] REsynthesize:
system also outputs warnings when editing the file, you first need to [https://github.com/deividgdt/
specific values occur, but Graphite specify the URL of the Graphite sys- resynthesize]
does not have this capability. You tem and then use regular expressions [4] Tools that work with Graphite:
will have to rely on third-party prod- to determine when to trigger an alert. [https://graphite.readthedocs.io/en/latest/
ucts if you need this feature. You also need to define an email han- tools.html]
Again, the alerting configuration is dler to take care of mailing the alerts. [5] Grafana: [https://grafana.com]
particularly easy in combination with In addition to Graphite alerts, you can [6] graphite-beacon:
Grafana, which comes with the Alert- use this tool to configure URL alerts [https://github.com/klen/graphite-beacon]
ing system used to create a ruleset. for web-based environments.
The ability to generate single- and
multidimensional rules is useful. The Conclusions The Author
Alerting system, in turn, comes with Holger Reibold, computer scientist, has worked
the query manager, which supports The Graphite developers like to as an IT journalist since 1995. His main interests
the integration of Graphite metrics classify their application as a are open source tools and security topics.

Figure 2: Grafana’s query editor supports the integration of Graphite data.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 27
TO O L S AlmaLinux Build System

Exploring the AlmaLinux Build System

Package Packer
The AlmaLinux Build System lets you build, test, sign, and release system at other, third-party Git reposi-
tories, which makes it suitable for
packages from a single interface. By Joe Casad many in-house DevOps development
settings.
When IBM announced that it and releasing them to public reposi-
was restricting access to Red Hat tories.” In other words, the goal is How Does It Work?
Enterprise (RHEL) source code and to assist with every phase of the
moving CentOS upstream, the distros package development process, rely- The AlmaLinux Build System auto-
that depended on RHEL and CentOS ing on automation to reduce human mates the process of building, sup-
source code were sent scrambling. It error and minimize manpower porting, and managing packages.
is still a little unclear whether IBM’s requirements. The vision is for something that is
moves are legal and consistent with The AlmaLinux Build System is a more than a build tool, with support
the GNU Public License (GPL), but free software project that is avail- for testing, signing, and releasing
the litigation to sort it out could able on GitHub. Other Linux dis- software packages.
take years, and in the meantime, the tributions are welcome to use the If the AlmaLinux project needed a
derivatives need a solution. AlmaLinux Build System as a tool build system to interact with source
One enterprise distribution that for building and managing pack- code originating from a Red Hat en-
weathered the storm quite smoothly ages. You can also point the build vironment, you might be wondering
was AlmaLinux [1] (see the box
entitled “Where Do They Get Their Where Do They Get Their Code?
Code?”). If you ask the AlmaLinux AlmaLinux was envisioned as a free alterna- IBM currently restricts access to some, but not
developers, they will say that one rea- tive to RHEL, which comes with a subscrip- all, RHEL source code. Some source code is
son for their success in navigating the tion fee and other corporate licensing available through the Red Hat Universal Base
transition to the post-RHEL era is the arrangements. AlmaLinux and other RHEL Image (UBI) [3]. AlmaLinux uses as much of
AlmaLinux Build System [2]. derivatives used source code from Red Hat the Red Hat UBI code as it can, but a majority
The AlmaLinux Build System repositories as the basis for building an of the code comes from the CentOS Stream
Lead Image © stylephotographs, 123RF.com

evolved from an earlier system independent distro. It is important to note project [4]. IBM did not eliminate all access
used by CloudLinux. (CloudLinux that Red Hat does not own the source code in to CentOS; they just moved it upstream, so the
is a contributor to the AlmaLinux the sense that the term own is used with pro- code does not include some of the final bug
project.) The developers refer to prietary software. Because Linux and most fixes and updates that go into the final version
their build system as “a project of the code included with it are open source of RHEL. AlmaLinux uses some code from the
designed to automate processes of and licensed under the GNU Public License CentOS Stream project and performs its own
(GPL), the code is available for others to use fixes and updates. They also pull code from
building distribution and packages,
and modify. other upstream sources when necessary.
testing packages, signing packages,

28 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 AlmaLinux Build System TO O L S

why they didn’t just use Koji [5], Q Terraform – an infrastructure-as- interface or text-based commands.
the freely available build tool as- code tool used to build simulated Support for command-line process-
sociated with Red Hat’s Fedora environments for package testing ing creates the possibility for script-
project. The answer given by the de- Q PGP – an encryption utility that ing and other custom automation
velopers is that, although Koji is an provides signing services for pack- scenarios.
effective tool, the AlmaLinux project age verification At the center of the system is the
had a much broader vision. For one Q Git – a source code repository Build System Master Service. The
thing, they wanted to integrate ad- system Master Service receives commands
ditional package formats (Koji is Git isn’t actually part of the build sys- from the user and sets the process
limited to RPMs). They also wanted tem itself, but it is an integral part of in motion, creating, restarting, and
to provide a complete, integrated the ecosystem, providing source code deleting builds and communicating
pipeline to manage a package from for building packages and communi- with the rest of the system via API
the build phase, to testing, to sign- cating with the build system through calls. Responsibilities of the mas-
ing the package, and finally to re- an API. ter service include requesting and
lease. The AlmaLinux Build System Figure 1 shows the complete sys- receiving source code from the Git
includes controls that allow the user tem at a glance. Users interact server and assigning tasks to the
to specify where to release pack- through either a graphical user build nodes.
ages, and it is one of
the first build systems
to support modular-
ity. A module is a
collection of packages
that occur together,
such as the packages
in a single applica-
tion or an operating
system component.
Support for modular-
ity lets you treat the
packages together,
thus saving steps
and streamlining the
configuration.
Like other build plat-
forms, the AlmaLinux
Build System is not a
monolithic application
but a combination of
back-end tools behind
a single, unified inter-
face. Some of the tools
incorporated into the
AlmaLinux Build Sys-
tem include:
Q Mock – a tool
for building RPM
packages
Q Pulp – a content
repository for or-
ganizing and dis-
tributing software
packages
Q NGINX – a web
server that serves
as an interface for
managing access to
the build system Figure 1: The AlmaLinux Build System at a glance.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 29
TO O L S AlmaLinux Build System

builds appears in the main screen

(Figure 2). A menu on the left of-
fers options for creating a new build,
new release, or new distribution.
Click on the Details link for a config-
ured build to view the build settings.
You can choose an architecture (Fig-
ure 3) or view the artifacts associ-
ated with the build (Figure 4).
To create a new build, choose New
build in the main view. In subsequent
menus, you can select a platform and
choose architecture(s), set options
for the Mock build tool, and specify
whether to build for a Secure Boot
system.
Once you have configured the build
settings, select the project (the pack-
ages) you would like to build (Fig-
Figure 2: Your first view shows the builds configured for the system. ure 5) and click on Create Build.

Another important component of First Look Testing and Later Steps

the build system is Pulp [6], which
provides artifact storage for newly- When you log in to the AlmaLinux The AlmaLinux Test System
built packages and other products Build System, a view of configured (ALTS) [8] included with the build
of the build process. According to
AlmaLinux Community Manager SBOMs
Jack Aboutboul, “the master service On May 12, 2021, the Biden administration help investigators identify and trace security
is the brain, and Pulp is the heart” released Presidential Executive Order 14028 risks that might affect the package. If a com-
of the build system. As you can see “Improving the Nation’s Cybersecurity” [9]. ponent used in building the package turns up
One of the important features of that order with a critical vulnerability, it will be easy to
in Figure 1, Pulp is essential to the
is the stipulation that software packages for spot the problem and to know that the pack-
later stages of the process, provid-
software used by the US government should age needs an update.
ing packages for signing and testing, include a bill of materials for all the code AlmaLinux was the first Linux distribution to
and forwarding finished packages provided in the package. This Software Bill notarize and provide an SBOM for all source
for release. Much of the power of of Materials (SBOM) is described as a “list and components. The AlmaLinux developers
the AlmaLinux Build System is in its of ingredients” for the software package. created an SBOM generation utility and inte-
ability to oversee the testing, signing, The idea is that providing an accurate list of grated it into the AlmaLinux Build System. You
ingredients used for building the package will can find the alma-sbom utility on GitHub [10].
and release phases of the develop-
ment process.

Getting the Code

The AlmaLinux Build System uses
the Gitea software development ser-
vice [7] to communicate with the
Git server. Gitea is described as an
all-in-one service for managing a Git
environment, including “code review,
team collaboration, package registry
and CI/CD.” The AlmaLinux team
has developed a gitea-listener tool
for interfacing with Gitea and the Git
repository. The AlmaLinux Build Sys-
tem also supports Fedora Community
Repository Platform format (COPR),
which makes it easy to add alterna-
tive repositories to the system. Figure 3: The AlmaLinux Build Systems supports several hardware architectures.

30 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 AlmaLinux Build System TO O L S

environment automates packet test- for scripting opens a range of possi- [https://github.com/AlmaLinux/alts]
ing in realistic conditions. ALTS first bilities for adapting the build system [9] Presidential Executive Order 14028:
launches a clean test environment for other projects. Q [https://www.whitehouse.gov/
(for instance, a Docker container) briefing-room/presidential-actions/2021/
using Terraform to recreate a real- Info 05/12/executive-order-on-improving-the-
istic setting that models actual pro- [1] AlmaLinux: [https://almalinux.org/] nations-cybersecurity/]
duction conditions. Once the envi- [2] AlmaLinux Build System: [https://github. [10] alma-sbom: [https://github.com/
ronment is in place, ALTS attempts com/AlmaLinux/build-system] AlmaLinux/alma-sbom]
to install the package, and, if the [3] Red Hat Universal Base Image:
installation is successful, begins a [https://catalog.redhat.com/software/ Joe Casad
series of integrity checks predefined base-images] Joe Casad is the editor in chief of Linux Magazine.
by the user. [4] CentOS Stream:
Results of the tests are then for- [https://www.centos.org/centos-stream/] This article was made possible by support
warded to the Pulp artifacts store [5] Koji: [https://koji.build/] from AlmaLinux OS Foundation through
in the form of test logs and reports, [6] Pulp Project: [https://pulpproject.org/] Linux New Media’s Topic Subsidy
and the results are then available to [7] Gitea: [https://github.com/go-gitea/gitea] Program (https://www.linuxnewmedia.
the user through the web interface. [8] AlmaLinux Test System: com/Topic_Subsidy).
Approved packages are then signed
and marked for release. The build
system lets you define and select
specific channels for the software
release, and the verification sys-
tem allows the receiver to trace the
authenticity back to the original
source code.

Conclusion
The stability and versatility of the
AlmaLinux Build System has given
the developers a head start on
achieving the project’s ambitious
goals while avoiding much of the
wheel spinning that often comes
with putting a distribution together.
AlmaLinux was recently chosen as
a standard Linux distribution for Figure 4: Viewing the artifacts associated with the build.
Fermilab and the CERN European
laboratory for particle physics. The
AlmaLinux project was also the first
enterprise Linux to offer a complete
Software Bill of Materials (SBOM)
for every package (see the box en-
titled “SBOMs”).
The AlmaLinux team is busy right
now using the AlmaLinux Build Sys-
tem to create, sign, test, and release
the next version AlmaLinux, but the
developers also want to sure make
the system is available to other us-
ers and other projects. The user in-
terface makes it easy to incorporate
other source code repositories, and
the testing, signing, and release com-
ponents support customization for
alternative projects and applications.
An API-driven design with support Figure 5: Adding a project to the build.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 31
TO O L S Authelia

Identity and access

management with Authelia

Bouncer
Add access controls to web applications that do not have their own user administration; however, this
useful gatekeeper requires a reverse proxy. By Tim Schürmann
To protect a private party from proxy (Figure 1) and checks all in- example, stipulate that passwords
hooligans, you could check the coming requests, just as a bouncer need to contain at least one uppercase
guests’ invitations at the entrance would check invitations. letter. To prevent attackers from simply
yourself, or you could hire a bouncer. When you access a web application brute-forcing the passwords, the num-
Authelia [1] acts as a bouncer for web in a browser, your request is first sent ber of login attempts can be limited.
applications to help you regulate ac- to the reverse proxy, which forwards You can only log in again after a speci-
cess to services that do not offer their it directly to Authelia for inspection. fied wait. Far more effective would be
own access controls. When a browser knocks on the door if the wait was automatically extended
Thanks to Authelia, developers do for the first time, it does not have an for each incorrect password, but that
not have to implement complex and Authelia session cookie. In this case, is beyond Authelia’s capabilities.
time-consuming user management in the tool redirects you to its own login To set the bar even higher for at-
their own web applications. Instead, page, where you first need to verify tackers, Authelia relies on two-factor
they can deploy Authelia upstream of your ID by providing a username and authentication. The second factor
their own software with two-factor password. is either one-time passwords, such
authentication and single sign-on as those generated by the Google
(SSO) by default. In other words, you Double Security Authenticator, push messages to cell-
just need to log in to Authelia to ac- phones, or hardware-based systems
cess several authorized applications. Authelia can retrieve the login data that comply with the FIDO2 WebAu-
Authelia requires that communica- from its own user database (a simple thn standard (Yubikey USB sticks).
tion with the web applications be YAML file) or consult an LDAP server. Push messages require users to have
protected by a reverse proxy. The Administrators can specify the pass- the Duo Push app by Cisco on their
software then connects to this reverse word structure in a policy and, for smartphones.
After successfully
logging in, Authelia
transfers a ses-
Lead Image © Gordon Bussiek, Fotolia.com

sion cookie to the

browser. The reverse
proxy submits all
following requests
to Authelia again.
As the browser now
has a session cookie,
Figure 1: The reverse proxy submits every incoming request to Authelia to authorize access explicitly. Authelia gives the

32 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Authelia TO O L S

reverse proxy the go-ahead to forward access rules that Authelia compares Listing 1: Access Rules
the request to the appropriate web against every incoming request. In
access_control:
application. By default, the session other words, the software also acts as
rules:
cookie then also applies to all sub- an access control system. - domain: wiki.example.com
domains of the domain controlled by The rules are stored in YAML format resources:
the application. A flow chart in the (Listing 1). The example is also based - '^/project/hamburg/.*$'
Authelia documentation [2] demon- on the online documentation [4]. It subject: 'group:hamburg'
strates the process: On receiving the allows all users from the hamburg policy: two_factor
first request, Authelia redirects the group to access the wiki.example.com methods:
browser to its own login page. domain; the path must comply with - GET
Because of this approach, Authelia the regular expression specified in re- - POST
networks:
does not need to know how a web sources. In the example, all the pages
- 192.168.1.0/24
application works, nor can it sniff the accessible below wiki.example.com/
transmitted data (in the form of pay- project/hamburg/ would be in the al-
loads). The tool itself only processes lowed scope. The requests must origi- example, the software cannot help
the authentication information. In nate from the 192.168.2.0/24 subnet you if you want microservices to
principle, any HTTP-based services and are only allowed to use GET and authenticate automatically. Also,
can be protected, such as microser- POST methods. some desirable functions for enter-
vice REST and GraphQL interfaces. Authelia is implemented in Go as prises are still missing, including
Authelia currently works with the a compact binary that can be used multidomain protection, although
proxies listed in Table 1. Apache and directly without installation and this feature is already on the project
Microsoft IIS are left out in the cold. without dependencies. The of- agenda.
As a general rule, all parties involved ficial packages on GitHub include The simple structure and clear con-
need to use HTTPS to prevent the ses- templates for a systemd unit and a figuration mean that Authelia can
sion cookie from being tapped. sample configuration. The develop- be set up quickly. This identity and
Authelia supports SSO on the basis of ers recommend using their official access management software is there-
trusted headers, which both the proxy Docker container, which only re- fore suitable for protecting small and
and the web applications must support. quires around 30MB of RAM [5]. medium-sized enterprises or retro-
Authelia also acts as an OpenID Con- Authelia is also designed for opera- actively protecting a domain’s web
nect 1.0 provider, with authentication tion in Kubernetes. You can start applications against unauthorized ac-
based on tokens. Although this function several containers in parallel there cess. Go developers in particular can
was still in beta when this issue went to configure the tool for scaling or integrate Authelia into their applica-
to press, it has already been received high availability. The official Helm tion and save themselves the trouble
with great interest. The ownCloud In- chart ensures a quick setup. of having to program their own user
finite Scale developers are planning to In production operation, the tool re- management feature. Q
integrate Authelia into their groupware quires further software components.
as an OpenID Connect provider [3]. It stores distributed session cookies
in RAM or a Redis database. Log Info
information, settings, and other data [1] Authelia: [https://www.authelia.com]
Ruleset generated at runtime are encrypted [2] Authelia architecture:
Imagine you want a team to have ac- and stored in a PostgreSQL, MySQL, [https://www.authelia.com/overview/
cess only to specific pages of a wiki. or SQLite database. To verify the prologue/architecture/]
To do this, you would create a list of identity of new users, Authelia sends [3] “Try to ship Authelia as the default IdP in
an email, so the ocis binary” by Michael Barz, October
Table 1: Supported Proxies administrators 27, 2023: [https://central.owncloud.org/
Proxy Standard Kubernetes XHR Redirect Request Method must provide t/try-to-ship-authelia-as-the-default-
Caddy Yes Partially Yes Yes an SMTP idp-in-the-ocis-binary/45662]
server. [4] Authelia access control:
Envoy Yes Yes Partially Yes
[https://www.authelia.com/overview/
HAProxy Yes Partially Partially Yes
authorization/access-control/]
NGINX Yes Yes No Yes Conclusions [5] Authelia via Docker: [https://www.authelia.
NGINX Proxy Yes No No Yes Authelia is not com/integration/deployment/docker/]
Skipper Yes No Partially Partially suitable for
Swag Yes No No Yes all use cases The Author
because of Tim Schürmann is a freelance computer scientist
Traefik 1.x Yes Partially Yes Yes
its mode of and author. Besides books, Tim has published
Traefik 2.x Yes Yes Yes Yes
operation. For various articles in magazines and on websites.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 33
TO O L S iRedMail

Fast email server deployments with iRedMail

Email the Easy Way

Setting up and maintaining an email service in the data center doesn’t have to be a nightmare. The
iRedMail open source solution lets you deploy a full-featured email server on a number of platforms
in a matter of minutes. By Rubén Llorente

Friends and workmates often tell shortcomings, however. Email have a third party host their email
me that email is obsolete as a means accounts are bound to receive il- service, and they end up purchasing
of communication. Everybody is us- legitimate messages containing mal- plans with email providers such as
ing some mobile messaging app or ware or unsolicited advertisement Google or Microsoft. Family busi-
another these days. Solutions such (spam). Therefore, a modern email nesses in particular have a tendency
as Signal Messenger or WhatsApp let service must be equipped with to use free email plans from big pro-
users send and receive documents smart filters capable of identifying viders such as these.
and messages, and they are compara- legitimate mail (colloquially known Setting up your own email service,
tively free from the spam and scam as ham) and stopping the rest. An- however, has been a solved problem
campaigns that plague the email other complaint against email ser- for quite a long time, so you have no
ecosystem. Thus, I am told, email is vices is that they are built by joining excuse for letting Google handle your
irrelevant. many unrelated components that email (or worse, having your employ-
People couldn’t be more wrong. are not trivial to configure. A typi- ees use a Gmail account to commu-
Email is vital for many businesses cal email service needs a web server nicate with your customers). Canned
because it allows them to deliver for hosting both a management solutions that build a functional email
messages to both customers, employ- interface for the system administra- server in a matter of minutes already
ees, and associates by standard, open tor and webmail for regular users. exist. Previously, I discussed Citadel
protocols that are not controlled by A Simple Mail Transfer Protocol [1], and now I want to introduce
Lead Image © Konstantin Inozemtcev, 123RF.com

a single organization. Servers can be (SMTP) daemon is needed to de- iRedMail.

configured to email error reports to a liver messages to users of different
system administrator with every inci- email services, whereas Post Office Enter iRedMail
dent, and email is still the most popu- Protocol (POP) or Internet Message
lar password recovery mechanism Access Protocol (IMAP) daemons let iRedMail is a for-profit operation
when somebody forgets the password users check email with clients such built on the freemium model as a free
to their favorite web forum. as Thunderbird or Mutt. “open source, fully fledged, full-fea-
This battle-tested communica- In the face of such complexities, tured mail server” [2]. The website
tion mechanism is not free of many small organizations prefer to lists three iRedMail-related products.

34 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 iRedMail TO O L S

Table 1: Supported Platforms

Platform Versions
CentOS Stream 8, 9
Rocky Linux 8, 9
Alma Linux 8, 9
Debian 11, 12
Ubuntu 20.04, 22.04
FreeBSD 13.x
OpenBSD 7.3

support tickets. It is worth noticing

that support tickets that require SSH
access are paid at a premium.
Figure 1: iRedMail Easy is a web-based deployment platform; however, you must perform iRedAdmin-Pro is the high-end ver-
a fresh install of a supported distribution on a machine you control and then create an sion, and its source code is available
account in the iRedMail system. to paying customers. It is clearly
marketed at email administrators who
intend to host the email services of
multiple different organizations or
customers. It includes features not
available in the free version, such as
the ability to assign different resource
quotas to each hosted domain, to
have different administration ac-
counts per hosting domain, and to
manage quarantined email with a
flexible system.

Getting Started
To build your iRedMail server, you
need a fresh install of a supported
Figure 2: iRedMail Easy uses Ansible to provision your email server. You need to give the platform. The list in Table 1 was valid
iRedMail company SSH access to your machine with superuser privileges, allowing them at the time of writing this article.
to set up the system for you. Keep in mind that support is not
equal for all platforms: For example,
The downloadable installer is the command their web deployer to turn my production iRedMail servers all
product I focus on in this article. The your machines into email servers in run on OpenBSD and, although all
free-tier product, licensed as free a similar fashion as the download- the core functionality works, certain
open source software under the GPL3, able installer.
is a set of scripts that turns a machine The website does
into an email server when run on a poor job trying
top of any of the supported operating to convince you
systems or Linux distributions. To do of the advan-
this, the scripts download and install tages of iRedMail
all the components required by the Easy over the
email server from the distribution’s downloadable
repositories and then configure them installer, but in
for you. practical terms,
iRedMail Easy (Figure 1) is a web- it looks like its
based deployment platform. To use only real benefit Figure 3: The steps to installing iRedMail on Rocky Linux 9 by the
it, you have to create an account [3] is getting com- downloadable installer are quite intuitive, but moderate knowledge
and give iRedMail the credentials mercial support is required for making some choices, such as which back end to
to log on to your servers over SSH and making it use to store user account information. OpenLDAP, MariaDB, and
(Figure 2). In this way, you can easier to submit PostgreSQL are supported.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 35
TO O L S iRedMail

Table 2: iRedMail Components

Tool Functionality
Postfix SMTP server
Dovecot IMAP server
Nginx Web server (optional)
iRedAdmin Web-based management interface (optional)
Roundcube Webmail (optional)
SOGo Webmail and groupware (optional)
Fail2Ban Brute force protection (optional)
Figure 4: Some components are optional. Fail2Ban integration in
Netdata System monitor (optional) OpenBSD is broken and does not work out of the box.
Amavis Content filtering
SpamAssassin Spam filtering integrations need some fiddling. iRedMail does not neces-
sarily support the most recent OpenBSD release, either.
ClamAV Malware detection
I recommend the downloadable installer, but I will skip
the detailed instructions for instal-
lation because they are documented
on the project’s website [4]. The
procedure will be a breeze for sys-
tem administrators and power us-
ers, but some basic IT knowledge
is required. For example, you are
asked which back end to use to store
user accounts (Figure 3) and which
optional applications you want to
install (Figure 4). A successful in-
stall gives you a server featuring all
the components you need for a small
system (Figure 5; Table 2).

Postmaster’s Office
Figure 5: A successful iRedMail install. Unless you go out of your way to
disable it, an
iRedMail install
will feature a
management
interface, reach-
able with a
web browser at
https://<yours-
erver>/ired-
admin, where
<yourserver>
is your server’s
fully qualified
domain name
(FQDN) or IP ad-
dress. From this
control panel,
you can perform
most of the basic
administrative
tasks, such as
Figure 6: iRedAdmin lets you create email accounts for users and add domains to the email server. creating new

36 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 iRedMail TO O L S

email accounts for users, adding do- Although the implementation deliv- is the behavior most users expect.
mains to the list of domains for which ered by the downloadable installer is Additionally, SpamAsassin does
you host email, managing quotas, and serviceable, it is a bit disappointing. not perform Bayesian filtering and
viewing the logs (Figures 6 and 7). In the default configuration, spam does not autolearn. Ideally, an email
The free version of iRedAdmin [5], is sent to quarantine instead of into server should be able to learn which
which is set up by the iRedMail the recipient’s Junk folder, which email users want to read and which
downloadable installer, falls a bit
short for all but the simplest sce- Email Connectivity
narios. Theoretically, an unlimited
An email server requires a reputable, static, publicly reachable IP address. You might manage to
number of domains may be hosted, make it work with dynamic IP addresses, but I don’t recommend it unless you are testing only.
but in practice you need to con- Each domain for which you host mail needs to have a mail exchange (MX) entry on its DNS re-
figure DomainKeys Identified Mail cords pointing to your server, such as:
(DKIM) email authentication sign-
linuxrocks.es. 3600 IN MX 10 linuxmag.operationalsecurity.es.
ing for each (see the “Email Con-
nectivity” box). DKIM keys cannot This record lets email servers trying to send mail to your users learn to which host they need to
be managed from iRedAdmin, so connect. In other words, if a Gmail user wants to send email to [email protected], Gmail will
you must configure them manually. look up the MX record for linuxrocks.es and discover that the associated mail server is at linux-
mag.operationalsecurity.es.
The free iRedAdmin lacks quarantine
Adding a sender policy framework (SPF) record lets other mail servers know that the owner of
management. By default, incoming
linuxrocks.es has authorized your server to send email in their name.
email may be prevented from reach-
ing the Inbox and stored in quaran- linuxrocks.es. 3600 IN TXT "v=spf1 a:linuxmag.operationalsecurity.es -all"
tine until the postmaster decides its Proper DKIM records sets up a mechanism that uses public key cryptography to certify that
fate. One reason an email might be your email has actually been sent from your server. The server’s public keys reside within
quarantined is if looks like it might a publicly available DNS record. Once DKIM is set up, your server will sign all outgoing mail.
carry malware or spam. The problem Servers that receive email from your server will then download the public key from the DNS
is, because iRedAdmin does not have records and verify the signatures of the email messages against it to determine whether
an interface for dealing with quaran- they are legitimate or forged.
tined messages, email that is quar- iRedMail sets up a DKIM engine on install, and outgoing mail is signed by default. Still,
you need to upload your public key to the DNS registry manually. It is worth noticing that
antined will never be checked and
iRedMail will use the same DKIM key to sign every message, so if you are hosting email for
can’t be released. Quarantine man-
both linuxrocks.es and linuxrules.es, both domains will be covered by the same key. I have
agement is a paid feature included in
found this does not work well (despite what the documentation says). Therefore, my advice
iRedAdmin-Pro. is to use a separate DKIM key for each domain. More information about DKIM and other DNS
records can be found in the documentation [6] [7].
Email
Filtering
iRedMail comes
with self-updat-
ing blacklists
and SpamAsas-
sin for sorting
out spam. Email
delivered from a
server listed on
a spam blacklist
(e.g., lists from
Spamhaus) are
rejected; other
email that looks
suspicious will
be classified as
spam, as well.
ClamAV is also
included to iden-
tify messages
carrying malware Figure 7: Both Roundcube and SOGo are available for iRedMail. Roundcube is a good option for people needing a
and isolate them. somewhat light webmail panel, whereas SOGo is a full groupware solution with CalDAV and CardDAV support.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 37
TO O L S iRedMail

they don’t to adapt the filters to their into an email server very fast, with Info
preferences. In a typical email server, most of the features you expect [1] “An overview of the Citadel BBS”
this is done by having the spam included. by Rubén Llorente, ADMIN, is-
filter take notice of which email is One of iRedMail’s strengths is its sue 57, 2020, pg. 38, [https://www.
manually moved into the Junk folder portability. Other free open source admin-magazine.com/Archive/2020/57/
by the users, and which email acci- software email appliances (e.g., An-overview-of-the-Citadel-BBS]
dentally placed in the Junk folder is Mail-in-a-Box) work on a specific [2] iRedMail: [https://www.iredmail.org/]
moved elsewhere by the users. distribution or operating system, but [3] iRedMail Easy portal:
The iRedMail instance you get from iRedMail can run on an astonishing [https://easy.iredmail.org/signup]
the downloadable installer can be number of platforms. Because of this [4] iRedMail documentation:
configured to behave properly with advantage, if your server fleet com- [https://docs.iredmail.org/]
autolearning spam filters that place prises instances of a given system [5] iRedAdmin free: [https://docs.iredmail.org/
spam mail in the Junk folder, but it (e.g., OpenBSD), the system admin- migrate.or.upgrade.iredadmin.html]
is a somewhat involved process [8]. istrator won’t have to set up an alien [6] Sign DKIM signature on outgoing email for
iRedMail Easy does the correct thing platform just for email. new mail domain:
out of the box and does not force On the other hand, if the out-of-the- [https://docs.iredmail.org/sign.dkim.
you to tweak spam filtering manu- box configuration does not work signature.for.new.domain.html]
ally. There is no reason at all why the for you, you will have to perform [7] Set up DNS records for your iRedMail
downloadable installer couldn’t do a number of manual steps that ne- server: [https://docs.iredmail.org/setup.
the same. My hunch is that iRedMail gate, up to a point, the advantage of dns.html]
Easy takes the proper approach and deploying an easy-to-roll solution. [8] Auto-learn spam/ham with Dovecot
the downloadable installer takes the Specifically, if you need to change imap_sieve plugin: [https://docs.iredmail.
crude approach because the iRedMail the default configuration of the org/dovecot.imapsieve.html]
organization wants to push you into spam filter, you will have to face a
using iRedMail Easy. quite involved process. Also, adding The Author
more domains to your mail host is Rubén Llorente is a
Conclusion doable from the iRedMail’s control mechanical engineer
panel, but adding DKIM keys for who ensures that the
If you need to deploy an email service those domains (which is a necessity security measures of
for a small organization (e.g., you these days) must be done manu- the IT infrastructure
need an email provider for a single ally from a console. In any case, of a small clinic are
domain), the iRedMail downloadable iRedMail still takes less effort than both legally compli-
installer is an option worth consider- building a full email stack manually ant and safe. He is also an OpenBSD enthusiast
ing. It will certainly turn a machine from the ground up. Q and a weapons collector.

38 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
TO O L S ScaleFlux

Computational storage that

supports storage operations

Smart
Assistant
ScaleFlux delivers performance gains of 50 percent and more with computing power built directly into the
network card to relieve the burden on the CPU. By Martin Loschwitz
ScaleFlux competes with NVMe- (e.g., database nodes, servers with several advantages. The computations
based drives by offering not only running Hadoop or Redis instances, or take place closer to the network traf-
store but compute functions, as well. members of an Elasticsearch cluster). fic, which offers a boost in terms of
However, relieving the load on the Several factors come together. In par- performance. Moreover, the NIC is
CPU and ensuring higher bandwidth ticular, individual steps such as com- optimized for this task, which also
and lower latency comes at a greater pressing and indexing data or cleaning tweaks performance. What is almost
price and begs one question: Is it up data records that are no longer as important is that tasks performed
worth the effort? needed require serious bandwidth on by the NIC no longer burden the host
the connected storage medium on the CPU, which can then take care of
Where Is the Bottleneck? one hand and high compute power on other things. NICs with the appropri-
the part of the CPU on the other. ate functions are now available from
Every server has a bottleneck that In the worst case, this leads to a all the major manufacturers, and sup-
limits its performance somewhere, single system slowing down an entire port is now implemented extensively
but the exact whereabouts of this setup, such as when the central data- on Linux (e.g., by the Data Plane De-
bottleneck tends to differ because base can no longer respond to incom- velopment Kit, DPDK).
the manufacturers of the individual ing queries or cannot respond quickly
components have been playing a cat- enough. Neither users nor administra- Computational Storage
and-mouse game for years. New CPUs tors like this one bit, because – espe-
usually come with new chipsets that cially on the end-user side – using a The computational storage approach
enable higher bandwidths for the con- service in this scenario is a pain. for storage devices has similar goals.
nected devices and RAM, which often The problem is not new, which ex- Its beginnings were pretty banal. For
makes the storage devices the bottle- plains why the hardware industry example, the principle of storage tier-
neck. The storage device manufactur- started looking for solutions years ing, which treats hot and cold data
ers then follow suit and launch faster ago. Most admins will be familiar differently, is well established. Hot
devices so that the network suddenly with the term “offloading” in a net- data resides on fast flash memory, cold
becomes the performance limit. The work context, which means that data on slow disks – or even on tape if
network in particular has seen major a network interface card (NIC) no they are intended for archiving. That
innovations in recent years, including longer simply processes the traffic hard disks are mentioned at all shows
Lead Image © sdecoret, 123RF.com

the now widely available 400Gb Eth- flowing through it, but also takes on the age of the approach. Because of
ernet, which has quickly passed the computing tasks. If you are using vir- their design, hard disks have far lower
buck back to storage devices. tual extensible LAN (VxLAN), packets bandwidth than flash memory and far
The situation can become particularly need to be tagged accordingly (Vx- higher latency. Installing flash mem-
critical when components interact, LAN tag) by the host CPU or by the ory – which was still extremely expen-
especially if these components happen NIC if it implements suitable func- sive at the time – alongside a hard disk
to be exposed to a heavy compute load tionality on its chip and which affords meant that the advantages of the two

40 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 ScaleFlux TO O L S

solutions could be combined, at least latency, and storage density. Addition- get the prefabricated drives in com-
for writes. However this approach is ally, computational storage drives putational storage drive (CSD) and
by no means sufficient for today’s ap- implement various features that are NVMe SSD+ (NSD) flavors, the later
plications (e.g., large databases and missing in normal NVMe devices, of which does not have the capacity
Hadoop and others), without even tak- such as the ability to perform atomic multiplier feature of the CSD – but
ing into account the throughput that storage operations. These features more on that later. In any case, one
current artificial intelligence (AI) solu- give the administrator more space thing is clear: ScaleFlux drives can
tions require. elsewhere, for example, by eliminat- run on any server that has free PCIe
In the meantime, other approaches ing the need for the database to take slots or can accommodate regular U.2
were vying for the users’ favor. care of atomic operations itself. NVMe devices, and this condition will
For example, data processing units Compared with the less popular ap- be true for the vast majority of off-
(DPUs) were popular for a while. proaches, such as those based on the-shelf standard servers.
These special components were also DPUs, typical computational storage The manufacturer’s specifications
used in servers and were intended to drives offer admins the option of de- on the drives’ capabilities make for
support the NIC. DPUs are still found ploying them in many more setups on pretty impressive reading. If you are
today, particularly in network de- the one hand. On the other hand, the looking to virtualize the devices’
vices, but the approach failed to catch lack of adaptation to specific applica- functions, you could go for a com-
on in general-purpose machines. One tions also means that it is not possible plete single-root I/O virtualization
problem is that the software needs to to squeeze every last ounce of perfor- (SR-IOV) implementation with a total
cooperate closely with the DPU; in mance out of every database – pre- of 15 different virtualization func-
fact, it has to be more or less com- cisely because the hardware compute tions. Hardware support for various
pletely coordinated with it, which routines are not adapted to the respec- security functions in line with the
makes cooperating with software tive programs. Trusted Computing Group (TCG)
from the open source world difficult. Opal 2.0 specification is also part
By way of an example, Facebook What ScaleFlux Offers of the hardware. The provider also
developed a key-value store named states a speed of 7.1GBps for sequen-
RocksDB that is capable of leverag- ScaleFlux is considered a pioneer in tial reads and 4.8GBps for sequen-
ing DPUs in appropriate setups, but it the computational storage industry. tial writes. The I/O operations per
can’t do so without specific custom- The company is older than you might second (IOPS) rates are also worthy
izations to the database. The prebuilt suspect. The people at ScaleFlux have of note. The claim is more than one
solutions for this scenario are only been working on computational stor- million IOPS with compression for
available from individual manufac- age drives since 2014, and by 2020, read-write workloads (ratio 70/30)
turers who require customers to pay their hardware was already well es- and 4KB block size enabled. That
quite heavily for the privilege, and tablished on the market. Since then, said, you cannot directly compare
this obviously stands in the way of ScaleFlux has continuously expanded these values with other standard
genuinely widespread use. its own portfolio and can point to NVMe devices; just make sure you
Today vendors such as ScaleFlux [1] success. Alibaba Cloud, which is the examine the ScaleFlux hardware’s
have entered the market to take the largest cloud provider in China, relies capabilities more closely.
principle further and build comput- on ScaleFlux hardware to accelerate
ing power directly into the network its in-house PolarDB, a cloud-native ScaleFlux Components
card. This approach eliminates some database with its own storage nodes
of the disadvantages of previous on which the capabilities of ScaleFlux This investigation starts with the
solutions, such as the need to use ad- hardware can be particularly well lev- hardware components that you as the
ditional hardware components. What eraged, according to Alibaba. admin adopt into your data center.
is particularly important from the What does ScaleFlux deliver, and how Unsurprisingly, NAND chips are used
provider’s perspective is that the solu- do the drives work in detail? A look for storing the data; these chips are the
tions need be transparent for both the at the company’s current portfolio go-to standard for the NVMe industry.
operating system and the user. provides clarity. Although they still However, the heart of the ScaleFlux
The close connection between the de- advertise their 2000 series drives, drives beats in the processor, which in
velopment of hardware and software, the far newer 3000 series is the real the case of the 3000 series (Figure 4)
which was still the rule for bridge highlight. The drive comes in three comes in the form of an ARM-based
solutions such as DPUs, should forms: as an add-in card (AiC) with a application-specific integrated circuit
therefore be eliminated. The solu- PCIe connector, as a solid-state drive (ASIC), which is one of the biggest
tion is achieved by the storage drives (SSD) with a U.2 interface, or as SFX differences compared with the previ-
themselves directly intervening in the 3000, which allows vendors to build ous series. The 2000 series still relied a
data stream and carrying out various their own hardware with ScaleFlux field-programmable gate array (FPGA)-
operations that improve throughput, features (Figures 1-3). You can also based chip, which turned out to be too

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 41
TO O L S ScaleFlux

Figure 1: Computational storage SSDs from ScaleFlux come in three Figure 3: … as M.2 memory modules for the corresponding slots.
form factors: as an AiC card for the PCIe slot, … © ScaleFlux The type and scope of the installed components are identical
across the different form factors. © ScaleFlux

drives is the ca- The SoC also implements the codecs

pacity multiplier, and is therefore even faster than a
as mentioned; generic ARM CPU. For this to hap-
beyond that, the pen, the ARM SoC in the memory
devices offer drive compresses the data before
the same set of storage in flash memory and only un-
basic features. packs the data again when requested
The foundation by the respective database. As the
is transparent manufacturer promises, this process
compression and is completely transparent from the
decompression of application’s point of view. The data-
the in-flight data base running at the top of this kind of
of a database. construct remains blissfully unaware
Figure 2: … as a regular NVMe with a U.2 interface and a 2.5-inch The goal of the of the data compression. It can still
form factor, and … © ScaleFlux exercise is obvi- read and write data as fast as a nor-
ous: Compression mal NVMe, thanks to the computing
low powered in the course of develop- increases the theoretically available power built in to the drive.
ment. To match its performance, the bandwidth, and greater data volumes On the application side, high-volume
3000 series also comes with a state- can cross the bus to the application at database queries and queries in which
of-the-art PCIe 4.0 interface, whereas the same time. The entire procedure is large volumes of data need to be pro-
the predecessor series had to make handled by the ARM SoC in the drive. cessed benefit from compression. The
do with the significantly
slower PCIe Gen 3.
The system on a chip
(SoC) is supported by
RAM and various special
components for acceler-
ating individual steps.
Although the computa-
tional components are
identical across all models
in the series, the number
of NAND chips and their
sizes differ. ScaleFlux
NVMe devices are cur-
rently available in sizes of
3.2, 3.84, 6.4, and 7.68TB.

Similarities
The central difference be- Figure 4: ScaleFlux NVMe drives offer typical NAND memory, as well as a dedicated ARM SoC with its own
tween the CSD and NSD RAM and special extensions for tasks such as data compression. © ScaleFlux

42 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 ScaleFlux TO O L S

manufacturer points to three advan- TCG Opal specifications. Encryption 50 percent of the drive’s space is
tages. First, read operations can be is again transparent from the operat- still available according to df, but
processed far more quickly and with ing system’s point of view; again, the the drive is actually already full, is
lower latency than with normal drives application is unaware that the data is unlikely in this respect. Whether the
because the compressed blocks can be encrypted on the storage medium. Be- value this feature adds is worth the
read as fast as with normal SSDs, but cause the drive handles the encryption extra cost to you is ultimately your
much more data can be read in the process very quickly thanks to smart decision.
same time. Second, in mixed read and codecs, there are no disadvantages in
write workloads, which are standard terms of performance. Made to Measure
in most data centers, compression also
further reduces latency because write- Differences Another thing common to both
to-read interference is lower. This term series is the integrated circuit (IC)
refers to the wait that occurs when, Against the background of compress- used. The manufacturer not only of-
say, the controller of an SSD switches ing drives, the limelight is again fers implementation support for it
between read and write commands. firmly on the capacity multiplier, but even helps with development, if
Again, because more data is read in which ScaleFlux massively advertises necessary. The firmware running on
parallel, fewer changes between read for its product. The keyword is “im- the ScaleFlux drives also plays a role.
and write mode are required overall, plicit” compression, which is the only CSware (Figure 5) offers the possi-
which improves overall latency. central difference between NSD drives bility to roll out special firmware for
Third, according to ScaleFlux, com- and the significantly more expensive specific purposes with the help of the
pressing the data prolongs the ser- CSD drives. manufacturer. Companies like Alibaba
vice life of the installed NAND chips The capacity multiplier is the manu- are likely to have made extensive
because lower volumes are written facturer’s approach to pretending use of this very option but are likely
to the data carrier all told. At this that the drive is far larger during to encounter greater interest from
point, it is worth recalling the way the provisioning process. This as- ScaleFlux than small companies with
flash chips basically work. At their sumption has certain advantages; for small platforms.
core, they use chemical reactions to example, the space gained through In any case, regardless of size, all
store data. However, these reactions compression can be put to good companies have the option of in-
cannot be repeated for an arbitrarily use by the operating system. On stalling the ARM SoC from the 3000
long period of time, which is precisely the other hand, some risks are as- series in their own use cases, which
why the drive writes per day (DWPD) sociated with the approach. If the is especially interesting if you want
number of a drive is already a deci- degree of compression on which the to take advantage of computational
sive factor for administrators during theoretical drive size is based can- storage without relying on ScaleFlux
the acquisition process. It indicates, not be achieved, the drive could fill equipment.
in roughly simplified terms, how of- up faster than the size specification
ten the drive can be completely over- would lead you to believe. At least What Others Say
written in total without fear of data in this scenario the drive would
corruption from defective blocks. As give the operating system correct Lower latency and higher bandwidth
people in the industry say, SSDs do usage figures. A situation in which can be achieved with ScaleFlux
not break down less often than hard
disks, but they are more predictable,
and this is quite true in essence.
ScaleFlux itself perhaps somewhat
over-extravagantly refers to this fea-
ture as the endurance multiplier, but
without providing any concrete num-
bers relating to the life expectancy
boost that the feature offers a NVMe
drive. Don’t bother looking for DWPD
data; however, you will find informa-
tion about the degree of compres-
sion that ScaleFlux looks to achieve,
which is said to be around a factor of
four on average – provided the data
to be processed can be compressed. Figure 5: The SFX 3000 chip is the heart of the ScaleFlux architecture. In combination with
The 3000 series models from ScaleFlux matching firmware, it is said to provide even better performance values than the standard
implement encryption in line with the version. © ScaleFlux

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 43
TO O L S ScaleFlux

hardware, as can a higher storage values for normal SQL (i.e., MariaDB Conclusions
density and, in turn, a lower wear or PostgreSQL) and Oracle are clearly
level factor for the SSDs. The ques- comparable, whereby Oracle particu- As usual, I was unable to elicit
tion is nevertheless how great are larly benefits from enabling compres- concrete prices from ScaleFlux, but
the benefits resulting from the use of sion on the large SSD. The figures are searches revealed that the manufac-
these components. Plenty of sources 220,000 IOPS compared with 330,000 turer’s devices are priced at about the
offer concrete figures online, includ- IOPS in this case. same level as normal NVMe drives
ing database specialist Percona, StorageReview.com has identified without the additional features.
which tested ScaleFlux in the context another use case that showcases the NVMe has become far cheaper since,
of both MySQL and PostgreSQL. benefits of ScaleFlux drives in a bet- so ScaleFlux hardware might seem a
Further figures, especially those for ter way: storage drives in the virtual little bit expensive. Technically, how-
Oracle, are provided by the US trade desktop infrastructure (VDI) context. ever, setups are likely to benefit con-
journal StorageReview.com [2]. The The values measured for the large siderably from the advantages these
test parameters are pretty much com- SSDs with compression enabled are devices offer.
parable: Random and sequential read sometimes more than 100 percent Any database administrator will be
and write operations were measured higher than those of the small SSDs delighted with the 50 percent and
with different chunk sizes of about without compression, but that also more performance gains they can
4KB and 64KB, as were read and makes sense. In the VDI context, the achieve by simply replacing the stor-
write operations by Oracle. proportion of data to be stored that age hardware. That the CPU also
Both providers arrive at test results can be meaningfully compressed is finds relief is a welcome side benefit.
that are coherent in themselves and likely to be significantly larger than Another thing is also clear: If you
coherent in relation to each other. in most database scenarios, which customize the drives’ firmware to suit
Random reads and writes at the 4KB traditionally work with fairly small your own application scenarios – with
chunk size produce similar results chunk sizes. a little help from the provider – you
across all drives when reading data, Although StorageReview.com exclu- can probably tease out even more im-
regardless of whether the 2:1 compres- sively compared ScaleFlux drives in pressive performance gains.
sion ratio was maintained during test- its tests, Percona additionally offered ScaleFlux is already turning its
ing. Both the 3.84TB and the 7.68TB a direct comparison with an Intel thoughts in a different direction.
CSD 3000 drives are in a range be- P4610 drive [3]. The comparison is Currently, the ARM SoCs do not yet
tween 850,000 and 900,000 IOPS, but realistic because the Intel device is include codecs for multimedia ap-
writing with the same block size leads one of the standard NVMe drives and plications, which in particular would
to significantly different results. therefore a standard solution in many benefit considerably from higher
Without data compression, the application scenarios. That said, the throughput and lower latency; the
drives achieve 380,000 to 420,000 Percona test was carried out a while offloading factor for the host CPU
IOPS, but well over 700,000 with back and therefore compares the Intel would also be significantly greater. In
compression. When reading with a drive with the CSD 3000’s predeces- perspective, ScaleFlux drives for the
block size of 64KB, the IOPS num- sor, the CSD 2000, which is probably multimedia sector are not only con-
ber drops to a meager 6,000 with- why the ScaleFlux drives were only ceivable but quite likely. Q
out compression and to 7,000 with slightly ahead of their Intel counter-
compression. However, the write parts in terms of speed in many indi- Info
processes are even more interesting. vidual tests. [1] ScaleFlux: [https://scaleflux.com]
With compression enabled, both the However, if you combine the test re- [2] “ScaleFlux CSD 3000 SSD Review” by
small and large ScaleFlux NVMe do sults from Percona and StorageReview. Charles P. Jefferies, January 9, 2023:
well at 6,000 IOPS. Without com- com and include the provider’s state- [https://www.storagereview.com/review/
pression, both devices return a fairly ments about the speed improvements scaleflux-csd-3000-ssd-review]
poor value below 2,000. In all the of the 3000 series, a conclusive overall [3] Testing the Value of ScaleFlux Computa-
cases described, the compression picture emerges. The speed advantage tional Storage Drive (CSD) for PostgreSQL,
used also had a positive effect on of the ScaleFlux 3000 series compared Percona whitepaper, 2021: [https://www.
latency, which was reduced by 30-40 with a setup without the special hard- percona.com/sites/default/files/Testing_
percent on average. ware is on average 50 percent greater the_Value_of_Scaleflux_for_PostgreSQL.pdf]
The compression factor has a less for MySQL, and even up to 200 per-
drastic effect in the case of database cent greater for PostgreSQL, depending The Author
applications. Here, the baseline of on the scenario. Thanks to compres- Freelance journalist Martin
the small 3000 series NVMe is at a sion, ScaleFlux drives, on average, Gerhard Loschwitz focuses
good 250,000 IOPS, whereas the large stored more than 200 percent of the primarily on topics such
NVMe with compression enabled is data compared with an Intel device as OpenStack, Kubernetes,
easily north of 300,000 IOPS. The given maximum usage. and Chef.

44 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
CO N TA I N E R S A N D V I RT UA L I Z AT I O N DIY Docker Images

Build and host Docker images

Master Builder
When facing the challenge of packaging your application in a container, take into account your needs in terms of
handling and security and investigate sensible options for hosting your own registry. By Martin Loschwitz

Many a developer looking to package SUSE in particular, will be relying on any effort. It is easy to imagine how
their own application in a container containers in the future, if only be- this trend will continue.
for the first time finds themself out cause it saves them a lot of work. For However, not only do the manufactur-
of their depth. Administrators who example, Red Hat need only maintain ers benefit from this approach, thanks
have not had much experience with its own software in containerized to the container format, the devel-
containers to date are in a similar form once to make it available on opers of smaller applications could
position if they are looking to con- RHEL 7, 8, and 9. deliver their own applications to the
tainerize small tools for their own use As long as a runtime environment users quickly and easily. They don’t
and deliver them locally. The ques- for operating containers is available have to deal with annoying details
tions are many. Although the required on a system, the underlying operat- like package managers and different
knowledge can be painstakingly ing system hardly matters. Contain- packages for different versions of a
gathered, it takes a long time, and ers also offer a very useful technical distribution. In this respect, contain-
it’s not much fun to boot. This article alternative for software that the large ers certainly are tempting.
rushes to the rescue. Besides the ba- distributors do not have in their own
sics of image building, I look at best portfolios. Although Red Hat cur- Questions and More
practices and continuous integration rently maintains various versions of
and continuous deployment (CI/CD) MariaDB or MySQL for its enterprise
Questions
mechanisms, as well as the question distributions, in the future they will The questions are always the same:
of a good DIY image registry. This simply point to the official container How do you turn the program source
much can be revealed in advance: images of the manufacturers instead code, which is available as a tarball,
The topic is not quite as complex as of doing the work themselves. into Docker containers that can be
Lead Image © Sebastian Duda, 123RF.com

many critics assume and claim. This effect can already be seen on the delivered by Docker Hub? Do practi-
desktop. Recently, Fedora announced cal CI/CD tools already exist that not
The Rise of Containers that it would no longer maintain Li- only facilitate container building, but
breOffice in package form but would also professionalize it and automati-
Containers are on the rise, whether point its own users to the official cally find errors? Is the use of Docker
you like it or not. For years, this LibreOffice Flatpaks. Under the hood, Hub even a good idea, or would a
magazine has been pointing out that Flatpaks are no more than containers, local registry, designed specifically for
the major distributors, Red Hat and for which Red Hat doesn’t have to put local images, be a better idea?

46 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 DIY Docker Images CO N TA I N E R S A N D V I RT UA L I Z AT I O N

Luckily for you, Docker developers package lists, which Apt relies on to image does not require its own ker-
include the tools you need to create ensure that the packets come from a nel. However, it does have to contain
and deploy local container images reliable source (line 2); add the pack- all the files that the desired program
with Docker, which fortunately does age sources for Docker to the local needs to run. After all, the standard
not involve complicated syntax or a list of package directories (line 3); host filesystem is a no-go area later
whole new format for the metadata and update the local package cache for the active container.
of the image to be built. To get your (line 4). Last but not least, run apt This lockout can be removed by bind
first custom image up and running to install the community edition of mounts and volumes later; however,
quickly, all you need is a local folder Docker on your system (line 5). A you will always want to build the
on a Linux system with some basic call to systemctl (line 6) should then container such that it has no depen-
tools and a Docker runtime environ- show active as the docker.service sta- dencies on software stored exter-
ment in place. tus. Assuming this is the case, the in- nally. The premise is that a Docker
stallation worked and the commands image is always self-contained; that
Preparation you need to build the image are in is, it has no dependencies on the
place locally. outside world.
In plain English, this means that Virtually every Docker image there-
before image building, a few prepa- Null and Void fore contains a reasonably complete
rations need to be made on the filesystem of a runnable Linux sys-
build system. You can use your own When building Docker images, you tem. How complete the filesystem is
computer for this process, although start with a working directory that is depends strongly on the application
in organizations that build many probably empty, so as the first step, and its dependencies. Some develop-
images, the norm in recent years is you need to create a new folder, ers intuitively go for tools like deboot-
to use a specially prepared system which won’t stay empty long, be- strap and build their own basic sys-
instead. This system does not even cause of the command you run in the tems, but this idea is not particularly
have to be a physical machine; a very next step, good. Even a basic installation of
virtual instance is quite okay. At the Debian or Ubuntu today includes far
latest, when a CI/CD toolchain comes mkdir nginx more than you really need for a con-
into play, the build process will no touch .dockerignore tainer. Additionally, completely DIY
longer take place on a local system images also need to be completely
anyway, although the development of to create a file that acts as an ex- self-maintained. Depending on the
the image will, including any neces- clusion list for files that Docker situation, this can involve a serious
sary test runs. The developer’s own doesn’t need to consider when amount of work.
preferences ultimately determine the building the image. Docker saw this problem coming
desired procedure. The rest of the process is a little and practically eliminated it with a
The general approach to getting a more complicated and requires some small hack. Instead of keeping the
new image off the ground is entirely knowledge of Docker images. Basi- entire contents of the container im-
independent of personal preference. cally, you have several options at this age locally, Docker uses the FROM
Docker itself ships the build function- point. You can either use a prebuilt command when building the image.
ality as a docker component. To use base image for your image or build it The command draws on a public
it, though, Docker must be installed yourself. image on Docker Hub as the basis
on the local system. The first step is Remember that a running container for the image to be created and only
to set up the community edition of on a system initially only contains a adds the components requested by
Docker locally. The following example filesystem. The Linux kernel runs var- the developer.
assumes a system with Ubuntu 22.04. ious functions (namespaces, cgroups) All major distributors maintain their
To begin, you need to install some to move the filesystem to a sort of iso- own micro-images for container
required software management pack- lated area of the system and operates building with their own distribution
ages (Listing 1, line 1); download it there. Unlike in full virtualization and make them available on Docker
the GPG key used to sign Docker’s or paravirtualization mode, a Docker Hub. The same applies to Red Hat

Listing 1: Docker Environment

01 $ sudo apt install apt-transport-https ca-certificates curl softwareproperties-common
02 $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
03 $ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
04 $ sudo apt update
05 $ sudo apt install docker-ce
06 $ sudo systemctl status docker

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 47
 CO N TA I N E R S A N D V I RT UA L I Z AT I O N DIY Docker Images

Listing 2: NGINX Dockerfile Enterprise Linux (RHEL), SUSE, application is therefore typically just
# Pull the minimal Ubuntu image
Ubuntu, Debian, Arch Linux, and the the application itself and its files,
FROM ubuntu particularly lean Alpine Linux, which along with the dependencies that are
# Install Nginx is optimized for container operation. not available in packaged form for the
RUN apt-get -y update && apt-get -y install nginx Distributors are very good at building base image you are using.
# Copy the Nginx config mini-images of their own distributions
COPY default /etc/nginx/sites-available/default and can do it far more efficiently than
# Expose the port for access
An Example
an inexperienced end user.
EXPOSE 80/tcp Distributors regularly maintain their After all this theory, it’s time to build
# Run the Nginx server
images, as well. When a new version the first container. I deliberately kept
CMD ["/usr/sbin/nginx", "-g", "daemon off;"]
of a base image is released, you just the following example as simple as
need to rebuild your own image on possible. It describes building a con-
Listing 3: File default the basis of the new image to elimi- tainer with Ubuntu 22.04 as the basis
server { nate security or functionality issues. (Figure 1) running NGINX, with the
listen 80 default_server; One practical side effect is that the web server serving up a simple HTML
listen [::]:80 default_server; local working directory for image page. The file shown in Listing 2,
building remains easy to understand named Dockerfile, is one of the two
root /usr/share/nginx/html; and clean. basic ingredients required. To experi-
index index.html index.htm; Another great feature of container enced container admins, the contents
building now comes into play: Dur- may seem less than spectacular, but if
server_name _; ing the build, CMD can be used to run you haven’t built a container yet, you
location / {
commands that, for example, add may be wondering what each com-
try_files $uri $uri/ =404;
packages to the distributor’s base im- mand does.
}
age. The content that the developer FROM is the previously mentioned pointer
}
needs to contribute to their own to a base image by a provider – Ubuntu

Figure 1: A preconfigured base image from one of the major distributors is recommended for building your container. A newcomer is
unlikely to be able to put together a leaner image without compromising some of the functionality.

48 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 DIY Docker Images CO N TA I N E R S A N D V I RT UA L I Z AT I O N

in this case. If you prefer a specific The usual way to solve the problem Listing 4: Workflow Config for GitHub
version, you can specify it after a in Docker is to subcontract a volume name: ci
colon (e.g., ubuntu:22.04). The RUN to the container at runtime where the on:
command initiates a command that respective files reside. However, for push:
Docker executes during the build this to work as intended, you need to branches:
within the downloaded base image. preconfigure NGINX in the container - "main"
In the example, it installs the nginx appropriately. You can use a static jobs:
package. COPY copies a file from the configuration for this, although you build:
local folder to the specified location would need to modify the Dockerfile runs-on: ubuntu:22.04
in the image. The example assumes accordingly. Alternatively, you can steps:
that a file named default is in the use variables to pass in the param- -
build folder and later will be in the eters in the shell environment from name: Checkout
NGINX site configuration in the im- which you launch the container as uses: actions/checkout@v3
age (Listing 3). the admin. In the Dockerfile, the -
Again in Listing 2, EXPOSE instructs developer would then define the vari- name: Login to Docker Hub
the runtime environment to expose able with an ENV statement and access uses: docker/login-action@v2
port 80 of the running container to it in the file itself with $<VARIABLE>. with:
the outside world to allow access. However, none of this hides the fact username: ${{ secrets.DOCKERHUB_USERNAME }}
Docker invokes CMD to start the con- that the example is quite rudimen- password: ${{ secrets.DOCKERHUB_TOKEN }}
tainer. In the example, it calls NGINX tary. In everyday life, especially with -
with daemon mode disabled so that more complex applications, you are name: Set up Docker Buildx
stdout remains open; otherwise, the hardly going to get away with such a uses: docker/setup-buildx-action@v2
runtime environment would termi- small number of commands, not to -
nate the container immediately. mention the problems that arise from name: Build and push
Next is building and launching the maintaining the image. For example, uses: docker/build-push-action@v4
image: the image built here has not yet been with:

published. Tests to check the func- context: .

file: ./Dockerfile
$ docker build . -t lm-example/nginx tionality of the image automatically
push: true
$ docker run -d -p 80:80 am-example/nginx are also not planned to date.
tags: ${{ secrets.DOCKERHUB_USERNAME }}/
All of this can be changed relatively
am-example:latest
The first command triggers the pro- quickly. The magic words are con-
cess. You need to call it directly from tinuous integration and continuous
the build directory. After doing so, development or automation and stan- complete application packages that
you will find the finished image in dardization of the image build and find their way in a fully automated
the local Docker registry. The second any testing. For example, an image manner into the Kubernetes target in-
command launches the image for test developer wanting to rebuild an im- stance at the end of a CI/CD pipeline
purposes. If docker ps displays the age just checks a new version of the and replace the workload running
NGINX container afterward, the pack- Dockerfile into Git, and Git handles there without downtime.
age build was successful (Figure 2). the rest automatically. When done, You don’t have to spend big money
the new image is made available on to implement CI/CD with Docker.
More Fun with CI/CD Docker Hub and can be used. GitHub is the obvious choice. It has
Of course, the number of ready-made comprehensive CI/CD integration
Granted, the example shown is un- CI/CD solutions for Docker is now for Docker, including the option of
spectacular and leaves out many practically approaching infinity, not automatically uploading the finished
package building options, as well as least because Kubernetes also plays images to Docker Hub.
options for running NGINX in a more a significant role in this business and Initially, much like this example, you
complex configuration. For example, has been something of a hot topic for have an almost empty working folder
in real life, an NGINX instance always the IT industry as a whole for years. with a Dockerfile and possibly the
needs an SSL certificate along with No longer just a matter of building in- required additional files. You first
the matching key. dividual images, the goal is to create add it to the Git version management

Figure 2: Assuming the build process worked, the container from the example can be launched and creates a working NGINX instance. © Haidar Ali [1]

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 49
 CO N TA I N E R S A N D V I RT UA L I Z AT I O N DIY Docker Images

system and then upload the reposi- Your Registry the domain.crt and domain.key se-
tory to GitHub. For the directory, you crets on the host through Docker.
then need to define the DOCKERHUB_ By the way, GitLab offers features You can do so with the first two
USERNAME environment variable and that are very similar to GitHub. If commands,
a personal access token (PAT) in the you don’t want to spend money on
DOCKERHUB_TOKEN variable. Next, add GitHub to create your own private $ docker secret create domain.crt certs/ U
an action to your directory, which is repositories, you can switch to a lo- domain.crt
an entire workflow in this example. cal GitLab instance instead. Also, if $ docker secret create domain.key certs/ U
For example, the .github/workflows/ you do not want to make your images domain.key
main.yml file in the repository might available to the public, you will need $ docker node update --label-add U
look like Listing 4. a private registry for your container registry=true <hostname>
Once the file is in the directory, any images.
changes checked into the repository That said, running the repository is which also add the contents of the
will trigger an automatic rebuild of not as easy as you might think at two files as passwords to Docker’s
the image by GitHub, which will first. Useful software for this task metadata. Before adding the Docker
then use the given credentials to was not available under a free license service, the last line creates a label for
check the image into Docker Hub. for a long time. Fortunately, several the node running the registry.
Once in place on Docker Hub, the providers now have suitable offer- Again, this example is very simple.
finished image can itself become ings on the market, and one of them For example, the option to secure
part of CI/CD pipelines that, for ex- is Docker itself. The command in access to images with a username
ample, control deployment within Listing 5 launches a local Docker and password combination is miss-
Kubernetes. registry. The command details are ing. Technically, this would not be a
Mind you, GitLab and GitHub are important. problem; the Docker documentation
just two of countless vendors trying The example assumes that the /mnt/ contains more information on the
to make a living with Docker CI/CD. registry/ folder exists on the host, subject.
Jenkins, the classic CI/CD tool (Fig- because it will be mounted to /var/ Running a registry with Quay (Fig-
ure 3), is also very much alive in this lib/registry/ later in the running ure 4) offers significantly more
environment, as are many others. container. You also need to create scope than the standard Docker

Figure 3: GitHub and GitLab now offer extensive CI/CD capabilities for Docker. Standard solutions like Jenkins help developers avoid strict
dependency on a specific Git solution. © Gary Stafford [2]

Listing 5: Local Docker Registry

docker service create --name registry --secret domain.crt --secret domain.key --constraint 'node.labels.registry==true' --mount type=bind,src=/mnt/
registry,dst=/var/lib/registry -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/domain.crt -e REGISTRY_HTTP_TLS_KEY=/
run/secrets/domain.key --publish published=443,target=443 --replicas 1 registry:2

50 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 DIY Docker Images CO N TA I N E R S A N D V I RT UA L I Z AT I O N

approach. The service, which was CD functions on board in the back- simplest option is to roll out Quay
developed by Red Hat to a large ex- ground (Figure 5). The project [3] in the form of a prebuilt container
tent, not only delivers images to cli- is available under a free license, but in Kubernetes.
ents but also has comprehensive CI/ the setup is not very intuitive. The
Conclusions: Not Too Tricky
As the examples show, building
Docker containers is not particularly
complicated. Even running a sepa-
rate registry for containers is quite
easy, all told. If you are planning
larger workloads that are based on
containers, you will inevitably have
to square up to the task of build-
ing images. The best idea is to use
only official Docker Hub images as
the basis for your own work. All
other approaches involve a huge
risk of working blindfolded and can
quickly turn into a nightmare. When
it comes to the practical process of
building containers, CI/CD tools will
help make the whole experience
more convenient. Q
Figure 4: Quay, a registry for container images, provides statistical data on individual
images in addition to the upload and download functionality. © Quay [4]
Info
[1] “Running the NGINX Server in a Docker
Container” by Haidar Ali, May 2022, Bael-
dung: [https://www.baeldung.com/linux/
nginx-docker-container]
[2] “Continuous Integration and Delivery of
Microservices Using Jenkins CI, Maven, and
Docker Compose” by Gary Stafford, Janu-
ary 2016:
[https://programmaticponderings.com/
2015/06/22/continuous-integration-and-
delivery-of-microservices-using-jenkins-ci-
maven-and-docker-compose/]
[3] Quay on GitHub:
[https://github.com/quay/quay]
[4] Quay: [https://quay.io]

The Author
Freelance journalist Martin
Gerhard Loschwitz focuses
Figure 5: Besides the registry functionality, Quay has a lot of additional tech on board that primarily on topics such
includes an extensive CI/CD toolchain that supports developers wanting to build images. as OpenStack, Kubernetes,
© Quay [4] and Chef.
Q

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 51
CO N TA I N E R S A N D V I RT UA L I Z AT I O N Kubernetes Backups

Dos and don’ts of backing up

Kubernetes storage

Securing
the Load
Containers offer great flexibility, but the data they contain often needs to be backed up. Stateful applications
that store their information in a container’s persistent volume can be backed up in a variety of ways, but not
all of them are easy. By Andreas Stolzenberger
In practice, switching to a consistent The databases are distributed redun- of applications have kept their SQL
scale-out architecture with stateless dantly across groups of containers back ends, which forces Helm charts
containers means that users have to and only stored in temporary directo- and operators to roll out the majority
rewrite their existing applications – not ries there. If a container fails, it loses of applications as a stateful set – on
necessarily the application logic, but the entire database, but redundant an architecture that was specifically
the data storage and structural compo- distribution prevents data loss. developed for stateless applications –
nents. Existing database technologies A newly started container synchro- whose core components include a
such as SQL can only handle stateless nizes with existing containers and SQL database with just one container
scale-out operations with major limita- adopts the redundant data of the and persistent memory.
tions. Object databases would be far failed container. The files need to One approach could be to migrate
better suited but entail a fairly exten- be backed up as objects that the ap- the old backup technologies to the
sive redesign of existing applications. plication regularly stores in Amazon containers at the same time, but it’s
I already described this problem in Simple Storage Service (S3) buckets not that simple. Classic client-server
detail in a previous article [1]. or comparable object storage in the backups do not work, precisely be-
In this article, I look at some methods active cluster. All high-level storage cause containers do not use a com-
for backing up your Kubernetes ap- functions, such as a rollback history plete operating system image with an
plications, preferably SQL databases. of object changes and backups, are init system nor their own filesystem
The main aim is to ensure that these handled by the object storage system, tools to run services in the back-
data backups can be restored quickly which explains why early versions of ground and monitor file changes. If
in the event of a failure. The over- container clusters did not even have users were to convert their contain-
view is deliberately restricted to the functions for assigning a persistent ers, they could stay with monolithic
architectures; you will always need to volume (PV) to a container for data services on virtual machines and save
adapt the code to suit your environ- storage. So much for the theory of the themselves all the work that Kuber-
ment and your applications. scale-out world. netes involves. Admins have to come
When users started to move existing up with different approaches to secur-
Backup and Restore applications that were not really suit- ing the persistent data of their Kuber-
Lead Image © Alexander Bedrin, 123rf.com

able for a scale-out architecture into netes applications.

Problems containers, the clusters had to intro-
An early idea of scale-out applica- duce functions to assign persistent Backing Up PVs
tions in containers was to change the storage there. Unfortunately, these
way in which applications store their PVs have reinstated some legacy To back up data from a persistent
data. Applications are distributed problems that platforms such as Ku- volume in Kubernetes, you should
across several nodes and do not have bernetes wanted to eliminate: backup understand how a PV works. Because
to write persistent data to any drives. and restore. Most of the current crop the container has no filesystem tools,

52 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Kubernetes Backups CO N TA I N E R S A N D V I RT UA L I Z AT I O N

it cannot access external storage re- CephFS and handles functions such the storage driver before creating a
sources. Instead, the host on which as snapshots. new DB pod with a connection to the
the container is running integrates the Before a pod on Kubernetes can ac- appropriate PV. A restore would be
storage with the container filesystem cess one or more PVs, it first needs to very simple: Upgrade the last work-
by means of a bind mount, which is order the PVs by issuing a persistent ing snapshot to a fully fledged PV
basically the same principle as when volume claim (PVC). The PVC speci- and start a new DB pod with this PV.
you use fies the PV’s parameters and access However, this operation is only pos-
modes. The most common class sible if the PV driver is capable of
--volume /host/dir:/container/dir comprises volumes of the read-write- taking snapshots and if backing up in
once (ReadWriteOnce) access type for this way makes sense.
to display a subdirectory of the local volumes that are exclusively available
filesystem in the Docker or Podman to only one container. These volumes Maintaining Containers
container. In short, the container does are generally used for databases.
not know anything about the external Read-only-many (ReadOnlyMany) vol- Snapshots have their limitations. If
volumes it is working with; they are umes can be accessed by several you use several Kubernetes clusters
simply part of the filesystem. pods at the same time, but only for with different storage back ends, a PV
In principle, Kubernetes understands read access. These are particularly snapshot will not work as a backup
two types of PVs: block and filesys- popular with scale-out web servers target. Of course, a portable backup
tem. Block PVs can be network drives and deliver the static HTML content. has other options: maintenance
that use the Internet Small Computer The third mode is read-write-many containers, which come with tools
Systems Interface (iSCSI), RADOS (ReadWriteMany) and allows multiple for writing the PV data to a backup,
Block Device (RBD), or Fibre Chan- containers to read from and write that can use a second PV acting as a
nel standard. In simple single-node to the volume. This mode obviously backup target. This process can be
setups, local logical volume manage- involves certain risks and is not sup- simple, such as:
ment (LVM) volumes are also fine. In ported by all storage drivers. As a
the case of block PVs, the Kubernetes rule, filesystem PVs are capable of rsync /<source> /<target>
node creates a filesystem such as XFS this mode.
on the drive and then maps it into the or
container. Alternatively, filesystem Stop First, Back Up
PVs can be used that rely, for ex- tar --czf <backup-file>.tgz /volume/*
ample, on NFS, GlusterFS, or CephFS. The myth of the simple and secure
PVs based on a network filesystem storage snapshot as a backup is still A full dump with tools such as mysql-
are theoretically easier to manage making the rounds on forums and dump is also recommended for DBs.
than block PVs. manuals. The story goes that users The backup PV for the maintenance
However, network filesystems always simply need to take an LVM or filesys- container can be a read-write-many
work without a write cache and are tem snapshot of their data partition to filesystem volume, which is accessed
therefore slower than block PVs when have a reliable backup. That is just as by the external backup software after
it comes to writing I/O, making them wrong today as it was 20 years ago. the dump and the backup data have
less suitable for operation with a da- A running database server – whether been processed downstream.
tabase. Block volumes in turn benefit MySQL, Microsoft SQL, PostgreSQL, The procedure here is just as simple.
from the fact that the Kubernetes or even SQLite – always has files The user first terminates the DB
host, which mounts the volume and open for write access. Snapshots of pod, starts the maintenance pod
manages the local filesystem, can filesystems with open files are po- with source and target PV, and runs
cache writes. Although good for per- tentially unusable, depending on the the backup tool in it. The mainte-
formance, it is bad for backups. state of the file during the snapshot. nance pod then terminates and a
Where exactly the volumes for the For virtual machines with databases new DB pod takes over. The catch
PVs come from is organized by the (DBs), a reliable snapshot backup is with this solution is that the longer
Kubernetes storage classes and driv- performed as follows: A script stops the backup takes, the longer the ap-
ers that are available from various the DB service, which writes all the plication is down. The maintenance
hardware and software manufactur- data from the cache (the DB) to the container also needs to have match-
ers to match their storage systems. tablespace. The script first synchro- ing restore tools in place so that it
Storage drivers have major functional nizes the filesystem cache before can also write the backed-up data to
differences. TopoLVM, for example, triggering the snapshot. The process a fresh PV in the correct format after
offers only a few functions and is needs to be similar with Kubernetes. a failure involving data corruption.
limited to single-node setups. Rook, The user stops the DB pod (and In the case of application-specific
on the other hand, manages Ceph deletes it in the process) and then backup tools, the user also needs to
storage clusters with RBD devices or assigns the snapshot of the PV via pay attention to the versions. The

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 53
CO N TA I N E R S A N D V I RT UA L I Z AT I O N Kubernetes Backups

tools must match the version of the mirrored DBs are ready to roll out. For such scenarios, it makes more
software used in the application Crunchy Data, for example, offers sense to roll out with separate defi-
container. a very powerful Kubernetes opera- nition files. The kind in this case is
tor for PostgreSQL DBs. It can set Deployment. Like StatefulSet, it
Continuous Backup up mirrored clusters with just a few describes the application pods, but
commands and offers ready-made without the embedded PVC template.
Continuous backup is another po- pods for backing up and restor- Instead, it references a PVC that you
tential approach that involves an ap- ing PostgreSQL clusters. The open create separately from the deploy-
plication or DB server reversibly for- source version of the Crunchy op- ment. An existing definition of a
warding each transaction to a backup erator (Figure 1) is available free stateful set can easily be rewritten
instance or a transaction log. How- of charge on GitHub [2], and the as a deployment by removing the
ever, the application also needs to manufacturer offers commercial volumeClaimTemplate block and in-
support this form of data replication. tools with support for production stead referencing the PVC described
Fortunately, all classic SQL DB serv- environments. in another file in the volumeMounts of
ers can handle a mirror mode. The Kubernetes supports various meth- the pods. You then need to declare
active DB server sends all changes ods for rolling out applications two deployments – one for the ap-
that need to be written directly to a with a connection to a PV. The kind plication itself and another for the
second instance by way of its own (i.e., the type of a Kubernetes API maintenance container – and then
SQL protocol. function call) is the StatefulSet. It reference the separately created
The user can then start additional describes a template for a persistent PVCs in both.
tools, such as a DB dump on the volume claim along with the ap- The complete backup process is as
backup node, without negatively af- plication pods. When a user starts follows: Delete the DB pod, start the
fecting the function and performance StatefulSet, it dynamically gener- maintenance pod, then roll out the
of the primary DB server. If the ates the PVC of the application. DB pod again. Of course, you can au-
backup server does not react imme- Therefore, if the user deletes the tomate this process with a tool such
diately, the source server caches the stateful set, they also delete the PVC as Ansible. Unfortunately, a lot of
incoming files and delivers them to and, where applicable, the associ- bad solutions are making the rounds
the backup instance after a delay. ated PV. However, for backup setups online that use the shell with shell:
Continuous backup allows every single such as the method described with kubectl <something> to try to control
transaction in a DB to be tracked, but the maintenance container, you Kubernetes resources and query their
it also makes the restore more difficult. would want to delete the applica- status. It makes more sense to use
In case of data loss in the active DB, tion rollout without jeopardizing the modules from the kubernetes.core
for example, where the DB application PV with the data or the associated collection and avoid error-prone shell
deletes the data undesirably because of claims. calls.
an error or an attack that encrypts or
overwrites data, the user needs to go
through the transaction log step by step
and find the last “good” transaction.
The additional overhead also means
that the user can only retroactively
determine the point in time up to
which the data states need to be re-
stored. In DBs with frequent write
and change access, a transaction log
will also occupy a large amount of
memory, which is why a combination
of point-in-time full backups and the
transaction log between the status
of the current DB and the last full
backup is recommended.

Tools and Configurations

In many cases, you do not need to
be overly creative when configuring Figure 1: Crunchy’s Kubernetes operator automates the process of rolling out PostgreSQL
a continuous backup scenario. A clusters. As an additional function, Crunchy can create backups by the operator of running
number of ready-made Kubernetes clusters.

54 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Kubernetes Backups CO N TA I N E R S A N D V I RT UA L I Z AT I O N

You can control Kubernetes resources that waits for the maintenance pod thing as a reliable backup at the
with the Ansible kubernetes.core.k8s to be completed after it has been push of a snapshot button without
module and enter the kind definition started. The pattern is: register pod additional work. Q
for Kubernetes directly into the An- state, until status = “Completed”.
sible code as definition:, but with The complete backup playbook first Info
the use of Ansible variables. Alter- deletes the DB deployment, starts the [1] “Kubernetes StatefulSet” by Andreas
natively, you can convert your exist- backup pod, waits until it has done Stolzenberger, ADMIN, issue 73, 2023, pg.
ing Kubernetes YML declarations to its work, and then restarts the DB 18, [https://www.admin-magazine.com/
Jinja2 templates and integrate them deployment. Archive/2023/73/Kubernetes-StatefulSet]
into the Ansible code with: [2] Postgres operator: [https://github.com/
Conclusions CrunchyData/postgres-operator]
definition: "{{ lookup(U
'template', U A reliable backup for Kubernetes
'kubernetes_deployment.j2') }} requires some preparation, which The Author
is no surprise to Kubernetes users, Andreas Stolzenberger
In the same way, state: absent re- because migrating existing applica- worked as an IT magazine
moves a deployment. Information on tions from virtual machines to pods editor for 17 years. He was
running pods, deployments, or val- controlled by Kubernetes already the deputy editor in chief
ues of a config map can be retrieved involves a great amount of work. If of the German Network
with the kubernetes.core.k8s_info you want to be independent of stor- Computing Magazine from
module and be bundled into an An- age backups, your best bet is to use 2000 to 2010. After that, he worked as a solution
sible variable by calling register. a maintenance pod to back up your engineer at Dell and VMware. In 2012 Andreas
In combination with the until loop data. The continuous variant also moved to Red Hat. There, he currently works
instruction, you can generate a wait- has its appeal but involves additional as principal solution architect in the Technical
ing loop in your backup playbook overhead. Of course, there is no such Partner Development department.
CO N TA I N E R S A N D V I RT UA L I Z AT I O N Local Azure Arc

Azure Arc agent on local machines

Cloud Contact
Manage your on-premises servers with Windows Admin Center in Azure. in your firewall. All you need for the
connection is an agent for Azure Arc
By Thomas Joos
(more on this later) on the desired
local servers.
Installing the Azure Arc agent on lo- services can be managed very easily
cal machines lets you integrate these from the Windows Admin Center. Registering WAC with Azure
machines with Azure and manage
them remotely. Neither a virtual pri- Connecting to Azure To register your WAC gateway with
vate network (VPN) nor port sharing the respective Azure subscription
in the firewall are required for this To connect the Windows Admin Cen- after installation, in WAC, click Azure
kind of access, and this approach also ter to Azure, you only need to set up hybrid center and choose the Register
supports remote PowerShell sessions the corresponding configuration on your Windows Admin Center gateway
and remote desktop protocol (RDP) the computer with the Admin Center link. WAC displays a window where
connections. In this article, I look into gateway. Once the connection has you first need to log in to Microsoft
how you can manage servers from been established, when you connect Azure; WAC then connects to your
wherever you are with Azure Arc. to the gateway from a web browser, subscription in a process that only
The ability to leverage Azure cloud you also have access to the Azure takes a few seconds. You then have to
resources locally from the Windows functions if you are authorized to do create a local account for the server
Admin Center (WAC) has been so. To set it up, click the gear icon in under Account in the WAC settings
around for quite a while now. It is the Windows Admin Center to call (which are available from the gear
particularly useful for backing up up the settings. When managing the icon at the top right) and one in
local data to the cloud with Azure gateways, you will find a wizard in Azure at the same time.
Backup. Azure Arc takes the opposite Azure, which you can use to log in. Once Azure and WAC are linked, the
approach and lets you manage your The various Azure services are then services for which Azure resources
on-premises servers with WAC in available on the network. can be used in the local data cen-
Azure. If you connect WAC to Azure, The Windows Admin Center gives ter are made available in the Azure
you can use it to manage Azure vir- you everything you need for these hybrid center when you connect to
tual machines (VMs) and synchronize services. After the install, you can a server in WAC. You can see the
Lead Image © Rungaroon TAWEEAPIRADEEMUNKONG, 123RF.com

VMs from Hyper-V with the cloud as then connect the WAC gateway to services that are already in use in the
Azure VMs for failover purposes. Azure. Doing so lets you use many Installed services menu. Azure Arc
The most interesting functions in Azure services in your local data cen- lets you connect to local servers with
Azure for supporting local networks ter (some of which are free of charge) an agent to Azure in WAC in just a
are Azure Backup, Azure Monitor, and connect your servers to Azure few steps. After a short wait, the local
Security Center, Site Recovery, and through Azure Arc, which means that server becomes available in the re-
much more, but local networks also you can manage the connected serv- spective resource group on the Azure
benefit from these Azure services. For ers in WAC on the Azure portal in the portal and you can manage, secure,
small and medium-sized enterprises same way as you would with the lo- monitor, and customize the server.
(SMEs), the advantage is that you cal Windows Admin Center. As men- At the same time, you can enable
can use Azure functions without hav- tioned previously, you do not need the Windows Admin Center for local
ing to run your own servers, and the a VPN and do not have to drill holes resources on the Azure portal, which

56 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Local Azure Arc CO N TA I N E R S A N D V I RT UA L I Z AT I O N

allows authorized admins to access top right by pressing the bell icon. On want to authorize. On the Azure por-
WAC in the cloud and use the Azure the server, you can now enter tal, click on the resource group with
Arc agent to manage local machines. your server and then on Access control
Basically, the same access options ex- appwiz.cpl (IAM) on the left. Here you define the
ist here as for management in the lo- authorizations that your account or
cal data center. With WAC now avail- to check whether the Azure Con- other users will have for the server.
able on the Azure portal, access can nected Machine agent is installed on Next, click the Add role assignment
of course be appropriately secured the server. The local machines can button in the Grant access to this re-
with authorizations from the Azure now be managed with Azure poli- source section. Select Windows Admin
Active Directory. After registration, cies, inventories are supported, and Center Administrator Login and click
the Azure hybrid center also lets you you can monitor both performance Next. On the next page, select Mem-
integrate Azure services in the local and security. Automatic management bers to specify the user accounts you
data center, such as Azure Backup for processes are now available, as is the want to authorize. Click on the Check
backing up local data in the cloud. ability to distribute updates from the and assign button to complete the pro-
Azure Update management center. cess. The Access control (IAM) | Check
Managing Servers Remotely Change tracking is another useful access | View my access button will
feature that can be used to track con- show the matching authorizations in
To establish the relationship be- figuration changes, and you can use the Role assignments window.
tween your local computers and extensions to intensify your collabora- Once you and the added users have
Azure Arc, you first need to register tion with Azure. For example, locally been assigned permissions, you and
WAC with an Azure subscription, operated SQL servers can be tied in they can access local servers from
as described above, then join your with Azure or comprehensive moni- the server’s resource group in the left
server in the Azure hybrid center toring scenarios can be implemented. pane with the Windows Admin Center
from Set up under Set up Azure Arc Additionally, the Azure portal now | Connect button. After a short time,
– this is also possible with a free offers remote maintenance for local a Windows Admin Center login win-
Azure subscription. Now select the servers (e.g., over SSH) by opening the dow appears on the Azure portal. You
connected Azure subscription, click portal.azure.com URL and clicking on need to authenticate with the admin-
the Add new option in Resource the All resources link. After starting the istrator account of the local server,
group, and enter the name of the desired server, you will see various tiles not with your Azure login credentials.
server (e.g., srv1). Choose an Azure in the lower area that let you perform At this point, you can also use Ac-
region (e.g., West Europe), then administrative tasks over the Internet. tive Directory logins with the syntax
click Set up. The server is then visi- <server or domain>\<username>.
ble in the Servers pane in Azure Arc Authorizing Remote
on the Azure portal (Figure 1). Conclusions
During this process, Windows Ad-
Administration
min Center installs an agent on your So that you can log on to the server In combination with Azure Arc, the
server to help you connect to Azure. locally in WAC, you first need to select cloud-based Windows Admin Center
You can view the current processes the user accounts in the cloud that you can be used for establishing a secure
connection to the local data center.
The setup is quick and easy and
even possible with the free version
of Azure. Small businesses will ben-
efit from simple administration and
support for more security, and larger
corporations can use the change
management feature and Azure Moni-
tor to monitor and control cloud re-
sources just as effectively as servers
in the local data center. Q

The Author
Thomas Joos is a freelance IT consultant and
has been working in IT for more than 20 years.
In addition, he writes hands-on books and
papers on Windows and other Microsoft topics.
Online you can meet him on [http://thomasjoos.
Figure 1: The local server is visible in Azure after successful integration in WAC. spaces.live.com].

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 57
S EC U R I T Y OSSEC

Intrusion Detection with OSSEC

Guardian
Angel

The OSSEC free intrusion detection and host-based intrusion prevention system detects and fixes security
problems in real time at the operating system level with functions such as log analysis, file integrity checks,
Windows registry monitoring, and rootkit detection. It can be deployed virtually anywhere and supports the
Linux, Windows, and macOS platforms. By Thomas Joos

As a host-based intrusion detection application lets you disable user ac- as network-based intrusion detection
system (HIDS), OSSEC [1] detects counts that are misused for attacks systems (NIDS) or firewalls to create
and reacts to security incidents in and supports alerting to ensure a a comprehensive security system.
real time. The software is capable of rapid response to incidents. The free Other editions are also available. For
detecting a wide range of security software basically focuses on moni- example, OSSEC+ offers additional
incidents, including attacks on file- toring systems and networks. functions, such as machine learn-
systems and directories, changes to In this article, I look at why the use of ing, but requires registration with the
system files and configuration files, OSSEC is a sensible step toward sig- manufacturer before use. This edi-
failed login attempts, and attempts to nificantly enhancing security on net- tion also integrates the Elasticsearch,
escalate privileges. The tool also de- works. Ultimately, OSSEC helps you Logstash, Kibana (ELK) stack. Central
tects changes to logfiles and network detect security incidents before virus administration and thousands of
attacks such as port scans, connec- scanners or other systems, which is rules, as well as role-based authoriza-
tion breaches, and distributed denial- particularly important in ransomware tions and a comprehensive reporting
of-service (DDoS) attacks. attacks, for example, because time is system, are restricted to the scope
In this article, I show you how to set a critical factor. of the commercial-grade Atomic OS-
up the server and the clients. You SEC version. The differences between
can also set up OSSEC as a Docker OSSEC Editions the various editions are listed on the
container. All packages are available download page.
directly from the download page [2]. The basic version of OSSEC is open
source and offers you a rich feature Typical Deployment
Taking Countermeasures set with log-based intrusion detec-
tion, rootkit and malware detection,
Scenarios
Photo by Rayner Simpson on Unsplash

OSSEC offers a range of counter- active response, compliance auditing, One practical use of OSSEC is system
measures to help you respond to file integrity monitoring, and system and application log analysis to detect
security incidents, such as blocking inventory. It is important to note that signs of security breaches or suspi-
IP addresses or hosts that exhibit OSSEC as a HIDS focuses on monitor- cious activity. If the system identifies
suspicious behavior and terminating ing individual systems. Therefore, this kind of activity, OSSEC notifies
processes that are unauthorized or you should use OSSEC in combina- you by email, Slack, or another con-
attempting attacks. Additionally, the tion with other security tools such figured notification method. At the

60 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 OSSEC S EC U R I T Y

same time, the software can carry out (WSL); obviously, this is not ideal. keep the communication between the
actions in its active response system, For macOS, the OSSEC install works systems private.
such as blocking users or IP addresses. in a similar way to Linux. You can In many cases, cloud providers also
File integrity checking is another use download the source code from the offer their own security products and
case. OSSEC monitors important sys- official website and compile the pro- monitoring tools. It is advisable to
tem files and directories for changes. gram or use a package manager such use these in combination with OS-
If something unexpected occurs, you as Homebrew. SEC to maximize the security of your
are notified and can actively com- When installing OSSEC, you need to cloud infrastructure.
bat attackers and malware before coordinate the components, which
the damage spreads, making this a include a central server, the OSSEC Installation and Setup
valuable tool, especially in the fight manager, and the OSSEC agents on
against ransomware. the systems to be monitored. The You can either install OSSEC or
For Windows users, OSSEC also manager collects and analyzes data download it as a virtual appliance;
supports monitoring the Windows from the agents and runs appropriate the former option is preferable. Serv-
registry. Changes in the registry can actions when the tool detects threats ers and agents can be downloaded
indicate a security breach or an un- or security breaches. Make sure you separately and connected to an OS-
wanted application. OSSEC tracks choose the right configuration for SEC server. In contrast to many live
these changes and alerts you to suspi- your requirements and set up the DVDs or simpler open source tools,
cious activity. communication between manager OSSEC is not easy to install, set up,
Rootkit detection is another important and agent correctly. Adjustments are and operate. You need to take time
feature of OSSEC. Rootkits are mali- made in the configuration file. to familiarize yourself fully with the
cious programs that try to hide deep The OSSEC server can also work product. As a general rule, it makes
in the operating system to remain un- without agents, but then it does not sense to reference the documenta-
detected. The tool searches for known read all the information. Agents can tion [3] when you are setting up and
signatures and behavior patterns to also be deployed on Windows servers managing the system.
identify rootkits and report their pres- to monitor the registry for changes. In the following discussion, I concen-
ence to you. The server can also read data from trate on the free OSSEC version 3.7.0
Finally, OSSEC has an active response system logfiles and aggregates all this install on Ubuntu 22.04. After the
functionality that reacts automatically information to carry out actions or mandatory Linux update, install the
to detected threats. For example, you deliver notifications in the event of required dependencies:
can configure OSSEC to block net- trouble.
work access for an IP address from OSSEC can be used in the cloud, as sudo apt-get update
which repeated failed login attempts well. In cloud environments, you can sudo apt-get -y upgrade
have been made. install the application on both virtual sudo apt-get install -y build-essential
machines and cloud-based servers. sudo apt-get install -y zlib1g-dev U
Parallel bookable services are avail- libpcre2-dev libevent-dev libssl-dev U
OSSEC on Various Platforms
able directly in the cloud for Micro- libsystemd-dev jq
OSSEC works across all platforms soft Azure, Amazon Web Services
and can be installed on Linux, Win- (AWS), and Google Cloud Platform The certificates and key files are then
dows, and macOS, which makes the (GCP). You need to install and config- downloaded and installed and the re-
environment a useful choice as an ure the OSSEC manager on a central pository is integrated:
additional security tool on hybrid instance or a dedicated server then
networks. Linux distributions such set up the agents on the individual wget http://www.ossec.net/files/U
as Debian, Ubuntu, CentOS, Red Hat virtual machines or cloud servers that OSSEC-ARCHIVE-KEY.asc
Enterprise Linux, and Fedora all sup- you want to monitor. Here, too, the wget https://github.com/ossec/ossec-hids/U
port OSSEC. You can use the package agents communicate with the central releases/download/ 3.7.0/U
manager for the install or compile manager and send information about ossec-hids-3.7.0.tar.gz.asc
from the source code. the system activities, which the man- gpg --import OSSEC-ARCHIVE-KEY.asc
On Windows, you must make sure ager analyzes before triggering appro-
that the software supports your ver- priate actions. Next, download the current OSSEC
sion and that all the required updates If you use OSSEC in the cloud, it is version, and unzip and execute the
are in place before downloading important to safeguard the communi- installation script:
and installing the OSSEC agent for cation path between the agents and
Windows from the official website. the manager. You need to use pro- wget https://github.com/ossec/U
You cannot install the OSSEC server tected connections such as encrypted ossec-hids/archive/3.7.0.tar.gz
itself on Windows unless you use virtual private network (VPN) tunnels gpg --verify ossec-hids-3.7.0.tar.gz.asc U
the Windows Subsystem for Linux or private network connections to 3.7.0.tar.gz

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 61
 S EC U R I T Y OSSEC

tar -zxvf 3.7.0.tar.gz && U (Solaris, FreeBSD, or NetBSD) if it de- OSSEC agents, the client IP addresses
cd ossec-hids-3.7.0/ tects an attacker. This step helps fend need to be added to the configuration
wget https://github.com/PCRE2Project/U off brute force scans, port scans, and file under <remote>, and the connec-
pcre2/releases/download/pcre2-10.40/U some other forms of attack. tion must be defined as secure:
pcre2-10.40.tar.gz You then have the option of adding IP
tar -zxvf pcre2-10.40.tar.gz U addresses to allowlists if you do not <remote>
-C src/external/ want OSSEC to check and block com- <connection><secure></connection>U
munication from some IP addresses. <allowed-ips>192.168.0.2</allowed-ips> U
No errors should appear in the build You will also want to enable remote <!-- OSSEC client -->
process at this point. syslog over UDP port 514 so that the </remote>
After starting the installation script, system can send logfiles to syslog
the next step is to select the language servers. Once you have made your If OSSEC detects an attack originating
in which you want to install OSSEC. choices, the wizard installs OSSEC from an IP address, the system blocks
English is preselected and I do not rec- accordingly. At the end of the instal- it for 10 minutes. If further suspicious
ommend changing this setting, because lation you will see a Configuration packets then originate from the IP
some of the translations were inaccu- finished properly message. address, OSSEC identifies them as re-
rate when I tried them out in the lab. peat offenders and blocks them for a
Configuring the OSSEC longer period. This can be defined in
Adapting the Environment the <active-response> section of the
Server same configuration file:
A wizard helps you customize the in- The next step is to tailor OSSEC to
stallation (Figure 1). The server selec- your requirements in the /var/ossec/ <!-- Active Response Config -->
tion installs the first OSSEC server on etc/ossec.conf configuration file, for <active-response>
the network; you can connect more example, in Nano: <repeated_offenders>30,60,120,240,480U
agents running on Linux, Windows, </repeated_offenders>
or macOS to the server later on. Next, sudo nano /var/ossec/etc/ossec.conf </active-response>
select the installation directory; the
default is var/ossec. You can customize the email configu- This example blocks potential at-
You can also enable email notifications ration and, as shown in Listing 1, in- tackers for a longer period on each
now by entering an email address. In tegrate the IP addresses of the services new attempt. If you make changes
many cases, the system finds the SMTP and clients in the <global> section. to the configuration file, you need to
server, but you can change the con- To read the syslogs from the various restart OSSEC.
figuration at any time. You will always
want to install the integrity check dae-
mon, which is also preselected, as is
the rootkit detection engine.
I also recommend adding active re-
sponse as part of the setup. This func-
tion lets the system run predefined
commands on the basis of incoming
events. For example, you can block
an IP address or block access for a
specific user. In the next step, it makes
sense to activate the firewall drop re-
sponse, which allows OSSEC to block
the host in iptables (Linux) or ipfilter

Listing 1: Store IP Addresses

<global>
<allow_list>127.0.0.1</allow_list>
<allow_list>::1</allow_list>
<allow_list>localhost.localdomain</allow_list>
<allow_list>127.0.0.53</allow_list>
<allow_list>10.0.0.2</allow_list>
<!-- OSSEC client -- >
</global>
Figure 1: The basic setup of OSSEC is handled by a text-based wizard.

62 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 OSSEC S EC U R I T Y

etc/ossec.conf file. At this point,

Connecting Agents Listing 2: Secure Communication with Server
it is important to enter the server’s <global>
The best way for OSSEC to collect IP address in the <ossec_config>
<allow_list>10.0.0.2</allow_list>
information from computers on the section:
<!-- OSSEC client -- >
network is to install agents. A cli-
ent for Windows is available from <ossec_config> </global>
the download page, which you can <client> <remote>
install from the graphical user inter- <server-ip><IP-address></server-ip> <connection>secure</connetion>
face. Connecting Linux computers is <allowed-ips>10.0.0.2</allowed-ips>
slightly more complicated but is also When managing the connected cli-
<!-- OSSEC client -- >
quickly done. ents, you can first display a list of the
</remote>
To launch the agent on a Windows connected computers and create the
server, you first need to complete authentication keys that the clients
the install, which can be scripted require for the connection. For Win- Conclusions
easily. A connection to OSSEC is dows clients, use the setup window; OSSEC is a powerful tool that can
then opened by the OSSEC agent on Linux, start the same tool as on detect and combat malware and cy-
manager where you enter the IP the server by typing: berattacks and can be run on a virtual
address or the name of the OSSEC machine – no physical hardware is
server and the authentication key sudo /var/ossec/bin/manage_agents required. As well as email notifica-
for the connection that you create tions, OSSEC can run actions and use
on the OSSEC server. It is important You can then enter the authentication Slack for communication – the project
at this point to enter the IP ad- code in the terminal by selecting I. It documentation provides useful help
dresses of the clients in the OSSEC generally makes sense to reboot the for setting up these features.
configuration file on the server, as server and the client after integrating In addition to the free OSSEC version,
already described. clients. To do this, run the following you might also want to try out OS-
To manage the agents on the server or on the devices: SEC+. You do need to register, and in
to create authentication keys, launch many cases OSSEC is fully up to the
the administration program on the /var/ossec/bin/ossec-control restart task in hand. Q
server by typing:
You can see whether a Linux cli-
sudo /var/ossec/bin/manage_agents ent has connected successfully by Info
typing: [1] OSSEC: [https://www.ossec.net]
Installing the Linux agent is basi- [2] OSSEC downloads:
cally the same as installing the sudo /var/ossec/bin/agent_control -lc [https://www.ossec.net/ossec-downloads/]
OSSEC server, but select the agent [3] OSSEC documentation: [https://ossec-docs.
installation variant and not server. If the connections between clients readthedocs.io/en/latest]
A wizard then appears, and you can and servers do not work, it is usu-
set up the agent in the same way as ally because of the firewall settings
the server. Specify the installation on the server. Make sure the firewall The Author
directory, enter the name of the OS- does not block communication be- Thomas Joos is a freelance IT consultant and
SEC server, and enable the integrity tween the server and the clients, has been working in IT for more than 20 years.
check daemon, the rootkit detection especially ports 1514 and 514. Addi- In addition, he writes hands-on books and
engine, and active response. On tionally, secure communication with papers on Windows and other Microsoft topics.
Linux computers, you can also con- the server must be permitted in the Online you can meet him on [http://thomasjoos.
figure the agent in the /var/ossec/ OSSEC configuration file (Listing 2). spaces.live.com].
Q

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 63
M A N AG E M E N T Restoring Hybrid Identities

Recovering from a cyberattack in a hybrid environment

Disconnected
Restoring identity is an important part of disaster recovery, since it lays the foundation for restoring normality
and regular operations. We look into contingency measures for hybrid directory services with Entra ID, the Graph
API, and its PowerShell implementation. By Evgenij Smirnov
The complexity of modern IT land- companies were forced to introduce the cloud account authoritative with
scapes becomes particularly apparent as an emergency measure during the password write-back, enabling more
in the event of emergencies caused pandemic. However, other services extensive password checks than the
by cyberattacks. It is not just the re- such as virtual private networks complexity conditions supported in
covery of the individual subsystems (VPNs), Microsoft Dynamics 365, AD and providing users with a simple
that needs to be considered when you Salesforce, or Box also work best self-service password reset (SSPR).
restore, but also the interactions. The with a cloud identity. Thanks to Cloud Trust, even Kerberos
failure of basic services such as au- authentication of a cloud account
thentication as the result of an attack Hybrid Identities on against local resources connected to
is a particularly serious worry. AD is possible.
Regardless of whether a company still
the Rise All types of hybrid identity are rela-
has its IT firmly anchored locally or How tightly the on-premises part of a tively easy to set up, and the provid-
is already on its way into the cloud, hybrid identity is tied in to its cloud ers supply both technical aids and
most directory services now have a counterpart can vary. Some organi- very good instructions. Microsoft has
hybrid design. The legacy Active Di- zations want to provide an online a particular interest in migrating iden-
rectory (AD) is the primary identity identity but keep the authentication tities to its own cloud as quickly and
Photo by Kelly Sikkema on Unsplash

store, and users, groups, and, increas- process entirely on-premises and rely painlessly as possible; the new privi-
ingly, computer accounts are synchro- on pass-through authentication (PTA) leged access strategy envisages the
nized to Entra ID (formerly Azure or locally installed instances of Active cloud as the “source of security” [1].
AD) or other cloud-based directories Directory Federation Services (ADFS). However, the successful cyberattacks
to enable seamless access to applica- Others synchronize the password of recent years all too clearly reveal
tions in the cloud. The prime example hash to the cloud (password hash the weaknesses of the strategy. On
is Microsoft Teams, which many synchronization, PHS) or even make the one hand, both components – on

64 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Restoring Hybrid Identities M A N AG E M E N T

premises and in the cloud – are re- rudimentary use (e.g., video tele- a reasonable time, even if relatively
quired in a hybrid identity landscape phony in Teams), they will typically weak passwords are in use.
to enable smooth operation of all ap- not waste time compromising these In known attacks, hackers have reset
plications. On the other hand, the re- services and simply encrypt or de- the passwords of all synchronized us-
lationship between the legacy identity stroy the applications and data oper- ers as a last resort before destroying
and the modern identity creates com- ated onsite. Your cloud tenant may be AD and Azure AD Connect. Although
pletely new attack vectors that exploit spared to a great extent. the data in the cloud applications
the peculiarities and functional weak- Now, though, you face a different are still intact, access is severely re-
nesses of both sides. challenge. Because AD is the lead- stricted or not possible at all for the
ing identity store on the hybrid net- time being.
Many Roads Lead to work, users and groups are available In other, more complex scenarios,
in the cloud but cannot be managed the attacker first penetrates the
Disaster there. If you enabled PHS, users cloud by phishing or manipulat-
A hybrid identity system can be at- can even log in to cloud services ing the multifactor authentication
tacked in many ways and be com- with the last known password, but (MFA) procedures and only then
pletely or partially disabled. The you cannot change the password if discovers that a synchronized AD
measures you need to take to get the password write-back, SSPR, or both account they have hijacked has far-
system operational again are just as have not also been enabled. On reaching authorizations. In this way,
varied. the other hand, if you rely on PTA they can obtain an identity that al-
A classic attack scenario focuses on or ADFS, your cloud users will no lows them to compromise the cloud
the local part of the organization. longer be able to log in because the tenant or at least steal data such as
The attacker typically takes complete primary authentication systems are email, documents, or chat histories
control of AD, ultimately to destroy it. no longer available. For this reason, from cloud applications to later
This positioning also opens the cloud both Microsoft and most experts phish other users (CEO fraud) or
component attack, because by taking recommend enabling PHS – at least threaten to damage the company’s
over the traditional AD, the attacker as an additional authentication pro- image and extort money.
also gains control of Azure AD Con- cedure. The frequently cited angst While you are restoring the on-prem-
nect or Cloud Sync agents and can prompted by synchronizing pass- ises infrastructure, you need to make
escalate into the cloud (Figure 1). words with the cloud is unfounded, sure your cloud services no longer of-
However, if the attacker discov- because hashing more than 1,000 fer the attacker a gateway before you
ers during their investigations times makes it uneconomical to re- reconnect the two parts of the hybrid
that the cloud services are only in construct the plain text password in environment.

Figure 1: Azure AD Connect is considered active even though the local AD has been compromised.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 65
M A N AG E M E N T Restoring Hybrid Identities

Graph PowerShell Is the compromised services, but the ap- changed such that users can no lon-
proach you use is decisive for the ger log in, but you still have admin-
Future next steps. Restoring on-premises istrative access to the client. In the
Although the MSOnline and AzureAD services after destruction or com- third case, you no longer have access
PowerShell modules are still available promise has a long history. For to the cloud tenant at all; initially,
and largely supported at press time, many applications – including AD you don’t even know how serious
the date for shutting down the service – specialist tools let you restore up the damage is, but users are report-
endpoint these modules reference to a relatively recent point in time ing that they can no longer log in to
has already been set. Some initial without the risk of reactivating the cloud services.
functions have even been disabled; malware introduced by the attacker. If your usual administrator accounts
you will find many references to this If this option is not available, you (including the break glass account)
with an online search. Therefore, this face a choice in most cases: no longer work, you need to check
article focuses on the cmdlets from Q Use a more or less fresh backup your documentation for applications
the Microsoft Graph Suite [2], which and follow up the restore (which whose identities may have sufficient
will be the only PowerShell interface must take place on an isolated net- permissions to create a new adminis-
to Microsoft 365, and in particular to work) by performing a comprehen- trator or reset the password of an ex-
Entra ID, supported in the future. sive analysis and cleanup. isting administrator. If you find such
The Graph modules come with an Q Revert to a backup from a little fur- an application, it may be your best
extremely large number of cmdlets. ther back, hope that it has not yet chance of regaining control over your
If possible, you will want to use been compromised, and remake tenant. At the same time, you have
PowerShell 7 to work with Graph the changes that have taken place probably also found the way for the
PowerShell. Even there, importing since the backup. attacker to hijack your client. In other
the entire Graph module, including Q Completely rebuild the directory words, the need for action is urgent
all the dependencies, takes several service because all of your back- because you need to shut down this
minutes and bumps up the RAM ups have been affected by the attack vector before proceeding to re-
utilization of the PowerShell session attack. store functionality.
to somewhere between 2 and 5GB. In all of these cases, you can be If all administrative access is lost,
If you already have experience with relatively sure that you have suf- the only thing that can help is a top-
Graph, it is a good idea to load only ficient control over the rebuilt AD priority support request to Microsoft.
the modules you really need. For environment as long as it remains Be prepared to have to prove tenant
basic Entra ID tasks, for example, isolated from the Internet (i.e., it ownership through documents such
you only require two submodules, cannot establish a connection to the as invoices or payment receipts and
and you can quickly set these up by outside world). not just, for example, the ability to
typing: Far tighter limits apply to disaster generate DNS records in custom
recovery of cloud services, because domains, although this is sufficient
Import-Module U the operator only allows access to for registration. In other words,
Microsoft.Graph.Authentication, U specific data, and in some cases this you depend on the support of your
Microsoft.Graph.Identity.DirectoryU access is read only. What you are management or commercial depart-
Management looking at in this case is restoring ment, whose ability to act is severely
the hybrid identity. For example, restricted by the attack and whose
If you rely on PowerShell 5.1 and assume you have backed up appli- nerves are on edge. It makes sense to
want to load many Graph modules cation data such as Exchange mail, include at least some of these docu-
or even the entire suite, you always SharePoint documents, Azure SQL ments in hardcopy in your emergency
need to increase the number of func- databases, and so on by some other folder.
tions available within the session, for means and can put them back into
example, the respective application as soon as Regaining Control
the identity is working again.
$MaximumFunctionCount = 32768 Most disaster recovery scenarios here Sometimes the attacked organizations
fall into one of the following catego- are lucky and their cloud identity
from the default value of 4096 to ries. In the first scenario, the cloud survives more or less unscathed.
something far higher, as shown. identities may have been compro- If you are facing this kind of situa-
mised but apparently not destroyed, tion and perhaps even have an up-
Recovery Methods and you still have access to a Global to-date clean backup of your local
Administrator (GA) account that is AD, you may have gotten off lightly.
The scope of this article does not active and still assigned the GA role. Remove the old synchronization ac-
allow for comprehensive instruc- In the second case, some cloud identi- count from the Entra ID directory
tions on how to regain control of ties have been destroyed or at least because its password may have been

66 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Restoring Hybrid Identities M A N AG E M E N T

compromised. When you rebuild the $uri = "https://graph.microsoft.com/beta/U the Sync has never run. If PHS was
Azure AD Connect synchronization, organization/$OrgId" enabled, all user accounts retain their
you will at least regain control over $body = @' last passwords, even after synchroni-
enabling your identities and their { zation has been deactivated, so that
passwords. If you were originally us- "onPremisesSyncEnabled": null the users – but unfortunately also the
ing PTA, this will also work as soon } attacker – can still log in.
as you set it up. Restoring ADFS, if '@ One thing to consider is that the vari-
this was your previous authentication Invoke-MgGraphRequest -uri $uri U ous portals and endpoints will take
method, is much more complex. -Body $body -Method PATCH varying amounts of time to change
On the other hand, in a situation in the displayed status of users and
which you urgently need access to or use the Microsoft Graph beta groups from Synchronized to Cloud
the cloud services but restoring the module, which still needs the Graph Managed (Figure 2). If your disaster
on-premises infrastructure takes time, module to authenticate and open the recovery plan allows it, you might
you need at least to enable your IT connection: want to delay further actions until
team to control cloud identities fully. the status is displayed uniformly
To do this, though, you first need $params = @{ everywhere.
to break the link to the on-premises onPremisesSyncEnabled = $null If you are working with ADFS, you
services. } need to change the authentication
The Graph API and PowerShell let Update-MgBetaOrganization U type of all federated domains from
you switch off synchronization. -OrganizationId $OrgID U Federated to Managed (Figure 3),
Please note that at the time of writ- -BodyParameter $params
ing, this is only possible through the Connect-MGGraph U
Graph beta endpoint. Connect to the In both cases, according to Microsoft, -Scopes Domain.ReadWrite.All, U
Graph API by typing: it can take up to 72 hours for the syn- Directory.Access-AsUser.All
chronization status to be completely Get-MGDomain
Import-Module U reset to Cloud Only. In practice, this
Microsoft.Graph.Authentication, U change usually takes place within 12 and proceed as follows,
Microsoft.Graph.Beta.Identity.U hours but is by no means even close
DirectoryManagement to real time. That said, you can see Update-MGDomain -DomainId <Domain> U
$OrgID = (Get-MgOrganization).Id on the portal that the command has -AuthenticationType Managed
taken effect after just a few minutes;
From here, either submit a REST the status of Azure AD Connect or for each domain that has an Authenti-
PATCH request to the beta endpoint, Cloud Sync very quickly reports that cationType of Federated.

Figure 2: Breaking down the hybrid identity takes different amounts of time in different parts of the Microsoft cloud.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 67
M A N AG E M E N T Restoring Hybrid Identities

Reconnecting Entra ID with [System.Convert]::FromBase64String($imid) is irretrievably lost but the Entra ID

[System.Text.Encoding]::U client has survived undamaged, the
the Local AD UTF8.GetString($chars) cloud identity is very useful for re-
From now on, you can and must if ($chars.Count -eq 16) { building AD.
manage your Entra ID client inde- [guid]::new($chars).Guid Finally, an important note: As soon
pendently of the local infrastructure. } as you set up the synchronization
Once the AD has been restored, you again and Azure AD Connect, or once
need to reconnect the two directories. If the output string can be clearly Cloud Sync has completed the match-
The procedure depends on whether read and evaluated, you can try to ing process, the local AD is once
you have restored the AD from a guess the attribute for which it is again authoritative for the passwords,
backup of the attacked AD or rebuilt a valid value – often user principle activation statuses, and group mem-
it with the known account data. Be- names (UPNs) or email addresses, berships of your users. Let your users
fore you continue, make sure that you but they can also be other unique at- know this early enough so that they
disable the Block Hard Match Take- tributes. If the string is unreadable, can change their local passwords be-
over security feature: you will probably find a GUID in the fore synchronization is activated.
second line, the objectGUID of the cor-
$uri = "https://graph.microsoft. com/beta/U responding AD object. Sander Berk- Very Old Backups
directory/onPremisesSynchronization" U ouwer describes matching in detail in
(Invoke-MgGraphRequest -Method GET U his blog [3], and in another blog post Of course, it becomes more difficult
-Uri $uri).Value.Features [4] he also discusses how matching if weeks – rather than days – have
can be established in a new forest – passed between discontinuing syn-
Also watch out for BlockCloudObject- in case you have completely lost your chronization and reconstructing the
TakeoverThroughHardMatchEnabled. If AD and must rebuild it. hybrid identity, specifically because
the value is set to true, you must dis- Entra ID saves some important at- the two environments have developed
able it with a corresponding PATCH tributes of the synchronized AD user, independently of each other in this
request against the same endpoint to which subsequently makes it possible time. The challenges are similarly
allow matching. to assign the ID, and the on-premises great if the backup instance that you
To ensure that everything works directory can also be rebuilt true to need to fall back to in the local AD is
optimally, you have to know which the original. The properties all have from a relatively long time in the past
attribute the matching was originally the OnPremises prefix and include – in the worst case, before the origi-
configured to use – lucky you if you the security identifier (SID), the dis- nal activation of synchronization.
documented your Azure AD Connect tinguished name (from which the If you are confronted with this kind
configuration. If this is not the case, organizational unit, OU) structure can of scenario, just try to keep calm and
you can read out the OnPremisesIm- also be derived), the sAMAccountName, take as much time as you need for the
mutableID attribute of your cloud user the UPN, and even the complex attri- initial comparison of the information
and try to convert it into a GUID or a bute OnPremisesExtensionAttributes, in the two directories. You need to
string in a script: which contains ExtensionAttribute<x> make sure that both the user data and
ranging from x=1 to 15. On the other the matching attributes, as well as the
$imid = (U hand, you will see fewer local at- group memberships and the license
Get-MgUser -UserId -Property U tributes for groups; the distinguished assignments that often go with them,
OnPremises-ImmutableID).U name and ExtensionAttribute<x> exactly match those in the cloud be-
OnPremisesImmutableID are missing here. That said, in a fore you re-enable synchronization.
$chars = U situation where the local directory You will benefit from the fact that

Figure 3: All domains have been switched to managed authentication.

68 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Restoring Hybrid Identities M A N AG E M E N T

Entra ID saves the many attributes imperative that you familiarize your- explained-user-hard-matching-and-soft-
described above in local objects. self with the Graph API and its Pow- matching-in-azure-ad-connect/]
Do avoid taking shortcuts or mak- erShell implementation. It is best to [4] “Attach a previously sync’ed Azure AD Ten-
ing any assumptions at this point. run through the scenarios described ant to a new AD Forest” by Sander Berkou-
Use PowerShell’s various capabilities in this article in a test AD linked to a wer, September 17, 2020:
to synchronize as precisely as pos- test tenant. Q [https://dirteam.com/sander/2020/09/17/
sible, create the missing objects and howto-attach-a-previously-synced-azure-
references, and update the changes Info ad-tenant-to-a-new-ad-forest/]
to object metadata that have oc- [1] Privileged access strategy:
curred since the attack on the other [https://learn.microsoft.com/en-us/
directory. security/privileged-access-workstations/ Author
privileged-access-strategy#strategic- Evgenij Smirnov has been working with com-
Conclusions assumption---cloud-is-a-source-of- puters since the age of 5 and delivering IT
security] solutions for almost 30 years. His Active Direc-
The risk of a cyberattack is part and [2] Microsoft Graph PowerShell: tory and Exchange background naturally led to
parcel of most networked IT environ- [https://learn.microsoft.com/en-us/ PowerShell, of which he's been an avid user and
ments these days. The effect of an powershell/microsoftgraph/?view=graph- proponent since its first release. Evgenij is an
attack on a hybrid identity landscape powershell-1.0] active community lead at home in Berlin, a lead-
depends, among other things, on how [3] “User Hard Matching and Soft Matching in ing contributor to German online communities,
well you are prepared for the various Azure AD Connect” by Sander Berkouwer, and an experienced user group and conference
scenarios. If Entra ID is an impor- March 27, 2020: speaker. He is a Microsoft Cloud and Datacenter
tant part of your hybrid identity, it is [https://dirteam.com/sander/2020/03/27/ Management MVP since 2020.
Q
M A N AG E M E N T Ralph Asset Management

Data center management with Ralph

Taking Control
The Ralph open source asset management system and configuration database keep things simple when it comes
to managing data centers, but without compromising flexibility. By Holger Reibold
Often the excellent products in the giving you a complete picture of the web server and stores its settings in
open source community for operating cash flows for these assets. Data cen- the /etc/ralph directory. The deb-
and maintaining IT infrastructures are ter and back-office functions optimize conf prompts help you configure the
specialist tools that can do one thing your environment. Moreover, Ralph database settings for a fresh install,
well, but nothing else. Therefore, an comes with a data center visualiza- and the ralphctl console command is
admin’s toolbox usually comprises tion tool that lets you map arbitrary available for Ralph management.
a large number of tools with partly objects. The interplay of these vari- To install Ralph, run the following
overlapping functionality. Bucking ous functions holds a promise of ef- commands on your Ubuntu 18.04 Bi-
this trend is Ralph [1], a well-de- ficient data center operation and a onic system:
signed and powerful asset manage- proactive approach to countering any
ment, data center infrastructure man- disruptions caused by infrastructure curl -sL https://packagecloud.io/U
agement (DCIM), and configuration problems. allegro/ralph/gpgkey | sudo apt-key add -
management database (CMDB) sys- sudo sh U
tem for data centers and back offices. -c "echo 'deb https://packagecloud.io/U
Quick Start
Whereas the IT asset management allegro/ralph/ubuntu/ U
department is responsible for the ef- At first glance, it seems that Ralph bionic main' > U
ficient management of IT devices can be used to solve various problems /etc/apt/sources.list.d/ralph.list"
throughout their entire life cycles, in data center administration and sudo apt-get update
the CMDB component, as the central optimization with a single tool, but sudo apt-get install mysql-server U
database, takes care of IT service man- it’s not quite that simple, because the nginx ralph-core
agement (ITSM). These two functional Ralph developers have imposed some
areas are supplemented by the DCIM restrictions on their environment for During the installation process, you
component, which provides functions some unknown reason. For example, need to specify the database settings.
for managing, monitoring, and plan- they only provide DEB packages for For an initial trial, it is perfectly
ning the data center infrastructure and Ubuntu 18.04 Bionic on the AMD64 okay to keep the default settings;
the IT systems installed there. DCIM platform. The Docker variant is still in you can adjust these later in the da-
in particular makes a significant con- the experimental stage. It is still un- tabase configuration file /etc/ralph/
tribution to the optimization and effi- clear when a version that is suitable conf.d/database.conf. The next step
Lead Image © ffzhang, 123RF.com

cient operation of data centers because for production can be expected. The is to extend the NGINX configura-
it also includes functions for planning experimental docker-compose configu- tion file (the developers provide the
the power supply, air conditioning, ca- ration can be found online [2]. required extension code on their web-
bling, and space utilization. Before installing, make sure Python site [3]) by editing the /etc/nginx/
Thanks to Ralph, you can track your 3.6 is installed on your Ubuntu sys- sites-available/default file and then
IT investments and their life cycles, tem. Ralph also requires an NGINX restarting the web server.

70 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Ralph Asset Management M A N AG E M E N T

Because Ralph interacts with a /var/log/ralph/ralph.log computers, and cell phones. IT asset
MySQL database, you need to gen- /var/log/ralph/gunicorn.error.log management dashboards visualize the
erate a matching database schema /var/log/ralph/gunicorn.access.log acquired data in real time.
and a Ralph superuser. Feeding the /var/log/nginx/ralph-error.log To create user- and task-specific
database with some demo data is /var/log/nginx/ralph-access.log views, use the functions of the Dash-
also a good idea, so you can get to boards menu. For a new dashboard,
know the environment first, To lower the bar in your evaluation pro- follow the Add link and assign it a
cess, the Ralph developers provide an Name and Description. For a new
sudo ralphctl migrate online demo [4]. Log in with ralph as dashboard to visualize your data, you
sudo ralphctl createsuperuser the username and password. Access to need to add and configure the data.
sudo ralphctl demodata the demo was restricted earlier in 2023. To do so, select Dashboards | Graphs |
Add, assign a name to the chart, and
but make sure the conditions are as Initial Overview select the model (e.g., data center).
close to your actual use case as pos- You can also specify the Aggregation
sible. Finally, you launch Ralph: After logging in, Ralph comes up with type and Chart type. The Params in-
a clear-cut dashboard that gives you put field is used to specify (in JSON
sudo ralphctl sitetree_resync_apps an initial overview of the number of format) the fields that will be pro-
sudo systemctl enable ralph.service monitored data center and back-office cessed in the chart. In principle, you
sudo systemctl start ralph.service assets, licenses, domains, and users. can also use the REST API to fetch
The software uses a modular ap- data. To use all of the charts, run:
Now the asset specialist is ready proach. The various functional areas
for use, and you can access the for the different asset types, domains, curl https://<IP address of Ralph system>/U
web interface of the local instal- users, and so on can be accessed api/gr | python -m json.tool
lation at http://localhost. Ralph from the menubar. Ralph provides full
generates a number of logfiles for data center and back-office support, You can view the details of the charts
troubleshooting: including printers, laptops, desktop as follows:

Figure 1: Use the Data Center menu to handle all data center-specific tasks, such as creating assets.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 71
 M A N AG E M E N T Ralph Asset Management

curl https://<IP address of Ralph system>/U pieces of information: the model; the updates. You can also create your
api/graph/1/ | python -m json.tool serial number, or alternatively a bar- own usage profiles (e.g., Load balanc-
code; and the service environment ing). Use the magnifying glass icon
The Data Center menu is used to (Figure 2). Before that, however, to search for existing entries or create
manage data center-specific aspects it makes sense to create the model your own.
(Figure 1). In particular, this is where variants to simplify the task of or- When you generate a new service,
you create the assets and the asset ganizing devices of the same type. you can add a comment or service
categories. The settings let you add You can specify a model type in the description in the Remarks input box.
various resources, including ven- dialog where you create assets by Then, in the Service env field, select
dors, services, environments, budget clicking on the green plus sign in the a supported type; you will also need
information, service environments, Model field. to specify the environment. Option-
configuration modules, configuration Additionally, assign a hostname, ally, you can use tags to specify a
classes, regions, access zones, report status, and barcode or serial number service more precisely. Ralph lists the
templates, custom fields, warehouse, (SN) to the new asset. When you cre- environments available for defined
and office infrastructures. Users can ate an asset, you can also specify its services in the Environment line.
access their profile settings, update location info and usage info. Once Pressing Save saves the new asset
their passwords, and edit their assets. you have created racks, you can also configuration.
specify the device cabinet, position,
Managing Assets and orientation. Entering the IP ad- Visualizing Components
dress for management access is im-
To manage servers and create new portant for server management. Ralph is good at visualizing the
ones, choose Data Center | All hosts. A key characteristic of an asset is its components of a data center. The
Ralph gives you a powerful search use, which you store in the Usage info functions can be found under Data
function in the menubar; you can use section of the asset configuration. The Center | DC Visualization. For ex-
it to search the asset inventory. Service env field is where you specify ample, you can use it to create a new
To add an asset, you usually use the further details for an environment. rack and populate it with servers.
Data Center | Hardware management The available options are produc- When you call up the visualization
function. Follow the Add data center tion, testing, and development. This function, Ralph shows you a new
asset link when you get there. To add information is particularly important rack. To customize the visualization,
a new server, you only need three for maintenance and for installing press Edit. You can drag the rack to

Figure 2: Adding a new hardware component.

72 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Ralph Asset Management M A N AG E M E N T

a new location, rotate it, and label it However, some limits apply. Various Pressing Save lets you save the Open-
by clicking on the pencil icon. Future vendors want you to set up adapters Stack configuration; you can now
plans are to expand the data center to establish the event stream with the track your cloud assets in the Data
layout to include additional columns endpoint in Ralph. According to the Center view.
and rows. developers, this is still the best way to
This type of visualization simplifies achieve near real-time synchroniza- Conclusions
rack management by adding multiple tion of cloud assets.
new racks to a representation in a With OpenStack as an example to At first glance, Ralph is an extremely
single action. To do this, you need to demonstrate the procedure, you need promising environment for managing
switch back to edit mode; the cursor to go to Cloud | OpenStack and select and optimizing data centers. In prac-
then turns into a green plus sign, and the OpenStack provider (Figure 3). In tice, however, the environment does
you can create any new racks you the configuration, enable cloud syn- not fully meet its potential or the
need. chronization; Ralph then generates an claims made by its developers, prob-
In rack management, the DC inspec- endpoint for fielding messages from ably because the developers who
tor is another tool for checking the the cloud. help project lead Marcin Kliks only
rack configuration. To access this Next, specify the Cloud sync driver. work on the project part time. Re-
tool, edit the cabinet element in the Thus far only OpenStack is sup- gardless of its various shortcomings,
rack list, navigate to the desired ported, and the driver is named Ralph still adds value to data center
system, and press Edit asset to open openstack.ocata. You can also use management. Q
its settings. This takes you to the the noop driver, which receives event
Asset dialog, showing basic, loca- messages but does not carry out any
tion, and usage information. You can actions. Info
now customize the type and number The Client configuration box takes [1] Ralph: [https://ralph.allegro.tech]
of asset settings in the scope of the input in the form of a JSON object, [2] docker-compose configuration: [https://
permissions by opening Settings | including the configuration for the cli- github.com/allegro/ralph/tree/ng/contrib/]
Permissions. ent library used by the selected syn- [3] Extension code:
chronization driver, such as: [https://ralph-ng.readthedocs.io/en/
Cloud Integration stable/installation/installation/]
{ [4] Ralph online demo: [https://ralph-demo.
Despite the many risks, cloud services "version": "3.0", allegro.tech/login/?next=/]
are a popular tool that lets many "auth_url": "http://10.0.0.1:35357/v3/",
companies outsource specific func- "password": "admin", The Author
tional areas. Cloud services need to "tenant_name": "admin", Holger Reibold, computer scientist, has worked
be viewed as part of your company’s "username": "admin" as an IT journalist since 1995. His main interests
infrastructure and integrated into } are open source tools and security topics.
the management
processes. Thank-
fully, Ralph also
lets you integrate
cloud services.
With Ralph, you
can synchronize
cloud assets with
providers such
as OpenStack or
AWS. It relies on
a simple hack:
Ralph provides an
HTTP endpoint
that retrieves
messages from
various cloud
platforms and
makes appropri-
ate changes to
cloud hosts and
other objects. Figure 3: As it stands, Ralph only supports interoperation and asset synchronization with OpenStack installations.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 73
 N U TS A N D B O LTS GitOps

Synchronizing repository changes with GitOps

Special Ops
GitOps applies DevOps practices through infrastructure automation of version control repositories. We
look at why it is so popular and why it is often used in the context of Kubernetes. By Artur Skura
In the DevOps world, everybody control system (VCS). Because Git is thing.” I will address it later in the
is talking about GitOps. Its increase currently very popular, it only makes article, but for now, suffice it to say
in popularity in recent years makes sense to use it instead of, say, SVN or that although Kubernetes is an excel-
it clear that it’s not just a passing Subversion; however, nobody forbids lent platform for GitOps, you can
fad, but a consistent trend offering you from creating a GitOps solution have a perfect GitOps system without
significant advantages over legacy with these legacy VCS options. Kubernetes, and in many scenarios,
approaches. Finally, just because Git allows you this option makes the most sense –
to revert operations doesn’t mean especially if your organization doesn’t
What GitOps Is Not you can as easily bring your infra- use Kubernetes at all.
structure to a previous state. One of
Before I move on to specific examples the complications is the persistent OpenGitOps Definition
of GitOps principles in practice, it is data stored in your infrastructure.
important to understand what GitOps Take, for example, an Amazon For some reason the industry pre-
is and, perhaps more importantly, Simple Storage Service (S3) bucket: fers a set of defined rules rather
what it is not. It’s probably easier to If you accidentally delete it, you can than more general statements. For
start with what it isn't, because many recreate it later when you discover example, most modern companies
people see GitOps as a panacea for all your mistake (provided someone else will tell you they are using the Agile
problems in managing applications hasn’t used that particular bucket project management methodology
and their infrastructure. Not only is name), but obviously the objects for their IT projects, but when you
this belief false – in fact, implement- that had been there will all have dis- look closer, you will realize most
ing a GitOps strategy usually involves appeared. Therefore, for persistent likely it is Scrum with all its deliber-
introducing additional steps and soft- data, and databases in particular, ate limitations. The case with GitOps
Lead Image © bambamstiger, Fotolia.com

ware into your usual workflow, which you need a specific approach that I is similar: Once you have a precise
may make it more complex – but if briefly explain later. In short, GitOps definition, you can easily check your
you don’t have experienced staff, doesn’t make backups and all other implementation to verify whether it
things can break more often. good practices redundant, and any meets the criteria or not. However,
You might also think that because negligence on these points can have the standard disclaimer applies: Al-
GitOps has “Git” in name, it has to grave consequences. though GitOps works great for many
be somehow related to Git. In fact, Last but not least, you might hear the organizations, it doesn’t mean it will
GitOps can be realized in any version opinion that “GitOps is a Kubernetes work for you, and in some cases

74 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 GitOps N U TS A N D B O LTS

some modifications might be per- Terraform, you are well aware of This approach offers several benefits:
fectly fine provided you understand the difference between imperative (1) It enhances transparency, because
the consequences. and declarative approaches. With an the entire configuration of the system
The OpenGitOps project, governed by imperative approach, the focus is on is codified and version controlled;
the Cloud Native Computing Founda- the commands or steps to achieve (2) it simplifies change management,
tion (CNCF), distills GitOps principles a particular state, for example, in a with changes to the system made by
into four fundamental concepts [1]: Bash script. With a declarative, you updating the configuration files in
1. Declarative Configuration: The describe the desired state of a sys- the Git repository, which triggers the
system’s desired state must be ex- tem in a high-level, human-readable automated deployment and manage-
pressed declaratively, which means format, such as CloudFormation. ment processes; and (3) it improves
describing “what” the system The consequence is that the “how” reliability and consistency, in that the
should look like, rather than “how” is very much hidden from the user automated processes ensure that the
to achieve that state. It’s a shift and only “what” is directly visible. system is always in the desired state,
from imperative scripts. The desired state, in a GitOps con- reducing (but not eliminating!) the
2. Versioned and Immutable Desired text, is a comprehensive blueprint likelihood of configuration drift and
State: The desired state is stored in of the system’s configuration. It en- human error.
a way that supports immutability, compasses almost everything, from
versioning, and a complete history, the infrastructure setup and net- Immutability and Versioning
ensuring traceability and resilience work configurations to application
and allowing for quick recovery settings, excluding persistent data. The second principle of OpenGitOps
and analysis of changes over time. This state is captured in files stored refers to immutability and version-
Note that the desired state is a and version-controlled in a Git re- ing in managing the desired state of
comprehensive description of the pository. The significance of this systems. This principle is crucial in
intended configuration of a system approach lies in its simplicity and ensuring that the infrastructure and
that includes everything neces- clarity; it provides a clear, versioned application deployment processes
sary to recreate the system or its history of the system’s evolution are both reliable and auditable. Im-
instances, excluding persistent and state over time. mutability, in this context, implies
application data. This data is ver- Consider a simple application deploy- that once a desired state is declared,
sion controlled, providing a tem- ment scenario: deploying a web ap- it cannot be altered retroactively. In-
plate for the system’s setup and plication onto a Kubernetes cluster. In stead, changes are made through new
configuration. a declarative model, the configuration versions, preserving the history of
3. Automated State Convergence: files (YAML or JSON) describe the modifications. This approach, when
Software agents continuously and desired state of the application: the combined with versioning, creates a
automatically compare the state of number of replicas, network settings, robust framework for tracking and
the system to the desired state. Any mounted volumes, and more. These managing changes over time.
discrepancies trigger automated ac- files do not contain any commands The version control system at the
tions to align the current state with to create these resources; they simply heart of GitOps (which, as previously
the desired state. describe how the end state should mentioned, is almost always Git)
4. Continuous Reconciliation: The look. When these files are committed serves as the perfect tool for enforc-
system continuously observes the to a Git repository, they become the ing this principle. When configuration
current state and attempts to ap- single source of truth for the applica- files that represent the desired state
ply the desired state. This ongoing tion’s deployment, which is one of of a system are stored in a Git reposi-
process ensures consistency and the reasons GitOps feels very natural tory, each change creates a new com-
resilience. with Kubernetes – but there are more mit. This commit acts as a snapshot
Note that in the GitOps context reasons. of the system at a specific point in
“continuous” refers to the ongoing Once the desired state is defined and time. The immutability of these com-
process of reconciliation between stored in a Git repository, automation mits ensures that each state of the
the desired and observed states. kicks in. Software agents, such as Ku- system can be revisited, understood,
Although it is not necessarily instan- bernetes operators or custom control- and audited. The version history be-
taneous, it is a constant, automated lers, continuously monitor the state comes a comprehensive log, detailing
effort to maintain alignment. of the system and compare it with every change, who made it, and when
the declared desired state in the Git it was made.
Declarative System repository. Whenever a discrepancy is In practice, managing a system’s con-
detected, these agents act to reconcile figuration with Git involves regularly
Management the differences, thus ensuring that the committing changes to the reposi-
If you are familiar with infrastruc- state of the system always matches tory. For instance, consider managing
ture-as-code (IaC) tools such as the declared desired state. the configuration of a web server.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 75
 N U TS A N D B O LTS GitOps

Initially, a configuration file is cre- can enter an unstable state (e.g., between development and operations
ated detailing the server’s settings because the dependencies that are in a GitOps-driven workflow.
and stored in a Git repository. When expected are now missing). Consider a Kubernetes environment
a change is needed, such as updating Edge cases and peculiar problems in which the deployment of applica-
the server software or changing secu- aside, GitOps enhances the stabil- tions is managed through GitOps. In
rity settings, the configuration file is ity and security of systems and im- this scenario, a software agent such
edited, and the change is committed proves collaboration: When you are as Argo CD or Flux is configured to
to the repository. This action doesn’t confident your work will not break monitor a specific Git repository con-
overwrite the previous state but in- the system, you are more likely to taining Kubernetes manifests. These
stead adds a new layer, preserving the contribute. manifests define the desired state of
old configuration for future reference the applications and infrastructure
or rollback. Automated Pulling within the Kubernetes cluster. The
This methodology offers several ad- agent is set up to check the repository
vantages. First, it provides a clear au- You have not yet reached the end, periodically for changes. Upon detect-
dit trail. Every change to the system’s though. The first two GitOps princi- ing an update, such as a new applica-
configuration is tracked, making it ples presented above are common to tion version or configuration change,
easier to understand the evolution many deployments, especially when the agent pulls these changes and
of the system and diagnose issues IaC is involved: If you use Terraform, uses Kubernetes APIs to apply them
when they arise. Second, it supports CloudFormation, or Pulumi, you are to the cluster. This process ensures
rollback capabilities. If a new con- most probably also using Git, but this that the state of the cluster always
figuration leads to issues, it’s often by itself is not GitOps, yet – unless matches the desired state declared in
straightforward to revert to a previous you use automated pulling. the Git repository.
version, reducing downtime and miti- In the GitOps framework, software The automated pulling of state decla-
gating risks – unless persistent data is agents are tasked with continuously rations by software agents addresses
involved, which requires a different monitoring the state of the system several challenges in modern soft-
approach. Third, versioning improves and pulling the latest configurations ware development and operations.
collaboration and transparency. from the Git repository. These agents It minimizes the risk of human error
Teams can work together on the same are designed to act autonomously, in deployment processes, enhances
configuration files, review changes, reducing the need for human inter- the speed and consistency of deploy-
and merge updates, ensuring that vention in the deployment and man- ments, and ensures a high degree of
everyone is aligned with the current agement processes. The agents con- alignment between the codebase and
state of the system. stantly check for updates or changes the operational environment. By auto-
Note that GitOps principles say noth- in the Git repository. When a change mating the synchronization between
ing of the implementation details, is detected, they automatically pull the desired state in the repository and
and it is up to you to decide what these changes and initiate the process the current state of the system, this
scenario fits your organization best. of applying them to the system. This principle of GitOps not only simpli-
A popular setup involves develop- automation ensures that the system is fies management but also ensures a
ment, testing and quality assurance always up to date with the latest con- more secure and stable operational
(QA), and production branches with figurations as defined by the develop- workflow.
relevant protection levels to prevent ment or operations team. By now you should have a good feel-
overwriting changes. Usually changes A common application of this prin- ing about why Kubernetes is so often
are introduced through merge or pull ciple is seen in the integration of used in GitOps setups: Because de-
requests and need to be approved these software agents within a con- ployments are “first-class citizens” in
by other members of the team. The tinuous integration and continuous Kubernetes, and features such as roll-
whole process is integrated with deployment (CI/CD) pipeline. In such backs are built-in rather than patched
change management adopted in a a setup, the CI/CD pipeline is respon- as an afterthought, automatic pulling
given organization. sible for the integration and testing feels natural and involves little risk
Does it sound easy? In theory, every- of code changes. Once the changes (as opposed to, say, database struc-
thing should work fine. In practice, it are merged into the main branch of ture changes).
often does, but when it doesn’t, you the repository, the software agents
have to overcome the temptation of take over. They detect the new com- Continuous Observation
proceeding with a “quick fix” in the mit, pull the updated configuration,
console, especially in the event of and proceed to deploy it onto the
and Application
an incident. Instead, learn always to respective environment, be it testing, Continuous observation in GitOps
use the defined workflow; otherwise, staging, or production. This seam- is implemented through software
you can create drift, and in especially less handover from CI/CD to soft- agents that perform two main func-
complex scenarios, the infrastructure ware agents exemplifies the synergy tions. First, they monitor the state

76 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 GitOps N U TS A N D B O LTS

of the system in real-time, which and operational status. In a Kuber- what enables GitOps to deliver on its
involves checking the configura- netes environment, for example, promise of fast, reliable, and consis-
tion and operational status of the tools such as Prometheus can be tent deployments. It ensures that any
system’s components, from infra- used to gather metrics and monitor changes made in the Git repository
structure to applications. Second, the state of the cluster, and custom are quickly and accurately reflected in
they compare this observed state scripts can be written to check the the live system, keeping the system in
to the desired state as defined in health and status of specific ap- a constant state of alignment with the
the Git repository. If discrepancies plications or services. These tools declared desired state.
are found, these agents work to and scripts feed information back to Those of you familiar with Kuber-
reconcile the differences, applying the software agents, enabling them netes may have already noticed that
changes as necessary to align the to make informed decisions about some of the observation mechanisms
current state with the desired state. whether the system is in the desired described here are either built in or
This process of continuous observa- state or if actions need to be taken to easily integrated with this popular
tion and application ensures that correct any deviations. container orchestration platform,
any drift in the system is promptly Alongside observation, the continu- which is one more reason to use it as
addressed and corrected. ous application is an integral part of a basis for a GitOps platform.
To facilitate this continuous observa- this principle. It refers to the ongoing
tion, a variety of monitoring tools process of applying the desired state A Simple Example
and scripts are employed. These to the system. In practice, this means
tools can range from simple scripts that whenever the Git repository is Too many words can sometimes
that check system health to more updated with a new or modified con- muddle the waters and complicate
sophisticated monitoring solutions figuration, the software agents auto- quite simple ideas. GitOps is not
that provide real-time insights into matically apply these changes to the rocket science and can be imple-
system performance, resource usage, system. This continuous application is mented in various scenarios. To
 N U TS A N D B O LTS GitOps

illustrate this point, I’ll build the repo. If any discrepancies are de- Conclusion
most basic GitOps platform with a tected, it uses the brutal but efficient
short Bash script. method of resetting, effectively rewrit- When you start learning GitOps, the
All disclaimers apply: This example ing Git history in the local repository number of names and tools seems
is only educational and lacks most to match remote changes. Finally, the overwhelming, with each one seem-
of the features you will find in real deployment happens – in this case ingly better than the other. Neverthe-
GitOps platforms; I have implemented by Docker Compose. Again, I would less, their main job is to synchronize
no error checking; pulling is based never do it in this way in a real-word the system with the changes intro-
on time rather than a webhook or an- scenario (unless a break in service duced in the repository as described
other mechanism that would fire the availability is acceptable, which is above. Whether you choose Flux,
synchronization off on each commit; quite rare nowadays). Argo CD, or another similar tool,
and so on and so forth. Nevertheless, the basic idea is the same, even if
Listing 1 illustrates the basic prin- What About the Data? implementation details may be quite
ciples relatively well. different.
The logic of the script is quite As previously mentioned, GitOps Do you have to use Kubernetes when
straightforward: After defining the principles, at least as defined by adopting the GitOps approach to
configuration and cloning the reposi- OpenGitOps, don’t directly apply your applications or infrastructure?
tory (which is only necessary on the to data. This statement basically Definitely not, but if you do so, you
first run), the script pulls the latest means that, especially for unstruc- automatically enjoy several benefits.
changes from the remote repository tured data, you need to take special One of the important advantages that
and compares them against the local care not to lose it. Snapshots, back- is not strictly related to the GitOps
ups, archival copies – you need to principles described earlier is secrets
Listing 1: Basic GitOps Platform make use of the optimal combina- management: If you use a Kuber-
#!/bin/bash tion of proven techniques to keep netes operator to synchronize your
your data safe, in line with the poli- local Kubernetes cluster, you basi-
# Initial configuration cies of your organization. cally eliminate the need for having a
REPO_URL="https://your-git-repository.git" What about structured data, separate deployment infrastructure
LOCAL_DIR="/path/to/local/repo" though? Wouldn’t it make sense to with workers, runners, or agents
COMPOSE_FILE_PATH="$LOCAL_DIR/docker-compose.yml" apply the GitOps paradigm to it? employing authentication secrets to
CHECK_INTERVAL=60 # arbitrary value in seconds Yes, of course! Over the course of make deployments, because every-
years, many projects (e.g., Liqui- thing you need (or nearly, depending
# Clone the repository initially if not present
base, Flyway, Atlas) have aimed on your setup) is already available in
if [ ! -d "$LOCAL_DIR" ]; then
at exactly this goal: not treating your cluster. Therefore, you not only
git clone $REPO_URL $LOCAL_DIR
fi data in databases differently from make your infrastructure more resis-
code from the GitOps point of view. tant to human error, you also make
# Pull latest changes and redeploy if there are new changes That is, after changes in a database it inherently more secure. Unless the
update_and_deploy() { structure have been approved, for attacker gets access to your VCS,
cd $LOCAL_DIR example by a proper merge or pull that is. Q
git fetch origin request, they are introduced into the
database through a proper agent,
# Check if there are new commits with versioning and rollback capa- Info
LOCAL=$(git rev-parse @)
bilities. (Note I am talking about [1] OpenGitOps definition of GitOps:
REMOTE=$(git rev-parse @{u})
changes in the database structure, [https://opengitops.dev/]
such as adding new tables, chang- [2] Byetbase on GitHub:
if [ $LOCAL != $REMOTE ]; then
git reset --hard origin/master # Sync with repo ing the names of columns, etc., and [https://github.com/bytebase/bytebase]
not about changing the stored data
# Deploy using Docker Compose itself. You could have millions of
docker-compose -f $COMPOSE_FILE_PATH down such operations per second, and Author
docker-compose -f $COMPOSE_FILE_PATH up -d following a GitOps workflow would Artur Skura is a senior DevOps engineer cur-
fi make no sense in most scenarios.) rently working for a leading pharmaceutical
} If you are interested in this approach company based in Switzerland. Together with a
and need a more robust framework, team of experienced engineers, he builds and
# Main loop
check the Bytebase project [2], which maintains cloud infrastructure for large data
while true; do
integrates several tools in one frame- science and machine learning operations. In
update_and_deploy
work, making it easier to manage da- his free time, he composes synth folk music,
sleep $CHECK_INTERVAL
done tabases such as Postgres in a GitOps- combining the vibrant sound of the ’80s with
oriented way. folk themes.

78 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
N U TS A N D B O LTS BGP Routing Protocol

Border Gateway Protocol

From A to B
We look at the Border Gateway Protocol, how it routes packets through the Internet, its weaknesses, and some
hardening strategies. By Benjamin Pfister
The Internet comprises a mix of au- Open Shortest Path First (OSPF) pro- this task in Europe, the Middle East,
tonomous systems (ASs) – networks tocol, or Enhanced Interior Gateway and parts of Central Asia.
and systems each under the adminis- Routing Protocol (EIGRP) for internal An official ASN assignment from the
trative control of a specific provider networks, BGP is the only Exterior RIR is required for a redundant Inter-
– that have officially registered num- Gateway Protocol (EGP). As such, it net connection by more than one car-
bers known as AS numbers (ASNs). is based on the path vector principle, rier (multihoming). Additionally, an
The Border Gateway Protocol (BGP), which has similarities to the distance IP address block independent of the
the latest version of which is BGP4, vector IGPs used to exchange routes provider must be assigned. Provider-
ensures accessibility between the within an autonomous system and independent (PI) or provider-aggre-
autonomous systems and is designed optimized for fast convergence times gatable (PA) address blocks are used
and optimized for handling high vol- to meet the stringent requirements for this purpose. However, this pro-
umes of routing information with a for low downtimes – right down to cess has become difficult because of
high level of stability. the millisecond range. However, even the scarcity of public IPv4 addresses.
Besides providers, large corporate smaller numbers of routes still need Smaller customers are normally as-
and government customers also have to be processed. signed addresses by the provider. If
to deal with BGP if they use or want Extensions to BGP make it multipro- the company or authority has its own
to use multihoming (i.e., connect- tocol capable (MP-BGP4); that is, AS and public address blocks, it as-
ing your own autonomous system to it supports IPv4 and IPv6. BGP can sumes the role of the local Internet
several providers). BGP is also used process and separate different types registry (LIR).
on some internal networks and forms of information and contexts in these Legacy ASNs are 16 bits in length and
the basis for multiprotocol label “address families.” According to decimal (ASPlain). Newer ASNs use
switching (MPLS) in wide-area net- information from the American Reg- 32 bits and are dot separated. This
work (WAN) structures, but can also istry for Internet Numbers (ARIN), format is known as ASDot [1]. ASN
be used for Ethernet virtual private as of 2023, a full BGP table on the 6541 in ASPlain becomes 0.6541 in
networks (EVPNs) or in combina- Internet contained around 940,000 ASDot notation.
tion with a virtual extensible local prefixes for IPv4 and 172,400 pre-
area network (VXLAN) in data center fixes for IPv6. Routers therefore need Route Selection and
networks. to have a large amount of physical
Today, BGP is capable of many more memory.
Attributes
functions than simply distributing IP BGP uses different types of attributes
prefixes. The protocol therefore has ASNs in Practice to influence the choice of the ap-
Lead Image © Rachael Arnott, 123RF.com

a wide range of options for policy- propriate route, distinguishing be-

based route selection. As already explained, a network tween transitive and non-transitive,
operator requires an ASN for the ex- normal and path attributes. When
Basics change of routing information. Public people start to talk about BGP, the
ASNs are assigned by Regional Inter- conversation quickly turns to peer-
In contrast to the various Interior net Registries (RIRs). The Réseaux ing, which means the neighborhood
Gateway Protocols (IGPs) such as the IP Européens Network Coordination connection between BGP routers and,
Routing Information Protocol (RIP), Center (RIPE NCC) is responsible for consequently, autonomous systems.

80 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 BGP Routing Protocol N U TS A N D B O LTS

phase, in which the peers wait for

keep-alive messages. If successful,
the Established phase is next, which
is where peering is set up and the
routers can exchange availability
information.
This process uses the interface on
the routers that leads to the target
IP address according to the routing
table. To be able to cover scenarios
with multiple paths without the risk
Figure 1: In BGP dual multihomed design, several providers are each connected to the of flapping (disconnection and recon-
customer’s AS by several connections. nection) of the BGP peering in the
event of a path error, a permanently
However, BGP routers do not simply The handling of the next-hop IP ad- accessible IP address should be used
use multicast to find their neighbors dress is also different. With eBGP, the as the source in such a case (Figure
when enabled, as is usually the case router always transmits the IP address 3). To do this, permanently active
with the IGP in an AS. of the outbound interface as the next loopback interfaces are used as up-
With BGP, the administrator on the hop in the accessibility information, date sources. These interfaces are
router must explicitly store the neigh- and the receiving router replaces this then also stored on the remote station
bors, including their IP addresses and with the outbound IP address of the as peer addresses and included in the
the remote autonomous systems in interface it used during redistribution. remote station’s routing to be able to
the respective routing process, and With iBGP, the IP address is adopted establish peering (Figure 4).
then reverse it on the peer router without change on forwarding. How-
(Figure 1). If router A in AS 64496 ever, this address must also be acces- Avoiding Loops
with an IP address of 192.0.2.1 wants sible for the target peer. If it is not,
to peer with router B in AS 64500 and you can configure the next-hop self Freedom from loops is one of the key
IP address 192.0.2.2, router A must parameter to achieve the same behav- requirements of any routing proto-
store the neighbor 192.0.2.2 with AS ior as with eBGP. col. In BGP, this is based on the AS
64500 and router B the peer 192.0.2.1 BGP uses connection-oriented TCP path – that is, a list of the transit ASs
with AS 64496. If the configurations on port 179 as the transport proto- through which a packet has already
do not match, peering will not take col, which must be allowed on the passed. Each autonomous system
place. transmission path. When establishing adds its ASN starting with an eBGP
peering, the connection setup goes peering (Figure 5). An ASN can oc-
Communication Process through different phases (Figure 2). cur several times in direct succession,
The idle phase begins with the idle but cannot be fragmented in different
A fundamental distinction exists be- status. Then a TCP three-way hand- places. If the ASN is fragmented in
tween external (eBGP) and internal shake with
BGP (iBGP) peering. iBGP peering the familiar
takes place within the same au- SYN, SYN/
tonomous system (e.g., to exchange ACK, ACK
prefixes between two BGP routers is used in
connected to different carriers). eBGP the connect
peering takes place between different phase. If this
autonomous systems, as in the previ- is successful,
ous example. the OpenSent
However, the two types of BGP differ phase is used
in terms of implementation, as can to synchro-
be seen, for example, in the multihop nize BGP
function, where the time to live (TTL) information
is influenced by the IP header. Several such as ASNs
hops to the peer can be the default and authen-
for iBGPs. With eBGP, you need to tication data,
configure this explicitly to support which is
peers located on different subnets. In normally fol-
this case, the TTL counter is incre- lowed by the
mented to reach the peer. OpenConfirm Figure 2: The BGP peering process comprises six phases.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 81
N U TS A N D B O LTS BGP Routing Protocol

Figure 3: BGP peering with loopback interfaces avoid session flapping if a connection fails.

the AS path, BGP drops this poten- does not scale well in larger environ- from OSPF to BGP). On this basis, the
tial path, identifying it as a loop. To ments. Confederations and route router transfers all paths from one
extend a path artificially, you have reflectors offer a remedy. Confedera- protocol to the other – in this case to
an option for concatenating the same tions are used to define subzones, but the BGP table. However, prefix lists
ASN with an AS path prepend. route reflectors are more common. can impose restrictions. If you do not
Because iBGP only has one ASN for When iBGP peers receive reachability want to announce routes that are too
all participating routers, this proce- information, the router forwards it to small, you can combine these routes
dure cannot be used here. To counter the eBGP peers and the route reflec- to create a summary route.
this, iBGP and eBGP behave differ- tors used for scaling, but not to the As mentioned previously, BGP uses at-
ently when forwarding availability other iBGP peers. tributes for path selection, which you
information. If an iBGP peer receives can manipulate with the use of route
information, it only forwards it to Publishing Routes maps. Table 1 lists the attributes in the
eBGP peers, but not to other iBGP order in which they are to be processed
peers. eBGP peers in turn forward Once the peering is in place, ac- and with a description in each case.
incoming reachability information to cessibility information needs to be Not all manufacturers use the Weight
both eBGP and iBGP peers. exchanged, as well as the originat- attribute, and it is only significant lo-
The iBGP’s behavior leads to some ing and redistributing options. With cally on a router for making outgoing
design restrictions in iBGP, which originating, you configure a network route selections. With regard to the
can be countered with several tools. to be announced in your AS. The BGP prefix, a higher weight for peering is
Normally, iBGP would require a full router searches the routing table for preferred to a lower one. The Local
meshing of all peers. However, this a known path to this network; it can Preference attribute is only exchanged
be known between iBGP peers. For example, a
through preference for selecting the outgoing
static routes route via a specific peer can be estab-
or dynamic lished within an AS.
routing pro- The Locally Originated attribute han-
tocols such dles local routes before routes learned
as OSPF. If from BGP. The length of the AS Path
successful, (i.e., the list of the ASs to be passed
the router through) is not checked until the next
transfers step. Routers prefer shorter paths. If
these net- a decision is not made on the basis of
works to the the AS Path Length, the Origin Code
BGP table. (i.e., the prefix source) is used. Local
During re- prefixes take precedence over prefixes
distribution, learned from EGP, followed by redis-
you specify a tributed paths.
source and a The Multi Exit Discriminator (MED)
Figure 4: The ASN entry in the RIPE NCC test database shows the target routing attribute is only used for eBGP be-
routing guidelines for import and export. process (e.g., tween two ASNs. Where several

82 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 BGP Routing Protocol N U TS A N D B O LTS

Figure 5: A simple example of the use of BGP route filters. These filters can be used for both inbound and outbound announcements.

peering instances to the same AS Challenges and Security to as peers in the BGP context [3].
exist, a router can use the MED to The practice of autonomous systems
specify a preferred path in the return The RIRs provide a database in distributing prefixes not assigned to
direction. In the next step, the router which AS operators store their their ASs is referred to as prefix hi-
prefers the path with the better metric guidelines, which define how AS jacking. The aim is to redirect data
to the BGP next-hop in IGP, which operators announce or accept routes traffic by manipulating the path
is why BGP is still based on IGP. to and from external autonomous selection.
Because BGP is designed for stabil- systems. From the outbound policy, On one hand, this arrangement can
ity, it would favor older entries over appropriate inbound route filters be used for man-in-the-middle at-
newer ones as the next path decision can be set for peering partners on tacks that sniff or manipulate data.
parameter. If a decision has not yet the partner’s side of the peering, On the other hand, denial of service
been made, the router compares the keeping BGP clean and making attacks that disrupt the availability
router ID of the next hop and gives prefix hijacking more difficult [2]. of services or entire networks are
priority to smaller numerical values. Besides, each AS operator provides also possible. Last but not least, it
Something similar is also used as the contact details in case of technical could simply be a misconfiguration.
last tie breaker. A lower numerical IP problems or misuse. Repeated cases of prefix hijacking
address is preferred over higher ones Of course, you cannot just blindly have been seen, which highlights
as a path. trust an arbitrary prefix from an the need for countermeasures.
These attributes show that BGP sup- arbitrary source. True to the motto However, incoming filters for route
ports complex guidelines for route “trust, but verify,” it is important to announcements cannot be used
selection that make the protocol any- check the incoming prefixes of the meaningfully or, if they are used,
thing but easy to handle. remote stations, which are referred are under many restrictions.

Table 1: BGP Attributes

Attribute Description
Weight Valid locally on a router to define the outgoing preference.
Local Preference Valid within the AS to define the outgoing preference.
Locally Originated Local before learned routes.
AS Path Length Number of ASNs in transit.
Origin Code IGP sources are preferred over EGP sources, which in turn prefer redistributed information.
MED Message to neighboring AS to route traffic for certain destinations by specified link.
eBGP Path over iBGP Path Paths by another AS are preferred over paths within the AS.
Shortest IGP Path to BGP Next-Hop Preferred path to the next BGP router determined from IGP information.
Oldest Path Older paths preferred.
Router ID Next hop with the smallest router ID is preferred.
Neighbor IP Address Next hop with the smallest IP address is preferred.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 83
N U TS A N D B O LTS BGP Routing Protocol

Signing with Public Keys However, the remote station also filters to restrict inbound and out-
needs to validate the ROA information bound route announcements, which
The Internet Engineering Task Force of the incoming prefixes. This check can be handled on the basis of the
(IETF) developed the Resource Public can lead to the Not Found, Unknown, aforementioned data from the routing
Key Infrastructure (RPKI) [4] as a po- Valid, or Invalid results. Valid means guideline of the responsible RIR. The
tential countermeasure. This framework that the check of the AS, the prefix, guidelines are not particularly flex-
defines which autonomous systems are and the corresponding length was ible. That said, it is always advisable
allowed to announce which prefixes. successful, whereas Invalid means to set route filters that block incoming
The attestation is based on the Interna- the opposite. Not Found or Unknown prefixes from external sources.
tional Telecommunication Union Tele- means that no ROA was found. RIPE For almost all peering instances, you
communication Standardization Sec- NCC provides sample configurations will want to avoid standard routes
tor’s (ITU-T’s) X.509 PKI framework. for manufacturers such as Cisco Sys- because they only make sense for end
RPKI uses the same hierarchy as for IP tems and Juniper. However, not all customers without multihoming. Ad-
address assignment to reflect the chain router platforms support RPKI. Ad- ditionally, BGP routers should filter
of trust and consequently ensure attes- ditionally, a potential compromise of private IP address ranges bidirection-
tation. The Internet Assigned Numbers the certification authority naturally ally with inbound and outbound
Authority (IANA) is the root element of would pose a risk. route maps because other routers do
the PKI. From there it passes through not route them on the Internet any-
the RIRs to the LIRs. BGPSec Validation way. These route maps usually match
RPKI can use different responsibility with prefix lists that can be checked
models. Large companies or providers RPKI is not the only hedging option. for a subnet mask and prefix length
can use a local PKI, also known as del- Also based on RPKI, BGPSec offers a in the accessibility information. To
egated RPKI, and the option of a hosted way of creating a chain of trust. The do this, they specify the number of
RPKI. In this case, the PKI is operated source of the routing prefix, the Origi- matching bits with the “less than or
by the RIR (the RIPE NCC in Europe). nator, uses RPKI to sign the informa- equal to” (le) and “greater than or
Each LIR can have ASNs and IP pre- tion, and all other routers of the AS equal to” (ge) operands (Table 2).
fixes attested by a certificate. Route path follow suit and sign with their Prefix lists offer the possibility of es-
origin authorizations (ROAs) can be private keys; therefore, each autono- tablishing an inbound and outbound
created in this way. ROAs include the mous system authorizes the trans- binding to the peer in question. On
authorized ASN, the IP prefix, and ferred prefix in the BGP update. this basis, you can apply inbound
the maximum length of the prefix, However, the use of RPKI and BGPSec and outbound prefix filters, which en-
which means a potential attacker also harbors risks for availability: ables pre-filtering in the distribution
does not have the option of announc- Peers also need to check the certifi- of prefixes. Route maps that use AS
ing this prefix with a more specific or cates issued by the PKI. The Network path ACLs, prefix lists, or community
higher prefix length, because a router Time Protocol (NTP) service must be tags for this purpose are slightly more
prefers longer prefix lengths and an available to check the validity period, granular in terms of classification.
attacker could otherwise leverage this as must HTTP-based services such The matching BGP attributes can be
fact to redirect data traffic for a more as Online Certificate Status Protocol adapted on this basis. The route maps
specific prefix. (OCSP) and certificate revocation lists use sequence numbers for sequential
Certificates offer an option for cryp- (CRLs) to determine a possible revo- processing of “classification” and
tographic verification of the prefixes. cation. However, routing information “adaptation.” They offer the option
If you do not set a maximum prefix is required in turn, which leads to a of changing BGP attributes, next-hop,
length, the AS can only announce the loop dependency. With BGPSec, pre- and community tags with a set com-
entire prefix and not specific parts of fixes cannot be aggregated to reduce mand on the occurrence of a positive
it. Email alerts at RIPE-NCC point out the number of prefixes transmitted. match result.
unauthorized announcements and The PKI operator can also withdraw BGP communities [5] can offer ad-
misconfigurations. sovereignty from the AS operator by ditional features and are a valuable
revoking the addition. They use numbers as tags,
Table 2: Prefix Examples certificates which, in turn, define an expected
Attribute Description used to sign behavior; therefore, a BGP directive
0.0.0.0/0 Default route only prefixes or re- can be controlled remotely. Both
fusing to issue well-known and custom communi-
0.0.0.0/0 ge 32 All host routes
a certificate. ties exist. Use of communities must
172.16.0.0/16 ge 24 All subnets in the IP range 172.16.0.0/16 with
Admins and be coordinated between the peers
a minimum prefix length of 24 bits
security man- involved. You can store special tags
192.168.0.0/24 ge 27 le 30 All prefixes that begin with 192.0.0 and agers can set in the communities for prefixes and,
whose prefix length is between 27 and 30
static route by doing so, communicate control

84 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 BGP Routing Protocol N U TS A N D B O LTS

commands to the peer for handling value, the receiving router does not important to intercept this data traf-
the data traffic for this prefix. Besides accept the segment. fic up front. One conceivable method
the filter and control options already Despite MD5 being considered vul- would be blackholing [6] on the
presented, it also makes sense to filter nerable for years because of pos- provider side, which means a router
out private ASNs in the AS path and sible collisions, changing the key drops all packets to a specific destina-
AS paths that are too long. with this method would result in tion into a null route.
the TCP session being terminated. This strategy would allow the up-
Securing the TCP Session Consequently, a new BGP session is stream provider to be informed by
created and prompts relearning of BGP that it needs to drop packets
Risks do not just occur at the BGP the routing information, which will to specific target IP addresses – also
protocol level. A potential attacker take some time. known as remotely triggered black
could attack the control plane by The TCP authentication options (TCP- hole routing (RTBH). The customer
sending masses of packets to the BGP AOs) method was developed as an sends a /32 prefix for IPv4 or /128 for
port, triggering a denial of service. optimized procedure and standard- IPv6 with the attacked target IP ad-
The first step is to restrict the commu- ized in RFC5925. This procedure en- dress and BGP community 666 to the
nication relationships for the socket ables the key to be exchanged with- peer. However, because peers do not
setup. Stateless packet filters – access out interrupting the TCP session and, accept these host prefixes by default,
control lists (ACLs) – can be used for as a result, the BGP session; avoiding specific coordination with the pro-
this purpose, specifically to make sure interruptions is particularly beneficial vider is required. In this case, the IP
that the target port for BGP (TCP/ for long-term TCP sessions such as address also is no longer accessible,
179) is only accessible from legitimate BGP. TCP-AO is only used to check but overloading of the connection
source IP addresses. the authenticity of the sender, with then stops and other services are no
However, this alone is not enough. no encryption of the user data, unlike longer affected.
It is also important to ensure the au- the Transport Layer Security (TLS)
thenticity and integrity of the data. or Internet Protocol Security (IPSec) Conclusions
Otherwise, an attacker could carry protocols. The TCP-AOs are based on
out blind insertion and replay or master key tuples (MKTs) for this pur- BGP is still a fundamental compo-
reset attacks. Blind insertion means pose. Management can be carried out nent of the Internet in 2024, although
an attacker attempting to inject false both statically and by an out-of-band not many people are familiar with
routing information or a session reset mechanism. The connection keys the background information. Rout-
(i.e., a termination) with a spoofed (traffic keys) are then derived from ing failures from misconfigurations
IP address on a router that is not the MKTs. or attacks, for example, can have an
secured by authentication. The big enormous effect, even if only in some
challenge, however, is that the TCP Black Holes areas because of the decentralized
sequence number must match the structure of the Internet. Because of
expected segment, which requires For some years now, attacks that the ever-increasing dependence on
both knowledge of the current session restrict or completely prevent the ac- online services, you need to keep an
and correct timing. If a session reset cessibility of services have been on eye on BGP and look into options for
occurs, a completely new setup is the rise. These attacks, known as securing it. Q
required, which means learning hun- denial of service (DoS) or, in the case
dreds of thousands of items of routing of multiple sources, distributed denial
information; the end effect is a denial of service (DDoS), can be launched Info
of service as it is happening. at either the network or application [1] ASDot and ASPlain:
Authentication by cryptographic level. Network-level DoS and DDoS [https://www.networkers-online.com/
procedures can provide a remedy. attacks are aimed at overloading tools/bgp-asn-4byte-converter/]
However, outdated procedures such network connections. Volumetric at- [2] Prefix hijacking: [https://www.youtube.
as MD5, which are now considered tack is another way of putting this. com/watch?v=IzLPKuAOe50]
insecure, are still mostly used. A sym- Attacks at the application level (e.g., [3] RFC7454: BGP Operations and Security:
metric key is available on both peers. on web servers) exploit vulnerabilities [https://datatracker.ietf.org/doc/html/
Each TCP segment contains a previ- in applications or specific application rfc7454]
ously calculated message authen- behavior to take down the servers. In [4] RPKI validation:
tication code (MAC). The recipient both variants, the first step is to de- [https://www.ripe.net/manage-ips-and-
checks this before accepting it on the tect the traffic pattern and then filter, asns/resource-management/rpki]
basis of the content in TCP headers, limit, or redirect the data traffic. [5] BGP communities: [https://www.youtube.
the content data, and the configured Because volumetric attacks in particu- com/watch?v=FMzPOZQawKI]
symmetric key. If calculations return lar are associated with high data rates [6] RFC7999: BLACKHOLE Community: [https://
a different result than the received that connections cannot handle, it is datatracker.ietf.org/doc/html/rfc7999]

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 85
N U TS A N D B O LTS DDos Defense

Building a defense
against DDoS attacks

One
Against All
Targeted attacks such as distributed denial of service, with thousands of computers attacking your servers until
one of them caves in, cannot be prevented, but they can be effectively mitigated. By Markus Stubbig
Cyberattacks come in many forms, same server with many clients. The shorter timeouts, more restrictive
such as secret spying on company resulting attack is called a distributed firewall rules, and general recommen-
networks, sabotage, and disruptive denial of service (DDoS). The attacker dations for software and libraries, to
actions. In a disruptive action, denial rounds up a huge number of comput- specific kernel settings. Rather than
of service (DoS), the attacker at- ers in the form of a botnet. standardized guidelines, numerous
tempts to overload a server with re- You are not powerless against DDoS recommendations are offered for
quests until it stops working. This at- attacks, but it is important to intro- hardening the operating system [1].
tack is easier said than done, because duce appropriate measures up front, All best practices are aimed at keep-
servers usually have more power in because during such an attack, the ing the attack surface as small as pos-
reserve than a single client can call Internet line is flattened by the flood of sible. If you don’t feel like typing the
up. The obvious idea is to attack the client requests and the server farm no numerous commands for these steps
longer responds. The simplest, albeit by hand, you can easily use a precon-
CDN most expensive, measure is to upgrade figured script [2]. However, only do
Many providers mention DDoS and content your infrastructure with more servers this if you understand the individual
delivery networks (CDNs) in the same con- and more bandwidth. If your budget lines of the script and they match
text. A CDN distributes its servers across as allows for this approach, you need your security policy. The result is a
many data centers as possible around the read no further. For everyone else, this fireproofed operating system that is
world. The aim is to locate the data closer to article describes various ways of ef- not so easy to mess around with.
the customer so the content is delivered lo- ficiently protecting your infrastructure. Trust is good, control is better. Soft-
cally and not sent halfway around the world.
The aim is not to provide protection ware auditors apply this principle to
This method saves provider bandwidth and
against huge attacks that hit at terabit check local computers for insecure
makes the services more responsive for the
customer.
per second rates, but simply to make settings and known vulnerabilities.
As a positive side effect, the CDN provides your servers more robust. The box If you don’t want to fire OpenVAS at
protection against DDoS attacks because the “CDN” describes why CDNs and DDoS your servers, you can use the leaner
attacker has to deal with many instances of are often mentioned together. Lynis [3]. The tool runs more than
Photo by Arisa Chattasa on Unsplash

the same service. If the attacker manages 200 tests and after a few minutes
to keep up the attack against data center A Hardening Operating presents its report in the terminal or
with its distributed army, the CDN provider by email. Everything that is suspi-
can simply switch to another data center and Systems cious or insecure is highlighted in
serve its customers from there. The differ- The first level of protection is pro- red. At the end of the report, the tool
ence is that DDoS is the attack and CDN is a vided by stricter settings on the provides specific recommendations
possible defense.
operating system side, ranging from for a more secure configuration.

86 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 DDos Defense N U TS A N D B O LTS

Protection at OSI Layers informs the CrowdSec community as the source instead of an IP address.
and all participants block the attack- For example, a block for Somalia
3 and 4
er’s IP address. This puts a quick end implemented by iptables is:
The major providers refer to pro- to a dictionary attack on the SSH ser-
tection Layers 3 and 4 of the Open vice. Conversely, you can also quickly iptables -A INPUT -m state --state NEW U
Systems Interconnection (OSI) model lock out your IP address if your tired -m geoip --source-country SO U
for network protection, which means fingers fail to type in your password -j DROP
blocking clients by IP address and correctly late at night.
geo-blocking, which ultimately only Although the Fail2Ban and CrowdSec Depending on the infrastructure or
consists of a long list of IPs. For ex- approach effectively prevents brute hosting offer, a firewall with a geo-
ample, the FireHOL blacklist contains forcing of user-password combina- function can be installed upstream
all addresses that have occupied a tions, it has two flaws. On one hand, of the server, which enables conve-
place on a blacklist in the past [4]. the CPU load increases if the server nient configuration of the security
Although this step will exclude the receives many requests and Fail- policy in a web GUI. In Figure 2, the
known troublemakers, the list does 2Ban has to parse large logfiles. On OPNsense open source firewall blocks
not take into account the individual the other hand, Fail2Ban can block access to individual countries by
attack situation, which is where dy- several clients with one IP address if clicking on the flag symbols.
namic blocking with Fail2Ban [5] they share a public address – which
comes into play (Figure 1). The tool is likely to be the case with most IPv4 Securing Websites
assumes that every IP address is be- Internet connections.
nevolent, but if an address appears Many websites offer a members-only
in the logbook on several occasions Setting Up Geo-Blocking area that can be accessed by pass-
from failed login attempts, it creates a word. As with an SSH service, the
block entry in iptables/nftables. The If you want to appeal to an interna- attacker tries typical username and
result is no communication with this tional audience, you have to grant ac- password combinations; this method
address. After a short wait, the client cess to all countries on your website. will eventually lead to success unless
with this address can join the game The reverse is also true: If you do not a form of website protection puts a
again and start its three attempts. Of have any trade relations with Asia stop to it.
course, the values and timeouts are and Africa, a geo-block can exclude The website protector works like this:
customizable and can be increased to almost 100 countries. An attacker in It sends a work order to the client’s
many hours for aggressive would-be Somalia will not see an SSH prompt web browser and only displays the
attackers. where they can try out password requested web content once the task
The community takes this one step combinations. has been completed. The activity can
further with CrowdSec [6]. The prin- Thanks to the geoip module and the vary from a simple I am not a robot
ciple is similar to Fail2Ban; all par- free country IP list from MaxMind, checkbox to, say, Select all squares
ticipants share the list of blocked ad- the Linux firewall becomes a geo- with traffic lights. Other watchdogs
dresses. When an unknown attacker filter in next to no time. The filter package the task in JavaScript, which
attacks its first server, this server command contains the country code the browser processes without human

Figure 1: Fail2Ban remembers the IP addresses of failed login attempts and sets up a firewall block for them.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 87
N U TS A N D B O LTS DDos Defense

Figure 2: The free OPNsense firewall filters access by country of origin.

intervention (proof of work). In all HTML code. The web protector this kind of task and had to click
cases, access to the web service is works as a benign man-in-the-middle on the right photos from a selection
slowed down so that a DDoS attacker and inserts its robot checks upstream (Figure 3). This activity is intended
can make significantly fewer requests of the actual web pages. Technically, to distinguish human visitors from
per second. The trick with web this takes the form of a reverse proxy scripts.
protection is that the client, not the that receives and analyzes the web The tool can be installed on Linux
server, is exposed to high load. The requests of all clients. Is this client with just a few lines:
client receives the difficult task and submitting an unusually high num-
the server only checks whether the ber of requests? Then a web chal- wget https://raw.githubusercontent.com/U
result is correct. lenge is issued that human visitors duy13/vDDoS-Protection/master/latest.sh
This DDoS protection does not re- have to answer. chmod +x latest.sh
quire you to tinker with the existing The reverse proxy is a piece of ./latest.sh
software that accepts
HTTP requests but has All components are then available
no content itself. For the under /vddos on the local filesystem.
response, the reverse In the best open source manner, the
proxy accesses another developers of vDDoS do not code
web server and deliv- everything from scratch but use ex-
ers the HTML code as a isting software with a free license:
proxy. As an intermediary Nginx for the web server, Gunicorn
in the data stream, the as the web gateway, and Flask as
proxy can inject all sorts the Python framework. You need
of things into the HTML to describe your websites and the
lines, such as captchas. desired protection method in the
configuration.
Captchas with vDDoS In Listing 1, vDDoS looks like an
HTTP/S server (Listen column) with
In addition to commercial different levels of protection (Secu-
providers with professional rity) when viewed from the outside;
web protection, you can it addresses several servers in the
find free applications on background (Backend). The starting
GitHub, such as the vD- signal is given by the vddos start
DoS software [7], which command. vDDoS then takes off
Figure 3: With the help of an image puzzle in vDDoS, presents captchas to its cli- and offers the configured websites
the website checks whether a human or a script ents. Every web visitor has from its IP address – with built-in
wants access. probably stumbled across DDoS protection.

88 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 DDos Defense N U TS A N D B O LTS

Search Engines and Crawlers system under attack and inform the detected. IPBan forwards the request
customer router over Border Gate- for blocking to the Windows firewall.
Whichever DDoS protection you way Protocol (BGP). The customer Although the software does not make
choose, you do not want to block the router immediately forwards the in- a Windows system secure, it does
crawlers of the major search engines. formation to the provider. The BGP reduce the attack surface on open
If the snooping bot from Google, Mi- routers now have a new host route services such as Remote Desktop or
crosoft, or DuckDuckGo encounters that no longer routes the traffic to virtual network computing (VNC).
a JavaScript challenge or Fail2Ban the target server but instead dumps
strikes, it will simply stop indexing it in the bit trash can. A black hole Test with a DoS Attack
your site. Fortunately, the operators is created in the routing table for
provide all the IP addresses under the attacked server. As soon as the Of course, you will want to test all
which their bots map out the Inter- onslaught subsides, the guard re- your defenses. If you do not cur-
net. You need to add these addresses moves the host route and makes the rently have a botnet in your arsenal,
to your personal whitelist to make server visible again. The idea be- you should at least unleash a single
sure that nothing stands in the way of hind this is known as remotely trig- client on your now protected serv-
the search bots. gered black hole (RTBH) filtering. ers. You do not need to access the
If you want to familiarize yourself Darknet to pick up the necessary
Sensor with a Black Hole with this form of DDoS protection, tools; instead, try GitHub or the
I recommend the community ver- Kali Linux distribution. The legal
If you have more than a few servers sion of FastNetMon [8]. For test framework must be established be-
in your administration area or do not purposes, FastNetMon also runs fore the first attack: The customer
want to integrate any additional soft- without BGP interaction and only or employer must consent to a de-
ware, you can get DDoS protection triggers a Bash script in the event liberate attack. It also makes sense
“off the web,” which does not refer of a DDoS alarm. In this phase, not to launch the attacks from your
to another provider in the sense of you can monitor your network and network to avoid ending up on one
as-a-service, but rather a monitor that set the threshold values. Once the of the blacklists mentioned above.
detects unusually high traffic flows on false positives stop, the FastNetMon A disposable virtual machine (VM)
switches and routers. server can become a BGP neighbor from a cloud provider is recom-
Routers count packets flowing in your autonomous system and mended, and because they are paid
through and report them in NetFlow send host routes. for by the minute, the financial out-
format to a central NetFlow collec- lay is very manageable.
tor. The collector receives the sta- Tools for Windows Servers To lay a simple siege, use the Siege
tistical information from all routers HTTP load tester, which can be in-
and, by doing so, obtains a precise The tools presented so far are ex- stalled with the package manager
overall picture of the utilization of clusively for Linux and Unix. If you on many distributions. The software
Internet links. This method works want to make your Windows server expects the URL of the web server
similarly with switches: They send a accessible over the Internet, you can as an argument and immediately
small percentage of the transported expand the Windows firewall by add- starts throwing GET requests at the
packets in sFlow format to the ing IPBan [9] or EvlWatcher [10] for server. Without DDoS protection,
collector, which can then deduce DDoS protection. Both products work the results on the screen flash by
the utilized bandwidth from the in exactly the same way as Fail2Ban. at breakneck speed. If protection is
samples. They monitor the login attempts and activated, activity stops after a few
If the number of packets or the block the source address after a few HTTP access attempts. Siege then
bytes transmitted per second ex- failed attempts. simply reports Resource temporarily
ceeds a defined threshold value, IPBan is particularly easy to set unavailable.
then it looks like the start of a DDoS up: Download the ZIP archive from If you want to carry out your own
attack. Now is the time to act. The GitHub, unpack, and start the execut- attack with a little more finesse, you
simplest case is an email alert. Au- able. In the open program window might want to use MHDDoS [11], which
tomated systems use the NetFlow IPBan informs you what it is cur- comes with more than 50 attack vec-
and sFlow data to identify the target rently doing and which logins it has tors that trick various commercial

Listing 1: vDDoS as a Reverse Proxy

Website Listen Backend Cache Security SSL-Prikey SSL-CRTkey
default http://0.0.0.0:80 http://10.1.1.84:80 no 5s no no
www.example.net https://0.0.0.0:443 http://10.1.1.84:80 no 307 /vddos/ssl/example.key /vddos/ssl/example.crt
login.example.net https://0.0.0.0:443 http://10.1.1.85:80 no captcha /vddos/ssl/example.key /vddos/ssl/example.crt

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 89
N U TS A N D B O LTS DDos Defense

DDoS defense systems, going far be- (thus far). Anyone expecting attacks attacker. If these tools and tricks in-
yond a simple flood of GET and POST in the multidigit gigabit per second volve too much manual work or you
requests. Command-line arguments range is well advised to use a com- anticipate massive DDoS attacks, the
let you specify how many requests mercial provider. Fun fact: This solu- same protections are also available
per second you want to throw at the tion works so well that even the bad from commercial providers. Q
server. guys with their illegal web portals use
it to protect themselves against even
Commercial DDoS Protection more evil villains. Info
[1] Linux hardening guide:
Numerous commercial providers also Conclusions [https://madaidans-insecurities.github.io/
offer turnkey DDoS safeguards with guides/linux-hardening.html]
support, logging, and alerting. These In addition to ransomware attacks, [2] OpenSCAP security guide for RHEL 7:
solutions range from cloud-based DDoS attacks pose major challenges [https://static.open-scap.org/ssg-guides/
approaches to hardware appliances, for companies. This foray through ssg-rhel7-guide-C2S.html]
which are usually integrated into the the defense arsenal from the open [3] Lynis: [https://cisofy.com/lynis/]
data center as firewall extensions. source world reveals a couple of [4] FireHOL Cybercrime IP feeds:
For example, industry giant Cloud- highlights. The basic service begins [https://iplists.firehol.org]
flare offers DDoS web protection with with a hardened operating system [5] Fail2Ban:
its application services, which works followed by a security check, which [https://github.com/fail2ban/fail2ban]
exactly like vDDoS when viewed is again followed by guard tools that [6] CrowdSec: [https://www.crowdsec.net]
from the outside. Under the hood, a keep an eye on logfiles, detect failed [7] vDDoS: [https://vddos.voduy.com]
reverse proxy receives and examines login attempts, and automatically [8] FastNetMon: [https://github.com/
the access requests and forwards the create rules for the local firewall. pavel-odintsov/fastnetmon]
legitimate traffic to the server with Finally, websites can be protected [9] IPBan:
the content. For this method to work, against a flood of requests with [https://github.com/digitalruby/ipban]
the DNS entry for your website must various image puzzles in the form [10] EvlWatcher:
point to the Cloudflare server. Note of captchas. [https://github.com/devnulli/EvlWatcher]
that your web server is only allowed In larger environments, the DDoS [11] MHDDoS:
to accept requests from Cloudflare, sensor is located well away from the [https://github.com/MatrixTM/MHDDoS]
otherwise the DDoS protection is server farm and receives traffic infor-
ineffective. mation from routers and switches. If The Author
The advantage is that Cloudflare and the throughput rates of individual cli- Markus Stubbig is a networking engineer who
comparable offerings seem to have ents are unusually high, the block is has worked in the IT industry for 20 years. His
more bandwidth in reserve than the sent to the provider routers by a BGP strong focus is on design and implementation of
attackers have been able to muster update, and the game is over for the campus networks around the world..

90 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Terminating OpenSSH N U TS A N D B O LTS

Automatically terminate OpenSSH sessions

The Clock Is Ticking

Disconnect OpenSSH user sessions after a certain period of inactivity with disconnected after 10 minutes of inac-
tivity, in exactly the way required by
the systemd-logind service. By Thorsten Scherf the Common Criteria and DISA STIGs.
The problem is, though, that these
When configuring a system, a large Compliance Undermined two options were intended for a com-
number of settings are required to meet pletely different purpose – checking
compliance requirements. Common The US Department of Defense (DOD) the SSH connection itself – and not
Criteria [1] is an international standard Defense Information Systems Agency for user session activity. It was only
for the security certification of com- (DISA) Security Technical Implemen- possible to terminate the session at all
puter systems. The standard defines the tation Guides (STIGs) [3] also stipu- by setting the ClientAliveCountMax=0
requirements as security targets. late these requirements for operating option in combination with an arbi-
Targets look different depending systems. The Guide for Red Hat En- trary value for ClientAliveInterval,
on the system you are using. For terprise Linux 8 [4] proposes imple- even if a user was inactive and even if
example, the requirements for a mo- menting these rules with specific the connection itself was intact. The
bile device differ from those for a configurations of the OpenSSH ser- positive result is merely a lucky side
desktop system, which explains why vice. Two statements, ClientAliveIn- effect and was never the intended to
protection profiles are different. The terval and ClientAliveCountMax, are way for this to work.
Protection Profile for general-purpose intended to help meet the compliance This “misbehavior” of the software
operating systems [2] clearly stipu- requirements: was fixed in the OpenSSH upstream
lates that user sessions must either version 8.2 at the end of 2020 [5].
be terminated or, alternatively, locked grep -i clientalive /etc/ssh/sshd_config Unfortunately, it also ruled out the
Photo by Jon Tyson on Unsplash

after a certain period of inactivity. ClientAliveInterval 600 option of terminating an SSH con-
However, recent OpenSSH versions ClientAliveCountMax 0 nection when a user is inactive. This
block a workaround frequently used new behavior is particularly annoying
to meet this requirement. We show Once you have made these changes in environments where the systems
you how to use the systemd-logind to your OpenSSH configuration, an need to meet specific compliance
service to solve this dilemma. SSH connection to this system will be requirements.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 91
N U TS A N D B O LTS Terminating OpenSSH

The OpenSSH upstream community for longer than allowed. For example, Info
has long seen feature requests [6] [7] if you want inactive user sessions to [1] Common Criteria: [https://www.
for the implementation of a configura- expire automatically after 10 minutes, commoncriteriaportal.org/index.cfm]
tion option that lets the SSH server you would use a value of 600 with [2] Protection profile for general-purpose
terminate inactive sessions. However, the new option: operating systems:
these requests have thus far been re- [https://www.niap-ccevs.org/MMO/PP/
jected. One of the reasons is that most # grep StopIdleSessionSec U -442-/#FMT_SMF_EXT.1.1]
shells support the TMOUT environment /etc/systemd/logind.conf [3] STIGViewer:
variable, which lets you set a timeout StopIdleSessionSec=600 [https://www.stigviewer.com]
for user input. That said, the approach [4] RHEL8 STIG: [https://www.stigviewer.
is fraught with a number of disadvan- After making these changes, remem- com/stig/red_hat_enterprise_linux_8/
tages and is easy to work around [8]. ber to restart the service by typing: 2021-12-03/finding/V-230244]
[5] OpenSSH patch:
# systemctl restart systemd-logind
systemd-logind to the [https://github.com/openssh/
openssh-portable/commit/69334996
Rescue For test purposes, set the value to ae203c51c70bf01d414c918a44618f8e]
After the upstream changes slowly 10 seconds and log in to this system [6] OpenSSH-RFE 1: [https://bugzilla.mindrot.
made their way into the various again with SSH. The command org/show_bug.cgi?id=3362]
Linux distributions, the outcry from [7] OpenSSH-RFE 2:
users was massive, of course. Af- journalctl -u systemd-logind [https://bugzilla.mindrot.org/show_bug.
ter all, Linux distributions such as cgi?id=1338]
Red Hat Enterprise Linux or SUSE reads and filters the system journal [8] Stackoverflow article on TMOUT:
Linux Enterprise Server are used by and shows how the user’s inactive [https://stackoverflow.com/questions/
corporations in compliance-critical session is automatically terminated 17397069/unset-readonly-variable-in-
environments. Because the OpenSSH after 10 seconds (Listing 1). bash/54705440#54705440]
upstream community was not really [9] systemd-logind:
willing to address the problem, al- Conclusions [https://www.freedesktop.org/software/
ternative solutions were sought. The systemd/man/latest/systemd-logind.
result now available is quite obvious Typical compliance requirements service.html]
when you think about it and is based stipulate that inactive user sessions [10] pam_systemd PAM module:
on the systemd-logind service [9]. must either be terminated or alter- [https://www.freedesktop.org/software/
This service is explicitly designed to natively blocked. Until recently, you systemd/man/latest/pam_systemd.html]
monitor users and their sessions and could use some of the OpenSSH [11] systemd-logind idle patch:
can detect the idle state of user ses- service’s own configuration options [https://github.com/redhat-plumbers/
sions, enabled with the use of a sepa- to do this. However, this behavior systemd-rhel8/pull/332]
rate PAM module, pam_systemd [10]. was deliberately changed in recent
This module takes care of registering versions of the software without
a user’s session with the systemd-log- providing an alternative approach The Author
ind service after login, which in turn, to terminating inactive SSH connec- Thorsten Scherf is the
creates a separate systemd slice unit tions. To remedy this shortcoming, global Product Lead for
for each new user and a scope unit the missing function has now been Identity Management and
each for any sessions belonging to the added to the systemd-logind systemd Platform Security in Red
same user and running in parallel. service. This option lets admins Hat's Product Experience
A patch [11] was released at the end define an interval after which inac- group. He is a regular
of 2022 to extend the systemd-logind tive user sessions are automatically speaker at various international conferences
service. Armed with the patch, you terminated. Q and writes a lot about open source software.
can now pass in
the new StopId- Listing 1: Terminated Session
leSessionSec con- Mar 7 05:06:07 kvm-04-guest19 systemd-logind[46596]: New session 5 of user root.
figuration option to Mar 7 05:06:17 kvm-04-guest19 systemd[1]: session-5.scope: Killing process 46282 (sshd) with signal SIGTERM.
the service, which Mar 7 05:06:17 kvm-04-guest19 systemd[1]: session-5.scope: Killing process 46285 (sshd) with signal SIGTERM.
ensures that a Mar 7 05:06:17 kvm-04-guest19 systemd[1]: session-5.scope: Killing process 46286 (bash) with signal SIGTERM.
user’s session ends Mar 7 05:06:17 kvm-04-guest19 systemd[1]: Stopping Session 5 of user root.
as soon as sys-
Mar 7 05:06:17 kvm-04-guest19 systemd[1]: session-5.scope: Succeeded.
temd-logind detects
Mar 7 05:06:17 kvm-04-guest19 systemd[1]: Stopped Session 5 of user root.
that the session
Mar 7 05:06:17 kvm-04-guest19 systemd-logind[46596]: Removed session 5.
has been inactive

92 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Performance Tuning Dojo N U TS A N D B O LTS

An army of Xeon cores to do your bidding

44 Cores
Explore low-cost parallel computing.
By Federico Lucifredi

I continuously explore options for

cost-effective (or just plain cheap)
parallel computing rigs, and while
next-generation, cutting edge hard-
ware is always interesting, I find that
retired options from yesteryear can
also show potential when their signif-
icantly lower cost is part of the over-
all assessment. Retrofitting a retired
Dell workstation with high-core-count
CPUs and the maximum allowable
RAM, I built a 44-core compute behe-
moth for less than $600 to run Monte
Carlo [1] simulations. Let me dive
into the details!

Bill of Materials
Table 1 details my hardware config-
uration. I found a refurbished Dell
Precision T7190 workstation [2] on
eBay in nearly perfect cosmetic con-
dition with a motherboard sporting
two LGA 2011-3 processor sockets –
which were both vacant (Figure 1).
The stock power supply is rated at
Figure 1: Inside view of the system before the build-out. Note how sockets are protected by 1,300W, more than sufficient for
plastic shielding plates. this project, but alas, one of the
CPU heat sinks was missing. The
Table 1: Shockwave Compute Server Specs description promised no ventilation
Component Spec shrouds or disks, but the unit came
Chassis and motherboard Dell Precision Workstation T7910 with four hard disks, one DVD-ROM
drive, and all the air shrouds, mak-
Power 1,300W
ing this a happy purchase ($159).
CPU 2x Intel Xeon Gold E5-2699 V4, 22 cores, 2.4GHz, 55MB of cache, LGA 2011-3
After temporarily installing a 10-
GPU, NPU n/a* core Xeon from the parts archive
Memory 256TB DDR4 ECC PC4-19200 2,400MHz and flashing the BIOS to its latest
Lead Image © Lucy Baldwin, 123RF.com

Storage 4x3.5-inch drive bays, slimline optical drive, LSI SAS 3008 12Gbps SAS revision with Dell’s very reasonable
(6Gbps SATA) bootable tooling [3] [4], I was able
Networking Intel I217 and I210 Gigabit Ethernet controllers, remote wake-up to install two newly procured CPUs,
Video NVIDIA Quadro which are Intel Xeon Gold E5-2699
v4 CPUs running at 2.4GHz and
HDMI, DP
each sporting 22 cores and 55MB
*Not applicable. of cache memory [5]. Fortunately,

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 93
N U TS A N D B O LTS Performance Tuning Dojo

Fixing an LGA 2011-3 Socket I had a second heat sink and fan on hand (Buying a
new one would have cost nearly as much as the work-
Identifying the issue with the CPU socket required some investigative
station itself!). This purchase set me back $250 for two
work of the bisection variety: installing one CPU, then installing the
CPUs, which were engineering samples (ES) verified to
other (both worked), then installing half of the RAM, then the other
run validation tests reliably at speed. Unfortunately, the
half (the second test failed), then continuing to divide this failed half
second socket also came with a bent pin, which sent me
until I identified the pair of DIMMs that were not working. However, the
on a two-week wild-side quest troubleshooting the CPUs
DIMMs themselves were working (swapped with another pair). Connect-
and its memory banks until I located it, cleaned it, and very
ing this picture back to the CPU pin was fortuitous: As I was re-seating
delicately and patiently bent it back into its original position.
a heat sink, I noticed some thermal paste out of place, and when I re-
(See the “Fixing an LGA 2011-3 Socket” box.)
moved the CPU, I found thermal paste in the socket – not a good thing,
even when things are working. I washed the thermal paste out with 70
percent isopropyl alcohol loaded in a Waterpik-type device I sourced on Total Recall
AliExpress for $20. Another $20 went to Amazon for a handheld USB mi-
Sixteen empty DIMM memory slots stared at me asking for
croscope [12] to examine the damaged area (Figure 2). Patient use of
attention. Raiding my lab’s archive, I found eight perfectly
an index card and tweezers enabled me to rectify the failure. The bent
suitable modules already in my possession (16GB DDR4
pin controlled the affected DIMM banks.
error correction code (ECC) PC4-19200 2,400MHz, exceed-
ing spec). A little bargain hunting led me to find another
eight modules on Amazon ($182 in total) with equivalent
specs manufactured by SK Hynix [6]. Collectively, the
16 modules combine to provide 256GB of RAM, half of
the maximum that could be installed without resorting to
more expensive load-reduced (LR)DIMM, which in turn
maxes out at 1TB. The current design provides almost 6GB
of RAM per core, and I retain the option to budget another
$2,000 to quadruple that amount if a workload is found
needing it – a very reasonable compromise.
I completed the setup with this newfangled technology
called software – Ubuntu 23.10 “Mantic Minotaur” provid-
ing the operating system layer, with the distribution in-
cluding all the necessary message-passing interface (MPI)
Figure 2: The bent CPU socket pin as seen under the microscope. and parallel processing tools one may want. The btop [7]

Figure 3: Forty-four cores humming along, but why are numbers 32 and 33 doing all the work?

94 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M
 Performance Tuning Dojo N U TS A N D B O LTS

Figure 4: Turning the system into a space heater with the right benchmark.

tool is everyone’s new favorite in-ter- cost calculation as much as, if not individually, but it is more interesting
minal system monitor, and it provides more than, the hardware cost itself. in this case to note that 2.4GHz is the
a first look at the completed system The 22 cores (44 threads) of each maximum speed they can all acceler-
(Figure 3). Note the inclusion of core CPU could turbo boost up to 3.6GHz ate to concurrently.
temperatures.
I already discussed BashTop in a
previous issue [8], but today I shall
focus on just one aspect of its CPU
view: What happens when all those
cores heat up? The system idles at
121W once booted up, so I will drive
up the temperature with the matrix
option of the stress-ng tool [9], an
ominous stressor known for its Intel
CPU heating properties [10]:

stress-ng --matrix 0 -t 1m --tz --times

The zero count syntax requests one

stressor to run on each CPU core in
the system: --times generates statis-
tics on userland and kernel time, and
the --tz option includes CPU tem-
perature data where available. The
CPU clock ran up from a cozy 1.2GHz
to 2.1GHz across the board, with all
cores pegged at 100 percent, eventu-
ally reaching a perfect 44 load aver-
age [11] (Figure 4). Temperature did
not exceed 72C at the hottest sensor
(good cooling on Dell’s part), but the
power draw tripled, rising to 367W
(Figure 5). The power-hungry nature
of the beast needs to factor in any Figure 5: Tripled power draw. The electric bill is the largest expense for this system.

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 95
N U TS A N D B O LTS Performance Tuning Dojo

At the time of this writing, an Ama- [4] Dell bootable DDDP: [https://www.dell.com/ [10] “Creating load for fun and profit” by
zon AWS m7g.12xlarge instance with support/kbdoc/en-us/000145519/how-to- Federico Lucifredi, ADMIN, issue 75, 2023,
48 virtual cores and only 192GB of create-a-bootable-usb-flash-drive-using- [https://www.admin-magazine.com/
RAM will cost almost $2/hr ($1.9584, dell-diagnostic-deployment-package-dddp] Archive/2023/75/Creating-load-for-fun-
US East, on-demand pricing), so you [5] Intel Ark: Xeon E5-2699 v4: [https:// and-profit]
could think of this new machine as ark.intel.com/content/www/us/en/ark/ [11] “Law of Averages” by Federico Lucifredi,
costing 12-1/2 days (300 hours) of products/91317/intel-xeon-processor- ADMIN, issue 11, 2012
AWS compute. Not bad! Q e5-2699-v4-55m-cache-2-20-ghz.html] [12] Low-cost USB microscope:
[6] Hynix 16GB DDR4 PC4-19200 2,400MHz [https://www.amazon.com/gp/product/
ECC REG DIMM: [https://www.amazon. B06WD843ZM]
Info com/gp/product/B01N6O511Z/]
[1] The Monte Carlo method: [7] btm(1) man page:
[https://en.wikipedia.org/wiki/Monte_ [https://manpages.ubuntu.com/man- Author
Carlo_method] pages/noble/en/man1/btop.1.html] Federico Lucifredi (@0xf2) is the Product Man-
[2] Dell Precision T7910 workstation: [8] “Next-generation terminal UI tools” agement Director for Ceph Storage at Red Hat
[https://i.dell.com/sites/doccontent/ by Federico Lucifredi, ADMIN, issue 64, and IBM, formerly the Ubuntu Server Product
shared-content/data-sheets/en/Docu- 2021, [https://www.admin-magazine. Manager at Canonical, and the Linux “Systems
ments/Dell-Precision-Tower-7000-Series- com/Archive/2021/64/Next-generation- Management Czar” at SUSE. He enjoys arcane
7910-Spec-Sheet.pdf] terminal-UI-tools] hardware issues and shell-scripting mysteries,
[3] Dell BIOS updates: [https://www.dell. [9] stress-ng by Colin King: and takes his McFlurry shaken, not stirred. You
com/support/kbdoc/en-us/000124211/ [https://manpages.ubuntu.com/ can read more from him in the new O’Reilly title
dell-bios-updates] manpages/jammy/man1/stress-ng.1.html] AWS System Administration.

Q
 Back Issues S E RV I C E

ADMIN Network & Security

NEWSSTAND Order online:

https://bit.ly/ADMIN-library

ADMIN is your source for technical solutions to real-world problems. Every issue is packed with practical
articles on the topics you need, such as: security, cloud computing, DevOps, HPC, storage, and more!
Explore our full catalog of back issues for specific topics or to complete your collection.

#78 – November/December 2023

Domain-Driven Design
Business experts and developers collaborate to define domain models and
business patterns that guide software development.
On the DVD: Fedora Server 39

#77 – September/October 2023

Secure CI/CD Pipelines
DevSecOps blends security into every step of the software development cycle.
On the DVD: IPFire 2.27

#76 – July/August 2023

Energy Efficiency
The storage share of the total data center energy budget is expected to double by
2030, calling for more effective resource utilization.
On the DVD: Finnix 125 (Live boot

#75 – May/June 2023

Teamwork
Groupware, collaboration frameworks, chat servers, and a web app package manager allow
your teams to exchange knowledge and collaborate on projects in a secure environment.
On the DVD: Ubuntu 23.04 “Lunar Lobster” Server Edition

#74 – March/April 2023

The Future of Software-Defined Networking
New projects out of the Open Networking Foundation provide a glimpse into the 5G
network future, most likely software based and independent of proprietary hardware.
On the DVD: Kali Linux 2022.4

#73 – January/February 2023

Databases
Cloud databases can be useful in virtually any conceivable deployment scenario, come in SQL
and NoSQL flavors, and harmonize well with virtualized and containerized environments.
On the DVD: Manjaro 22.0 Gnome

W W W. A D M I N - M AGA Z I N E .CO M A D M I N 79 97
S E RV I C E Contact Info / Authors

WRITE FOR US
Admin: Network and Security is looking • unheralded open source utilities
for good, practical articles on system ad- • Windows networking techniques that
ministration topics. We love to hear from aren’t explained (or aren’t explained
IT professionals who have discovered well) in the standard documentation.
innovative tools or techniques for solving We need concrete, fully developed solu-
real-world problems. tions: installation steps, configuration
Tell us about your favorite: files, examples – we are looking for a
• interoperability solutions complete discussion, not just a “hot tip”
• practical tools for cloud environments that leaves the details to the reader.
• security problems and how you solved If you have an idea for an article, send
them a 1-2 paragraph proposal describing your
• ingenious custom scripts topic to: [email protected].

Contact Info
Editor in Chief While every care has been taken in the content of
Joe Casad, [email protected] the magazine, the publishers cannot be held re-
Managing Editors sponsible for the accuracy of the information con-
Rita L Sooby, [email protected] tained within it or any consequences arising from
Lori White, [email protected] the use of it. The use of the DVD provided with the
magazine or any material provided on it is at your
Senior Editor
own risk.
Ken Hess
Copyright and Trademarks © 2024 Linux New
Localization & Translation
Media USA, LLC.
Ian Travis
No material may be reproduced in any form
News Editor whatsoever in whole or in part without the writ-
Amber Ankerholz
ten permission of the publishers. It is assumed
Copy Editors that all correspondence sent, for example, let-
Amy Pettle, Aubrey Vaughn ters, email, faxes, photographs, articles, draw-
Authors Layout ings, are supplied for publication or license to
Dena Friesen, Lori White third parties on a non-exclusive worldwide
Amber Ankerholz 6 basis by Linux New Media unless otherwise
Cover Design
stated in writing.
Joe Casad 28 Dena Friesen, Illustration based on graphics
by rendeeplumia, 123RF.com All brand or product names are trademarks
Ken Hess 3 of their respective owners. Contact us if we
Advertising
Thomas Joos 14, 56, 60 haven’t credited your copyright; we will always
Brian Osborn, [email protected]
correct any oversight.
phone +49 8093 7779420
Rubén Llorente 34 Printed in Nuremberg, Germany by Kolibri Druck.
Publisher
Martin Gerhard Loschwitz 40, 46 Brian Osborn Distributed by Seymour Distribution Ltd, United
Kingdom
Federico Lucifredi 93 Marketing Communications
Gwen Clark, [email protected] ADMIN (Print ISSN: 2045-0702, Online ISSN: 2831-
Benjamin Pfister 80 Linux New Media USA, LLC 9583, USPS No: 347-931) is published bimonthly by
4840 Bob Billings Parkway, Ste 104 Linux New Media USA, LLC, and distributed in the
Dr. Holger Reibold 10, 24, 70
Lawrence, KS 66049 USA USA by Asendia USA, 701 Ashland Ave, Folcroft PA.
Thorsten Scherf 91 January/February 2024. Application to Mail at
Customer Service / Subscription Periodicals Postage Prices is pending at
Tim Schürmann 34 For USA and Canada: Philadelphia, PA and additional mailing offices.
Email: [email protected] POSTMASTER: send address changes to Linux
Artur Skura 74
Phone: 1-866-247-2802 Magazine, 4840 Bob Billings Parkway, Ste 104,
Evgenij Smirnov 19, 64 (Toll Free from the US and Canada) Lawrence, KS 66049, USA.
Andreas Stolzenberger 52 For all other countries: Represented in Europe and other territories by:
Email: [email protected] Sparkhaus Media GmbH, Bialasstr. 1a, 85625
Markus Stubbig 86 www.admin-magazine.com Glonn, Germany.

98 A D M I N 79 W W W. A D M I N - M AGA Z I N E .CO M