How to master

embedded Linux system

maintenance in 2024

by .

1
Dear reader,
Welcome to the synthesis of our 2023 study on embedded Linux system maintenance!
Embark on a journey with us as we delve into the ambitions, challenges, pains, and
perspectives of equipment manufacturers (OEMs) in their pursuit of developing and
maintaining industrial Linux-based devices.

This exploration has been driven by The Embedded Kit team with the desire to better
understand the experiences and hurdles faced by OEMs in the realm of embedded
Linux maintennce. These insights will serve as the bedrock for adapting our products
to bring more value to our users.

It took dozens of interviews with R&D managers, lead software engineers from com-
panies of all sizes, and hundreds of responses to quantitative surveys combined with
extensive online research to have a global overview, which we are excited to distill and
share with you through this synthesis.

You will find in this report:

• An overview of the challenges encountered by peers, delineated by their level of
maturity
• Solutions and best practices to streamline embedded Linux maintenance
• Decision matrices designed to help you select solutions pertinent to your unique
context
• Insights and prospects on the future of embedded Linux maintenance

Before we embark on our Linux journey, we’d like to thank, once again, all the people
who shared their precious experience and insights. And to those who haven’t had the
chance to share their perspectives on this topic, feel free to reach out!

Enjoy the ride,

The Embedded Kit team

2
Summary
PART 1 - State of embedded Linux system maintenance 4
- Where do your peers stand?

Teams with little in-house knowledge of embedded Linux 4

Mature teams with Linux-based products on the market 5
Embedded Linux experts 6

PART 2 - Solutions & best practices 7

OTA update 7
Continuous integration 9
Automated testing 9
Automated testing tool comparison 11
Cybersecurity 12
DevSecOps platform 13

PART 3 - Perspectives 14

PART 4 - Conclusion 15
PART 1
State of embedded Linux system
maintenance - Where do your peers
stand?
Our study led us to gather experiences from various types of profiles, from embedded software developers to R&D managers,
via tech leads and software architects, all working in companies manufacturing equipments. Through these discussions, we’ve
identified three different levels of maturity when it comes to embedded Linux system development and maintenance.

Teams with little in-house knowledge of embedded Linux

With very limited prior experience with Linux, these teams find themselves somewhat lost when
developing their Linux-based system and aim to develop business applications that add value
to their products. Maintenance isn’t a priority. This profile is prevalent among startups and small
to medium-sized industrial enterprises with small development teams, in particular companies
whose products have traditionally been based on microcontrollers (MCUs).

Quickly focus on business app Time-to-market Budget

Main challenges
Product stability Shortage of in-house skills

Seeking assistance from external firms

Preferred solutions
Using off-the-shelf distributions

Expert advice #1 - Do you need to migrate to Linux and MPU?

You need to validate the needs of MPU and Linux by confirming which features (which also include
your global product roadmap with future features) will be requested. Do you need to get access to
a better development framework? Do you plan to integrate multimedia usage? Advanced graphical
UI? Or do we need more power for calculation or an AI algorithm?...

Pierre GAL, Head of Product Development at The Embedded Kit

Samir BOUNAB, CEO of The Embedded Kit

Expert advice #2 - Build your own MPU team!

Hiring a good Linux engineer is complex (but not impossible). But ramping up your current team
is another very interesting way to explore. It will empower your team, open new opportunities for
them, and help reduce the attrition rate. In that case, starting with an off-the-shelf distribution and
receiving support from an external Linux expert in the beginning may be helpful. And we can help
you with that at The Embedded Kit ;)

4
Mature teams with one or more Linux-based products on the market
Even if their Linux systems are already in place, these teams still face challenges related to long-
term maintenance and cybersecurity.
This profile is mainly found in mid-sized enterprises and, in some cases, large corporations that
have been working with microprocessors for a considerable time and have a dedicated, small-sized
team.

Rationalize development & maintenance efforts

Main challenges
Reduce dependence to providers Platforming

In-house development Open source & community support

Preferred solutions
Externalization Off-the-shelf distribution & tools

Expert advice #1 - Team retention

Keeping your Linux teammates is a key success factor. Such profiles love tech, new developments
and may be bored by a pure maintenance period. To manage that properly, keep knowledge as
wide as possible, ramp up junior on maintenance activities, and keep high-value tasks for experts.

Julien BERNET, Cybersecurity team lead

Pierre GAL, Head of Product Development

Expert advice #2 - Platforming

When you start building a range of products, thinking about platforming is a key topic.
OEMs are already used to build the hardware side. That’s, however, less common on the software
side, but it’s still as important to mutualize development efforts and anticipate maintenance ef-
forts. Products like Welma Yocto Linux can help you with platforming, as it has been designed for
this use case.

5
Embedded Linux experts
With large, well-trained teams capable of handling embedded Linux development and
maintenance in-house, this profile often corresponds to extensive teams within large corporations
or mid-sized enterprises with a strong Linux culture.

Maintain stable systems Be cybersecure

Main challenges
Reduce dependence to providers Platforming

In-house development Open source & community support

Preferred solutions
Off-the-shelf distribution & tools

Expert advice #1 - Team organization

Managing a team comprised of multiple Linux experts poses challenges, as each expert holds their
unique perspectives on the system, quality standards, future-proof solutions, and personal habits.
To address this, it’s crucial to establish a well-structured organization that promotes shared respon-
sibilities and actively involves all team members.

Pierre LECOMTE, Solution Director & Product Management Lead

Pierre GAL, Head of Product Development

Expert advice #2 - Avoiding dependency and vendor lock-in

Many large organizations utilize renowned software solutions that may result in a vendor lock-in
effect, leading to long-term costs. Consider the level of dependency and the associated risks when
engaging with a new supplier or adopting a new product.

6
PART 2
Solutions & best practices

1 OTA update

Remote software updates play a crucial role in maintaining and enhancing the security and
functionality of connected devices. By prioritizing reliability, implementing the A/B update
schema, and addressing potential challenges, we can ensure that these updates are seam-
less and robust, providing an improved experience for both users and equipment manufac-
turers.

Besides the established A/B update model, various update schemes, such as the golden im-
age and differential incremental update, exist. However, we won’t delve into those details
today.

What is OTA update?

OTA update, which stands for “over-the-air updates,” refers to the capability to remotely up-
date the root file system, firmware, bootloader, and/or applications and services through a
wireless network (like Wi-Fi or mobile telecommunications). Due to language convenience,
the term OTA has become standard, even for non-wireless update cases.

Why should you use OTA update?

OTA update is a crucial aspect of modern device management as it enables seamless secu-
rity and device enhancements without the need for physical intervention on your device, as
was the case a few decades ago when using a hard disk or a flash drive to manually down-
load new firmware on each device.

Resisting power and network loss: a top priority

One of the primary goals of OTA updates is to ensure that devices can withstand power
and network interruptions, which can lead to abrupt, uncontrolled software shutdowns. In
scenarios where thousands or even hundreds of thousands of devices are in use, avoiding
a 10% failure rate due to updates is paramount. To achieve this, we prioritize resistance to
power and network losses.

How does OTA update work? The A/B update schema, a classical and proven approach
To achieve robustness in remote updates, we employ the A/B update schema. This ap-
proach involves having two copies of updateable software components: Part A and Part B,
which are crucial for maintaining system integrity during the update process.
When a device boots up, it
initially runs a bootloader
responsible for managing the
starting process. The boot-
loader’s role is to hand over
control to one of the partitions,
either Part A or Part B, typically
containing the kernel and Root
File System (Root FS). This be-
comes the active partition.

During an update, the new software version is This is why we have seamlessly incorporated them
installed on the inactive partition (usually Part as OTA update options within Welma, our Linux
B). If the update process is interrupted, the active distribution. This integration provides device man-
partition (Part A) remains unchanged, ensuring ufacturers with a ready-to-use, robust method to
continued functionality. A subsequent reboot di- keep their devices up to date.
rects execution to the updated partition, making it
the new active partition. This approach guarantees While we have focused on the embedded part of
that the device can roll back to a known, function- this solution, there is another aspect to consider:
ing state if the update on the inactive partition fails the update service and fleet management solu-
to validate correctly. tion that assists in managing and deploying these
updates remotely. Mender.io stands out as one of
A key feature of this update strategy is the ability the only solutions on the market that, in addition
to maintain two different versions of software on to the local update system, provides a cloud-based
each partition, ensuring the system can always device management platform for handling update
revert to a previous functional version in case the campaigns.
new one encounters errors. Initially, in version N of
the software, partition A is active while partition B
is inactive, running on version N-1. However, upon It’s evident that different strategies and technical
updating the system to version N+1, partition A solutions exist for updating embedded Linux sys-
becomes inactive with version N, while partition tems. It’s crucial to define the right strategy based
B takes over, becoming active with the brand-new on specific needs and the system environment.
version N+1. This cycle continues with subse- For example, a platform in a regulatory context
quent updates, with partition A becoming active may not be suitable for remote updates. On the
under version N+2 when updating to version N+2, other hand, a highly connected system with higher
while partition B remains inactive under version cybersecurity risks should allow for quick and ro-
N+1. This pattern repeats, ensuring a reliable and bust updates. More specific scenarios may require
smooth update process for end-users while main- a customized approach, such as in a constrained
taining service continuity. connectivity network where an incremental update
may be necessary to optimize the amount of data
If you need to implement an OTA update solution exchanged over the network.
on your system, the A/B update schema is a reli-
able mechanism to maintain system integrity.

OTA update solutions

OTA update solutions such as SWUpdate, RAUC,
and Mender offer a simplified and efficient ap-
proach to handling firmware updates on embed-
ded Linux systems. These tools make it easier for
device manufacturers to ensure the security and
functionality of devices in the field.
8
2 Continuous integration

Incorporating continuous integration (CI) practices into your development workflow

ensures that changes are continuously tested and integrated. Automated testing helps
identify issues early in the development process, reducing debugging and validation time.
CI tools and platforms make it easy to set up testing pipelines for your embedded projects.

Focus on reproducible builds

Ensuring that your system’s build process is consistent and reproducible is crucial for
maintaining uniformity and reducing errors during development. To achieve this, using
version control (e.g., Git) and pipelines for automated builds (with platforms like GitLab,
GitHub, or Jenkins) is a key success factor. This approach allows implementing regular
automatic builds to receive prompt feedback, validate pull requests, and generate archives
for release builds.

While this way of working has become a standard in software development processes,
implementing such a continuous integration system in an embedded Linux context to build
a complete operating system remains complex to deploy properly.

3 Automated testing

In the realm of embedded products, automated testing is pivotal. It streamlines

development, enhances quality, and minimizes errors, making it essential for reliability and
functionality.

What is the pyramid of tests?

Testing embedded products can be challenging, but it is essential to ensure the reliability
and functionality of these complex systems. To comprehend the hierarchy of tests, we turn
to the well-known Pyramid of Tests, which is a framework for organizing testing efforts in an
IoT/embedded project.

9
• Unit testing: At the base of the pyramid lies unit testing. It involves the automated validation of
specific components, algorithms, or functions within the code. Unit tests evaluate individual parts of
the software with various parameters to ensure they function as expected.

• Component testing: Moving up the pyramid, we encounter component testing. This level involves
testing larger components or subsystems, such as an embedded system driver.

• Integration testing: Integration tests examine how different components behave when assembled.
This level of testing focuses on the interaction between various layers and services in an embedded
solution, such as OTA updates, which require coordination between cloud platforms, firmware
uploads, and data reporting.

• System testing: At the peak of the pyramid is system testing. This level employs end-to-end testing,
simulating user interactions with the complete IoT system. System tests validate all business features,
providing a holistic view of the system’s functionality.

What does on-board automated testing mean?

Automated testing in embedded comes in two flavors: “on-board” and “off-board”. On-board automated
testing refers to testing on the real hardware or in the actual cloud environment. It is performed
automatically but takes place within the real integration environment. Onboard tests are ideal for
checking system behaviors that are specific to the actual deployment environment.
What’s the difference between manual testing and automated testing?
It’s crucial to differentiate between manual testing and on-board automated testing. Manual testing
remains essential for various aspects of product development. While on-board automated testing offers
numerous benefits, manual testing is irreplaceable for specific tasks that require human intervention
and expertise. The key is to strike a balance between manual and automated testing.

What are the benefits of automated testing?

Reducing development-to-testing lag: One significant challenge in embedded development is the
time gap between feature development and testing. In traditional testing methods, this gap can be
quite substantial, making it more difficult to fix bugs or issues that emerge during testing. The aim of
automated testing is to close this gap, allowing teams to discover issues earlier in the development
process, making bug fixing quicker and more efficient.

• Behavioral testing: Automated testing covers standard behavioral tests, akin to what manual testing
accomplishes. This includes verifying that the software meets its expected functional requirements.
• Edge case management: Automated testing is capable of managing edge cases that can be difficult
and time-consuming to address manually. This ensures that the software is robust and reliable under
various conditions.
• Non-regression testing: While some non-regression testing can still be performed manually,
automated testing can handle a substantial portion of these tests, ensuring that new changes do not
break existing functionality.
• Endurance testing: This type of testing examines how the system performs over time under normal
usage. Automated testing is particularly valuable for such long-duration tests.
• Robustness testing: Testing the system’s robustness, especially in edge cases, is best achieved with
automated testing.
• Performance measurement: Automated testing allows for comprehensive performance
measurements, which are difficult to achieve manually. This includes assessing how well the system
performs under various conditions and loads.

10
Select the right automated testing tool for your system

In-house tool
development

Technical
capabilities for Perfectly adapted Feature-rich but Feature-rich but Feature-rich and
to your needs not for embedded only for embedded optimized for
embedded systems testing Linux embedded testing

CI integration Adapted to your Yes Yes Yes

needs

Syntax Adapted to your Verbose Ramp up required Accessible

needs

Cost Free Free Free One time fee

No No embedded Ready-to-use but Set of embedded

Off-the-shelf testing features missing interface testing features
available testing features provided

Flexibility
High High High High

Customer support None Community-based Community-based Access to dev team

Watch our webinar on “How

to elevate product quality
with automated testing”

+ link to webinar
11
2 Cybersecurity

System security

Security measures at the system level are essential to protect against unauthorized
access or tampering. They include vulnerability monitoring, providing a device identity
using X.509 certificates, and using secure elements for pre-provisioned identities when
using cloud services based on robust certificate management.

OS security

Robust security measures must be implemented at the operating system level to

protect against vulnerabilities and threats. Start with creating a secure image. Disable
unnecessary interfaces to reduce potential attack vectors. Implement secure boot
mechanisms to help verify the authenticity of critical components like the bootloader
and kernel, as well as an authenticated read-only file to prevent unauthorized writes
and system or application modifications. Safeguard secure keys to ensure recovery in
case of failure. Finally, employ encryption for data storage.

Application security

Security practices should extend to the application layer to protect sensitive data and
functions. Use the principle of least privilege and restrict applications using tools like
SELinux, AppArmor, or containers to isolate them from the system.

See our article on “How to build

a secure by design Linux-based
product”

12
 5 DevSecOps platform

The maintenance process in software development is a dynamic and crucial phase that
extends beyond the initial release of a product. It involves a meticulous strategy to ensure
the ongoing stability, security, and performance of the software. Long-Term Support (LTS)
versions, periodic updates, and the management of Common Vulnerabilities and Exposures
(CVEs) are integral components of this process.

Why keeping your embedded Linux system healthy for the long run is important
As devices become more and more part of global systems—like high-end devices
connected directly or indirectly to IoT hubs, business apps, ERPs, and industrial tools—
it’s super important to make sure they don’t have any weak points. That’s why long-term
maintenance activities are crucial to prevent vulnerabilities and make sure your product
stays top-notch throughout its whole life.

The challenges of embedded Linux system maintenance

Despite the importance of long-term maintenance activities and processes, equipment
manufacturers often face challenges in implementing effective methodologies. Challenges
include the need for a flexible system versioning strategy, a platforming approach to
optimize the time spent on maintenance for multiple products, CVE and LTS update follow-
up, and the perception that maintenance is not always an added value activity. To address
these challenges, a robust methodology and adapted tooling are essential.

Key concept: LTS versions

LTS, or long-term support, is a key concept to grasp in software development. It denotes the extended
support provided for all the software layers in your system, akin to what the Yocto Project offers on
various kernel versions. This support spans several years, simplifying the task of maintaining stabil-
ity and reliability in embedded systems. With a maintenance period lasting three to four years, LTS
versions serve as the rock-solid foundation for uninterrupted software operation, delivering crucial
updates, security patches, and bug fixes. This stability is especially vital in fields like medical devices,
where software maintenance may stretch over a decade or more. The migration strategy from one LTS
version to another, involving routine minor updates and periodic shifts to major LTS versions, strikes a
balance between embracing new features and upholding a robust foundation.

Build a reference DevOps platform from the get go

At The Embedded Kit, we’ve mapped our reference DevOps platform in our commitment
to ensuring the long-term health and security of our customers software projects.
Anchored in Yocto BSP source code management, it orchestrates a seamless build process,
incorporating SonarQube analyses for source code integrity. At its core is CVE scan, our
Linux vulnerability scanner that meticulously detects vulnerabilities in the SBOM. The
platform facilitates a sophisticated annotation process, allowing for nuanced analysis and
differentiation between actual vulnerabilities and false positives.
Automated testing, with Pluma, ensures that every release undergoes rigorous validation,
mitigating the risk of major regressions in the system.

With a holistic approach that encompasses development, analysis, and testing, our
reference DevOps platform serves as a robust foundation, empowering developers to
navigate the complexities of long-term maintenance with efficiency and precision.
13
PART 3
Perspectives

In the rapidly evolving landscape of embedded Linux system development and maintenance, several key
considerations shape our approach and outlook for the future.

Yocto Project: the best solution for tailor-made embedded Linux systems
We firmly believe that Yocto remains the optimal choice for equipment manufacturers’ embedded Linux
developments, offering unparalleled advantages in custom configuration, optimization, cybersecurity,
documentation, and strong community support. Nevertheless, it’s important to acknowledge that Yocto
can present complexity in its management and implementation. While the benefits are substantial, ad-
dressing the intricacies of Yocto may require dedicated expertise and resources to ensure a smooth and
successful integration into your projects.

Ownership and control

The need for ownership and control over embedded systems is a fundamental principle guiding more
and more OEM strategies. Having complete control over their systems provides the flexibility to adapt
to changing requirements and technological advancements. In a world where the lifespan of embedded
systems can extend for 10 to 20 years or more, this long-term view is critical. It ensures that systems
remain viable and secure throughout their extended lifecycles without the need to migrate to another
system because of a provider deficiency.

Cybersecurity is a priority
We are convinced that the landscape of embedded Linux development and maintenance will soon un-
dergo a transformation due to newly published cybersecurity regulations. These new regulations, such
as the Radio Equipment Directive (RED) and the Cyber Resilience Act (CRA), make security restrictive
for embedded systems, depending on their level of criticism. As such, it’s important to start integrating
robust cybersecurity measures into embedded systems, safeguarding them against evolving threats and
regulatory compliance demands.

Off-the-shelf hardware + software packages

Looking ahead, we see the future of embedded systems as a convergence of hardware and software
packaged offers. OEMs will increasingly seek comprehensive solutions that include both hardware and
software components, allowing them to accelerate their development processes. This trend empowers
OEMs to focus on their core business expertise while relying on integrated hardware and software pack-
ages to handle the low-level intricacies.
That’s exactly what The Embedded Kit is trying to achieve in partnership with renowned silicon vendors
and SOM makers like STMicroelectronics, NXP, boundary devices, Aaeon, etc.
PART 4
Conclusion

This exploration helped us identify how to simplify embedded Linux development &
maintenance challenges faced by equipment manufacturers.

That’s why we created The Embedded Kit, a comprehensive toolkit for embedded
systems. The Embedded Kit offers everything device-makers need to build,
connect, test, and secure a custom embedded Linux system. Its four off-the-shelf
products (Welma Yocto Linux, Kamea IoT, Pluma automated testing and CVE Scan)
were designed specifically for equipment manufacturers, to ease and accelerate their
product development while giving full source code control.

Want to see a demo? Or just share some feedback? We’ll be happy to chat with you :)

____________

This report is based on community contributions from all over the world. We would like
to thank, once again, the people who took part in our exploration, and the witekians
who have shared their knowledge, who have been working side by side with equipment
manufacturers for years to build, customize and maintain their connected solutions.

15
theembeddedkit.io