DRIVER ARCHITECTURE IN A LINUX WORLD

John W. Linville, Dux Linux

LVL7 Systems, Inc.
13000 Weston Parkway, Suite 105, Cary, NC 27513, USA
[email protected]

Abstract
Vendors of modern networking hardware realize that the devices they provide require a great deal of
software in order to function effectively. As a result, many hardware vendors provide large amounts of
software for use with their devices. However, the prevalent use of the VxWorks 1 operating system has
allowed hardware vendors to become careless with the architecture of the driver software they provide.
Often the driver software provided by hardware vendors is difficult to use in a protected environment
like Linux .2 This paper begins by discussing strategies for making use of such drivers in a Linux
environment, and continues by discussing how hardware vendors might architect Linux -friendly driver
software. Finally, the paper discusses how vendors can provide first-class Linux support for their
hardware and why it is in their best interest to do so. This paper is a must read both for system
integrators new to Linux and for driver software architects unfamiliar with the Linux environment.

1 Introduction products by offering as much software support for

exercising the features of their hardware as possible.
In the modern world of advanced networking, sim- Today, most networking hardware vendors provide
ple hardware just does not exist. Requirements for a large base of software to support their products.
both speed and functionality have increased expo- Such software typically contains hardware access
nentially for many years. In many segments of the routines, application support libraries, and even ser-
marketplace, designs relying on a system’s CPU to vice tasks that perform maintenance services related
forward frames, to learn addresses, or even to run to supported applications of the hardware. However,
simple protocols are at such a disadvantage that al- most hardware vendors are not in the business of
most no one in those market segments produces such selling software. As a result, hardware vendors tend
a system. Modern networking designs require hard- to have limited software development resources. It
ware that is capable of off-loading basic networking is not surprising that these vendors tend to support
functions from the CPU. Whether that hardware is only the most ubiquitous operating system platforms
a programmable co-processor or a configurable net- with their driver software.
working ASIC, complicated software is required to

For some time, VxWorks has been the domi-

bond that hardware and the system’s CPU into a nant operating system for embedded networking de-
cohesive networking system. vices. Consequently, the primary focus for software
In order to encourage the use of this advanced but support by networking hardware vendors has been

complicated hardware, networking hardware vendors VxWorks . More recently, Linux has become a
provide driver software for use with their products. strong player in the world of embedded systems and
This software is generally not a formal device driver particularly in the embedded networking arena. At

as defined by an operating system, but rather an first glance, VxWorks and Linux may seem to
informal collection of software designed to ease the have much in common with one another. For ex-
use of the driven hardware. Nevertheless, this soft- ample, both support one or more of the industry
ware is often incorporated directly into commercial standard Posix APIs. However, these two operat-
end products. This situation has provided hardware ing systems are distinctly different in many ways.
vendors with the opportunity to differentiate their Supporting one rarely implies support of the other.
1 VxWorks is a registered trademark of Wind River Systems, Inc.
2 Linux is a registered trademark of Linus Torvalds.
This situation is aggravated by the fact that many erally insufficient for control of these devices. Other
hardware vendors take advantage of the less restric-


forms of hardware access generally require execution
tive “features” of the VxWorks environment when in kernel context, and vendor-provided conglomerate
implementing their informal driver software. The


drivers tend to be poorly segregated between hard-
Linux kernel enforces a strict driver model for ware access and higher-level functionality. The solu-
hardware access and it segregates different software tion seems obvious: package the conglomerate driver
components into separate address spaces. In con-


in a way that allows it to execute within the context
trast, VxWorks does not enforce any strict driver of the kernel.
model or memory access protection.3 The software A vendor-provided driver tends to have a top-level
provided by many hardware vendors makes no dis- API that is intended for use by developers in cre-
tinctions between hardware access routines, applica- ating a working system. Developers will expect to
tion support libraries, or system service tasks. The use this API when developing their networking ap-
lack of such distinctions is incompatible with the pro-


plications. However, in this model the actual con-
tected environment provided by Linux . As Fig- glomerate driver will reside in a separate address
ure 1 illustrates, the result is a software conglomer- space from the rest of the application. This makes
ate that can be very difficult to integrate effectively


traditional linking impossible. One solution is to
into a Linux system. provide a library for the application to link against
that mimics the published API of the conglomerate
driver. Underneath, this doppleganger 4 library will
communicate to the kernel-resident code by estab-
lished means, such as an ioctl() call to an actual

Linux device driver. Figure 2 illustrates the imple-

mentation of this approach.

FIGURE 1: Typical conglomerate driver

2 The Naı̈ve Approach

Linux is a protected system. The Linux kernel

uses the memory management hardware available on
modern CPUs not only to protect tasks from each
other but also to protect the kernel from errant user-
land tasks. Furthermore, in order to ensure system
integrity the kernel allows only “root”-owned pro-
cesses to access physical address ranges. Even those FIGURE 2: The Naı̈ve Approach
processes are required to use specific programming
interfaces in order to map physical address ranges In practice, this approach works reasonably well to
into a process’s virtual address space. Other types provide access to a conglomerate driver’s functional-
of hardware accesses are restricted entirely to code ity from userland applications. However, the cost of
running in the kernel’s context. This group includes switching between user and kernel contexts can accu-
interrupt handlers, DMA transfers, and any other mulate to be quite large if driver APIs are called re-
hardware access that is not memory-mapped. These peatedly or in rapid succession over a period of time.
protections have great value, but they serve to make Also, many complex conglomerate drivers make use
it very difficult to control hardware of any complex- of application callbacks. Since there is no simple
ity entirely from userland. way to call userland functions from within the ker-
As asserted earlier, modern networking hardware nel’s context, this method either precludes the use
tends to be both highly functional and highly com- of callbacks or requires one to invent a means of
plex. Simple memory-mapped register access is gen- simulating them appropriately for the given driver
3 Wind River Systems, Inc. offers a product called VxWorks AE which does offer a protection domain model. However,
use of VxWorks AE is not as prevalent as is the use of its predecessor. Therefore, all references to VxWorks within this
document should be interpreted to refer to the original VxWorks .
4 In German folklore, a doppleganger is the ghostly double of a living person—especially one that haunts its fleshy counterpart.
and application. Similarly, other means of applica- ware practices, it can cause problems due to non-
tion ←→ driver communication that rely on shared preemptibility in the kernel’s context, and it puts
memory may only be available with some difficulty, extra code into the kernel’s context where it is more
if at all. difficult to debug and more dangerous in the event
Perhaps the biggest problem with this approach lies of failure. To the näive, this approach seems like
a great idea. However, experience shows that it is

with the fact that standard Linux kernels are not

preemptible. While not generally a problem with rarely satisfactory. There must be a better way.
properly written code, this can be quite a big prob-
lem with code that was not written with this environ-
ment in mind. Such software will generally not make 3 A Better Strategy
any effort to bound its own execution time. Thus,
improperly written code will very easily starve all of

There are potentially many reasons why one may
the other processes in a system. Since VxWorks wish to preserve the code a vendor provides as a sin-
allows any or all tasks to be preemptible, vendor- gle conglomerate. This is particularly true if the con-
provided conglomerate drivers generally fall into this glomerate driver is an existing piece of developed and
category of improperly written code. tested software. It has already been demonstrated
Note that non-preemptibility is a problem even if that it is undesirable to run a conglomerate driver
processes execute at a very low priority. Even though in the kernel’s context. However, it is also possi-
threads executing in the kernel’s context receive no ble to run most or all of a conglomerate driver in a
special consideration when it comes to scheduling, user’s context. This strategy can produce a work-
once they are scheduled they will continue to hold the able system without major changes to an existing
CPU until they voluntarily relinquish it. Since it is conglomerate of software.

presumable that every thread will be scheduled even- As discussed earlier, the Linux kernel restricts
tually, this is a potential problem with any thread ex- many kinds of hardware accesses to only code run-
ecuting within the kernel’s context. There is a well- ning in the kernel’s context. Complex networking
known kernel patch maintained by Robert Love that hardware will generally require more than simple
will tend to solve this particular problem.[1] How- memory-mapped register accesses in order to func-
ever, not everyone will be using this patch at the tion properly. If the body of an existing conglomer-
present time or even in the near future.5 ate driver is to reside in a user’s context, some means
A less tangible problem with this approach relates must be devised to account for these non-memory-
to the amount of code running in the kernel’s con- mapped hardware accesses. The general approach
text. Vendor-provided conglomerate drivers tend to to providing this type of access is to implement a
be rather large. These conglomerate drivers tend small formal device driver that executes in the ker-
to provide a lot of functionality and are often tar- nel’s context. This small driver provides the required
geted at multiple devices within a family of hard- functionality (e.g. DMA transfer setup) through a
ware. It is generally accepted that the number of small suite of custom ioctl() calls. As depicted in
bugs in a piece of software is proportional to the size Figure 3, generally only a few small changes to the
of that software. So, the potential for bugs in driver conglomerate driver are required to make this strat-
code is fairly large. By placing code in the kernel’s egy work.
context, one makes it more difficult to debug that
code and makes the consequences of bugs in that
code greater than if the same code were running in
a user’s context. By itself, this should be enough

to dissuade the experienced Linux developer from

pursuing this approach.

The Linux kernel reserves most direct hardware ac-

cess to code running in the kernel’s context. Vendor-
provided conglomerate drivers do nothing to seg-
regate hardware access routines from library rou-
tines and service tasks. The simple solution seems
to be to package the conglomerate driver in a way
that lets it run in the kernel’s context. Unfortu-
nately, this approach is quite expensive in terms of
context switches, it precludes certain common soft- FIGURE 3: A Better Strategy
5 This paper will avoid advocating the use of kernel patches that are currently outside the mainstream.
This strategy avoids many of the problems found the diagram. So, this strategy is somewhat “upside-
with the previous approach. No special “shim” or down” with regard to accepted norms. Of course,
doppleganger library is required to allow the appli- this is not a technical issue. However, the “upside-
cation to access the conglomerate driver, and call- down” nature of this strategy can lead to much con-
backs, two-way sharing of pointers, and other com- fusion. Confusion leads to errors, errors lead to bugs,
mon software techniques are easily available to the and bugs lead to failures.
body of the conglomerate driver’s code. Thread pre- As we have already seen, running a conglomerate
emption is not a problem since code running in a driver in the kernel’s context has many problems.
user’s context is always preemptible. Debugging is Still, it is very tempting to preserve existing code as
possible using standard means,6 and crashes do not a single entity. With slightly more effort, it is possi-
automatically crash the whole system. In general, ble to run a mostly intact conglomerate driver in a
this is a much better strategy than the previously user’s context while still producing a workable sys-
described approach. tem. However, this strategy is not without costs.
However, this strategy is not a panacea. It may be Context switching is expensive, and this strategy
very difficult to isolate those services that must run requires more context switching than is desirable.
in the kernel’s context from the rest of the conglom- Also, this strategy is “upside-down” when compared
erate driver. Presuming that the conglomerate driver to how things are normally done. Confusion leads to
was not architected with this strategy in mind, it is failures and is best avoided if at all possible. While
likely that any simple partitioning will be less than this strategy is better than the first approach, it
optimal. It is very likely that some hardware accesses seems clear that another solution must be found.
will occur more frequently than truly required and
that others will be a small part of a larger atomic op-
eration that has no other need to run in the kernel’s 4 The Best Solution
context. Hence, this strategy is still prone to exces-
sive amounts of expensive context switching or non- The discussion so far has enumerated and demon-
preemptible execution in the kernel’s context. This strated the problems with maintaining a conglomer-

situation is exacerbated if the networking hardware

ate driver as a single entity in a Linux environment.
needs to be accessed through a Linux net driver.7


The cost of excessive context switching becomes sig-
The Linux networking stack is part of the kernel. nificant, and the costs associated with debugging dif-
As such it executes within the kernel’s context. Con-


ficulties, system complexity, and design unconven-
sequently, Linux net drivers execute within the ker- tionality are significant as well. While there may be
nel’s context in order to provide access to network-


defensible reasons for attempting to do so, using a
ing hardware through standard Linux networking conglomerate as-is can be quite expensive. It seems
APIs. If the current strategy is in use and the net-

likely that the approaches described so far introduce
working hardware needs an associated Linux net as many problems as they solve.
driver, then either: a) the net driver needs to have So, what is the correct solution? All of the problems
a means of synchronizing hardware access between discussed so far stem from an architectural conflict:

itself running in the kernel’s context and the con- the Linux kernel restricts most hardware accesses
glomerate driver running in a user’s context; or, b) to code running in the kernel’s context; and, con-
the net driver needs to be able to exchange incoming glomerate drivers do not adequately separate hard-
and outgoing frames with the conglomerate driver ware access code from other conglomerate compo-
running in a user’s context. Either of these possibil- nents. The deceptively simple answer is that one

ities is expensive with regard to context switches. In must remove this conflict between the Linux ar-
particular, option b) requires as many as three sepa- chitecture and the architecture of the conglomerate
rate context switches for any given frame received at driver. This means dismantling the conglomerate
the CPU. Obviously, this is more context switching driver and reconstructing it in a form compatible

than is desirable. with the Linux driver model.

This strategy also has an intangible problem. Most How does one begin? As was discussed earlier, the
software architectural diagrams show the operating typical conglomerate driver contains not only basic
system and device drivers at the bottom and any hardware access routines, but also application sup-
applications at the top. With this strategy, the ac- port library routines and even services tasks. In
tual hardware driver lives somewhere in the middle of many cases the hardware access routines will need
6 Unlike code running in the kernel’s context, code running in a user’s context is easily debugged using software debuggers

and other common debugging means.

7 Use of the Sockets API to communicate over networking hardware will require the implementation of a Linux net driver
for that hardware.


to run within the kernel’s context. However, the Linux -specific code to the “other” library or by cre-
other components almost always need to run in a ating a doppleganger library that provides the same
user’s context. The task at hand is to separate symbol definitions as the hardware access library but

the two application-oriented components from the uses Linux -specific code to drive the hardware ac-
system-oriented hardware access component. The cess code running in the kernel’s context. The end
result should look something like the depiction in result is a functional solution in nearly every case.
Figure 4 below. In most cases, however, this solution is not for the
faint hearted. Once a conglomerate driver has ex-
isted for some time, the lack of architectural disci-
pline amongst conglomerate driver developers tends
to lead to a spaghetti-like mess of hardware ac-
cesses within service tasks, application callbacks
from within interrupt handlers, and other question-
able practices that can be difficult to translate to

the Linux environment. Sorting-out such a tangle

of software can require a great deal of time, energy,
and motivation. An intimate knowledge of both the
conglomerate driver and the hardware it drives is
required as well. Still, in the author’s opinion the re-
sulting “purified” driver is well worth the cost. That
is especially true if quality, performance, or main-

tainability are desirable attributes for one’s Linux -

FIGURE 4: The Best Solution based product.8
One starts by sorting the pieces of the conglomerate Unfortunately, this solution has a greater upfront
driver into a hardware access group and an “other” cost than the approaches discussed so far. One must
group. Sometimes this will be a straightforward pro- be willing to dig deeply into code that most devel-
cess, but often it will require quite a bit of code anal- opers consider a “black box.” This can require a
ysis. At this point it will likely be necessary to be- large investment of development time and resources.
gin breaking existing files into hardware access and Properly separating a conglomerate driver into hard-
“other” versions of the original files. Also, many ex- ware access and “other” libraries requires not only
isting functions will need to be modified either to an intimate knowledge of the conglomerate driver

remove unnecessary hardware accesses or to formal- and the hardware it drives, but also enough Linux
ize those accesses in a way that is supportable in knowledge and experience to accomplish the task.

the Linux environment. The end result should be Covering both of those competencies can be diffi-
two distinct object libraries. The hardware access cult, and the investment of time and resources can
library should have no dependencies on the “other” be expensive. However, the payoff is in better per-
library, while the “other” library should at most re- formance, easier debugging, and greater stability. In
quire access to the functions defined in the hardware the long run this is the cheapest of the alternatives
access library. In particular, there should be abso- discussed so far.
lutely no use of shared global variables between the
two libraries and no calling of functions defined in the
“other” library from functions defined in the hard- 5 Driver Architecture 101
ware access library.
With the former conglomerate driver broken into So far, the discussion has been about how to make

these two distinct libraries, one has the basic ingre- use of an existing conglomerate driver in a Linux
dients necessary for implementing this solution. The environment. This discussion has been pertinent be-

hardware access library can be packaged to run in the

cause the prevalent use of VxWorks in the em-
Linux kernel’s context using a wrapper that im-

bedded networking arena has fostered the prolifera-
plements one or more of the standard Linux driver tion of poorly architected conglomerate drivers. But
interfaces. The “other” library is used for linking drivers would not require difficult adaptation for

against applications requiring access to the hardware Linux if they were developed properly in the first
controlled by the former conglomerate driver. The place! A set of driver libraries that are architected

interface between the hardware access library and in a Linux -friendly way should work equally well

the “other” library is implemented either by adding with VxWorks or any other operating system.
8 Hopefully this applies to everyone. . .
How does one structure a driver library so that it

vendor may want to make its hardware as attrac-

works equally well with both Linux and operat-

tive as possible to those using Linux . How might
ing systems like VxWorks ? A proper driver archi- a vendor do this? A good first step is to ensure that

tecture should provide for a hardware access layer. the vendor’s driver architecture is Linux -friendly.
This layer should encapsulate all of the atomic oper- A vendor might do this either by purifying what ex-
ations one would want to perform on the hardware. ists now or by making sure that the driver is archi-
Once defined, this layer should not be violated for tected correctly in the first place. As well, the vendor
any reason whatsoever. Also, any dependencies on should conduct the exercise of running the driver un-

the hardware access must be “one-way”—calls from der Linux in order to ensure that the driver actu-

the hardware access layer to higher layers are specif-

ally works in the Linux environment. These exer-

ically not allowed. In Linux , the hardware access cises will serve to guarantee that the Linux market
layer will be run within the kernel’s context. In an is available to the vendor’s hardware.
unprotected operating system, this layer will be just Michael Tiemann,9 CTO of Red Hat, Inc., has sug-

another part of the driver library. gested that Linux should be viewed as a platform
A driver architected in this way should work reason- rather than simply as an operating system.[3] What
ably well either in a protected operating system like


does this mean for a hardware vendor? For one thing,
Linux or in an unprotected operating system like


this means that the vendor should implement stan-

VxWorks . A driver library architected in this fash- dard Linux device drivers as appropriate for the
ion from the beginning should introduce neither ex- capabilities of the vendor’s devices. Moreover, a wise
tra runtime overhead nor extra complexity in either vendor will determine how to take advantage of ex-

a protected or unprotected environment. Further- isting Linux functionality10 as well as how to ex-
more, this basic design framework is more in line with tend it in ways that are appropriate for the vendor’s
modern software design practices and should lead to hardware. In this way, the vendor’s hardware be-
fundamentally better software with fewer bugs and comes accessible to any software already available for

better stability. There really is no excuse for not fol- Linux that is related to the functions provided by
lowing this basic design framework when architecting the vendor’s hardware. It should be clear that this
a new hardware driver. would benefit both the hardware vendor and the sys-

As demonstrated, there are a number of ways to tem integrator using Linux .

approach the problem of adapting an existing con-

That Linux will be a significant embedded operat-

glomerate driver to a Linux environment. However, ing system for at least the near future seems certain.
this problem should not exist in the first place! A

Linux users are more likely to use hardware with
driver library architected in a Linux -friendly way

good Linux support. This is especially true since

will work equally well in either a protected environ-

hardware that is well supported by Linux is un-
ment like Linux or an unprotected environment


likely to require much custom software to make use
like VxWorks . And, due to the better design prin-

of it. The benefits of embracing Linux should be

ciples in use, a driver architected in such a way will obvious.
tend to be better software. Clearly this is the way
hardware driver libraries should be done.


7 Conclusion
6 Embracing Linux

For some time, VxWorks has dominated the mar-
Not that long ago, Linux was an operating system ket for embedded networking operating systems.
only for desktop and server computers. However,


This market dominance has allowed networking
in the past few years Linux has become quite a hardware vendors to concentrate only on supporting

significant player in the world of embedded systems VxWorks with the software they provide to drive
and particularly in the embedded networking arena.


their hardware. Unfortunately, the unprotected na-

Some have suggested that Linux will become the ture of VxWorks has fostered the development of
dominant operating system in the embedded systems

driver software that is difficult to use with a pro-

market.[2] What is certain is that Linux will con- tected operating system like Linux . There are ways
tinue to be a significant player in the embedded net- to use a more-or-less intact conglomerate driver with

working arena for the near future.

Linux , but the results are generally sub-optimal.
Given the growing popularity of Linux in the em- Following simple rules, it is possible to re-architect
bedded arena, it seems reasonable that a hardware an existing conglomerate driver in order to make
9 Michael Tiemann was a co-founder of Cygnus, the first company with a business built around Open Source technologies.
10 Linux has extensive support for networking e.g. routing, bridging, VLANs, LAGs, etc.
 References

it more compatible with Linux -like operating sys-

tems. The results achieved by using a “purified”
driver can be quite satisfactory. However, conduct- [1] Andrews, Jeremy, 2001, Interview: Robert Love,
ing such a purification requires a larger commitment KernelTrap, http://kerneltrap.org/. . .
of resources than the average organization is willing . . . /node.php?id=1.
to make. As usual, much better results are achieved
[2] Sprackland, Teri, 2002, Embedded Market Ac-
by doing the job right the first time.
cepts Linux, Nikkei Electronics Asia,
A properly architected driver will work equally well http://www.asiabiztech.com/nea/200205/. . .
with either protected or unprotected operating sys- . . . /cmpu 183147.html.
tems. There should be neither a runtime penalty nor
a development penalty for doing things correctly in [3] Tiemann, Michael, 2002, How Linux Will Revolu-
the first place. Such a driver will open a much wider tionize the Embedded Market, LinuxDevices.com,
market for the hardware it drives. Why would one http://www.linuxdevices.com/articles/. . .
build a driver any other way? . . . /AT7248149889.html.