Cooperative Hybrid Objects Sensor Networks

Contract Number: INFSO-ICT-224327

CHOSeN

Project Report

DELIVERABLE NO D4.3.3

DELIVERABLE TITLE Application Middleware and Gateway Software

AUTHOR Matthias Berning (KIT)

Dawud Gordon (KIT)

DISCLOSURE LEVEL Consortium confidential

VERSION V1.0 final

Copyright CHOSEN Consortium

2008-2011
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

Table of Contents

1 Abstract.............................................................................................................................................3
2 Concepts............................................................................................................................................3
2.1 Software Architecture of the Middleware Node.................................................. 4
2.2 Distributed Middleware .......................................................................................................... 4
3 Hardware Reference ....................................................................................................................5
3.1 Middleware Node ......................................................................................................................... 5
3.2 CHOSeN RF Interface ............................................................................................................... 6
4 Middleware Software...................................................................................................................6
4.1 Program Core.................................................................................................................................. 6
4.2 FAT Driver.......................................................................................................................................... 6
4.3 Ethernet Bridge to Aeronautics Application Software .............................. 6
4.4 Graceful Reboot ............................................................................................................................ 7
4.5 BASIC Interpreter....................................................................................................................... 8
4.6 Web Server ....................................................................................................................................... 8
4.7 Subscription Management & QoS Control................................................................ 9
5 Distributed Middleware..............................................................................................................9
5.1 Distributed Command Protocol ........................................................................................ 9
6 Conclusion.....................................................................................................................................11

Copyright CHOSEN Consortium 2008-2011

Page 2
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

1 Abstract
This report describes the current implementation and integration status of the CHOSeN
Middleware Application for the system prototype. The task of the middleware is partitioned into
special hardware (the Middleware Node), Middleware Software (on the Middleware Node) and
Distributed Middleware (on all CHOSeN Nodes). This concept is further explained in the next
chapter.

The first part focuses on the Middleware Node, which provides gateway functionality between
the onboard networks in the aeronautic and automotive domain and the CHOSeN network. In
addition it is host to the central part of the Middleware software, which is responsible for data
stream management. The second part describes all software modules of the Middleware and
Distributed Software that were already implemented on the hardware. Most of these modules
are drivers for the hardware interfaces and network protocols. The last chapter delivers an
overview on the implemented part of the Distributed Command Protocol. This protocol is used
for communication between the Middleware Node and the Distributed Middleware.

2 Concepts
This chapter will give a short overview on the concepts of the CHOSeN Middleware Application
and protocols. As depicted in Figure 1, the middleware solution for the CHOSeN project is a
distributed embedded software/hardware application running on dedicated node extensions.
The middleware application is distributed over these middleware nodes, but is also distributed
to the wireless sensor nodes where smaller code-portions implement the necessary operations
for the middleware application.

Figure 1: The CHOSeN Network Stack

The middleware nodes will interface with the wireless sensor network under development within
the project, as well as the preexisting vehicle communication systems of the respective
industrial partner. In this fashion they also serve as network bridges within the automobile,
allowing seamless communication between the different modules of the CHOSeN aeronautic

Copyright CHOSEN Consortium 2008-2011

Page 3
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

and automotive applications. For more details, please refer to the complete specification in the
CHOSeN Deliverable 2.4 Software and Middleware System Integration.

2.1 Software Architecture of the Middleware Node

In Figure 2 the software architecture and hardware/software interfaces of the middleware node
platform are shown. The middleware uses a FAT32 driver over the SPI to interface with the
microSD card. The communication is a 4 wire protocol, which is dictated by the SD Card SPI
Mode specification and allows for standard file-based read and write operations.

The middleware platform is present in several instances per demonstrator, where each
middleware node represents the bridge between a wireless sensor node cluster and the system
bus, and also a distributed instance of the middleware software. Of these instances, one of
these boards will provide access to the CRF ALIX processing board which does not interface
with the bus, but only has access to the system through a middleware node directly. This
connection is implemented serially and bridges packets in a proprietary CHOSeN data format.

Figure 2: Middleware Node Software Architecture

The middleware communicates with a wireless cluster using a chosen receiver which is
connected to the board serially over UART. The transceiver functions as the cluster head for
the cluster and dictates the network parameters based on input from the middleware. For this
reason two separate communication interfaces are necessary over the serial line, one for
forwarding data packets in both directions and another for configuring the wireless
communication stack using the configuration interface on the node (See Deliverable 2.4
Software and Middleware System Integration for more information).
Finally, these middleware nodes provide an interface to the on-board bus systems of the two
demonstrators. Although these are shown as being a single unit in Figure 2, they are actually
two separate hardware modules, one for each of the buses respectively. Each bus interface is
also supported by a software driver, providing a layer of abstraction to the middleware over a
bus-independent interface.

2.2 Distributed Middleware

The portion of the middleware running on each node of the sensor network is called the
Distributed Middleware. Its main duties are management of data subscriptions and publications,

Copyright CHOSEN Consortium 2008-2011

Page 4
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

and the configuration of the lower communication layers. For this purpose, the Distributed
Middleware will exchange information with the main middleware service using the wireless
network and proprietary communication protocol. Therefore only the generic UART interface
driver is needed for communication with and over the CHOSeN Transceiver.

3 Hardware Reference
This chapter will give a short overview on the hardware platform used for the central
middleware nodes. The nodes will host the Middleware Software Application and provide
gateway functionality between the CHOSeN RF Network and the network to the central
processing unit in the plane and the car respectively.

Figure 3: Picture of the Middleware Node

3.1 Middleware Node

The platform depicted in Figure 3 is based on a PIC32MX340F256H processor which runs the
middleware application and an embedded HTTP web server. All components of the node can
be clearly identified on the figure above and in the board-layout in Figure 4. Starting from the
left side, the USB-Connector is used in combination with the voltage regulator as power-supply
for this prototype. This type of connector can be found on many embedded platforms, but can
also be easily replaced to fit the on-board power system. The next component is the RJ45
connector currently used to provide Ethernet access for the node. On the right side of the
connector are the ENC28J60 Ethernet controller and the embedded microcontroller which
implements the network stack according to IEEE 802.3. This can be quickly adapted to
interface the ADFX bus used in aeronautic applications. Next to the microcontroller, which is
also runs the CHOSeN Middleware Application, is the microSD-card interface. The flash
memory provides persistent storage for sensor data, configuration files and user interfaces
based on HTML websites. The last major part on the board is CONAN connector on the right,
which provides connectivity to the RF interface. This will be provided by the new CHOSeN
Transceiver, as described in the next chapter. For prototyping purposes, an Akiba node is used
as RF frontend.

The specific drivers needed for all parts of the board have been implemented for the current
version of the hardware. Further work will focus on the integration of the new RF frontend and
the addition of a CAN Interface for automotive application.

Figure 4: Board Layout of the current Middleware Node

Copyright CHOSEN Consortium 2008-2011

Page 5
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

3.2 CHOSeN RF Interface

The wireless sensor network is accessed by using the scalable baseband transceiver, which is
currently developed in the scope of the CHOSeN project. The new module, which is prototyped
on an FPGA board at moment, will provide a protocol processor for new MAC protocols, an
autonomous power management unit, a digital baseband module and the RF frontend. The
transceiver will be plugged onto the gateway using a modular connector. A driver for the device
was not implemented yet, due to the early state of the module.

4 Middleware Software
The Middleware Software running on the middleware node is based on different distinct
modules, grouped around the main program core. The functionality of the core modules is
described in the next chapters.

4.1 Program Core

The main code of the CHOSeN gateway node contains the main() function which in turn
contains the main program loop. This loop iterates continuously throughout the entire program
execution, triggering code from the different software modules at regular or irregular intervals.
This is the piece of code that is called at program start, which then coordinates the rest of the
software modules discussed in this chapter. Each module has an initialization function to be
called from the main program before the program loop starts. Once the program loop has
begun executing, all function calls to other software modules originate from this code or as a
result of an interrupt handled by this code.

The user defined interrupt handling functions are also found in this code, the processor
supports prioritized interrupts, with either high or low priorities. The interrupt handling functions
for both interrupt priorities analyze the interrupt to determine its origin before passing execution
to the specific modules needed to process that interrupt.

General hardware initialization occurs before the main loop within the main program code. The
software-programmable oscillator is set to 41.666667MHz. Global variables are also initialized,
and hardware modules are configured.

4.2 FAT Driver

The software for the CHOSeN gateway provides code for accessing SD, respectively SDHC
(high capacity), cards over SPI. Intuitively, other compatible standards for cards can also be
driven by the software, e.g. microSD or miniSD. The current hardware of the node has got a
slot for microSD cards. The media might be using FAT12, FAT16 or FAT32 as file system; the
code supports all three of them. The code also provides functionality to disconnect a plugged in
card during operation.

The file system provides easy access to a persistent storage, which is especially needed for the
web server described in chapter 4.6. Apart from that, the SD card can also be used as
intermediate buffer for all data that cannot be delivered immediately.

4.3 Ethernet Bridge to Aeronautics Application Software

Microchip offers an IEEE 802.3-based (Ethernet), or IEEE 802.11b-based (WiFi), IPv4
networking stack, which is detailed in Microchips documentation. The Ethernet part is
implemented in software but also external networking controllers, e.g. the ENC28J60 or its
successors ENC424J600/ENC624J600, can be incorporated as replacements for the software-
based Ethernet part. We have aligned the other software packages and our own code around
the TCP/IP stack in a modular way to be able to quickly upgrade to new vanilla releases from
Copyright CHOSEN Consortium 2008-2011
Page 6
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

Microchip. The stack is essentially the heart of the bridge between the CHOSeN middleware
node and the aircraft application software based on AFDX. To reduce the size of the binary
loaded onto the PIC32 only the features described in the following list, are compiled in. The full
feature list is available on the Microchip website.

PHY&MAC:
The lowest two layers (ISO/OSI layers 1 and 2) of the network stack provide Ethernet
according to IEEE 802.3. In the current configuration of the gateway, these layers are
provided by the ENC28J60 Ethernet controller instead of software running on the PIC32MX.

ARP & Announce:

The ARP protocol is running on top of Ethernet. The ARP protocol is needed to resolve the
MAC address based on the IP address in the network. In addition, Microchip provides an
implementation of their Announce protocol. The Announce protocol uses Ethernet
broadcasts to announce the gateways IP address and NetBIOS name into the local network
segment on a regular basis.

IPv4 & ICMP:

The current Ethernet stack supports only IP in version 4 in the networking layer and its
related protocols e.g. ICMP on the same layer or TCP on the transportation layer. The IP in
the current version 6 is not needed for this application

TCP & UDP

On the transport layer (ISO/OSI layer 4) Microchips Ethernet stack supports either unreliable
datagram transmission using UDP or reliable message transmission using TCP or SSL. For
the CHOSeN gateway, only UDP and TCP are implemented.

HTTP & SNTP:

On the application layer (ISO/OSI layer 5 to 7) the gateway supports HTTP and SNTP. The
former provides the basis for the web server and all communication over the network. The
latter is used for time synchronization between all nodes in the Ethernet network. The time
resolution provided by SNTP is 232 second (~0.25 ns). The timeserver needs to be provided
by the central processing unit in the network, e.g. the OBU in the car controller network.

Settings & Operation:

The TCP code listens on port 80 for new TCP socket connection requests from clients. Upon
receiving a new connection request, a connection is opened on the condition that there are
less than MAX_HTTP_CONNECTIONS already open. If this is not the case, the TCP socket
connection is refused. The HTTP connection limit is currently set to 3 connections. Once a
socket is established, the connection is passed to the HTTP code for processing. After the
file or the HTTP status is returned the connection is closed. Since the current website does
not require a stay-alive HTTP connection and connections are closed immediately after
responding to the request, only in rare cases would HTTP connections be denied because
the maximum number of connections has already been reached.

4.4 Graceful Reboot

This small module allows the software stack to be rebooted in a safe manner. The module
primarily cares for preserving file system consistency on an attached microSD media. The
reasoning is the following: when the gateway is operating, there is probably at least one file
handle constantly open. Each open file is associated to a file buffer in main memory of the
bridge. These file buffers are not synchronized most of the time. Therefore the Graceful Reboot
code takes care to first flush file buffers on the SD media and close all open file handles before
rebooting the Middleware node.

Copyright CHOSEN Consortium 2008-2011

Page 7
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

4.5 BASIC Interpreter

For simple queries to the sensor network, a BASIC is implemented in the software of the
Middleware node. It can be accessed directly through the Client API, or more intuitively, trough
the console provided by the web server (see next chapter). The interpreters software stack is
fully integrated into the software stack of the gateway as an optional module. The interpreter
can be configured and selected for in-compilation at compile time of the software stack.
Currently, the code execution unit of the interpreter only supports unstructured programming
analogue to C64 BASIC. On the other hand, the code analysis unit largely supports all three
programming paradigms. The code analysis unit consists of three sub-modules, namely the
scanner, the parser and the semantic code analysis. The scanner and parser fully support all
three paradigms. The semantic code analysis currently supports only unstructured
programming.

Figure 5: Structure of the BASIC Interpreter

The node offers a Client API for different interfaces to the interpreter. All commands that cannot
be handled locally by the client are sent to the gateway. The API is defined by a set of Common
Gateway Interface (CGI) files on the web server. Clients can issue HTTP POST requests to the
web server to send data to it and affect its internal state, whereby the Uniform Resource
Locator (URL) in each request has to point to the proper CGI handler file for the request. Any
data given to the web server have to be properly encoded as parameters in the POST request
according to HTTP. The bridge acts as requested in the following and afterwards sends any
output data back via an HTTP response. For simplicity and runtime performance, the
interpreter's output data is handed back in plain format instead of facilitating a data description
language, e.g. XML, JSON or YAML. The client may also issue HTTP GET requests to the
Client API to only retrieve a resource specified by the included URL. With regard to the POST
and GET request semantics, the Client API conforms to REST semantics put on top of HTTP.
When abstracting from the HTTP-based Client API to web services, this concept is commonly
called RESTful Web Services.

4.6 Web Server

The web server is based on the Microchip implementation but was heavily rewritten to fit the
CHOSeN platform. All references to a file within the code were rewritten or removed, including
file pointers, placeholders and byte counters. The new file references point to the websites new
storage location on the SD card using the file level operations provided by the FAT system
code. As of the current status of the project, only HTTP GET is supported by the web server,
allowing a web client to download a static file from the SD card to his or her local workstation. In
the case that no card is plugged in, the web server returns an error that the requested source
could not be found.

GUI for Debugging:

The debugging GUI consists of a website delivered by the web server, which provides a
console for direct or script-based programming, based on the BASIC interpreter. The
Copyright CHOSEN Consortium 2008-2011
Page 8
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

console is implemented in a single-threaded Java Applet which connects to the bridge via
HTTP on top of TCP/IP. This means that client and bridge can be remotely connected when
both parties have access (either direct or indirect, e.g. via Network Address Translation
(NAT), a proxy or a Virtual Private Network (VPN) solution) to the Internet. The user interface
of the console is depicted in Figure 6.

Figure 6: Picture of the Debugging Console

The BASICs console's core is a parser based on the JavaCC parser generator which
analyses console input based on a context-free grammar. The console supports issuing of
single statements or a complete BASIC program which is kept in local memory of the applet.
Programs are immediately launched after transmission to the CHOSeN gateway and are
subsequently executed until they terminate. Any output of the interpreter is immediately
transferred back to the console and displayed on the user interface.

4.7 Subscription Management & QoS Control

The high-level function of the middleware application will be the management of the
communication in the wireless sensor network. For this purpose, a subscription module
provides an interface for application software, to request and publish sensor data in the
network. To prioritize the data streams, another module provides a schedule for packet
transmissions and manages the network parameters. To fulfill this task, it will also initialize and
configure the Distributed Software via the RF channel.

These services provided by the middleware, as defined in CHOSeN Deliverable 2.4, are
currently in testing state. The next step will be the implementation and integration into the given
platform.

5 Distributed Middleware
The Distributed Middleware will run on the sensor nodes in the network and will communicate
with the central portion of the software to manage the subscription and publication of the sensor
values, as well as the optimization of the network parameters. Therefore a specialized
command protocol is needed for communication between the two software parts.

5.1 Distributed Command Protocol

The distributed command protocol is based on the STuples concept detailed in the CHOSeN
Deliverable 3.1 Interface Specification. This means that the messages interchanged
between the distributed and the middleware node, consist of several concatenated parts taken
from a set of predefined data structures. It should be noted that all these packets are sent as
data packets, although information containing the MAC and network parameters are
transmitted. Some of the information can be processed by lower layers in the protocol stack, but
is always transferred to the middleware layer.
Copyright CHOSEN Consortium 2008-2011
Page 9
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

Message Data Types:

There are three different types of messages defined for the Middleware Protocol. They are
used to identify the sender of a message. The platform implements one message type for
sensor nodes, one type for the middleware node and one for external sources. These
external sources e.g. the application running on the on-board unit can request information
from the Middleware. The resulting requests are relayed by the Middleware Node to the
sensor network and the answer is transmitted back to the application.

Internal Context Data Types:

Internal context data types describe values and properties which are internal to the wireless
sensor network. This can be context information, or values from within the system, but also
values that are intended for other parts of the system. These values contain the timestamp of
the sending node, which is used for synchronization of clocks and sensor data. The node
reliability is an indicator ranging from 0 to 100 representing the number of previous
messages that were transmitted successfully for a certain node. When this tuple is contained
in a message originating from a sensor node, the value indicates the link quality to the
Middleware Node. The last three values contain information concerning the battery runtime
for energy constrained nodes. As an universal indicator for the battery charge level, the
battery voltage of each node is transmitted. Based on this information and the mean current
draw, each node can calculate its remaining battery lifetime. If this value drops below a
programmable threshold (e.g. 24h), the nodes can issue a message containing the battery
alert data type.

Control Data Types:

Control tuples are used to deliver instructions to the networked nodes. These instructions
can range from direct action requests to reconfiguration parameters. The set signal strength
data type is used to configure the power at which a node transmits. This is heavily
dependent on the hardware being used, as the hardware must provide a level of
configurability which supports the protocol. In the current state of the CHOSeN project, this
hardware feature is not yet implemented in the Middleware Software. Each node can send a
message containing the request timestamp tuple to get the current time on the target. This is
especially useful to synchronize the local time with the network time. The protocol offers two
possibilities to send nodes into sleep state. The first one is the sleep period, to set a specific
idle time. On the other hand the node can be ordered to sleep until a Wake-On-Radio event
occurs using the sleep WOR command. An acknowledgement or short ACK tuple is
implemented to confirm the reception of the previous command. The node will be equipped
with LEDs as simple physical interface. Although this feature probably wont be used in an
energy constrained productive use, a blink LED command is available for development and
debugging. All nodes in the network will be organized in groups for management and
addressing. Therefore several commands deal with the assignment of the 2 bit group IDs.
The request group ID tuple with length 0 is used to poll a node for a list of all groups it is
member of. The answer uses the same tuple with a length of one byte. The commands
subscribe group ID and unsubscribe group ID are used to assign the nodes to these groups.
Two additional tuples are used to address the command if they are contained in the
message. The names address node and address group speak for themselves in this context.
The last implemented tuple is named application start and is sent from the application to
request initialization parameters from the network. The answer to this message is application
specific but will be handled by the Middleware Software.

The tuples specified above are composed to messages. Their order is not fixed, exept for the
message type which should go first in the set. In general all tuples can be used in all message
types, although they may have different meanings depending on the source. This includes the
possibility that a piece of data can be ignored if there is no behavior defined.

Copyright CHOSEN Consortium 2008-2011

Page 10
CHOSeN Consortium confidential
Deliverable: D4.3.3 Application Middleware and Gateway Software V1.0

6 Conclusion
This document provided a functional overview and a status report on the implementation of the
Middleware and its three major components. These are the special hardware (the Middleware
Node), the Middleware Software (on the Middleware Node) and the Distributed Middleware (on
all CHOSeN Nodes).

So far the hardware for the Middlware Node is almost finished concerning the essential parts
and the overall layout. These are the central processing unit, Ethernet and wireless network
interfaces, mass storage and power supply. The next revision will only add minor points,
including connectors for the platform specific communication busses. This will also influence the
further development of the firmware running on this node, called the Middleware Software. The
current implementation includes drivers for all peripherals on the board, as well as a web server
for HTTP communication and debugging. Mass storage is also included via file system drivers
for the microSD-card interface. The counterpart to the central Middleware Software is the
Distributed Middleware installed on all nodes in the network. A specialized Distributed
Command Protocol was designed and implemented. The protocol is used to disseminate the
network and node status information in the network, as well as issuing commands to the
Distributed Middleware. The details for the different parts can be found in the corresponding
chapters of this document. Further information on the overall concept is described in the
associated deliverables of the CHOSeN Project.

Copyright CHOSEN Consortium 2008-2011

Page 11