DOC TOR A L T H E S I S

Department of Computer Science, Electrical and Space Engineering

Architectural Design Principles

Hasan Derhamy Architectural Design Principles For Industrial Internet of Things

EISLAB

ISSN 1402-1544
For Industrial Internet of Things
ISBN 978-91-7790-128-0 (print)
ISBN 978-91-7790-129-7 (pdf)

Luleå University of Technology 2018

Hasan Derhamy

Industrial Electronics
Image on the cover:
Silver Fern (Cyathea dealbata, or from Māori kaponga or ponga ), Auckland,
New Zealand, 2009.

The Fern is an example of a naturally occuring fractal. Nature is an inspiring

model for architecting complex technological solutions. Through fractals
nature efficiently architects infinitely complex systems.
 Architectural Design Principles
For Industrial Internet of Things

Hasan Derhamy

Department of Computer Science, Electrical and Space Engineering

Luleå University of Technology
Luleå, Sweden

Supervisors:
Jens Eliasson and Jerker Delsing
Printed by Luleå University of Technology, Graphic Production 2018

ISSN 1402-1544
ISBN 978-91-7790-128-0 (print)
ISBN 978-91-7790-129-7 (pdf)
Luleå 2018
www.ltu.se
 To Nana and Grandad and my family

iii
iv
 Abstract

As society has progressed through periods of evolution and revolution, technology has
played a key role as an enabler. In the same manner that mechanical machines of the
1800’s drove the industrial revolution, now digitalized machines are driving another one.
With this recognition of a fourth industrial revolution, the Industry 4.0 initiative was
founded. One of the drivers of Industry 4.0 is the Industrial Internet of Things (IIoT).
The IIoT is a consequence of widely present computing ubiquity and interconnected-
ness. Software has become a crucial tool of almost all industries from bakeries and arts
to manufacturing facilities and banking. Programming is now a required competence
and used by a variety of professions. It is not only about algorithm development, it has
become more about engineering and integrating existing designs and tools. This impacts
the way software is architected and drives a large body of research in the area.
Software solutions are becoming more distributed, not only over multiple processes,
but over heterogeneous hardware and business domains. Computing platforms could be
mobile or geographically separated over large distances, exposing the solutions to network
disturbances, performance degradation and security vulnerabilities.
Hence, IIoT introduces complexity on a scale previously unseen in the software in-
dustry. Software architecture must accommodate these heterogeneous domains and com-
petencies and handle the increasing levels of complexity.
This thesis proposes an architectural style for designing IIoT software architectures.
The popular Service Oriented Architecture (SOA) style is not sufficient to define a com-
plete architecture for IIoT applications. SOA fundamental principles are defined as loose
coupling, lookup and late binding. The proposed architecture style extends these SOA
principles with autonomy, specialization, data at its source and first person perspective.
It preserves the benefits of SOA that models functionalities as reusable services with
standardized interfaces. Thus, the proposed style helps to capture the heterogeneity of
IIoT (e.g. systems, capabilities, domains, competencies etc.), while handling challenges
imposed by it. The style also captures resource constraints of IIoT platforms; distri-
bution of application logic across IIoT; dependence between services within IIoT; and
presentation of the solution in various stakeholder perspectives.
The IIoT generates large amounts of data that is subsequently stored, analysed,
archived and eventually fed back into the product life cycle. Centralization of data has
well known challenges. This thesis proposes a method of information extraction based
on the principle of data at its source. Such data preserves implicit context, reducing the
burden of semantic data within the system. Desired information is expressed through
dynamic (runtime) queries. Using the queries, a path is created to retrieve the requested

v
data. It alleviates the need for data to be stored in intermediary nodes; data remains at
the source. Thus, IIoT applications extract information and present it to requesting sys-
tems without redundant source related context. This helps with issues of data ownership,
access control and stale data.
Another IIoT challenge tackled in this work is decentralization of Manufacturing
Execution System (MES). It is motivated by a need to mitigate the impact of vulnerable
shared networks on the factory floor; and by business requirements to reduce dependence
on local factory infrastructure. This thesis explores a solution where functions of MES
are distributed to the workstations that enables them to operate autonomously. Such
autonomous workstations utilize the proposed Intelligent Product, Workflow Manager
and Workflow Executor systems. Thus, MES can be decentralized to edge nodes as
envisioned by Industry 4.0.

vi
 Contents
Part I 1
Chapter 1 – Introduction 3
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Research Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Thesis scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 Thesis structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 2 – Software, Systems and the IIoT 13

2.1 Internet of Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Industry 4.0 and RAMI 4.0 . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Software Architecture Practice . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Fundamental Architectural Styles . . . . . . . . . . . . . . . . . . . . . . 19
2.5 Advanced Architectural Styles . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.7 Systems of Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Chapter 3 – Architecting IIoT Software 29

3.1 Protocol Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Information Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 Towards Decentralized MES . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.4 Challenges of Software Engineering in IIoT . . . . . . . . . . . . . . . . . 35
3.5 Principles for Software Architecture design in IIoT . . . . . . . . . . . . 37
3.6 The Principled Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 39
3.7 The Architectural Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Chapter 4 – Contribution 53
4.1 Summary of Appended Publications . . . . . . . . . . . . . . . . . . . . . 53

Chapter 5 – Conclusion and Outlook 57

5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

References 63

vii
Part II 71
Paper A – A Survey of Commercial Frameworks for the Internet of
Things 73
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
2 What is an Internet of Things Framework . . . . . . . . . . . . . . . . . 76
3 Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4 Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Paper B – Translation Error Handling for Multi-Protocol SOA
Systems 95
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
2 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . 99
3 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5 Implementation and Results . . . . . . . . . . . . . . . . . . . . . . . . . 107
6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
8 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Paper C – IoT Interoperability - On-demand and low latency Trans-
parent Multi-protocol Translator 115
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
3 Arrowhead Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4 Proposed SOA-based translator . . . . . . . . . . . . . . . . . . . . . . . 123
5 Translator Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6 Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
9 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Paper D – Orchestration of Arrowhead services using IEC 61499:
Distributed Automation Case Study 141
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
2 IEC 61499 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
3 Arrowhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4 Case study: Conveyor Loop with Distributed Control . . . . . . . . . . . 147
5 Application of Arrowhead architecture . . . . . . . . . . . . . . . . . . . 147
6 Application integration in IEC 61499 . . . . . . . . . . . . . . . . . . . . 150
7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

viii
Paper E – Service Oriented Architecture Enabling The 4th Genera-
tion Of District Heating 155
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
2 The Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
3 Structure and Behavioral Models . . . . . . . . . . . . . . . . . . . . . . 161
4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Paper F – Protocol Interoperability of OPC UA in Service Oriented
Architectures 171
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
2 OPC Unified Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 175
3 Proposed Translation Solution . . . . . . . . . . . . . . . . . . . . . . . . 176
4 Application Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Paper G – In-network Processing for Context-Aware SOA-based Man-
ufacturing Systems 189
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
3 Proposed solution - Expression of Interest . . . . . . . . . . . . . . . . . 196
4 Proposed solution - Dynamic Service Provisioning . . . . . . . . . . . . . 197
5 Wheel loader ball bearing monitoring . . . . . . . . . . . . . . . . . . . . 200
6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Paper H – Workflow Management for Edge Driven Manufacturing
Systems 205
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
3 Arrowhead Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
4 Work flow handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
5 Edge Automation Services . . . . . . . . . . . . . . . . . . . . . . . . . . 213
6 Use case implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
7 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Paper I – System of Systems Composition based on Decentralized
Service Oriented Architecture 223
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
2 Challenges for Industrial Information Distribution . . . . . . . . . . . . . 228
3 Related Works and Technologies . . . . . . . . . . . . . . . . . . . . . . . 228
4 Proposed solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

ix
 5 A Graph analysis of Arrowhead Framework . . . . . . . . . . . . . . . . . 231
6 Building Systems of Systems using graph queries . . . . . . . . . . . . . . 235
7 Implementation of a System of System Composer . . . . . . . . . . . . . 239
8 Demonstration of proposed solution . . . . . . . . . . . . . . . . . . . . . 246
9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Paper J – Software Architectural Style for the Industrial Internet
of Things 251
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
2 Software Architecture Practice . . . . . . . . . . . . . . . . . . . . . . . . 256
3 Advanced Architectural Styles . . . . . . . . . . . . . . . . . . . . . . . . 258
4 Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
5 Systems of Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
6 Challenges of Software Engineering in IIoT . . . . . . . . . . . . . . . . . 266
7 Principles for Software Architecture design in IIoT . . . . . . . . . . . . 268
8 The Principled Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 271
9 The Architectural Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

x
 Acknowledgments

I would like to start by saying this has been an inspiring and rewarding experience.
I enjoyed researching into Industrial Internet of Things and working on cutting edge
technologies. I have been challenged to think and argue scientifically and in the process
extended my knowledge. My research has been motivating, exciting and satisfying, and
I appreciate the support and opportunities. I am thankful to all, even though not all are
mentioned below.
I enjoyed working with and would like to express my gratitude to my supervisors Jens
Eliasson and Jerker Delsing for their guidance and patience. I value the freedom given
to me in my research and the thought provoking, visionary discussions and feedback.
I appreciate the opportunity to work on large IoT projects, and to collaborate with
academic and industrial partners. I enjoyed the challenging debates, constructive criti-
cism and professional opinion.
I would like to thank Volvo Trucks team Frida Schildauer, Mattias Andersson and
Patrik Gustafsson for valuable feedback and discussions.
I had a great team at the EISLAB, and I enjoyed interesting discussions, opinions
and conversations.
My warm gratitude has my family for their support and patience.

Alhamdulillah Rabbil ‘Alameen

(All praise is due to Allah the Lord of the worlds)

Luleå, May 2018

Hasan Derhamy

xi
xii
 Part I

1
2
 Chapter 1
Introduction

“You got so much to say, say what you mean, mean

what you’re thinkin’, and think anything.”
Cat Stevens,
“Can’t keep it in”, 1972

1.1 Introduction
The world of computer systems, including software and hardware, are in a constant state
of evolution. Software is taking a growing proportion of the responsibility for overall
system functionality. It has become a crucial tool of almost all industries from bakeries
and arts to manufacturing facilities and banking. As systems gain more software aspects,
managing software complexity and heterogeneity becomes a key challenge [1]. In the late
80s and early 90s, it was already well recognized that software architecture is a key
to manage the complexity. This early work laid the foundation for the area of software
architecture [2]. It is the study of the form and structure of software abstractions intended
to satisfy a set of functional and non-functional requirements. The field of software
architecture has grown and been applied to many different computing challenges. It is
fully introduced in Chapter 2.
The Industrial Internet of Things (IIoT) is a logical consequence of widely present
computing ubiquity and interconnectedness. With simple beginnings in 1982, Carnegie
Melon University researchers connected a vending machine to the Internet allowing staff
to query the state of Coke in the machine (empty, warm, cold) before walking to buy
a soda [3]. In this sense, the vending machine was the first Internet of Things (IoT)
device. It was actively sensing its environment, storing and processing information and
had connectivity to its stakeholders. The current trend of IIoT (digitalization) seen
around the world is an evolution of this simple scenario.
Technological and business opportunities and challenges within IIoT have given rise
to a “zoo” of software solutions. In this context “zoo” implies variations of size, domain,
methods, forms (web app, smart app, libraries, tools), solutions and more.

3
4 Introduction

Computing domains in industry are commonly grouped into operational and informa-
tion technology (OT, IT). They are shaped by different requirements and environments:
one focused on control (automating mechanical processes), real-time requirements, relia-
bility and long life cycle etc.; the other focused on data management, process efficiency,
business process management and frequent updates etc. This impacted the development
of hardware, communications and software. The hardware platforms acquired different
properties. In OT domain, deterministic operation is built into the platform that is also
tightly integrated to the physical world (e.g. A2D/D2A and DIO ports). In IT domain,
platforms are high performance processors with large memory and storage. Similarly,
communication networks diverged in their properties. In OT domain, communication
commonly is deterministic, requires hard wired addressing and secured by dedicated or
restricted access. In IT domain, Internet stack is the prevailing communication stan-
dard, with global addressing scheme, high bandwidth links and cryptographic security.
Software development specialized to address demands of each domain. In OT domain,
software emphasises deterministic modelling abstractions, machine-to-machine interac-
tions and robustness. In IT domain, software realizes high level abstractions, reconfig-
urable functions and human oriented interfaces. This resulted in differences in software
properties, capabilities, scale, tools, technologies and competencies etc. i.e. architectural
approaches.

Software within the two domains are put together to achieve overall business goals.
As shown in Figure 1.1, the IT and OT have divided ISA 95 [4] pyramid and taken
responsibility for the different levels. OT is at the manufacturing control layer (level 1
and 2), responsible for sensing, manipulating, monitoring and controlling the production
processes etc. IT is at the manufacturing operations, business planning and logistics
layers (level 3 and 4), responsible for plant scheduling, inventory management, logistics,
workflow, recipe control etc.

Figure 1.1: ISA 95 pyramid with Information and Operational Technology domains shown.

Building on existing ISA 95 functions, the challenges of Industry 4.0 can be ap-
1.1. Introduction 5

proached in following ways:

1. Implement it in IT Shown in Figure 1.2, this will leverage powerful tools, methods,
processing power and architectural approaches. Communication, synchronization with
physical production process (level 0), determinism, real-time operations are the challenges
to be resolved. (An extreme case of this could be cloud based manufacturing with direct
access to production process).

Figure 1.2: Implementation of Industry 4.0 with Information Technology realizing ISA 95 func-
tions.

2. Implement it in OT Shown in Figure 1.3, this will leverage nearness to physical

production process, physical security and deterministic modelling approaches. Legacy
OT communication networks may limit flexibility. The challenge is to handle software
complexity (level 3 and 4) using architectures of OT. The OT will need new concepts
such as Cyber Physical Systems (CPS) [5]. (An extreme case of this could be peer-to-peer
OT platforms coordinating all layers).

3. A fusion of IT and OT Shown in Figure 1.4, this will leverage both IT and OT
strengths. Such a fusion is an IIoT system that is capable to realize Industry 4.0 fea-
tures. It requires multidisciplinary teams and architectural vision. Such an architecture
has to accommodate heterogeneity of systems, capabilities, domains, competencies etc.
Creating such an architecture is the challenge.
The IT and OT domains are decomposed and rearranged together into autonomous
nodes. The system is highly distributed with horizontal integration. IIoT pushes away the
pure vertical integration and enables a holistic view of the software architecture on each
node. Enabling technologies and methods for this change are Edge [6] and Fog [7] com-
puting. The technological heterogeneity of such applications is another challenging issue
[8]. For example, within a single IIoT application several communication protocols can
be employed. This requires communication interoperability (as defined in [9]) solutions
6 Introduction

Figure 1.3: Implementation of Industry 4.0 with Operational Technology realizing ISA 95 func-
tions.

Figure 1.4: Implementation of Industry 4.0 with fusion of Information and Operational Tech-
nology. ISA 95 functions are realized as IIoT applications.

like translators [10] between different request/response and push/pull protocols such as
HTTP, CoAP, MQTT, AMQP, XMPP and OPC-UA. The cause of the communications
heterogeneity is from the mixture of computing platforms in the IIoT. It includes small
8-bit micro-controllers (MCU), 32-bit Digital Signal Processors (DSP), 32-bit or even
64-bit single board computers, general purpose PCs and high end server grade hardware.
This huge variation of hardware capability entails a large variation of communications
technologies, applications spaces and requirements. Therefore, the solutions will tailor
the communications according to the functional and non-functional requirements. This
means that multi-protocol solutions are inevitable. Existing approaches may utilize a
proxy (reverse, forward or injection) [11], a gateway [12], or something more advanced
such as an ESB [13]. All of these solutions face well known challenges of centralization
or pre-configuration. This increases infrastructure costs and reduces flexibility. IIoT
1.1. Introduction 7

introduces complexity on a scale previously unseen in the software industry. Software ar-
chitecture must accommodate these heterogeneous domains and competencies and handle
the increasing levels of complexity.
What is software architecture and how does it impact implementation design? A
brief introduction is made here and a more detailed discussion is presented in Chapter 2.
Unconstrained software can satisfy most user requirements in an almost unlimited number
of ways [14]. Software is made through a series of design decisions until implementation in
code. Each design decision made adds some constraint to the solution and each constraint
cascades to downstream design decisions. Early decisions, such as architectural, are
fundamental; they have a higher and longer lasting impact on the overall solution. Hence,
software architecture should only constrain critical (load bearing) parts of the solution
[2]. Thereby allowing as much flexibility for solution variation as possible. This practice
is in-line with development methods such as Agile [15].
Software design has major implications on migration toward Industry 4.0 [16]. When
designing new solutions legacy systems, infrastructure and procedures must be taken into
account. For industrial automation this is acute because some equipment life-cycles can
span well over a decade. Software architecture must be designed to accommodate legacy
technology. On the other hand, software architecture must also minimize constraints
on how future (unknown) technologies can be integrated. Therefore, the software ar-
chitecture will evolve while the components may have limited ability to change. As the
architecture adapts for changing requirements and technologies, its architectural style
acts as a anchor through this evolution. An architectural style is the language or man-
ner in which the concrete architecture is described. It utilizes abstract concepts that
communicate intent and rationale of the software architecture. What would a suitable
architectural style be for IIoT engineering?
Architectural styles such as Service Oriented Architecture (SOA) [17] and Multi-
Agent Systems (MAS) [18] can be used to engineer IIoT applications. Research by
Colombo et al. in [8] reported the use of SOA as a promising architectural style for
cloud based process monitoring and control. IIoT frameworks and platforms such as the
Arrowhead Framework [19], the Far-Edge [20] and Axcioma [21] utilize SOA, MAS or
other component based architectures. Systems and Systems of Systems (SoS) theory was
introduced in the Arrowhead project [22] to supplement SOA abstractions. Introduction
of new concepts, such as local automation clouds, may be required [23] to enable the
merging of IT and OT abstractions and architectures.
The Reference Architectural Model for Industry (RAMI) 4.0 [24] also creates new
opportunity to rethink the principles and styles that make up software architecture in
the fourth industrial revolution [25]. Proposals such as the RAMI 4.0 attempt to allow
more flexible integration between Product Life-Cycle Management (PLM)/Enterprise
Resource Planning (ERP) and the shop floor. Traditional vertical integration through a
layered architecture introduces overhead. This may be overcome as advancements in edge
computing enables more functions, such as Manufacturing Execution System (MES), to
exist in lower layers of the architecture. This has blurred the boarders between IT and
OT domains, such that physical and logical entities contain functionality of both IT and
8 Introduction

OT domains. These functions then cannot rely on a pure layered abstraction to separate
their concerns. For example, MES functions of tracking and execution may run on edge
computers [26] that are also running process control algorithms.
Decentralizing MES functions such as workflow management creates challenges like
logic and information fragmentation and others. Workflow management is a traditional
research area in industrial automation and production [27]. The fourth industrial rev-
olution has pushed research of IIoT enabled workflow management [28, 29] and supply
chain [30]. Industry has also seen the potential of IIoT in supply chain management
[31, 32, 33, 34]. A workflow (or business process), is the combination of activities and
information toward the creation of new value [35]. Workflow patterns are split into four
broad categories; Control, Resource, Data and Exception Handling workflows. Web2.0
architectures such as REST and hypermedia have been proposed for data driven workflow
automation [36].
There are many challenges in IIoT engineering, among them: delegation of higher level
of ISA 95 functions into factory edge computing; mechanisms for composition and co-
ordination; interoperability across a plethora of protocols; information sharing between
machines; engineering of IIoT applications (architecture); and more.
This thesis addresses the challenges of communications interoperability, information
extraction, workflow management and IIoT software architecture design.

1.2 Problem formulation

The use of software architecture to guide early design decisions has been used since the
early 1990’s. Architectural decisions have many downstream impacts as solutions develop
and evolve. Recent years has seen a rapid increase in the number of reference architectures
designed to guide IIoT application development [37, 38]. Commercial reference architec-
tures also exist such as Microsoft Azure [39] and Intels platform reference architecture
[40]. Platforms and frameworks of IIoT claiming to provide solutions for IIoT have in-
creased [41]. For example, Thingworx is an IIoT integration platform that has strong
mash up capabilities, device management and information routing [42]. Thingsquare is
a lightweight cloud based integration platform for connecting IoT devices together [43].
IoTivity is an IoT platform targeting device-to-device integration rather than device-to-
cloud integration [44]. But what are the architectural basis of these platforms? In what
way/direction have they constrained the solution space?
IIoT encompasses cloud based platforms as described above, networked embedded
systems and systems in between. Enterprise software architecture has been required to
handle rapidly changing user requirements and general purpose logical manipulations.
This has resulted in a rich architecture portfolio from layered and object oriented to
MAS and SOA Embedded systems including OT, on the other hand, are closely coupled
to hardware. Tradtionally, their software architecture has focused on processor hardware
abstraction, operating system design and execution models. Complexity is introduced by
realtime operation, determinism, signal processing algorithms and etc. The application
layer has been lgithweight and single purpose. Due to their specialization and constrained
1.2. Problem formulation 9

application domain, embedded systems do not have such rich architecrual repertiore
as enterprise systems above. As embedded and OT systems become networked and
participating in IIoT the architectures must mature.
The research questions posed, led the author from practical interoperability, towards
information sharing in industrial distributed systems and workflow management. The
questions are as follows:

Q1 What possibilities and limitations are there for efficient and interoperable multi-
protocol IoT networking?

The area of IoT has evolved to incorporate a plethora of messaging protocol standards,
both existing and new, emerging as preferred communications means. It appears unlikely
that IoT applications will come to consensus on a single standard protocol. The variety of
protocols and technologies enable IoT to be used in many application scenarios. However,
this creates vertical silos and reduces interoperability between vendors and technology
platform providers. There are existing approaches to tackle the interoperability challenge:
Protocol middleware tends to move the interoperability problem rather than solving it
and impacts application design; and protocol proxies have scalability issues, requiring
configuration effort and introduce processing overheads. The challenge here is to make
an interoperability solution that supports transparent translation, is scalable, has secured
interfaces, can perform error reporting and conducts QoS monitoring and control.

Q2 What are the possibilities for ancillary information extraction within SOA-based
heterogeneous networks?

Communications within a production line system will often be limited to only op-
erational communications required to complete the work. This is to avoid operational
performance degradation through network or data access congestion. But this means that
non-operational data access, such as Key Performance Indicator (KPI) data, is either re-
stricted or prohibited all together. KPI are critical to understanding and optimizing the
production line processes. Domain engineers today complain of the need to utilize soft-
ware support/engineering in order to change KPI generation. This question asks if it is
possible to enable domain engineers at the edge to extract non-operational information.

Q3 How can Manufacturing Execution System functionalities, such as execution process

logic and tracking, be distributed across Systems of Systems?

The MES is traditionally a centralized group of capabilities related to factory level

control and execution. The high level capabilities of the MES include: planning, schedul-
ing, tracking, execution and quality assurance etc. The challenge is the decentralization
of MES. It is motivated by a need to mitigate the impact of vulnerable shared networks
on the factory floor; and by business requirements to reduce dependence on local factory
infrastructure.
MES can be decomposed and distributed to edge layers and rely on local communi-
cation between computing systems.
10 Introduction

In answering these research questions, it became clear that the design of the overall
system is the key toward interoperability, information extraction and functional distri-
bution. This leads to the next question.

Q4 What are the underlying architectural principles and styles that can guide design of
Industrial Internet of Things applications?

IIoT applications are made up of highly interconnected and ubiquitous computing

systems that are embedded into their physical environment. It is a multidisciplinary do-
main requiring knowledge and/or awareness of materials, physics, electronics, embedded
software, communications/networking, security, enterprise software and data storage/an-
alytics etc. As each constituent system is updated, IIoT application evolve in a piecemeal
manner. Software without some form of clear architecture leads to brittle applications
that resist change. The challenge is how software architecture can accommodate het-
erogeneous domains, competencies and handle the increasing levels of complexity. This
question asks what principles and styles should be used to design IIoT application archi-
tectures.

1.3 Research Methodology

During licentiate work, the author approached this research using Action Design Research
(ADR) method [45]. ADR is an evolution of Design Research (DR) [46] and Action
Research (AR) [47]. It is most commonly used in information systems research which
focuses on the use of technology for betterment of organizations. ADR puts emphasis on
both technological artefacts and organizational change. This requires strong definition
of the artefact followed by continuous feedback from users. The measure of success is
the organizational learning and artefact generalizability. This approach was not utilized
for the rest of the thesis work as it became clear that the research was not in direct
interaction with users. The research focus has been on machine-to-machine integration
and the functional distribution of application logic.
Experimental computer science and engineering research [48] provided an alternative
approach. This approach targets implementation and experimentation. Research and
learning is made through a series of experimental software proposals, prototypes and
evaluation. In this research, the learning and questions are focused on how design can
impact solution flexibility within the context of IIoT and manufacturing or process in-
dustries. To this end, a series of experimental prototypes have been implemented and
their architecture retrospectively analysed.
The results have been presented, reviewed and evaluated in discussion with industrial
domain experts (production engineer, software architects, business project managers)
and research peers, subsequently the solutions were refined addressing the critique and
shortcomings. A chronological order of prototype and experiments is listed below:
1. Wheel loader smart ball bearing monitoring application
2. Ultra wide band distance measurement and location
1.4. Thesis scope 11

3. SOA-based on-demand multi-protocol translator

4. Festo conveyor belt production line wireless control
5. OPC-UA protocol translation - modified smart ball bearing monitoring
6. State information extraction prototype
7. Service Oriented Nutrunner (legacy protocol adaptation)
8. Workflow manager and executor for shop floor MES tracking and execution
9. System composer prototype (extension of state information extraction)

Prototyping and analysis have contributed to the understanding and development of

the architecture guidelines presented in this thesis.

1.4 Thesis scope

Software architecture and the IIoT are broad areas of research. Engineering IIoT appli-
cations is a multidisciplinary activity. Knowledge of a variety of disciplines is required.
Architecture of IIoT applications demand competencies in many computer system top-
ics such as: computer platforms, software engineering, physics of constrained systems
(battery life and RF etc.), networking and information modelling etc. Focused experi-
mentation, prototyping and developing of relevant testbeds is required to produce usable
results. This research work was performed in close collaboration with other researchers
and industrial partners with different IIoT competencies.
In the big picture, this research is directed toward increasing efficiency and sustain-
ability of industrial production. Industry 4.0 and IIoT promote such goals by fostering
research into technologies, methods, concepts and tools that will improve operational
efficiency, flexibility, fairness, reduce costs, improve sustainability, encourage reshoring
and improve quality of life etc. This thesis tackles software architectures of IIoT appli-
cations. These days, software is a crucial tool of almost all industries. The result of this
thesis targets efficiency by increasing reuse and promotes competition through choice;
helps with issues of data ownership and accessibility; and availability of production.
The work in this thesis investigates software engineering methods for IIoT application
development. The scope of research focused on communication protocol interoperability;
constrained device service orchestration; data extraction with SoS compositions; decen-
tralized workflow management; and extending SOA principles for IIoT applications. In
this context the relevant research areas that are not within the scope of this thesis are
security architectures and key management, algorithm development, artificial intelligence
and machine learning, big data analytics, communication protocol development, resource
management and sensor/actuator technologies.

1.5 Thesis structure

This compilation thesis consists of two parts. Part I introduces the research area, de-
scribes the research methodology, and formulates the main research questions of the thesis
12 Introduction

work. Part II consists of ten appended papers that contain the core research contribu-
tions of this thesis. All appended papers have been published, accepted or submitted
for publication in peer-reviewed scientific journals or conferences. The appended papers
have been reformatted to match the layout of this thesis without changing their contents.
Part I is divided into five chapters. Chapter 2 introduces background technologies,
methods and domains and gives overview of theoretical foundations for software archi-
tecture. In Chapter 3 the research questions are explored, proposed approaches are pre-
sented and prototyped systems are described. Software architecture style is presented,
discussing proposed architectural principles and views. Chapter 4 summarizes the re-
search contributions of the appended papers to this thesis. Chapter 5 draws conclusions
and presents and outlook for future work.
 Chapter 2
Software, Systems and the IIoT

In this chapter, the background technologies and domains are introduced. Relevant
concepts are defined as how they are used and what they mean in the context of this
thesis

2.1 Internet of Things

The first thing that had a networked presence was the soda machine in the Carnegie
Melon research facility [3]. The term Internet of Things was first coined by Ashton
with regard to Radio Frequency Identification (RFID) and the Internet [49]. The vision
according to [49] was for a computing world with knowledge acquired without human
intervention. Computers that are enabled with sensing capabilities are able to understand
the world without the limitations of human created data. With RFID it is possible to
sense the environment and use this information across supply chains and throughout
a manufacturing process. IoT has evolved into a more generic concept of networked
intelligent objects that able to maintain a digital presence on the Internet or an Intranet.
According to [50] the IoT requires two definitions, for smaller and larger deployments.
With a smaller scenario defining IoT as:

“An IoT is a network that connects uniquely identifiable ‘Things’ to the

Internet. The ‘Things’ have sensing/actuation and potential programmability
capabilities. Through the exploitation of unique identification and sensing,
information about the ‘Thing’ can be collected and the state of the ‘Thing’
can be changed from anywhere, anytime, by anything.”[50]

And a larger deployment defining IoT as:

“Internet of Things envisions a self-configuring, adaptive, complex net-

work that interconnects ‘things’ to the Internet through the use of stan-
dard communication protocols. The interconnected things have physical or
virtual representation in the digital world, sensing/actuation capability, a

13
14 Software, Systems and the IIoT

programmability feature and are uniquely identifiable. The representation

contains information including the thing’s identity, status, location or any
other business, social or privately relevant information. The things offer
services, with or without human intervention, through the exploitation of
unique identification, data capture and communication, and actuation capa-
bility. The service is exploited through the use of intelligent interfaces and
is made available anywhere, anytime, and for anything taking security into
consideration.”[50]

The underlying difference between these two definitions is complexity. Complexity in

terms of the scale of devices and management in the scenarios. Within the context of
this thesis, the definition of IoT that will be: any network of things, virtual or physical,
with capability to communicate using the Internet stack, able to compute logic and may
have some form of sensing and actuation.
The origins of IoT are in industrial applications, but the practical focus has tended
toward home automation and consumer electronics. Personal Area Networks (PAN)
with the focus on smart phones, watches and appliances, are a key driving force here.
Home security systems and home energy efficiency are another two driving areas for IoT.
Choice and flexibility are of utmost importance along with usability and ergonomics.
Resilience and security gain less attention from consumers and therefore developers.
Privacy concerns are now being raised as the amount of digital data produced by IoT
devices increases. However, for most consumer applications, privacy, confidentiality and
authenticity are of less concern when compared with industrial use cases. The Industrial
IoT is evolved from adapting IoT technologies for the industrial world.

2.1.1 Industrial IoT

Industrial domain is different from consumer domain. It has specialized communications
protocols, security requirements, QoS and device life cycles [51]. Initially, IIoT applica-
tions referred to any application that relied on an Internet- (or Intranet-) capable sensor
and actuator networks. Many web, industrial automation, embedded and wireless sensor
network applications are usually grouped under the umbrella of IIoT. While IoT has
maintained its position covering domain of home automation and consumer electronics,
IIoT extended it to include industrial domain with all its implications. The key enabler
of both is connectedness (networked). The IIoT integrates with a wider array of commu-
nication protocols. For example, the OPC UA protocol [52] is tagged for standardization
within Industry 4.0 and Apache Kafka [53] already has momentum for analytics engines.
IIoT based low power wireless networks may require real-time performance with time
triggered variations of IEEE 802.15.4. Hence, IIoT applications must handle changes in
and introduction of new communications protocols.
Security is an important aspect for networked systems, such as IIoT. In [54], Delsing
discusses business benefits and secure models for IIoT in industrial automation. The open
connectedness of IIoT applications make them vulnerable and require protection from
various threats. Some of these vulnerabilities can be mitigated if secure IIoT models are
2.2. Industry 4.0 and RAMI 4.0 15

used. The traditional security triad of confidentiality, integrity and availability still apply
to the IIoT, and now privacy must also be included. Traditionally, industrial computer
networks rely on network segregation with highly controlled network access [55] or an “air
gap” between factory floor and IT networks [56]. This includes using firewalls to control
what connections are allowed to pass between network segments, for example network
traffic entering and leaving the factory may be fully denied. An IoT application may be
deployed through a public app store, with terms and conditions often unread. However,
IIoT applications have monetary consequences and could expose commercially sensitive
information. Authenticity and terms and conditions must be taken seriously. Hence,
mash-up or app store approaches must provide means of allowing commercial certainty,
that security vulnerabilities will be not exposed.
IIoT applications operating in continuous production require QoS agreements and
monitoring. QoS refers to the non-functional requirements of an application. The QoS
concerns could be battery life time, bandwidth, round trip delay, redundancy, backup,
resilience, recovery or more. QoS is an important issue for IIoT therefore it must comply
stricter limits. Whereas IoT has relaxed QoS concerns, to the degree they may not exist.
The life-cycles of IIoT applications differ to those of IoT. IoT application life-cycles
have less verification steps and a more rapid turn over, with faster adoption of new
technologies. By comparison IIoT applications must pass thorough testing, simulation,
validation and verification prior to deployment. The deployment environment of IIoT ap-
plications requires integration with areas, such as legacy systems and devices, simulators,
intelligent robotics, big data, analytics and augmented reality etc. In addition, IIoT must
not introduce cyber security vulnerabilities to other areas, such as robotics. Therefore,
a software architecture style must account not only for IIoT but also the surrounding
domains. The next section introduces Industry 4.0 and some of the fields surrounding
IIoT.

2.2 Industry 4.0 and RAMI 4.0

Industry 4.0 is the German initiative [57] that aims to bring about challenges of the
fourth industrial revolution. It undertakes to bring together advances in digital technolo-
gies. Starting with a focus on the smart factory concept. It now includes concepts such
as big data and analytics, CPS, cloud, fog and edge computing, IIoT, augmented real-
ity, intelligent robotics, additive manufacturing, simulation and cyber security. Each of
these fields are being developed within their own areas. But combining the technologies
together and creating a cohesive technological environment is a challenge.
Industry 4.0 has proposed the Reference Architecture Model for Industry (RAMI)
4.0 [24]. It is a reference architecture that joins life-cycle, software concerns and the
automation pyramid into a cube view. Figure 2.1 shows this RAMI 4.0 cube. The
RAMI 4.0 shows the need for recognising the multi-dimensional relationship between
engineering domains within industrial automation. RAMI 4.0 makes explicit the need
for all hierarchy components to share some burden of the software layer distribution.
Industry 4.0 proposes the administration shell for I4.0 (Industry 4.0) components [58].
16 Software, Systems and the IIoT

The administration shell is method of “wrapping” equipment with required software to

satisfy the vertical layers of the cube. I4.0 component can be at any part of the hierarchy
of the cube, i.e. the work centre or equipment could each have their own administration
shell.

Figure 2.1: The RAMI 4.0 cube. With the vertical axis representing software concerns, the
horizontal axis representing life cycle stages, and the diagonal axis represents automation hier-
archy.

The RAMI 4.0 cube has some interesting properties for software architecture. The
three axis of RAMI 4.0 highlight the multi-dimensional nature of Industry 4.0 and IIoT.
From the vertical axis, the software layers are separated across the automation hierarchy.
Whereas, from the diagonal axis, the automation hierarchy levels contain each of the
software layers. Before diving deeper into the implications of the RAMI 4.0 on software
architecture, the next section describes software architecture practice and origins.

2.3 Software Architecture Practice

The practice of software architecture was defined to manage the increasing complexity
of software applications. Software design challenges evolved from algorithm design to
include communication structures, information storage and retrieval and interactions
with physical sensory and actuation capabilities. This section discusses what software
architecture practice is, its styles and how it is used to manage complexity.
Software architecture is used to model high level design that satisfies stakeholder re-
quirements. The architecture gives guidance for application design and development. It
2.3. Software Architecture Practice 17

is also used as a framework for technical and managerial decision making [2]. The archi-
tecture is not only used for design and implementation decisions, but also cost estimation,
requirements validation and design analysis. Software architecture is a fairly recent re-
search area with fundamental scientific papers published in the late 80’s and early 90’s.
As computing technology (hardware and networking), software practices and user ap-
plication spaces have evolved, so too have architectural practices. Recent research into
software architecture has focused on blue prints and patterns [38]. These reference archi-
tectures specify component entities and methods to tackle spatial distribution, streaming
and discreet information flow, and close coupling to physical world.
At its essence software architecture follows the analogy of construction architecture
(as in civil engineering). Perry and Wolf [2] introduce it as the presentation of different
views for the purpose of communication. Similarly, construction projects have structural,
plumbing, electrical and aesthetics views. These views highlight key aspects (i.e. load
bearing walls) of the design and are utilized by different stakeholders in the validation
and implementation of the project. Perry and Wolf [2] propose a three view approach;
process, data, and connector views. The process view describes the operational con-
struction of transformation activities, the data view describes the information and how
it changes during the life of the application and finally, the connector view describes the
interconnection between transformation activities.
Capturing early design decisions and rational guides the overall architectural style. An
overall architectural style is established dependant on how the architecture has described
these early design decisions and rationale. The concept of style again draws a parallel in
the construction world. For example, an architect may suggest the home design to follow
a Victorian style. This constrains and guides the subsequent design decisions, such as
flooring, lighting selections, may etc. Once a style has been built into the home, it will
influence future development or extension to the home. Similarly, once an architectural
style has been selected for an IIoT application it will influence future development or
extension of the application.
In a relatively short period of time, architectural concepts matured. Kruchten intro-
duced the 4+1 view model of software architecture [59]. This model was later adopted
by the Rational Unified Process (RUP). As shown in Figure 2.2, the ”4+1” views are
logical, development, process and physical view, plus a scenario view that extends across
each of the former views.
The logical view is an entity model of the design. For example this could be an object
based decomposition. Here, application data and information models could be captured,
however, because the focus is on end-users or domain engineers, it may lack the needed
depth for software development. The development view is a detailed break down of the
structural aspects of the organization of the code. This assists with division of work
between developers and dependency matching of development resources. The process
view is an execution view of the design, looking into concurrency and synchronization
aspects, capturing their transformations and relationships. The physical view shows
the distribution of the software over the executing hardware. Capturing certain non-
functional aspects of the design and guiding the deployment requirements and planning.
18 Software, Systems and the IIoT

Figure 2.2: The 4+1 architecture with views mapped to the RAMI 4.0 software layers. The
business, communication and integration views are not represented in this architecture. On the
other hand, RAMI 4.0 does not have a development view.

Interestingly, Krutchen did not utilize separate connection view and data view as
Perry and Wolf did. The 4+1 architecture does however pin-point that each view targets
a different audience, this is an important element of architecture practice. Users are
interested in functionality presented in the logical view. Developers would split workload
and design code repositories based on the development view. Technical and business
integrators tune performance and scalability using the process view. System engineers
plan topology, communications and deployments based on the physical view. Kruchten
further elaborates that each view has its own set of elements, patterns, rationale and
constraints. Within each view different architectural styles can be used. The 4+1 style is
a meta style that allows freedom to adapt the architecture to a minimal set of constraints
for the intended usage. Scenarios are used as a shared validation vision - to make sure
that requirements are satisfied by each of the architectural views.
The World Wide Web (WWW) which morphed into a key technology over the 90’s
lacked a clear definition of its architectural style. Without a clear style, changes to the
WWW architecture lacked consistent stylistic principle and were be based on short term
solutions to immediate issues. In his dissertation, “Architectural Styles and the Design
of Network-based Software Architectures” [60], Fielding defines an architectural style
for the WWW and web application development. This study addressed this issue and
provided guidance on adapting and evolving WWW protocols and techniques. The the-
sis introduces software architecture as an abstraction of run-time elements. It describes
that an architecture is not only a static structural description of a system, but rather a
description of a dynamic running system. The resulting Representational State Transfer
(REST) architectural style has become a common approach to web application develop-
ment. REST combines a number of architectural styles, including Client-stateless-Server
with caching and proxying; layered; mobile code; and uniform interface. Each style uti-
2.4. Fundamental Architectural Styles 19

lized by REST introduces a constraint to the freedom of design decisions, thus, providing
a foundation for the rationale of the design. The REST style definition can be used to de-
termine which application architectures conform to the web and those that do not. Thus,
a web application that is developed by several different teams, maintains architectural
continuity and avoids embrittlement.
Before discussing advanced architectural styles, the next section provides a high level
overview of some fundamental ones. These styles are often used as a basis for creating
or as part of the advanced architectural styles.

2.4 Fundamental Architectural Styles

An overview of fundamental architectural styles is described here. It is not the inten-
tion to give an exhaustive list nor to classify or compare styles here. Full description
and classification of the styles below can be found in Garlan and Shaws work on “An
Introduction to Software Architecture” [61] and in Fieldings dissertation [60].

2.4.1 Pipes and Filters

The Pipes and Filters architectural style is suitable for applications with sequential pro-
cessing requirements. Examples are software compilers and its usage in the Unix oper-
ating system. Because pipes connecting the filters are streams of type-less binary data,
the filters can be re-arranged and interconnected as needed. That means, filters can be
re-arranged and interconnected as needed. There are extensions of this style with typed
pipe and filter, this however removes one of the benefits - complete interchangeability of
filters.

2.4.2 Layers and Tiers

Software layering is one of the well used architecture styles. It defines that the design
should be decomposed into horizontal layers with each lower layer providing abstraction
to an upper layer. Each layer has a specific responsibility and well defined interface.
Layered architectures have strict rules that the dependencies must only be in a top
down. A change or removal of an upper layer should not mandate a change in the lower
layers. This has been used effectively in communications and the Internet stack [62].
The tiers architectural style is classed together with layers as it follows a similar
pattern with the exception that each layer runs on a different platform. Sometimes
referred to as n-tiers, it is used extensively in web applications. Commonly, tiers will
be divided into user interface, firewall/load balancing, controller (web server) and data
store (database).
20 Software, Systems and the IIoT

2.4.3 Object Oriented Design

Object oriented organization is yet another highly utilized architectural style. This style
breaks the design down into types (classes) and instances (objects). Objects are respon-
sible for managing their own internal representations and hiding that representation from
peer objects. Interaction between objects is achieved through function or procedure calls.
This style is used for both architecture design and Object Oriented Programming (OOP).
Class diagrams and object relation diagrams make up some of the architectural artefacts.
In the context of architecture, inheritance of objects is less important and not usually
shown Distributed objects is an extension of this style for networked systems requiring
interaction through object function and procedure calls.

2.4.4 Components
The Component architectural style is an intuitive grouping together of logic and func-
tionality. Components can be composed into applications because they use standard
interfaces. A component is self contained logic that can be compiled into a modular
library, this is the main difference to object oriented design. Hence a component may
contain many objects. This style is very flexible and has been used to a great extent for
automation and embedded systems engineering.

2.4.5 Repositories
Repositories are a style used for data driven applications. Often used where shared
access to memory is required or where intelligent software are interdependent through
information. For example, a black-board architecture can be used for a compiler with
multiple concurrent processes handling the multi-step transformation. Repositories are
often used with the n-tier style for web applications.

2.4.6 Event Driven

The event driven architecture style uses implicit invocation of remote functions or meth-
ods. It is possible to maintain late binding between the components, rather than explicitly
binding at design or compile time. Components must announce events that are answered
by an appropriate function registered by another component. The announcer does not
know if any function is registered for the event or which component actually handles it.
Depending on the eventing implementation, data exchange can be problematic. If the
framework does not allow data to be passed with the event, then some form of shared
repository is required. Event driven architectures have gained a lot of attention due to
the benefit of function reuse and late binding between components. For example, the
Enterprise Service Bus (ESB) appears to follow an advanced form of the event driven
style.
2.5. Advanced Architectural Styles 21

2.4.7 Interpretor
The interpretor style has been used for programming language development. The Python
Virtual Machine (VM) or the Javascript engine are examples of common interpretor based
solutions. The Java VM uses a compiled byte code, this makes Java a hybrid language.
First the Java code is compiled to byte code, then the byte code is interpreted by the VM.
The architecture is flexible to changing command sets or for providing a full abstraction
over hardware/firmware changes. The interpretor style is used for many Domain Specific
Languages (DSL) and scripting languages. Underlying methods, functions and compo-
nents can be mapped to languages that are suitable for stakeholders as an application is
moved to new usage domains.

2.5 Advanced Architectural Styles

After looking at some of the founding and fundamental architectural styles used, we now
look into some advanced styles. These styles tend to create hybrid style of many other
styles. Their principles have a broad coverage and are intended to be applied to software
engineering in general, rather than a specific usage. In addition, they may take into
consideration development methodologies, such as agile based methods.

2.5.1 Service Oriented Architecture

Service Oriented Architecture is the use of software interfaces called services to create
distributed computing and facilitate remote system interaction and data exchange. SOA
grew from the use distributed objects and OOP for distributed software systems. It has
been highly successful as an enterprise architecture much of its success can be attributed
to the flexibility and scalability of the architecture. There is much literature on this
subject but the work from Erl in [63] is a reference authority. It defines eight principles of
service oriented design. These principles are the characteristics which should be followed
by developers and met by the software in order for the application be classified as a SOA-
based application. These are defined in the enumeration below with a short description
of each.
Principles of Service Oriented Design:

1. Standardised service contract Are used for services to express their capabilities
and method of interaction.

2. Service loose coupling Means to reduce dependencies between service provider

and consumer. Allowing independent implementation and evolution of provider and
consumer systems.
22 Software, Systems and the IIoT

3. Service abstraction Refers to including only essential data in a service contract

and hiding the implementation detail. This abstraction enables independent evolution of
the implementation from the service contract.

4. Service reusability Requires that logic is broken down into agnostic functionality,
which can be reused within different contexts. This principle is dependent on the service
modelling approach used.

5. Service autonomy Means that service should control their own logic and environ-
ment. It is essential for reliable and consistent behaviour. The scope of the autonomy is
dependent on the service capability.

6. Service statelessness As opposed to autonomy, statelessness advocates moving

state information to an external store such as a database or hardware. Too much state
information will reduce service scalability as the number of consumers increases. Stateless
services are also more robust to restarts, as state information is not lost.

7. Service discoverability Requires that services, in addition to the service con-

tract, have a meta description allowing evaluation of fitness for use. Service reuse and
composition is founded on strong service discoverability.

8. Service composability Services are reused in different compositions depending on

application requirements. This requires some reconfigurability of the services.
A service is the definition of a communication contract which includes the abstract
functional description (the purpose of the service), its interaction pattern (how to utilize
the service), and its structure and semantic (how to interpret the payload and make it
meaningful).

2.5.2 Microservices
The microservices architectural style is an evolution of SOA. It is generated through
experience of architects and engineers overcoming the pitfalls in the SOA style. The
argument for moving from monolithic applications to a Microservices application are the
same as for moving to SOA. As stated by Newman in [64], Microservices are software
applications with a single purpose. Newman goes on to argue that SOA lacks clarity
on service design. In particular guidance on service size and application decomposition
are two areas of ambiguity. In addition, Microservices also raise the notion of smart
endpoints and “dumb” pipes [64]. This is in response to heavy middleware such as ESB
that are more intelligent than the services.
2.5. Advanced Architectural Styles 23

The principles of the Microservices can be summarized as:

1. Single responsibility A core principles of microservices is the service size selection.

The single responsibility principle guides architects to not bunch too many capabilities
into a single service. It is Martin who proposes the single responsibility principle [65],
as a means of decomposing software systems into modules. Avoid mixing requirements
from different responsibility domains. Microservices apply this very literally and reduce
the service provided to a single responsibility service. So, as domain requirements evolve
Microservices can also evolve without requiring testing against adjacent (unchanged) do-
main requirements. The single responsibility principle promotes evolutionary design.
This is because as new responsibilities appear or existing ones disappear, microservices
can be added, deleted or changed with no (or little) impact on other services.

2. Smart endpoints and “dumb” pipes With implementation of “dumb” pipes it

means reducing or removing dependence on intermediary infrastructure. This principle
is very much inspired by the successful approach used by the Web. Specific layers in the
pipes are only responsible for particular functions such as addressing, packet/datagram
routing and security. Otherwise, the pipes only act as the interconnection fabric. Smart
endpoints also entail design for failure. This means that smart endpoints are developed
with resilience to service call failure and that active monitoring of smart endpoints, data
stores and caches can catch operational or business relevant metrics

2.5.3 Web 2.0

The first Web 2.0 [66] conference was held in 2004. The Web 2.0 is an evolution of
the WWW, acknowledging a significant shift in the direction of software applications.
O’Rielly and Battelle introduced 7 principles of the Web 2.0 [67] as:

1. Web as a platform
2. Harnessing Collective Intelligence
3. Data as the next Intel Inside
4. End of software release cycle-the ”perpetual beta”
5. Lightweight programming models
6. Software Above the level of a Single Device
7. Rich user experience

These principles are in many ways reflect a smaller set of changes significant to soft-
ware development. The web as a platform directs software vendors away from building
applications as platforms for locking consumers in with. Harnessing collective intelligence
is acknowledging the success of utilizing user input and participation over publishing. The
Web 2.0 users produce huge amounts of data. Data driven applications then are sources
of revenue. The perpetual beta is linked to the Web as a Platform as software appli-
cations are on-going services rather than packaged artefacts. Lightweight programming
models boost the perpetual beta through quick development/release cycles. Software
24 Software, Systems and the IIoT

built with a single computer in mind does not take advantage of the Web as a Platform
and perpetual beta. A Rich user experience is defining, if user collaboration is desired.
Additionally, user experience becomes the user lock-in and service differentiation.
These principles lead to 8 design patterns for the Web 2.0:
1. The Long Tail
2. Data is the Next Intel Inside
3. Users Add Value
4. Network Effects by Default
5. Some Rights Reserved
6. The Perpetual Beta
7. Cooperate, Don’t Control
8. Software Above the Level of a Single Device
As can be noted, some of these design patterns are directly brought from the princi-
ples. Showing that the principles must first be elaborated before patterns can be defined.
The Long Tail means to not focus only on the central applications of the Web, but on the
users of the web. Target applications to the larger masses on the web. Network effects
by default promotes making privacy settings open by default. Web users must opt-out of
the data sharing. Some rights reserved is indicating the need to make applications easy
to adopt. The pattern Cooperate, Don’t control, means to utilize and build on shared
web services and re-use of data services.
Some of these principles and design patterns are highly relevant to web software
engineers, whilst others are specific to the Web. The future of software development
is bringing about a loosely coupled set of software applications that can, each, be in
perpetual beta, whilst being developed by ad-hoc combinations of software developers,
in a cooperative manner.

2.5.4 Mashups and Service Mashups

Mashups as a concept are the combination of web resources including data and API
resources, to create new web applications. It is generally for user interface creation,
information sharing and aggregation and content publishing [68]. It is a design pattern
born of the Web 2.0 principles. For example Trendsmap [69] is a web application that
utilizes the Twitter and Googlemaps APIs to show Twitter activity by location. Users are
able to view what the trending Twitter topics are in particular cities. Service mashups are
similarly user generated combinations of web services to create new web applications. IoT
platforms such as ThingWorx [42] utilize mashups for user interface design and service
orchestration. A service mashup can pipe data between services allowing user driven
processing, filtering, aggregation and presentation. Node-red [70] is Javascript based
graphical environment that can mash together different services into a composition. Its
structure is similar to typed pipe and filter diagrams. The IFTTT (“if this then that”)
platform [71] is an IoT mashup cloud that supports applets and services that can connect
heterogeneous things together. Using chained conditionals it is a strong commercial
mashup platform.
2.6. Systems 25

2.5.5 Multi Agent Systems

Multi-agent systems (MAS) are sought to address challenges of heterogeneous, dynamic
systems with distributed nature and requiring distributed decision making [72, 73].
MAS is a network of communicating agents. Agents communicate their preferences,
negotiate sub-goals and manage their intentions to realize individual or overall system
goals [74, 72].
An agent is a computer system. It is placed in an environment and can autonomously
act in this environment. The actions of an agent are directed to achieve delegated goals.
It operates independently (autonomously) without direct intervention. An agent has
ability to act in a designer’s/user’s best interest while interacting with other humans,
machines or agents. While autonomy is the key principle, an intelligent agent can also
exhibit reactivity, pro-activeness and social ability.
Reactivity. Agents are able to perceive and react to changes in their environment (in
a timely fashion). This implies both soft and hard real time requirements. This might
impact decision making process (as it can be time consuming and require processing
power), constraining agent’s intelligent behaviour.
Pro-activeness. Agents can take initiative to achieve their goals. In other words,
agents can employ goal directed behaviour.
Social ability. Interaction, negotiation and collaboration underpins the social ability.
MAS then represents a network of self-interested systems cooperating and competing
to accomplish their goals. The theory of agents and MAS is a step in the
“... steady move away from machine-oriented views of programming to-
ward concepts and metaphors that more closely reflect the way in which we
ourselves understand the world.”[73]
MAS is often used to implement Game Theory algorithms [75]. MAS applications,
though, are vast from economics and social sciences to industrial automation and power
systems.
Agent architectures tend to focus on agents’ decision-making process such as Proce-
dural Reasoning Systems and Practical Reasoning [73]. MAS is more about interactions
between agents, e.g. FIPA specification etc.

2.6 Systems
As defined by the US Military DoD JP 1 02 a System is:
“A functionally, physically, and/or behaviorally related group of regularly
interacting or interdependent elements; that group of elements forming a
unified whole”[76]
While ISO/IEC 42010 defines a system as:
“A collection of components organized to accomplish a specific function
or set of functions”[77]
26 Software, Systems and the IIoT

These two definitions provide a basis for understanding what makes up a system. In
this thesis, a system is a key building block for understanding an architecture. Here we
say, a system captures a fully autonomous entity with an independent set of functions,
life-cycle and purpose (objective). A system may or may not communicate and collab-
orate with other systems. However, in our case, many of the systems will need to work
collaboratively in order to achieve their own goals. Systems thinking means that when
designing a system, all aspects of the life of the system must be considered. This means
that all software and hardware aspects; such as logical, functional, structural security,
mechanical housing, spatial location, electronic and human interaction are captured un-
der a single entity. Spatial and temporal boundaries have been the intuitive delineation
lines between systems. However, as the definitions above indicate, functional boundaries
must also be considered. Add to this, managerial (change/configuration management),
political and economic boundaries, will capture strong lines of separation between sys-
tems.

2.7 Systems of Systems

Moving onto System of Systems extends this theory with goals and objectives that require
participation of multiple component systems to be achieved. In an SoS, the primary
modelling unit remains a single system with self-centred goals. The SoS introduces a new
set of goals that are only achievable through the collaboration of participating systems.
An SoS is itself a system because it satisfies the properties of a system. However, a system
does not satisfy the properties of an SoS, therefore a (pure) system is not an SoS [78]. This
is primarily because the management and life-cycle aspects of an SoS are quite different
to those of a system. Compared with systems an SoS is much more un-predictable life
cycle that is prone to change. Any assumptions regarding the component systems of
the SoS must be moderated with fall-backs and mitigation plans. Management and
development of an SoS is also rather different to that of a system. Control over change,
budget and overall direction of a component system may not be under the control of the
SoS team. Hence, there is only limited influence over the systems team, and so careful
synchronization and analysis of change is required.

2.7.1 SoS Design principles

Work has been done to understand the design principles of SoS architecture. In [79],
Maier describes principles of heuristic design for SoS. They are:

1. Stable Intermediate Forms While designing SoS, incomplete forms of the SoS
should be designed and put into action. As stated, an SoS is made up of component
systems. Utilizing a sub-set of these component systems to create a partially functioning
but stable SoS will act as a proof of concept and a platform for early learning. This
principle matches very well with concepts of Agile development and iterative design.
2.7. Systems of Systems 27

2. Policy Triage The SoS engineering team does not have control over the system
development or modes of operation. They must triage the situation and choose carefully
where to exert influence on systems teams and where/when to adapt the SoS to the system
choice. This is captured as over-control - leading to failure due to lack of authority, or
under-control - leading to failure due to elimination of an integrated SoS.

3. Leverage at the Interfaces The component systems of an SoS are often designed
and managed independently of each other. Furthermore, the SoS designers have limited
influence over internal architecture of the systems. This leads to SoS design having a much
greater emphasis on interfaces between systems rather than the design of component
systems.

4. Ensuring Cooperation An SoS is based on independently operating systems.

There is no guarantee that these systems should cooperate. There is also a challenge of
influence upon each of the component systems. Therefore, the incentives for cooperation
must be managed. In addition, barriers (costs) for cooperation must be reduced. Again,
the Internet is a prime example, with light weight communications protocols, combined
with great value of interconnectedness.

2.7.2 SoS Types

SoS can exhibit emergent behaviour [80]. In some cases it is desirable and other it can
be damaging. Depending on the class of SoS, emergence may be controlled (or planned
for). In Maier’s work [79], he describes a taxonomy of three SoS types:

1. Virtual Virtual SoS lack centralized management and purpose. Behaviour is not
created with intention, rather it emerges from the resulting SoS. The Internet is an
example of a virtual SoS; There is not enforcement of Internet standards and there is no
centrally agreed reason for the Internet. The Internet Engineering Task Force (IETF)
[81] must utilize principle 4 ensuring cooperation to create the Internet standards.

2. Voluntary Voluntary SoS can also be referred to Collaborative SoS. Involve cen-
trally agreed purpose of an SoS, but do not subscribe required adherence to the SoS. SoS
management does not have any coercive power on individual systems. This is perhaps
the most challenging and common form of SoS within the context of IIoT.

3. Directed In the case of directed SoS, purpose and management is fully centralized
and has significant control over component systems. The total SoS design lends itself
closely to Monolithic assumptions regarding behaviour control.
28 Software, Systems and the IIoT

2.8 Summary
In this section the Industrial Internet of Things and some software architecture ap-
proaches were discussed. Architectural styles like SOA, MAS, Web2.0 and microservices
all have contributing principles toward a software design and development for the IIoT.
However, there are certain challenges of IIoT that can be handled by one style and not the
other. Pure SOA usage in a physical environment lacks the descriptive need for modelling
physical entities with multiple, inter-dependent services. Pure MAS has high resource
demands from nodes that are constrained in battery life and communications capability.
Web2.0 focuses predominantly on web based commerce and interaction. There is little
attention to low bandwidth or niche requirements of industrial use cases. The principles
of Microservices build up many advantages to SOA and can be useful to IIoT. They
however, lack the same expressiveness required for understanding the physical nature of
dependent services in IIoT nodes.
In the next chapter, a consistent argument for a new architectural style is presented,
based on defining a set of principle that are able to cope with the challenges of IIoT
software architecture design. SoS is used along with SOA as the basis for modelling
interacting IIoT nodes. Functionality is spread amongst the nodes and applications
emerge from the resulting SoS. SoS is a flexible notion that lends itself well to environment
with highly varying and heterogeneous capabilities. For example, a company or society
can be modelled as a SoS with humans making up the component systems. An employee
will have a set of personal objectives and capabilities, for example, objective of earning
an income and capability of developing software. Employees in a company work in a
collaborative centrally administered organisation. Products and services are the result
of the emergent value of the collaborating employees.
 Chapter 3
Architecting IIoT Software

In this chapter the novel solutions to the research problems are elaborated. The first
section presents the communications protocol interoperability solution and its properties.
In section two, the information extraction design and solution is presented. Section
three describes the decentralized MES solution for workflow execution, tracking and
management. The fourth section provides a detailed proposal for guiding architecture
development for IIoT applications.

3.1 Protocol Interoperability

This thesis proposes an on-demand multi-protocol translator. It is able to translate
between Internet application protocols such as HTTP, CoAP, MQTT and OPC UA. It is
a SOA-based software system that bridges service exchanges between non-interoperable
peers. The solution supports, transparent, scalable and secure translation. Additionally,
it can perform error reporting and QoS monitoring. Semantic and syntactic translation
have not been addressed but could be added in the future.
The translator is transparent to the systems being bridged when it is used within
a SOA framework such as the Arrowhead Framework. Some form of interconnection
manager, i.e. Arrowhead Orchestrator, must identify the protocol mismatch and generate
a translation bridge request. On receiving the request the translator instantiates a new
temporary translation bridge with suitable protocols. No other configuration is required.
The translator uses the architecture shown in Figure 3.1. Each protocol must be
translated to/from an intermediate format. This intermediate format provides a com-
mon (internal) exchange between protocols. It is verbose and helps to minimize the
information loss between two protocols. So any protocol combination can be achieved
through the intermediate format.
Each temporary translation bridge consists of two protocol spokes (interfaces) and a
hub. The hub is responsible for error reporting and QoS monitoring of the individual
bridge. In case of translation or transmission errors the hub should communicate with
external reporting/log servers. A QoS agreement can be communicated when generat-

29
30 Architecting IIoT Software

Figure 3.1: The internal architecture of the proposed protocol translator. The upper persistent
system will provision dynamic instances as required. Each instance have a custom protocol
pairing and service configuration.

ing the request for protocol bridging. The hub shall monitor the bridge for breach of
QoS agreement and raise error warnings if needed. Each protocol spoke shall be pro-
tected against malicious users through access control policy enforcement. The protocol
spokes are responsible for all protocol and service related behaviours. Such as, active
polling/publishing when bridging from a request/response provider pattern to a pub-
lish/subscribe consumer pattern.
The multi-protocol translator is an important component of the Arrowhead Frame-
work [19]. It constitutes a core support system providing interopability and thus seamless
operation of IIoT applications.

3.2 Information Extraction

In this section the proposed system composer is described. It is responsible for creating
SoS specifications for extracting information from a SOA-based network of systems. It
must process information extraction queries and create system interconnections that can
satisfy the query. These system interconnections link independent systems into typed
pipe and filter trees, with one or more branches. Here, this linking is referred to as
system composition and results in the creation of an SoS. The system composer provides
this capability through an information extraction service. The solution here satisfies the
following qualities:
1. Agnostic of the SoS communication behaviour (event driven or polled)

2. Executable and human readable information extraction specification

3. Supports fine grained access control

4. Not dependent upon globally centralized data stores

5. Data sources, analytics and sinks remain loosely coupled

The system composer uses graph models to analyse the environment and find in-
terconnections between systems that will extract the required information. The graph
3.2. Information Extraction 31

is described using the proposed graph taxonomy. A graph model based on this taxon-
omy can be analysed/queried to retrieve functional, communication and security related
edges and nodes. The nodes in the graph taxonomy are: devices, systems, service types
and interfaces, objects, users, attributes and operations. The edges in the graph taxon-
omy are: hosted by, provided by, requires, offered by, supports, implemented by, aliases,
represents and defined.
The graph model is queried first for functional path between data source and the
data sink. Figure 3.2-a shows the resulting functional bipartite graph. If the functional
graph is connected, then the communications graph can be queried. Using functional
nodes as anchors, the graph model is queried for interface nodes. Figure 3.2-b shows the
resulting communications bipartite graph. If the communications graph is connected,

Figure 3.2: System composer graph query results. a) functional graph, b) communication graph

then the security graph can be queried. Using the communications nodes as anchors,
the graph model is queried for user, object and operation nodes. These nodes must be
connected through attribute nodes that satisfy the access control policy. If the policy
has been satisfied, then the communications bipartite graph can be used to form the SoS
specification (system interconnection rules).
As shown in Figure 3.3 the system composer interacts with external registries and
information stores to acquire data regarding the active environment. The data retrieved
is processed by a graph updater using the proposed SoS graph taxonomy. The updater
pushes the new graph data to the graphing module. This recurring process is either
periodic or event driven depending on service interface design of the external systems.
The graphing module uses a graph database (in this case a Neo4J) to store and query the
graph model. The graphing module receives requests from a query parser. These requests
are graph queries and could be SPARQL or Open Cypher requests. The graphing module
runs the received query across the graph model held in the database and returns the
result. The query parser processes the results from the graphing module and builds a
set of interconnection (orchestration) rules. These interconnection rules make up the
SoS specification and this is sent to an external SoS manager, such as the Arrowhead
Orchestrator. In parallel the query parser will have responded to the application system
32 Architecting IIoT Software

that originated the information extraction query. This application system will have
originally made its request through the information extraction service provided by the
system composer.

Figure 3.3: Information extraction. The System Composer internal architecture.

System composer was prototyped and applied to wheel loader and Nutrunner case
studies. It has been proposed to be included to the Arrowhead Framework as a core
support system.

3.3 Towards Decentralized MES

This section presents the proposed solution toward decentralization of MES.
The MES functions of production tracking, execution and workflow control are de-
centralized to the factory floor. The other functions of detailed scheduling, planning,
supply chain and quality assurance are assumed to be available and out of scope. The
proposed architecture divides the production tracking, execution and workflow control
amongst three systems: 1) workflow manager, 2) workflow executor and 3) intelligent
product. The decomposition was motivated by the different development and oper-
ational life-cycles of these functions. In Figure 3.4 the SoS level diagram shows the
relations between the proposed systems.
The workflow manager is an IIoT system that is operating on the factory floor on
edge computing. A workflow manager system is expected to run within a workstation
support one or two operators. It is a generic system and with lightweight bootstrap
configuration it can be cloned across different workstations. The workflow manager
interfaces with the MES recipe and tracking function to pull local scope workflow and
push work results. It provides service interfaces for workstation equipment and execution
functions to pull workflow steps and push results. The workflow progress can be queried
locally and the MES tracking function is kept up-to-date.
3.3. Towards Decentralized MES 33

Figure 3.4: The proposed MES workflow management solution and the interactions between its
constituent systems.

The proposed workflow manager implements the activity diagram in Figure 3.5. When
a new product arrives at the workstation, the workflow manager retrieves the product ID.
The workflow manager queries the intelligent product for the local production specifica-
tion (workstation related information). The workflow manager filters the local workflow
and provides an operation list to the workflow executor. The workflow manager receives
the operation report from the workflow executor and sends this back to the intelligent
product. The product then leaves the workstation.

Figure 3.5: The proposed workflow manager activity diagram.

The workflow executor behaves as an instructor to the workstation equipment.

It owns all service interfaces to equipment at the workstation. It will request from
the SoS manager the identities of available equipment. Workflow manager provides a
workflow state machine for a given product. The state machine specifies what operations
34 Architecting IIoT Software

to perform to produce the given product. Workflow executor delegates jobs to workstation
equipment as specified in the state machine. It utilizes its SoS to carry out the workflow
state machine.
The workflow executor is a software system running on edge computing. It does
not interface with services outside the local workstation scope. Its integration with
(legacy) equipment interfaces means that it may have customization dependant on the
workstation.. The workflow executor block diagram is shown in Figure 3.6. It illustrates
the dynamic interfaces to equipment and operation step state machine engine. The
equipment interfaces grow and shrink based on the interconnection rules returned from
the SoS manager (Arrowhead Orchestrator).

Figure 3.6: The proposed workflow executor architectural block diagram.

Intelligent product is an IIoT system. Initially it is a software system containing

workflow, recipe, product specification, tracking and other information related to pro-
duction. At some stage of production this software system moves to the physical product
(hence, intelligent product), carrying full production history and results. At this stage
it must have sufficient computing power and networking capability. It continues its op-
erational life even after leaving the production facility, continuing to store and provide
information about the product. So it becomes a truly IIoT system.
Within the Far-Edge project, at this stage, part of this concept was developed. Intelli-
gent product software has workflow recipe and production tracking. This is implemented
using Hyperledger smart contracts. As a physical (unintelligent) product approaches
a workstation, a barcode or RFID is scanned and the workflow manager retrieves the
workflow from the smart contract. As work is completed, the workflow manager sends
tracking results to the intelligent product through another smart contract. In its current
form, the intelligent product is part of the overall workflow management solution and
part of the pilot.
The interconnection of these systems must be organized through engineering data.
3.4. Challenges of Software Engineering in IIoT 35

For example, in [82], a plant description system is used to engineer the workstation
prior to any workflow execution. The workflow and Arrowhead orchestration are closely
related. There are still design questions on how to best delineate functions of both.
These questions are ongoing research and discussion within both the Productive 4.0 and
Far-Edge project consortium.
Proposed systems have been prototyped and are part of the autonomous workstation
pilot at Volvo Trucks facility.

3.4 Challenges of Software Engineering in IIoT

As already discussed in Chapter 2, the complexity of software is increasing as embed-
ded/control systems meet with more abstract computer science. This section summarizes
the challenges this complexity poses to current and future engineers. The architecture of
the software must help with tackling the challenges outlined below.

1. Intersection of concerns IIoT applications cross several engineering domains.

This collision of disciplines and thinkings requires careful consideration regarding where
functionality should be placed. Depending on the method of separation of concerns,
this challenge can be less significant. A traditional vertical layered architecture enables
independence of technical domains. Whereas a cross cutting architecture (slices) will de-
mand multidisciplinary development teams. Either way, this challenge will remain with
the question who should be concerned with a particular functionality? This challenge is
closely linked to the functional decomposition but also includes non-functional concerns.

2. Functional decomposition A large part of the architecture modelling is the func-

tional decomposition - what capabilities are required and who provides them. IIoT
applications are fundamentally spatially distributed and often have limited computing
power or communications links. Thus, a capability may need to be split into two smaller
sized, or two capabilities be combined into one. For example, if the communications
link cannot support streaming, then a Fast Fourier Transform (FFT) may be required
on the sensor. In this way, functional decomposition has impacted IIoT device processor
selection. Vice-versa, the available processing power may mean that FFT is not possible
at the sensor, and so an intermediary node may perform the FFT. This functional flexi-
bility shows that the architecture must enable functionality with freedom of movement.
Functional interface design also impacts the ease of movement of deployed functions.

3. System interdependence Managing dependencies between systems is a growing

challenge as intelligent software and functionality is spread out between production pro-
cesses. If the chain functional dependence between a set of systems breaks, then all up
36 Architecting IIoT Software

stream systems could be interrupted. System life-cycle is another issue with system inter-
dependence: Change management that requires synchronization amongst interdependent
systems increases deployment effort (cost of change).

4. Data modelling The intersection of different domains exacerbates the data mod-
elling issue for IIoT. Architects may need to utilize semantic web with highly typed
representations of data, or plain text /binary data over a low power wireless link. With
evolving techniques and technology this challenge is always moving. This challenge is in
semantics of data, usage of data, storage of data and serialization of data.

5. Communications bottlenecks Highly connected machines cause scalability issues

in the communications lines. A centralized messaging mechanism suffers from single point
of failure and bandwidth limitations. Point-to-point communications can be exhausting
for constrained devices (constraints being in bandwidth, energy, processing power or
memory). In case of degraded performance, routers and network devices have high im-
pact on IIoT applications.

6. User interactions Lines of separation between computers and humans are not
clear cut. User interact with machines with augmented reality, robotics and etc. So-
lutions cannot target a single display and information must flow between augmented
reality and traditional displays seamlessly. Virtual commissioning and human-machine
co-working also require fluid movement of information.

7. Systems of Systems design and definition Development of individual equip-

ment requires engineering knowledge. However, combining these systems into business
value may not need deep engineering knowledge. In fact domain knowledge is required to
extract value from a SoS. Therefore this challenge highlights the user (domain engineers)
ability to drive change in a SoS function.

8. Legacy equipment integration This challenge is especially acute for an industry

that has traditionally seen equipment life-cycles of up to a decade. These legacy systems
impact functional decomposition, security vulnerabilities and migration strategies.

9. Data security A big challenge with so much networked data being made avail-
able and with increasing operational dependence on access to this data. Also, due to
networks of multi-vendor applications open traditionally secure environments to sloppy
3.5. Principles for Software Architecture design in IIoT 37

development or malicious code. The lines of attack for social engineering increases. This
challenge deals with: access control, forged credentials, social engineering and networks
of multi-vendor applications

10. Protocol heterogeneity This is caused due to intersection of domains with

differing communications requirements. It is evident through the number of protocols
proliferated within industry that convergence onto a single protocol capable of handling
all communications requirements is not possible.

11. Determinism Deterministic behaviour is something that OT cannot ignore. Op-

erator safety is dependent on this. Included in this challenge is real-time communications.
These factors encompass many non-functional requirements and shape the functional de-
composition.

12. Development practices OT engineers are accustomed to high cost well planned
stage-gate projects. While, IT developers prefer iterative time-boxed projects. Although
there still remain differences between IT and OT, the software life-cycle is still becoming
shorter.
IIoT solutions must support evolutionary development and have fast mean time to
failure. Life-cycles of IIoT systems can vary greatly. At any given time, systems will
be at different stages in their life-cycle. The challenge for architects is to delineate
between systems so that sporadic disruptions do not interrupt overall operation of the
IIoT application.
The challenges listed here are not application requirements. They are general issues
that architects must be aware of and will influence architectural considerations even
before requirements are considered. In order to address these challenges from a software
architecture perspective, first a set of architectural principles must be defined.

3.5 Principles for Software Architecture design in

IIoT
Architectural principles are fundamental to the reasoning of early design decisions. A
selected architectural style must satisfy the principles proposed in this section. In turn,
the concrete software architecture must conform with the architectural style. Each step
toward a concrete architecture constraints the final solution or the available means to
the ends. In Figure 3.7 the relation between architecture abstraction concepts is shown.
Although, the number of constraints introduced by the architecture principles are small,
they have long lasting impact. Common solutions to problems can be captured in ar-
chitecture patterns or reference architectures [83]. As opposed to a style that does not
38 Architecting IIoT Software

specify component instances, an pattern will have instances of components that will re-
solve a specific problem or set of problems. For example, Bell documented common SOA
patterns in [84]. Each concrete architecture should be able to track its reasoning through
the layers up to an architectural principle.

Figure 3.7: Software architecture abstraction layers. Small changes in the principles at the core
echoes through all the layers.

We begin elaborating the principles by adapting existing architectural principles.

SOA, as presented in Chapter 2, has several principles that we use as a starting point.
Here, the principles are defined in a manner independent of the architectural style (SOA).
So for example, services, systems, components and agents are referred to as decomposition
elements.

1. Loose coupling (Adapted from SOA) Firstly, each decomposition element is inde-
pendent of the internal design of other elements. Meaning that, for example, logic, intel-
ligence, implementation technology or in-fact (internal) composition can change without
impacting other elements. This helps to improve maintainability and changeability of
the elements.

2. Searchable (Adapted from SOA) Loosely coupled decomposition elements can

discover each other. The location of both the physical process and the address designa-
tion can be moving and need not be static. This supports a means of scaling through
transparent caching or proxies. Deployment and (re-)adjustment are two major factors
this principle supports.

3. Run-time binding (Adapted from SOA) Functionality provided by different de-

composition elements are linked at runtime. This is usually achieved through messaging
interfaces of remote loosely coupled functon. Compile time binding means that any
change to the location or function of the underlying library forces a re-compile of the
whole system. Of course software libraries are important aspect, however, applications
with large library dependencies increase the cost of change.
3.6. The Principled Decomposition 39

4. Autonomy Decomposition elements independently make decisions regarding life-

cycle and capability offerings and dependencies. Closely linked to the principle of loose
coupling, autonomy requires that the decomposition elements exhibit resilience to exter-
nal failures. Usage, spatial and life-cycle are the main areas of autonomy.

5. Specialization This principle suggests that each element should have a limited
number of concerns. The decomposition elements can be responsible for varying amounts
of functional or business logic. An extreme of this principle would be as with microservices
and limit each element to a single responsibility. This principle must be balanced with
other principles such as autonomy.

6. Data at its source This principle guides the architecture to avoid centralized
data stores where possible. Data is a commodity with value. Access must be controlled
and in some cases charged. This principle undertakes to direct privacy by design as a
characteristic of the architecture prior to any development. There are mechanisms to
allow cache and proxy, whilst allowing data source origin having certain controls over the
data.

7. First person perspective This principle guides the architecture to ”empathise”

with the decomposition element being described. The architect must think from the
point of view of the decomposition element. Not as simply as limited components of
the software application. The decomposition elements have their own agency. Apply-
ing this principle supports achieving autonomy, privacy by design, loose coupling and
specialization.
In Table 3.1 these seven principles are mapped to the challenges they address. In Fig-
ure 3.8 the challenges and clustered around respective design principles. This illustrates
the implicit influence and connection of these challenges. The design decisions driven
by the principle will address the respective challenge and to some degree the related
challenges in the cluster.
This section has presented the principles for design of a software architecture in the
context of IIoT application development. The practice of architecture in the IIoT requires
holistic consideration of the decomposition elements. The principles direct architects to
think about IIoT applications at different levels of abstraction. It encourages architects
to investigate methods of reducing security vulnerabilities without relying on encryption
or authentication. Primarily, the principles lead architects to think of the decomposition
elements with respect and care. Recognizing that the software will have a journey, build
a relationship with developers and leave an impact on its users.

3.6 The Principled Decomposition

Here, architecture decomposition is defined as the process of splitting and joining software
capabilities into implementable groupings and abstract logical groups.
40 Architecting IIoT Software

Table 3.1: Mapping of principles to the challenges they address

Challenge Principles
Intersection of Concerns Autonomy, First person perspective
Functional Decomposition Autonomy, Specialization,
First person perspective
Systems Interdependence Loose coupling, Autonomy, Specialization
Data Modelling Autonomy, Data at its source
First person perspective
Bottlenecks Data at its source
User Interactions Loose coupling, Data at its source
SoS Design/Definition Loose coupling, Searchable, Run-time binding
Legacy Equipment Loose coupling, Autonomy
Data Security Data at its source, First person perspective
Protocol heterogeneity Autonomy, Run-time binding
Real-time Communications Autonomy, Specialization
Development Practices All Principles

Figure 3.8: IIoT Software challenges clustered on software architecture principles

Architectures such as SOA and MAS address decomposition aspects outside the scope
of the style. SOA uses architecture patterns [85] to guide application decomposition. In
MAS, design methodologies are used to guide the decomposition process, for example,
system oriented methodologies, such as Tropos [86] and Prometheus [87], propose agent
task and role decomposition. The approach proposed here, guides architects on how to
apply the principles to decomposition in a generic manner. Therefore without introducing
specific patterns, architects can begin to shape the overall architecture. To this end, it is
assumed that the requirements of the IIoT solution have been gathered and are presented
as a set of feature requirements. These features document both functional and non-
functional requirements, although they are more focused on functional requirements.
3.6. The Principled Decomposition 41

These features are used by the architect to decompose the application/solution. The
objective of the architect is to map these features into elements, connectors and modules.
Along with the rationale, these architectural components will be presented in different
views, dependant on the intended target stakeholders. Each of these steps should be
recorded and kept.

Autonomy A high level solution can be modelled as a black box with all capabilities
required to satisfy the requirements - with no detail on how it achieves this. This is insuf-
ficient and a developer will come up with an architecture based on limited information.
Studying the feature life-cycle, usage and spatial separations, this black box is broken
down into groups of capabilities. This first step of functional decomposition resulted in
a number of black boxes representing groups of capabilities that have similar demands
of autonomy. The next step is to look into specialization.

Specialization The architecture is now black boxes decomposed based on; life-cycle,
spatial distribution and decision making autonomy. Each black box of capabilities could
have quite different objectives or implementation requirements (i.e. legacy equipment).
Through each block box, split the capabilities based on the level of relatedness. Creating
specialization is at the architects discretion, and there is no definite way to determine
relatedness. However, where multiple processes are involved, the lines may be drawn
based on equipment scope boundaries. This will result in a higher granularity of black
boxes that are more meaningful to developers.

Data at its source The black box groupings of capabilities can now be related by data
usage and storage. The architect should guide (constrain) the solution to utilize data at
its source. This is done by introducing direct(ed) relationships between the capability
black boxes. Where a it is not possible to maintain data at the source, then additional
(non-functional) cache/proxy/archive capabilities should be added. The architecture now
has hierarchical groupings of capabilities decomposed by autonomy and specialization and
connected through directed data relationships.

First person perspective The black boxes can now be viewed as systems. Each
system has a set of capabilities that it is responsible to fulfil. By putting themselves into
the perspective of the system the architect questions how changes in the environment may
impact capabilities. Looking outward at other systems, what are my functional and data
dependencies? Do I have any fail safe contingencies or are there any mitigating factors
for dependency failure or loss? How will others make use of the capabilities I am offering?
Looking inward, what is my life cycle? What mechanisms do I have for optimizing my
life time? How can I trust other systems and how can I prove who I am? What are
my security vulnerabilities or strengths? By applying the first person perspective, the
architect will have captured capabilities as concrete service interfaces. This principle
has the added benefit that an architecture that rationalizes such considerations can be
utilized by engineering teams to identify and better guide implementation technology
42 Architecting IIoT Software

choices. In addition, this step must address the security constraints, keeping data safe
and safeguarding against malicious interactions.

Loose coupling, Searchable, Run-time binding From an architectural perspective

these three principles ultimately guide the interface implementation architecture. Ability
for SoS composition is ultimately down to these principles. The architect must the
decomposed systems to ensure they take action according to these principles.

3.7 The Architectural Style

In the previous section the functional artefacts of the architecture are now decomposed.
Each individual system could utilize a different internal architecture; such as agent archi-
tecture, object modelling, pipe and filter, layered architectures or any other. Additionally,
the intended high level collaboration between systems needs to be elaborated. In this
section the architectural style will be developed through elaborating the views and the
decomposition at each view. Because decomposition is done by utilizing architectural
principles, the style will naturally exhibit these principles. Before examining the style we
start with a short recapping of what an architectural style is. Referring back to Figure
3.7, architectural style fits between principles and reference architectures and patterns.
There is no concrete delineation between a style and a pattern/reference. It is possible
for a reference architecture to be more abstract than a style. But over all an architectural
style should only contain the expressions required in order to create the form, structure
and decomposition of an architecture. This includes; elements, connectors and views that
are used in creation of a concrete architecture. A style provides context and a fundamen-
tal framework for analysing the rationale behind architecture decisions. While a pattern
or reference architecture intends to describe reusable solutions to common problems, a
style is a way of thinking or an approach to solution design. Therefore, an architectural
style does not specify any component instances. On the other hand reference architec-
tures often describe precise component details or predefined capability separations. The
rationale behind a reference architecture or pattern may not always be communicated or
become lost due to “copy paste” reuse.
The proposed architecture style is illustrated in Figure 3.9. It shows the discussed
challenges, principles and decomposition. It builds upon such theories and architectures
as systems, SoS and SOA. The Arrowhead Framework has applied SoS and SOA to great
effect with industrial pilots [19]. SOA has additionally been used for decades in IT and
is a new trend for OT in industry (research has been on-going). The decomposition
method has already been presented. They style then consists of views and elements.
The elements are built up from SoS and SOA notation and experience from working on
the Arrowhead Framework. Each view defines a form of the architecture. Drawing on
RAMI 4.0 gives a starting point for understanding the stakeholder views.These views are
described individually in the following sub-sections. The proposed architectural style is
not born in vacuum and builds on existing architectural styles, such as layering, n-tier,
3.7. The Architectural Style 43

client-server and pipes and filters. Depending on the specific application demands, the
proposed architecture style can also be combined with other architectural styles.

Figure 3.9: The proposed architectural style for designing IIoT applications.

An on-going example will be used to illustrate the proposed architectural style. The
example is from a pilot use case of the Far-Edge H2020 project. It involves two primary
cases; plug and produce and autonomous work station. For a high level introduction to
the use case: 1) When moving equipment between workstations there is need to create
automated configuration of the equipment and work stations. Such that domain specialist
resources can make factory floor changes with out assistance from IT resources. 2) There
is a need to make the work station able to operate with little or no factory wide IT
infrastructure. It is autonomous and decoupled from factory wide IT applications such
as MES.

3.7.1 Views
The Figure 3.9 illustrates the views as they reflect the interests and concerns of all stake-
holders. A view must capture important design decisions relevant for the stakeholders
(including the architect). Views are a critical method of creating focus on relevant
44 Architecting IIoT Software

concerns. The RAMI 4.0 layers define 6 areas of concern; business, functional, data,
communication, integration and asset. Roughly speaking, IT has focused on the upper 3
layers. This is in contrast to OT that focuses on the lower 3 or 4 layers, often providing
a thin service layer for interaction with IT infrastructure. Industrial middleware [88] has
positioned itself somewhere within the middle 4 layers, generally abstracting the physical
assets and remaining flexible to business processes. Suggested here, is to capture the
different RAMI 4.0 layers as application network views. Each layer captures the point
of view distribution across relevant RAMI4.0 hierarchical components. The views in this
architecture style can be as simple as a catalogue of concerns.
As already mentioned, systems and SoS make up the core modelling elements. The
concept of systems map well to capturing the concerns of each view. IIoT elements
are modelled as systems, such that, physical and logical independence is provided for
each element. Battery, processor, RAM, flash, networking (mesh vs tree), actuation and
sensing, are at the discretion of the domain architects and developers of each individual
system. This provides the architects of each IIoT system with the capability requirement
from business, function, information, communication, integration and asset concerns.

Business view
The business view captures concerns regarding business value
generation, regulatory compliance and domain or organization
norms.

The business view must capture the networked distribution of domain logic across the
IIoT device hierarchy. This requires a catalogue of requirements that capture value gener-
ating activities, regulation compliance activities and any organizational norms. For each
requirement a matching capability must be identified so that a set of business capabilities
are defined that will full fill all business requirements. Finally, the business capabilities
should be mapped to IIoT device hierarchy groupings. At this stage there is no direct
mapping to systems, only to the system classes (i.e. product, field device, work centre,
enterprise and etc.). This means that requirements and capabilities at the business layer
are visually allocated a specific layer of the IIoT hierarchy. The decomposition of the
capabilities with the subsequent mapping to hierarchy have applied the decomposition
technique defined in Section 3.6. This means that visualization and decomposition of
business logic leads to understanding infrastructure requirements and functional require-
ments. Thus it is the starting point for the functional view, which must support the
business operations and objectives.
During creation of this view/form, the architect must ask themselves:

1. What are the business considerations each component group are responsible for?
2. What are the dependencies between constraints (therefore dependency between
groups)?
3. Are there any direct and indirect impact of the constraints on the overall design of
each group?
3.7. The Architectural Style 45

For example, a regulatory requirement may require execution of certain business op-
erations. Figure 3.10 captures the capability mapping required to provide regulatory
compliance to safety nut tightening. That is, the safety nut, controlling the wheel align-
ment, must be tightened to a specific torque and follow through angle. This value must
be recorded and kept on record by the truck manufacturer. This view then shows how the
responsibility for performing the logical operations are distributed amongst the differing
hierarchical components to achieve the objective. With this view, the architect is able
to validate the business rule for wheel alignment safety nut tightening.

Figure 3.10: The business view is made up of simple black boxes representing each layer of
the IIoT device hierarchy with the allocated business capabilities decomposed from the business
concerns.

Functional view
The functional view captures concerns regarding system identi-
fication and capability assignment.

The business view captured the capabilities required to deliver value, stay within
regulations and any organizational norms. It has also allocated those capabilities to
the IIoT device hierarchy. Next the business capabilities are turned into functional
requirements. Domain knowledge is required to turn business capabilities into functional
requirements. In addition user requirements, such as interaction requirements, must
be captured. These requirements are all captured in a catalogue that will enable the
business capabilities and user interaction needs. For each catalogued requirement, a
functional capability must be defined, such that it satisfies the business capability at the
required IIoT device hierarchy level. The functional capabilities must be decomposed
according to the method in Section 3.6. Decomposition of the functional capabilities will
result in capability groupings that can be referred to as systems. Hence, the component
systems of the architecture have been identified, mapped to the IIoT device hierarchy
and their capability offerings/dependencies defined. The functional view has captured
46 Architecting IIoT Software

the distribution and relationship of how objectives across the networked systems are
satisfied. Each system provides capabilities as services that can collaborate to complete
business objective. The systems provide and consume these services so the functional
dependencies between systems are identified as service interfaces. This is the view that
will often be used when communicating the solution back toward the customer and/or
system users.
While creating this view, the architect should be asking questions such as:
1. What functionality needs to be implemented to support the business capabilities?
2. How will users interact or work together with the systems to deliver the business
capabilities?
3. How will systems interact with one another to deliver the business capabilities?
4. Are there any systems with too many dependencies (i.e. high impact if failure
occurs)?
Following on from the earlier business rule regarding safety nut tightening, the systems
and services required to satisfy the business capability are shown in Figure 3.11. Of
course, these functional capabilities can also be reused for other business capabilities.

Figure 3.11: The functional view consists of the component systems and capability services with
dependency identification.

Information view
The Information view captures the concerns of what form data
takes and meaning it has.
The functional capabilities and dependencies have created service exchange paths
between systems. It is the business capabilities and user interaction requirements that
3.7. The Architectural Style 47

have motivated the systems and service exchanges. The data that is generated by the
execution of a capability or required in order to perform a capability has not been defined
yet. The Information view captures what data is required/generated, what the meaning
of the data is and what format it takes.
While creating this view, the architect should be asking questions such as:

1. What does the capability do? How does its execution effect the meaning of the
information?
2. Which capability produces the information?
3. Which capability requires the information?
4. What are the dependencies between information?

The proposed architectural style does not dictate the elements or methods used in
the view. Any ontological language could be used to describe the information or an
external ontology could be referred to here. It is also possible to perform conceptual
data modelling here using techniques such as entity-relationship modelling [89]. In Figure
3.12 an example Entity Relationship Diagram (ERD) is shown. It defines the conceptual
model of the information required to make use of the nut tightening capability. It also
defines the information generated by execution of the nut tightening capability. The
generated information is shown to relate to the product entity. This view has defined
what information is used by the functional capabilities and where the source and sinks of
the information are. Using this model developers can understand exactly the information
that must be stored for the safety nut tightening of the wheel alignment.

Figure 3.12: This Entity Relationship Diagram shows the information used and generated by
the Nutrunner as it executes the nut tightening capability. It also relates the results back to
product being worked upon.
48 Architecting IIoT Software

Communication view
The Communications view captures concerns of system interac-
tion patterns and methods.

Until now, the individual systems have been identified. Their capabilities and service
interfaces defined. The format and meaning of information exchanged between systems
specified. But how does the information move between systems? In which direction is
service exchange initiated? The communications view captures how information is passed
between the systems.
While creating this view, the architect should be asking questions such as:

1. How does information move between systems?

2. In which direction is service exchange initiated?
3. What are the suitable interaction patterns?
4. How secure does the information need to be during transit?
5. Are there any real time communications required?
6. How are communication QoS considerations handled?

To move information between systems it must be serialized, compressed, encrypted,

sent on the “wire”, decrypted, decompressed and de-serialized. On top of this, interaction
patterns must be selected. Common ones being request/response, publish/subscribe
or repository (shared memory). Interaction patterns and methods do not need to be
homogeneous across the distributed architecture. Communication requirements can differ
considerably across the distributed system. For example, a constrained IIoT device may
require a specialized protocol with small library foot print or packet size. This view will
show the communications network between the systems with allocation of capabilities
to a bus. For example, CoAP bus and HTTP bus would be represented as separately
although they use the same IP network. Network segregation with protocol gateways or
translators should be shown in this view. Figure 3.13 shows the functional view with the
communication lines added.

Integration view
The Integration view captures concerns of interfacing between
the virtual and real worlds.

Thus far, the individual systems interconnected with each other through communica-
tions buses with known information formats and capabilities. Due to the nature of IIoT
applications, many of the systems will have sensors and actuators to perform real world
interactions. The integration view captures what hardware interfaces are required by the
systems in order to interact with the physical world.
While creating this view, the architect should be asking questions such as:

1. Do the systems require A2D or D2A conversion?

2. Are there any legacy interfaces to be supported (i.e. serial interfaces)?
3.7. The Architectural Style 49

Figure 3.13: The Communication view specifies the communications patterns and methods as a
shared bus between systems.

3. What support for digital inputs/outputs are required?

4. Does the system need an integrated sensor?
Not all questions need to be answered. The architecture should show the important
constraints or provide a selection of possible constraints. This can give valuable freedom
to designers for integration selection, so this decision is at the discretion of the architect.
For example, Figure 3.14 shows the touch screen interface to the display system and the
legacy nutrunner equipment with a serial interface. The databases shown here could be
represented in the system view as an internal component of the system, or in this case
as an outside store with a direct interface.

Figure 3.14: The Integration view highlights important real world or legacy equipment integra-
tion points.

Asset view
The Asset view captures concerns of physical world devices.
System requirements for integration and communication have been formed. The Asset
50 Architecting IIoT Software

view is where architects will identify important devices for running the systems on. In
addition legacy devices are shown here with the legacy communications bus defined in
the integration view. Additionally, the asset view should show networking appliances
and network interfaces such as Ethernet, Wi-Fi, IEEE 802.15.4 or etc.
While creating this view, the architect should be asking questions such as:

1. Is the IIoT device battery powered or mains powered?

2. Is it possible for the IIoT device to utilize power harvesting?
3. How secure must the system be?
4. How much cryptographic processing will be required?
5. Will the system need to store a lot of information?

The device requirements such as battery power, cryptography hardware acceleration,

processing power and volatile and non-volatile memory should be specified. Again, in this
view the architect should concentrate on important constraints in order to allow designers
flexibility to select optimal decisions based on first hand implementation information.
The safety nut tightening scenario is shown in Figure 3.15. Not all hardware are shown,
only those that require special attention. In this case a high level of detail is capture for
the devices shown.

Figure 3.15: The Asset view is communicates hardware specifics - this view can be very detailed,
or leave more flexibility to designers.
3.7. The Architectural Style 51

System view
The System view captures concerns of a single system with all
the software layers present.

Thus far the architecture style has focused on network views of the IIoT application.
This is no accident, as IIoT applications are highly distributed. For each of the IIoT sys-
tems defined so far this view must be developed. The system view is an implementation
view that can be utilized for estimation and development. Therefore, it will be utilized by
software and hardware engineers in building the system. It must capture the system con-
cerns related to: asset, integration, communication, information, function and business.
This view will capture the system architecture, it could utilize a object oriented, layered,
pipe and filter, event driven or agent architectures etc. It it could be some suitable com-
bination. For example, a suggestion could be use a hybrid of two styles; 1) a layered
style captures the asset, integration and communication concerns and 2) an object style
captures the information, function and business concerns. Hardware Abstraction Layers
(HAL) create a platform upon which high layer objects can be manipulated. The layered
style indicates clear dependencies and responsibilities. Object oriented modelling pro-
vides flexible definitions and changing interactions between components. This is shown
in Figure 3.16.

Figure 3.16: The System view is utilized by the architect to communicate a model of the system
that can guide implementation.
52 Architecting IIoT Software
 Chapter 4
Contribution

The appended publications are in chronological order and will be presented in this
manner. This section presents a summary of the paper and the contributions made by
the author.

4.1 Summary of Appended Publications

Paper A: A Survey of Commercial Frameworks for the Internet of Things
Authors: Hasan Derhamy, Jens Eliasson, Jerker Delsing and Peter Priller
Published in: Proceedings of IEEE Emerging Technologies for Factory Automation
conference 2015, Luxembourg.
Summary: This paper surveyed currently available commercial frameworks which sup-
port development of Internet of Things. The investigation identified that there are two
main approaches to IoT frameworks; 1) data-centric - cloud based frameworks, and 2)
device centric - automation based frameworks. The cloud based frameworks offered very
good multi-protocol support due to a centralized software bus approach. While on the
other hand, device centric frameworks concentrated on a single protocol and implemen-
tation, management and control of devices.

Contribution: The author was responsible for surveying frameworks and reporting
on findings.

Paper B: Translation Error Handling for Multi-Protocol SOA Systems

Authors: Hasan Derhamy, Pal Varga, Jens Eliasson, Jerker Delsing and Pablo Punal
Pereira
Published in: Proceedings of IEEE Emerging Technologies for Factory Automation
conference 2015, Luxembourg.
Summary: In this paper the initial investigation into interoperability for Industrial In-
ternet of Things was performed. It was identified that interaction pattern differences,

53
54 Contribution

exemplified by error handling, could differ significantly between protocols. In addition,

robust systems require strong error handling. Proposed here was a method for translat-
ing error mismatches by means of defining behaviors between protocol translations.
Contribution: The author contributed to defining the error handling technique and the
implementation and testing.

Paper C: IoT Interoperability - On-demand and low latency Transparent Multi-protocol

Translator
Authors: Hasan Derhamy, Jens Eliasson and Jerker Delsing
Published in: IEEE Internet of Things Journal, 2016.
Summary:This paper presented a full Interoperability solution for the IIoT. It pro-
posed a transparency architecture, and software architecture for a scalable and efficient
protocol translator. The proposed translator is a participant in a SOA-based service
composition. Using a hub-and-spoke architecture, the translator is extensible to more
advanced translation, i.e. security, syntax and semantics, and extensible to fan-in and/or
fan-out translation.
Contribution: The author was responsible for transparency design, service provisioning
and orchestration, software architecture, implementation and testing.

Paper D: Orchestration of Arrowhead services using IEC 61499: Distributed Automa-

tion Case Study
Authors: Hasan Derhamy, Dmitrii Drozdov, Sandeep Patil, Jan van Deventer, Jens
Eliasson and Valeriy Vyatkin
Published in: Proceedings of IEEE Emerging Technologies for Factory Automation
conference 2016, Berlin.
Summary: In this paper a case study of using Arrowhead SOA framework in an IEC
61499 manufacturing line was investigated. The paper is a work-in-progress showing in-
termediate results of using a Services based loading and unloading point collaborating
with a IEC 61499 based processing station. The translator was transparently employed
here with advanced orchestration in order to connect the IEC 61499 software application
with the services operating in a constrained environment.
Contribution: The author was responsible for the service modeling and design, for ser-
vices implementation and Arrowhead framework integration.

Paper E: Service Oriented Architecture Enabling the 4th Generation District Heat-
ing
Authors: Jan van Deventer, Hasan Derhamy, Khalid Atta and Jerker Delsing
Published in: The 15th International Symposium on District Heating and Cooling,
2016, Seoul.
Summary: This paper proposes a new approach for the next generation of district
4.1. Summary of Appended Publications 55

heating. The approach combines SOA-based data services with advanced optimization
services. This provides a new platform for achieving efficiency gains which results in cost
savings.
Contribution: The author has contributed toward applying the Arrowhead framework
and describing how it operates within the district heating use case.

Paper F: Protocol Interoperability of OPC UA in Service Oriented Architectures

Authors: Hasan Derhamy, Jesper Rönnholm, Jerker Delsing, Jens Eliasson, and Jan
van Deventer
Published in: Proceedings of IEEE Industrial Informatics, Emden, Germany, 2017
Summary: This paper extends the translation concepts of earlier works with support for
OPC UA. OPC UA is a sophisticated communications protocol that combines transport,
security and information into a single protocol. For this reason, performing translation to
and from it with pure transfer protocols such as HTTP, CoAP and MQTT is a challenge.
Contribution: The author has contributed to writing this paper and to the research
in mapping the communications protocols. The author was also responsible for building
the demonstrator and testing the prototype implementation.

Paper G: In-network Processing for Context-Aware SOA-based Manufacturing Systems

Authors: Hasan Derhamy, Jens Eliasson, Jerker Delsing, and Jan van Deventer
Published in: Proceedings of IEEE Industrial Electronics Conference, Beijing, China,
2017
Summary: This paper proposes the use of graph modelling and queries to create an
effective Information Centric Networking over lay. Three key aspects: 1) Graph Query
Language usage, 2) Dynamic System Provisioning, 3) Composition of Systems. The
paper uses Neo4J graph database and the Open Cypher graph query language. Open
Cypher uses a ASCII art form of graph query and supports property graphs. Meaning
that both nodes and edges can have properties associated.
Contribution: The author was responsible for the researching and proposing the SOA
approach to in-network processing, prototyping proof of concepts and proposal.

Paper H: Workflow Management for Edge Driven Manufacturing Systems

Authors: Hasan Derhamy, Mattias Andersson, Jens Eliasson and Jerker Delsing
Accepted to: Proceedings of IEEE Industrial Cyber-Physical System, St Petersburg,
Russia, 2018
Summary: This paper presents an edge automation workflow system that allows dis-
tribution of MES data to the factory floor. It utilizes Industry 4.0 concepts such as
administration shells and the RAMI4.0. The Workflow manager is a system that will
retrieve the production order recipe from a smart product or its digital twin. It then
communicates with Cyber Physical Systems or executors to complete the workflow and
56 Contribution

return the full results to the smart product. It uses SOA and SoS so that it is able
to maintain a local archive or send the results to the cloud for permanent storage. It
is based on work with Volvo Trucks Corperation for the Far-Edge and Productive 4.0
projects.
Contribution: The author was responsible for the research, designing the workflow
manager and executor architecture, prototyping the demo systems, testing and writing
the paper.

Paper I: System of Systems Composition based on Decentralized Service Oriented Ar-

chitecture
Authors: Hasan Derhamy, Jens Eliasson and Jerker Delsing
Submitted to: IEEE Internet of Things Journal, 2018
Summary: This paper extends the proposal on using graph theory for composing sys-
tems of systems. It presents a formal graph description of the Arrowhead framework and
service definitions. Utilizing the formal graph relations, paths are then found for logical
interconnection, communication interconnection and security interconnection. The secu-
rity graph model is based on the NGAC model.
Contribution: The author is responsible for the graph model of the Arrowhead Frame-
work, the graph queries for discovery of SoS paths and for injection of local bridging -
which connects disconnected graphs (i.e. where protocol interoperability is present). The
author is also responsible for the text.

Paper J: Software Architectural Style for the Industrial Internet of Things

Authors: Hasan Derhamy, Jens Eliasson and Jerker Delsing
Submitted to: IEEE Internet of Things Journal, 2018
Summary: This paper presents a software architectural style for decomposition of IIoT
applications. It proposes architectural design principles that are autonomy, specializa-
tion, data at its source and first person perspective. Traditional SOA principles of loose
coupling, searchability and run-time binding were also adopted. The proposed archi-
tectural style utilizes SoS and SOA to guide the capability decomposition process. The
architectural style was exemplified on a case study of safety-nut tightening for truck
wheel alignment.
Contribution: The author is responsible for the principle idea and formulation of pro-
posed architectural principles and style. The author conducted the case study and was
main contributor to the text of the paper.
 Chapter 5
Conclusion and Outlook

The thesis presents findings in the domain of IIoT regarding communication protocol
interoperability, information extraction, decentralized MES, software capability decom-
positions; and proposes an architectural style based on SOA and SoS principles.
The overall research approach has been through cycles of identification, concept
proposition, prototype implementations and final reflection and retrospective analysis.
This approach is repeated through small projects, each building on the result of the
previous. The first project was tackling the challenge of communication interoperability
within IIoT. The solution involved dynamic translators being injected to the communi-
cation path. The dynamic aspects of protocol translation were identified to be useful
for information extraction in IIoT Systems. Building on this, system composition was
proposed to create SoS and further developed on a use case. This then lead to propos-
ing an approach for workflow management within IIoT. This ultimately resulted in the
proposed architectural principles and style.
The next sections summarizes the research contributions and outlines a vision toward
the future.

5.1 Conclusion
The contribution of this thesis is towards enriching the body of knowledge within the
domain of software architecture in IIoT. The challenges and requirements of IIoT on
software are studied and analysed, resulting in a proposed set of architectural principles.
Several prototype implementations have been developed. All findings contribute to the
research community, where they can be further utilized.
IIoT applications are made up of highly interconnected and ubiquitous computing sys-
tems that are embedded into their physical environment. An IIoT application is likely
to evolve in a piecemeal manner as each constituent system goes through a distinct life
cycle. Also, technological advancements are rapid and so change is frequent. Software
without adherence to some form of clear architecture leads to brittle applications that
resist change. Therefore, a well defined architecture is required to guide software devel-

57
58 Conclusion and Outlook

opment and integration. On the other hand, a highly prescriptive software architecture,
without clear rationale, results in a resistance towards adoption of new technologies.
The proposed architectural style utilizes systems, SoS and SOA principles. The style
promotes clear rationale behind design decisions at all levels. It provides a framework
for introducing change and handling system fragmentation and complexity.
When making changes, the designer should be guided by layers of architecture. Where
required, the designer can break compliance with the reference architecture, so long as the
architectural style is adhered to. The style can also be modified so long as the principles
are then preserved. However, architectural principles must avoid changes or there must
be a strong rationale behind the alterations. Changes at this layer of the architecture
will have wide impact on all application systems.
The research contributions are summarized below.
Q1 What are the possibilities and limitations for efficient and interoperable multi-
protocol IoT networking?
The results of this thesis show that it is possible to have multi-protocol interoperability
without the use of fixed gateways or proxies. The use of an on-demand multi-protocol
translator allows for seamless communication between two systems incapable of direct
messaging. It is possible to maintain advanced error detection, trust bridging and QoS
monitoring. The options become severely limited when the Confidentiality, Integrity
and Availability (CIA) model of security is applied to translation enabled information
exchange. Some of these issues can be mitigated if the translator can be treated as
a trusted peer. This adds the benefit of being able to build trust between two peers
who do not trust each other, but both trust the translator. In addition, the use of
standards, such as IETF COSE for object level encryption, can hide the contents of
sensitive information from the translator. The translation enabled information exchange
can introduce communication delay. The internal architecture of the proposed translator
has been designed so to reduce this delay. There are further possibilities to reduce
this delay by using a pipe-line architecture throughout the software library layers. This
research was presented in papers B, C and F.

Q2 What are the possibilities for ancillary information extraction within SOA-based
heterogeneous networks?

This thesis shows that it is possible to dynamically extract information from its source
within an IIoT network. Based on the usage of SOA and SoS theory, it is possible to 1)
visualize the IIoT elements, 2) select data sources, data processing and data sinks and 3)
re-orchestrate communications paths. A graph model is proposed to capture the SOA and
SoS elements. Where a graph path is disconnected between collaborating systems a local
bridge can be injected, connecting the systems. The local bridge could be a translator (in
the communications path), or a data manipulator (i.e. filter or trigger in the functional
path). The results show the feasibility of dynamic manipulation or creation of an SoS
during run-time by domain engineers. Because of local information access, centralized
datastores (warehouses) are avoided. This research was presented in papers G and I.
5.1. Conclusion 59

Q3 How can Manufacturing Execution System functionalities, such as execution process

logic and tracking, be distributed across Systems of Systems?

Interoperation of SoS is dependent on the goals of individual systems and the overarching
SoS goals. This thesis approaches this challenge by defining three systems operating at
the edge to co-ordinate workflow management, execution and tracking. An intelligent
product is responsible for tracking progress through the manufacturing process. A work-
flow manager and a workflow executor exist in each workstation. The workflow manager
is responsible for interaction with the intelligent product and co-ordination with inven-
tory and execution systems. The workflow executor is responsible for interaction with
equipment and the step wise execution of the operations state machine. The capability
decomposition guided by architectural principles promotes system reuse across differ-
ent workstations and supports system development as workstation requirements change.
This research was presented in paper H.

Q4 What are the underlying architectural principles and styles that can guide design of
Industrial Internet of Things applications?

The IIoT encompasses a vast variety of requirements and solutions. It is a multidisci-

plinary domain requiring knowledge and/or awareness of materials, physics, electronics,
embedded software, communications/networking, security, enterprise software, data stor-
age/analytics and more. Therefore, architecting a solution cannot rely on purely com-
puter science nor on engineering. The architect must rationalize design decisions based
on a principled approach.
The proposed principles are autonomy, specialization, data at its source and first
person perspective. These are combined with traditional SOA principles of loose coupling,
searchability and run-time binding.
Autonomous components tend to show resilience to faults and share less internal
state. Component specialization supports evolutionary update to individual systems and
reduces the cascading impact of change. Components developed with first person per-
spective anddata at the source, retain ownership of data and require less pooling of data,
thereby reducing data vulnerability. Furthermore, data at the source reduces the seman-
tic overhead of describing full information context when using data. This research was
presented in paper J.

During this research, several IIoT applications have been developed applying and
refining individual principles that contributed to the proposed architectural style. De-
veloped interoperability solution is able to support multi-protocol IIoT networking with
run-time service (translator) injection. Through implementation of IIoT applications
such as Wheel loader, UWB localization, Festo conveyor belt and Nutrunner, ideas and
principles for information extraction were developed and tested. The proposed graph
model allows information extraction at its source within SOA-based heterogeneous net-
works. The developed SoS graph model of a wheel loader correctly responds to Open
60 Conclusion and Outlook

Cypher graph queries, building the functional, communication and security paths. The
developed system automatically generates SoS compositions. This lessens required IT
competency on the factory floor, eliminates human error and reduces development time.
Furthermore, the proposed approach can be used to validate SoS specifications. For
instance, functional paths will indicate whether all required systems and services are
present in the current network. Principles of application capability decomposition were
applied to autonomous workstation use case. This work tackled the challenges imposed
by business driven requirements to manufacturing; it is an effort towards decentralized
MES. The proposed workflow manager and executor enables workstations to handle local
MES functionality and reduces integration efforts on the factory floor. The workstation
with these capabilities becomes self-contained, requiring near zero local infrastructure
support (MES, ERP, PLM, Internet). It lessens reliance on shared networks that can
be exposed to degraded performance impacting operation of the factory floor. Within
the settings of Industry 4.0, self-contained workstations can be relocated to remote sites
with minimum integration efforts.
The result of this research can be utilized by industry and academia to create soft-
ware architecture for IIoT applications. When compared with mechanical or electrical
engineering, software engineering is extraordinarily constraint free. However, the IIoT
introduces architectural constraints such as processing power, communications and bat-
tery life among others. Choices of where logic is implemented can sometimes appear
arbitrary. Therefore, software architecture is often a recommended practice; imposing
guidelines on design decisions regarding distribution and placement of logic. The con-
ducted research approaches this challenge by studying the rationale behind application
capability decomposition within IIoT. The thesis proposes an architectural style that
promotes design of systems that can handle the complexity imposed by IIoT.
The body of knowledge acquired during this research can be utilized in teaching
a new generation of engineers (from any domain). It is crucial for students as future
engineers and software architects to understand architectural thinking and principles.
Developers must be able to approach their implementation with understanding of the
architectural vision. This hopefully will result in a consistent, correct and stable software
implementation.

5.2 Future work

Computer systems are being programmed to allow delegation of complex human activ-
ities, relieving humans from many everyday mundane activities. Technology advance-
ments and growing computing capability drive the trends in ubiquity, interconnection,
intelligence, delegation and human orientation [73]. It can be said that the IIoT is a
result of that advancement.
In the future, software and computer engineering will involve smaller teams producing
niche systems with specific goals. The current issue with this approach is the fragmen-
tation of the solution, system heterogeneity, interoperability and the cost of integration.
Standard interfaces are one way to handle interoperability and fragmentation. However,
5.2. Future work 61

intelligent systems will need to negotiate content alongside interfaces. Self organisation
rather than managed organisation will be required for a functioning IIoT. It would be
interesting to see potential of machine learning and artificial intelligence, particularly in
self organizing systems within IIoT.
Another area of future work can be directed towards timing issues in SOA-based IIoT
applications. Due to the messaging overhead, SOA applications generally experience
longer delays. Timing and delay are a relative measure and a matter of perspective; an
additional delay need not be seen as an issue. It would be interesting to see, how applying
the principle of first person perspective may affect the design of a system, such that it is
less sensitive to delays introduced by other systems.
An interesting research direction world be in native platform support for SOA-based
IIoT applications. The proposed architectural principles may drive design of such plat-
forms supporting high level human oriented abstractions.
In computing, trust is usually modelled as a binary characteristic. A challenging task
would be to investigate the potential of IIoT systems to operate within a continuum of
trust. How do the architectural principles assist in creating graded trust collaboration?
What would the technology look like to achieve this?
It is possible that, government regulation could be required for architectural planning
of future software systems with certified sign-off in much the same manner as housing
architecture.
62 Conclusion and Outlook
 References

[1] M. Hölzl, A. Rauschmayer, and M. Wirsing, “Software-intensive systems and new

computing paradigms,” M. Wirsing, J.-P. Banâtre, M. Hölzl, and A. Rauschmayer,
Eds. Berlin, Heidelberg: Springer-Verlag, 2008, ch. Engineering of Software-
Intensive Systems: State of the Art and Research Challenges, pp. 1–44. [Online].
Available: https://doi.org/10.1007/978-3-540-89437-7 1

[2] A. W. D.E. Perry, “Foundations for the study of software architecture,”

ACM SIGSOFT, pp. 40–52, October 1992. [Online]. Available: http:
//delivery.acm.org/10.1145/150000/141884/p40-perry.pdf?key1=141884&key2=
2495967421&coll=GUIDE&dl=GUIDE&CFID=44366193&CFTOKEN=42763085

[3] “The ’only’ coke machine on the internet,” Carnegie Melon University. [Online].
Available: https://www.cs.cmu.edu/∼coke/history long.txt

[4] B. Scholten, The road to integration : a guide to applying the ISA-95 standard in
manufacturing. Research Triangle Park NC : ISA, 2007.

[5] E. A. Lee, “Cyber physical systems: Design challenges,” in 2008 11th IEEE In-
ternational Symposium on Object and Component-Oriented Real-Time Distributed
Computing (ISORC), May 2008, pp. 363–369.

[6] P. Garcia Lopez, A. Montresor, D. Epema, A. Datta, T. Higashino, A. Iamnitchi,

M. Barcellos, P. Felber, and E. Riviere, “Edge-centric computing: Vision and
challenges,” SIGCOMM Comput. Commun. Rev., vol. 45, no. 5, pp. 37–42, Sep.
2015. [Online]. Available: http://doi.acm.org/10.1145/2831347.2831354

[7] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing and its role in the
internet of things,” in Proc SIGCOMM, 2012.

[8] A. Colombo, T. Bangemann, S. Karnouskos, J. Delsing, P. Stluka, R. Harrison,

F. Jammes, and J. L. Lastra, Industrial Cloud-Based Cyber-Physical Systems - The
IMC-AESOP Approach. Springer International Publishing, 2014.

[9] “What is interoperability?” [Online]. Available: http://www.himss.org/library/

interoperability-standards/what-is-interoperability

63
64 References

[10] H. Derhamy, J. Eliasson, and J. Delsing, “Iot interoperability - on-demand and

low latency transparent multi-protocol translator,” IEEE Transactions on Services
Computing, 2016, submitted.

[11] A. P. Castellani, T. Fossati, and S. Loreto, “Http-coap cross protocol proxy: an

implementation viewpoint,” in 2012 IEEE 9th International Conference on Mobile
Ad-Hoc and Sensor Systems (MASS 2012), vol. Supplement, Oct 2012, pp. 1–6.

[12] M. Jung, J. Weidinger, C. Reinisch, W. Kastner, C. Crettaz, A. Olivieri, and Y. Boc-

chi, “A transparent ipv6 multi-protocol gateway to integrate building automation
systems in the internet of things,” in 2012 IEEE International Conference on Green
Computing and Communications, Nov 2012, pp. 225–233.

[13] B. Negash, A.-M. Rahmani, T. Westerlund, P. Liljeberg, and H. Tenhunen,

“Lisa: Lightweight internet of things service bus architecture,” Procedia Computer
Science, vol. 52, pp. 436 – 443, 2015, the 6th International Conference on Ambient
Systems, Networks and Technologies (ANT-2015), the 5th International Conference
on Sustainable Energy Information Technology (SEIT-2015). [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S1877050915008108

[14] F. P. Brooks, Jr., “The mythical man-month,” SIGPLAN Not., vol. 10, no. 6, pp.
193–, Apr. 1975. [Online]. Available: http://doi.acm.org/10.1145/390016.808439

[15] A. Cockburn and J. Highsmith, “Agile software development, the people factor,”
Computer, vol. 34, no. 11, pp. 131–133, Nov 2001.

[16] J. Delsing, O. Carlsson, F. Arrigucci, T. Bangemann, C. Hübner, A. W. Colombo,

P. Nappey, B. Bony, S. Karnouskos, J. Nessaether et al., “Migration of scada/dcs sys-
tems to the soa cloud,” in Industrial Cloud-Based Cyber-Physical Systems. Springer
International Publishing, 2014, pp. 111–135.

[17] F. Jammes, B. Bony, P. Nappey, A. W. Colombo, J. Delsing, J. Eliasson,

R. Kyusakov, S. Karnouskos, P. Stluka, and M. Till, “Technologies for soa-based
distributed large scale process monitoring and control systems,” in IECON 2012-
38th Annual Conference on IEEE Industrial Electronics Society. IEEE, 2012, pp.
5799–5804.

[18] F. G. Quintanilla, O. Cardin, and P. Castagna, Product Specification for Flexible

Workflow Orchestrations in Service Oriented Holonic Manufacturing Systems.
Cham: Springer International Publishing, 2014, pp. 177–193. [Online]. Available:
https://doi.org/10.1007/978-3-319-04735-5 12

[19] J. Delsing, Ed., Arrowhead Framework: IoT Automation, Devices, and Maintenance.
CRC Press, 12 2016. [Online]. Available: http://amazon.com/o/ASIN/1498756751/

[20] “Far-Edge Project,” Jan 2018. [Online]. Available: http://www.faredge.eu/

References 65

[21] An extendable component-based interoperable open model-driven architecture.

Remedy IT. [Online]. Available: http://www.remedy.nl/en/axcioma

[22] “Arrowhead framework wiki,” May. 2016. [Online]. Available: https://forge.soa4d.

org/plugins/mediawiki/wiki/arrowhead-f/index.php/MainPage

[23] J. Delsing, J. Eliasson, J. van Deventer, H. Derhamy, and P. Varga, “Enabling iot
automation using local clouds,” in 2016 IEEE 3rd World Forum on Internet of
Things (WF-IoT), Dec 2016, pp. 502–507.

[24] “Reference Architectural Model Industrie 4.0 (RAMI4.0) - An Introduction,”

April 2018. [Online]. Available: https://www.plattform-i40.de/I40/Redaktion/EN/
Downloads/Publikation/rami40-an-introduction.pdf

[25] “The Fourth Industrial Revolution - What it means and how to respond. World Eco-
nomic Forum,” Jan 2018. [Online]. Available: https://www.weforum.org/agenda/
2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/

[26] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and chal-
lenges,” IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637–646, Oct 2016.

[27] W. M. van der Aalst and K. M. van Hee, Workflow Management: Models, Methods,
and Systems. The MIT Press, 2000.

[28] M. Brettel, N. Friederichsen, M. Keller, and M. Rosenberg, “How virtualization,

decentralization and network building change the manufacturing landscape: An in-
dustry 4.0 perspective,” International Journal of Mechanical, Industrial Science and
Engineering, vol. 8, no. 1, pp. 37–44, 2014.

[29] E. Sardis, V. Anagnostopoulos, A. Voulodimos, and T. Varvarigou, “On surveillance

and workflow control system design,” in 2010 IEEE International Conference on
Progress in Informatics and Computing, vol. 1, Dec 2010, pp. 267–271.

[30] B. Tjahjono, C. Esplugues, E. Ares, and G. Pelaez, “What does industry 4.0 mean
to supply chain?” Procedia Manufacturing, vol. 13, pp. 1175 – 1182, 2017. [Online].
Available: http://www.sciencedirect.com/science/article/pii/S2351978917308302

[31] K. Alicke, D. Rexhausen, and A. Seyfert. (2017) Supply chain 4.0 in

consumer goods. [Online]. Available: https://www.mckinsey.com/industries/
consumer-packaged-goods/our-insights/supply-chain-4-0-in-consumer-goods

[32] J. Steding. (2018) Supply chain management trends in industry 4.0. [Online].
Available: https://www.soneticscorp.com/4-supply-chain-management-trends/

[33] Frost and Sullivan. (2017) Digital supply chain - iiot impact on manufacturing
supply chain. [Online]. Available: https://ww2.frost.com/frost-perspectives/digital-
supply-chain-iiot-impact-manufacturing-supply-chain/
66 References

[34] i Scoop. (2018) Logistics 4.0 and smart supply chain management in
industry 4.0. [Online]. Available: https://www.i-scoop.eu/industry-4-0/supply-
chain-management-scm-logistics/

[35] W. M. P. van der Aalst, A. H. M. ter Hofstede, and M. Weske, “Business pro-
cess management: A survey,” in Proceedings of the 1st International Conference on
Business Process Management, volume 2678 of LNCS. Springer-Verlag, 2003, pp.
1–12.

[36] B. Balis, “Hypermedia workflow: A new approach to data-driven scientific work-

flows,” in 2012 SC Companion: High Performance Computing, Networking Storage
and Analysis, Nov 2012, pp. 100–107.

[37] M. Weyrich and C. Ebert, “Reference architectures for the internet of things,” IEEE
Software, vol. 33, no. 1, pp. 112–116, Jan 2016.

[38] H. P. Breivold, “A survey and analysis of reference architectures for the internet-of-
things,” ICSEA 2017, p. 143, 2017.

[39] “Microsoft azure iot reference architecture,” Microsoft Corporation, Tech. Rep.,
2016.

[40] “The intel iot platform - architecture specification white paper internet of things,”
Intel Corporation, Tech. Rep., 2016.

[41] H. Derhamy, J. Eliasson, J. Delsing, and P. Priller, “A survey of commercial frame-

works for the internet of things,” in 2015 IEEE 20th Conference on Emerging Tech-
nologies Factory Automation (ETFA), Sept 2015, pp. 1–8.

[42] Thingworx industrial innovation platform - ptc. PTC. [Online]. Available:

https://www.ptc.com/en/products/iot/thingworx-platform/

[43] Thingsquare docs. Thingsquare. [Online]. Available: http://www.thingsquare.com/

docs/

[44] Home iotivity. IoTivity. [Online]. Available: https://www.iotivity.org/

[45] M. Sein, O. Henfridsson, S. Purao, M. Rossi, and R. Lindgren, “Action design

research,” vol. 35, pp. 37–56, 03 2011.

[46] J. C. Jones, Design Methods, 2nd Edition. New York, NY, USA: John Wiley &
Sons, Inc., 1992.

[47] “Overview of action research methodology.” [Online]. Available: http://web.net/

∼robrien/papers/arfinal.html
References 67

[48] N. R. Council, Academic Careers for Experimental Computer Scientists and

Engineers. Washington, DC: The National Academies Press, 1994. [Online].
Available: https://www.nap.edu/catalog/2236/academic-careers-for-experimental-
computer-scientists-and-engineers

[49] “That ’internet of things’ thing,” Tech. Rep. [Online]. Available: http:
//www.rfidjournal.com/articles/view?4986

[50] R. Minerva, A. Biru, and D. Rotondi, “Towards a definition of the internet of things
(iot),” Tech. Rep., May 2015.

[51] S. Jeschke, C. Brecher, T. Meisen, D. Özdemir, and T. Eschert, Industrial Internet

of Things and Cyber Manufacturing Systems. Cham: Springer International
Publishing, 2017, pp. 3–19. [Online]. Available: https://doi.org/10.1007/978-3-319-
42559-7 1

[52] S.-H. L. W. Mahnke and M. Damm, OPC Unified Architecture. Springer-Verlag

Berlin, 2010.

[53] “Apache kafka.” [Online]. Available: http://kafka.apache.org/

[54] J. Delsing, “Local cloud internet of things automation: Technology and business
model features of distributed internet of things automation solutions,” IEEE Indus-
trial Electronics Magazine, vol. 11, no. 4, pp. 8–21, Dec 2017.

[55] E. D. Knapp and J. T. Langill, Industrial Network Security: Securing critical in-
frastructure networks for smart grid, SCADA, and other Industrial Control Systems.
Syngress, 2014.

[56] D. Meltzer, “Securing the industrial internet of things,” Tech. Rep., June 2015.

[57] H. Lasi, P. Fettke, H.-G. Kemper, T. Feld, and M. Hoffmann, “Industry 4.0,”
Business & Information Systems Engineering, vol. 6, no. 4, pp. 239–242, Aug 2014.
[Online]. Available: https://doi.org/10.1007/s12599-014-0334-4

[58] “Structure of the Administration Shell,” April 2018. [Online]. Avail-

able: https://www.plattform-i40.de/I40/Redaktion/EN/Downloads/Publikation/
structure-of-the-administration-shell.pdf

[59] P. B. Kruchten, “The 4+1 view model of architecture,” IEEE Software, vol. 12,
no. 6, pp. 42–50, Nov 1995.

[60] R. T. Fielding, “REST: architectural styles and the design of network-based

software architectures,” Doctoral dissertation, University of California, Irvine, 2000.
[Online]. Available: http://www.ics.uci.edu/∼fielding/pubs/dissertation/top.htm

[61] D. Garlan and M. Shaw, “An introduction to software architecture,” Pittsburgh,

PA, USA, Tech. Rep., 1994.
68 References

[62] H. Zimmermann, “Osi reference model - the iso model of architecture for open
systems interconnection,” IEEE Transactions on Communications, vol. 28, no. 4,
pp. 425–432, April 1980.
[63] T. Erl, SOA Principles of Service Design (The Prentice Hall Service-Oriented Com-
puting Series from Thomas Erl). Upper Saddle River, NJ, USA: Prentice Hall PTR,
2007.
[64] S. Newman, Building Microservices: Designing Fine-Grained Systems, 1st ed.
O’Reilly Media, February 2015.
[65] “The single responsibility principle - 8th light.” [Online]. Available: https:
//8thlight.com/blog/uncle-bob/2014/05/08/SingleReponsibilityPrinciple.html
[66] T. O’Reilly and J. Battelle, “Web squared: Web 2.0 five years on,” Tech. Rep., 2009.
[67] T. O’Reilly, What is Web 2.0? Design Patterns and Business Models for the Next
Generation of Software. O’Reilly Media, September 2009.
[68] D. Merrill, “Mashups: The new breed of web app,” Tech. Rep., 2006.
[69] “Real-time twitter trending hashtags and topics.” [Online]. Available: https:
//www.trendsmap.com/
[70] “Node-red.” [Online]. Available: https://nodered.org/
[71] “Ifttt helps your apps and devices work together.” [Online]. Available:
https://ifttt.com/
[72] S. Bussmann, N. R. Jennings, and M. Wooldridge, Multiagent systems for manufac-
turing control: A design methodology. Germany: Springer-Verlag, 2004.
[73] M. Wooldridge, Introduction to Multiagent Systems. New York, NY, USA: John
Wiley & Sons, Inc., 2001.
[74] S. D. J. McArthur, E. M. Davidson, V. M. Catterson, A. L. Dimeas, N. D. Hatziar-
gyriou, F. Ponci, and T. Funabashi, “Multi-agent systems for power engineering
applications - part i: Concepts, approaches, and technical challenges,” IEEE Trans-
actions on Power Systems, vol. 22, pp. 1743–1752, 2007.
[75] K. L.-B. Yoav Shoham, Multiagent Systems - Algorithmic, Game-Theoretic and Log-
ical Foundations. Cambridge University Press, 2009.
[76] “Department of defense dictionary of military and associated terms,” Washington,
DC, USA, Tech. Rep., 2015.
[77] ISO/IEC/(IEEE), “ISO/IEC 42010 (IEEE Std) 1471-2000 : Systems and Software
engineering - Recomended practice for architectural description of software-intensive
systems,” 2007.
References 69

[78] “Systems engineering guide for systems of systems,” Washington, DC, USA, Tech.
Rep., 2008.

[79] M. W. Maier, “Architecting principles for systems-of-systems,” INCOSE

International Symposium, vol. 6, no. 1, pp. 565–573. [Online]. Available:
https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2334-5837.1996.tb02054.x

[80] J. S. Osmundson, T. V. Huynh, and G. O. Langford, “Emergent behavior in

systems of systems,” INCOSE International Symposium, vol. 18, no. 1, pp.
1557–1568. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/j.
2334-5837.2008.tb00900.x

[81] “Internet engineering task force.” [Online]. Available: https://www.ietf.org/

[82] O. Carlsson, C. Hegedus, J. Delsing, and P. Varga, “Organizing iot systems-of-

systems from standardized engineering data,” in IECON 2016 - 42nd Annual Con-
ference of the IEEE Industrial Electronics Society, Oct 2016, pp. 5277–5282.

[83] P. Reed, “Reference architecture: The best of best practices,” Tech. Rep., 2002.

[84] M. Bell, SOA Modeling Patterns for Service Oriented Discovery and Analysis. Wiley
Publishing, 2010.

[85] T. Erl, SOA Design Patterns, 1st ed. Upper Saddle River, NJ, USA: Prentice Hall
PTR, 2009.

[86] P. Bresciani, A. Perini, P. Giorgini, F. Giunchiglia, and J. Mylopoulos, “Tropos:

An agent-oriented software development methodology,” Autonomous Agents and
Multi-Agent Systems, vol. 8, no. 3, pp. 203–236, May 2004. [Online]. Available:
https://doi.org/10.1023/B:AGNT.0000018806.20944.ef

[87] L. Padgham and M. Winikoff, Prometheus: A practical agent-oriented methodology,

I. Group, Ed., 2005.

[88] F. Gosewehr, J. Wermann, W. Borsych, and A. W. Colombo, “Specification and

design of an industrial manufacturing middleware,” in 2017 IEEE 15th International
Conference on Industrial Informatics (INDIN), July 2017, pp. 1160–1166.

[89] Z. M. Sikora, Entity Relationship Modelling. London: Macmillan Education UK,

1997, pp. 46–55. [Online]. Available: https://doi.org/10.1007/978-1-349-14693-2 5
70 References
 Part II

71
72
 Paper A
A Survey of Commercial
Frameworks for the Internet of
Things

Authors:
Hasan Derhamy, Jens Eliasson, Jerker Delsing, Peter Priller

Reformatted version of paper originally published in:

Conference paper, IEEE Emerging Technologies and Factory Automation, Luxembourg,
2015.

© 2015, IEEE, Reprinted with permission.

73
74
 A Survey of Commercial Frameworks for the
Internet of Things

Hasan Derhamy, Jens Eliasson, Jerker Delsing, Peter Priller

Abstract

In 2011 Ericsson and Cisco estimated 50 billion Internet connected devices by 2020,
encouraged by this industry is developing application frameworks to scale the Internet of
Things. This paper presents a survey of commercial frameworks and platforms designed
for developing and running Internet of Things applications. The survey covers frameworks
supported by big players in the software and electronics industries. The frameworks are
evaluated against criteria such as architectural approach, industry support, standards
based protocols and interoperability, security, hardware requirements, governance and
support for rapid application development. There is a multitude of frameworks available
and here a total 17 frameworks and platforms are considered. The intention of this
paper is to present recent developments in commercial IoT frameworks and furthermore,
identify trends in the current design of frameworks for the Internet of Things; enabling
massively connected cyber physical systems.

1 Introduction
For more than a decade the Internet of Things (IoT) has boosted the development of
standards based messaging protocols. Recently, encouraged by the likes of Ericsson and
Cisco with estimates of 50 billion Internet connected devices by 2020 [1], attention has
shifted from interoperability and message layer protocols towards application frameworks
supporting interoperability amongst IoT product suppliers.
The IoT is the interconnection of ubiquitous computing devices for the realization of
value to end users [2]. This definition encompasses ”data collection” for the betterment
of understanding and ”automation” of tasks for optimization of time. The IoT field has
evolved within application silos with domain specific technologies, such as health care, so-
cial networks, manufacturing and home automation. To achieve a truly ”interconnected
network of things” the challenge is enabling the combination of heterogeneous technolo-
gies, protocols and application requirements to produce an automated and knowledge
based environment for the end user.
In [3], Singh et al. elaborate on three main visions for the IoT: Internet Vision,
Things Vision and Semantic Vision. Depending on which vision is chosen the approach
taken by a framework will differ and provide a better result for those applications. As
surveyed by Perera et al. in [4], there are many existing IoT products and applications
available. These however are based on proprietary frameworks which are not available

75
76 Paper A

for development of customized applications. The frameworks presented in this survey are
all targeted as a basis for development of IoT applications.
This paper presents a survey of highly regarded commercial frameworks and platforms
which are being used for Internet of Things applications. Many of the frameworks rely
on high level software layers to assist in abstracting between protocols. The high level
software layer provides flexibility when interconnecting between different technologies
and is well suited for working in cloud environments. In some cases the frameworks look
into standardizing interfaces, defining a software service bus or simply opting to choose
a single network protocol and set of application protocols. This is further discussed as
follows; in Section 2 introduces the concept of frameworks and defines three categories
of frameworks used in this survey. Sections 3 and 4 then introduces the frameworks and
platforms studied, grouped by application area. In Section 5 a discussion of a comparative
analysis of the frameworks and platforms is presented. The survey finishes with a few
concluding remarks in Section 6.

2 What is an Internet of Things Framework

Framework in the context of this report is a set of guiding principles, protocols and
standards which enable the implementation of Internet of Things applications. It can
but does not need to be an active participant of the overall IoT system. Frameworks
can enhance IoT application development by; rapid implementation, interoperability,
maintainability, security and technology flexibility.
To achieve rapid implementation many of the ”boiler plate” tasks can be computer
aided or removed completely. For example, a web service will not be concerned with han-
dling the routing of messages, this is taken care of by underlying protocol or framework,
such as IP the stack. The Internet protocols are based on layered architecture. Inter-
operability must extend across each layer. For example data packets can be forwarded
between nodes on the same network so long as they all support the same network proto-
col, such as IP. In the context of service oriented architecture, interoperability requires
that applications are able to discover service providers and consume the services in order
to perform the system functions. This requires shared understanding of interface and
shared understanding of data semantics.
A key criteria for opening closed systems to the Internet is security. Authentication,
authorization, confidentiality, integrity and privacy are of most interest to IoT applica-
tions. To be able to upload data to an untrusted cloud service provider and then share
the data with different service consumers requires complex security functions.

2.1 Internet of Things Approaches

The approach used by a framework will determine its suitability for different application
spaces and impact information latency, data collation, feature interdependency, module
design and network topology. This section will provide an overview of the high level
approaches employed in current frameworks.
2. What is an Internet of Things Framework 77

Many frameworks take a data centric or data driven approach. Utilizing a global
cloud, they focus on enabling collation, visualization and analytics on data. This archi-
tecture is well suited for applications such as asset tracking, logistics and predictive main-
tenance [5]. In some cases the framework will allow creation of local hosted instances [6]
but do not detail a method of interconnecting multiple cloud instances within the frame-
work. This approach is suitable for providing data as a service but will generally leave the
implementation specifics of the end-points to application developers. The framework sim-
plifies the operation of end-points to only feeding data back to a central repository which
will then implement complex security authorizations and usage tracking. The increasing
computing power at the edge of networks is not leveraged in such frameworks and can
introduce inefficiencies in bandwidth and latency. Figure 1 illustrates the concept of a
global cloud through which IoT applications connect and communicate.

Figure 1: Global cloud concept

A smart objects approach makes the endpoints active participants within the frame-
work. The end points are included as key aspects of the framework which means the
focus of the framework is on interconnecting the end-points. This approach is well suited
for distributed automation tasks which require a high level of device independence, such
as home and building automation and manufacturing.
While the devices can be very small they are often directly addressable from the
Internet, see [7]. Because of this focus on automated end points and functional behavior,
many of these frameworks do not go into the specifics of cloud integration and so do
not provide good support for data collection. The data is produced by end points and
consumed by end-points, which within this context will usually have some predefined
understanding of the end-point pairing. The implementation of the data in the cloud is
left to each IoT designer with minimal support from the framework.
In both of the approaches mentioned, many features such as end-to-end security and
layered interoperability suffer due to ad-hoc development either at the end-points or
within the cloud. A third approach becoming more prevalent, is taking into account
the need to satisfy real-time automation requirements while not hindering the value of
78 Paper A

semantic big data and data analytics. Figure 2 illustrates the local cloud concept with
independent local operation and shared global functionality.

Figure 2: Local cloud concept

Frameworks based on an approach which enables standards based development of end

points and also of data warehousing, will enable secure interoperation of modularized
and distributed applications. Key to the framework is leveraging intelligence at the end
points and utilizing global cloud and local cloud concepts [8]. Enabling specialization
of clouds for the requirements of the user or end points, whilst enabling data and event
movement between clouds. This approach scales peer-to-peer networks into integrated
cloud solutions.

3 Frameworks
3.1 IoT Frameworks for home automation
Home automation has been a key area for development of IoT-based applications. With
the reduction in costs of manufacturing of IoT enabled devices, there are three major
frameworks trying to gain support from device manufacturers and application developers.
IoTivity - backed by Intel and Samsung, AllJoyn - backed by Qualcomm, LG and Sony,
and Thread - back by ARM and Google.

IPSO Alliance

The IPSO Alliance framework is interested in standardizing semantic description in the

IoT and supports a resource based object model [9] for other frameworks to build on.
Using SenML and either XML or JSON encoding the IPSO alliance fills only part of the
framework stack. But other frameworks have chosen to work with the IPSO framework
and indeed IoTivity and OMA-LWM2M do build on IPSO Alliance work. The IPSO
3. Frameworks 79

specification builds on top of IETF CoRE standards such as 6LoWPAN, CoAP and
SenML [9].

IoTivity
The IoTivity framework is developed by the Open Interconnect Consortium initially
targeting IoT in smart homes and looking to further expand to other IoT silos. It is
based on CoAP and its key building blocks are the Connectivity Abstraction (CA) layer
and a Resource Introspection (RI) layer. The framework is being extended to HTTP and
other communication protocols are supported through protocol plug-ins [10].
The IoTivity stack is shown in Figure 3. The thin block stack supports resource
constrained devices. IoTivity makes use of D/TLS for security. Another interesting
feature of IoTivity is Soft Sensor concept [10], which supports processing raw sensor
data at intermediate or edge nodes.

Figure 3: IoTivity framework stack

As described in the Things Manager [10], IoTivity adopts a similar approach to the
IPSO Alliance by using a object and resource based model. LWM2M and IoTivity find
convergence here with the IPSO Alliance model and possible opportunity for interoper-
ability.

AllJoyn
The AllJoyn framework, developed by the AllSeen Alliance, is designed for enabling
interoperability for home automation [11] and industrial lighting [11] applications. The
AllJoyn core operates as a software bus [12] between devices. Devices must implement
a bus attachment responsible for message marshaling and serialization. Constrained
devices use a thin library [12], and do not have a bus attachment, so must connect to an
AllJoyn router. Thin devices work at the messaging level, while full devices communicate
by remote procedure calls. Running such frameworks introduces overheads which limit
real-time performance and participation of resource constrained and low power devices.
AllJoyn core library provides authentication and encryption for end-to-end secu-
rity [12]. Authentication is provided by means of Simple Authentication and Security
Layer (SASL) security framework defined by the D-Bus specification [13]. It supports
both point-to-point ”session” keys and point-to-multi-point ”group” keys. Thin Client
80 Paper A

devices [12] don’t support this security. It is transport agnostic [12] and is currently
running on WiFi, Ethernet, serial, and Power Line.

Thread

The Thread groups framework defines a protocol stack based on Nest’s early implemen-
tation of the smart thermostat; it uses the IETF IP stack, UDP and builds up additional
security and commissioning functionality [14]. The Thread protocol can address devices
directly and is able to perform peer-to-peer communications in a mesh network. It is
seen as an evolution from the traditional ZigBee stack to an IP based stack [14].
Being built atop the standard IEEE 802.15.4 radio allows them to make use of already
mass produced ZigBee chips [14]. There is limited information available as Thread is only
available to member companies and the Thread group will be providing a certification
process.

3.2 The Arrowhead Framework

A light framework aimed at enabling the Industrial Internet of Things and to improve in-
teroperability between applications. It is based on Service Oriented Architecture (SOA).
It is made up of three core systems; Service Register, Orchestration, and Authorization.
Supporting SOA-based design principles including; standardized service contracts, ser-
vice loose coupling, service abstraction and service autonomy. In order to be Arrowhead
compliant, applications must register services they produce with an Arrowhead service
registry. Then use the Arrowhead authorization service to accept or reject consuming
applications. The Orchestration system allows dynamic reconfiguration of the service
consumer and provider end-points.
The framework can operate with a hierarchical set of core systems allowing a single
machine to operate its own Arrowhead cloud, allowing local authorization and orches-
tration rules. Inter-cloud service discovery is supported meaning that local clouds can
consumer outside services or provide data as a service to outside consumers.
A sample Arrowhead network is shown in Figure 4. In this network two embedded
devices provide one service each to a third device which then provides a single service to
an operator’s computer. All security, discovery and connection rules are supported by
the Arrowhead framework.
The Arrowhead framework prescribes a method of documentation in order to enable
native interoperability between services, and uses service transparency to enable inter-
operability between services which have used different semantics and communications
protocols. Applications are responsible for implementation details such as messaging
and business logic. Once the base services are developed or 3rd party services are chosen,
Arrowhead provides a framework for governing and interconnecting the services.
3. Frameworks 81

Figure 4: Arrowhead Core Services and network example

3.3 OMA - LWM2M

The Open Mobile Alliance - Light Weight Machine to Machine (LWM2M) framework has
been developed for the purpose of monitoring, provisioning and managing connections of
networked devices [15]. LWM2M is based on CoAP and defines a layer above CoAP. The
initial release addressed the interfaces between two devices in the areas; Bootstrap, client
registration, device management and service enablement, and information reporting [15].
The framework uses the standard CoAP to UDP binding and also defines a new binding
for CoAP to SMS. The LWM2M stack is shown in Figure 5, other than the addition of
SMS, LWM2M utilizes IETF standards.

Figure 5: OMA - LWM2M framework stack

The framework defines a resource and object model whereby the LWM2M client holds
objects which each contain some resources. These objects can be instantiated by either
a remote LWM2M server or by the client itself. Once instantiated the objects can be
operated upon via the interfaces mentioned above.
As shown in the diagram of the LWM2M stack the security layer is handled with
DTLS [15]. It is , but not recommended, possible to send the data without security.
Access control is specified at the object layer with the use of an Access Control Object
Instance [15]. They define what operations are allowed on a per Object Instance case. So
within a given LWM2M client a single object instance will hold resources with different
access control rights depending on the LWM2M server performing the access.
82 Paper A

The framework unifies the CoAP server and client model into LWM2M server and
LWM2M client which allows bi-directionally RESTful communications [15]. For example,
a LWM2M server is able to POST to a LWM2M client and vice versa. This means the
application development architecture has a more flexible concept of client/server.

3.4 Other frameworks

Other frameworks include many groups either targeting niche or specific aspects of the
IoT, or are looking to provide oversight and guidance to the development of the IoT tech-
nologies. The IPSO Alliance targets semantics and standardizing smart object interaction
by defining the information model and RESTfull message exchange. The Industrial In-
ternet Consortium (IIC) is looking to provide leadership and guidance to IoT framework
organizations to bring direction towards convergence in IoT requirements, best practices
and overall architecture. The Smart Energy Profile 2.0 (SEP2.0), originally from the
ZigBee Alliance, is a well-defined specification for smart meters and energy management;
it is now standard of the IEEE-SA. SEP2.0 is based on IP and uses a RESTfull inter-
face. It fully defines governance aspects such as, encryption, authentication and key
exchange [16]. The AXCIOMA [17] framework developed by Remedy IT uses Object
Management Group (OMG) technologies and standards such as CORBA and DDS. AX-
CIOMA has defined the need for supporting Request-Response and Publish-Subscribe
patterns and supports both these interfaces. The framework is targeting real-time and
embedded networks.

4 Platforms
4.1 Cloud-based IoT Platforms
Centralized platforms offer a simple method of integrating sensors into IoT applications.
By employing a global cloud approach, Cumulocity, ThingWorx and Xively provide an
integration platform for organizations to build IoT applications on. They recognize that
many commercial organizations will be interested in gaining value from the data provided
by embedded sensors.

Cumulocity
In the Cumulocity platform, sensor nodes are clients which connect to the cloud through a
RESfull HTTPS API. Sensor nodes are modeled as objects with properties and methods
for access and manipulation. Commands are pulled by devices from the Cumulocity
server [18]. Depending on the pull frequency there will be a delay from the issued
command until the device receives it. Constrained networks not operating on HTTPS
use an ”agent” [18] to connect to the Cumulocity server. A server side ”event” language
syntactically resembling SQL scripting [18] loads triggers to be performed in reaction to
events. Cumulocity only supports RESTful HTTP/S.
4. Platforms 83

ThingWorx

The ThingWorx platform targets application integration through model driven develop-
ment. It composes services, applications and sensors as data sources and interconnects
these through a virtual bus. The framework is transport agnostic and has been ported
to run with CoAP, MQTT, REST/HTTP and Web Sockets [6]. Treating other clouds as
data sources ThingWorx integrates with other cloud providers such as Xively and web
services such as Twitter and weather services. Communication between devices, services
and applications must be routed through the ThingWorx bus, thereby not enabling peer
to peer communication. Using a Mashup builder, organizations are able to quickly con-
nect data sources to dashboards, for tracking and monitoring assets and gathering data
from many data sources to perform data analytics in real-time [19].

Xively

Third in this group is the Xively platform, formerly known as Pachube. This platform
similar to the previous two provides a central message bus which routes messages between
devices of different protocols. The components which make up the Xively architecture
can be seen in Figure 6.

Figure 6: Xively platform architecture

The message bus combined with the Xively API for MQTT, HTTP, and Web Sockets
to provide an interoperability layer. It is a data driven platform with ability to give fine
grain access to data streams and data feeds. Based on the client server model, they have
a centralized method of device configuration where each device has a virtual presence
and when a device comes online it uses its serial number and some form of mutual
authentication to receive its configuration parameters setup on the Xively server. The
framework has additional services which allow for Business Services, Systems Integration
and Business Opportunities for companies and assist with governance of the network.
84 Paper A

4.2 Device to device platforms

In this section two platforms are described: Echelon’s IzoT platform and the ThingSquare
platform.

IzoT
The IzoT platform is made up of a communication stack intended for peer-to-peer com-
munications [20] consisting of several proprietary high level protocol services which run
on top of UDP [20]. Supporting priority messaging and end-to-end acknowledgements
on unicast and multicast messages, the communication stack can support multiple si-
multaneous messaging on unconstrained devices. It has built in discovery and interface
publishing, and can run on many networks including 6LoWPAN, free topology twisted
pair, WiFi and potentially any medium which can support UDP sockets [20].
IzoT supports symmetric and asymmetric key encryption and authentication. Using
a proprietary communication stack limits the ability for IzoT to be adopted widely for
general IoT applications.

ThingSquare
The ThingSquare platform is founded from the development of the Contiki OS and
is strictly based on IETF communications stacks. Their offering includes cloud based
device governance and boot-strapping, but is limited in terms of cloud based application
integration and data analytics. The focus of the framework is enabling automation,
control and monitoring of smart objects through the Internet. Contiki OS boasts the
smallest IP stack according to ThingSquare [21]. There operating system has been ported
to run on many of the current IoT micro-controllers [21].
The ThingSquare framework uses cloud based services for device management, au-
thentication and authorization [21] of new devices to the network. The ThingSquare
framework will only allow authorized users and devices to register and control other
ThingSquare devices. A simplified network is shown in Figure 7, along with the network
stack supported.

Figure 7: ThingSquare network and stack

4. Platforms 85

4.3 Platforms for Cloud to Gateway Integration

Intel, Microsoft and IBM have all formed strong offerings in the cloud to device IoT
market. This group of platforms offer platform support in the cloud and also offer
solutions in intelligent gateways, however do not provide much application development
support for end-points.

Intel
Intel have partnered with Wind River and McAfee to produce an IoT framework which in-
cludes hardware for things, intelligent gateways, cloud and Platform as a Service [22] [23].
Intel’s hardware technologies, Wind River’s Operating Systems (OS) and McAfee’s secu-
rity products can be utilized in different layers of the IoT from embedded to the cloud.
VxWorks [24] can scale down to a 20kb foot print for use in constrained embedded
systems, while Wind River’s gateway OS, based on Linux, can support many application
environments including Lua, Java, and OSGi [23]. Whitelisting binaries means that
binaries without the correct signature cannot be executed on a device [23]. Role-based
access control is used to provide a learning mode to generate security policy rules [25].

Microsoft
Microsoft supports the IoT at three layers. The Microsoft Azure Cloud provides an excel-
lent platform for developing and integrating distributed applications using its proprietary
Enterprise Service Bus. Device connectivity and governance is supported by Microsoft
Azure Intelligent Systems Service (ISS) [26]. Microsoft StreamInsight is a platform for
in-memory data analytics and processing [27]. It allows IoT applications to process data
without the latency involved with traditional databases. It can be run as a local Web
service or in the cloud as a hosted service in Azure. The device layer is supported by
Microsoft Windows embedded [28] and the .net microframework (.netmf) [29].
Microsoft Research have developed the HomeOS platform [30] as a multi-protocol
home automation server. It is a virtual OS running on a COTS computer and providing
inter-connection between multi-vendor COTS devices [31]. Although not yet, commer-
cially available, it occupies the same space as AllJoyn, IoTivity and Thread [31].

IBM
There are many offerings from IBM and by combining them it is possible to run an
end-to-end industrial or consumer IoT systems with MQTT-based communications and
enterprise middleware. The key offerings are BlueMix application server, WebSphere en-
terprise integration middleware, MobileFirst application development platform, Informix
database, and the MessageSight MQTT broker. These different IBM products are shown
in Figure 8.
IBM BlueMix is based on Cloud Foundry [32] application server, offering additional
management capabilities. The IBM IoT Foundation is hosted on Bluemix and provides
86 Paper A

Figure 8: IBM product framework

governance of IoT devices. IBM WebSphere MQ offers proven and robust enterprise ap-
plication integration and supports MQTT networks by integrating WebSphere with the
IBM MessageSight Appliance [33]. Previously known as IBM WorkLight Foundation [34]
MobileFirst is a platform enabling machine to machine connections through mobile de-
vices. It is a platform for web application and native mobile application development
and provides hosting for those applications.
IBM MessageSight is a communications appliance which can handle high volumes of
MQTT communications. Built with performance in mind, the IBM MessageSight appli-
ance can process over 350K MQTT v3.1 messages per second and can publish over 15M
messages per second to consumer applications [35]. The IBM MessageSight appliance
can be run in a Virtual environment [36].

5 Discussion
The frameworks and platforms have been introduced and a high level over view of their
key features has been described. Now we will discuss the frameworks within the context
of the evaluation criteria. The evaluation criteria are; the framework approach, industry
support, underlying protocols, security, applicability to constrained devices and support
for rapid application development.
Architectural approaches were introduced in section 2.1 and the frameworks can be
categorized as in Table 1. This categorization is indicative of the target application space
for the frameworks.

Table 1: The three approaches discussed against the frameworks and platforms studied
Approach Frameworks, Platforms and Protocols
Global Cloud Cumulocity, Xively, ThingWorx, IBM, Microsoft, Intel, LWM2M
Peer to Peer IPSO, Thread, Thingsquare, IzoT, SEP 2.0, AllJoyn, IoTivity
Local Cloud Arrowhead

Cumulocity, Xively, ThingWorx, IBM, Microsoft and Intel are serving the global cloud
5. Discussion 87

approach to running IoT Applications. They provide the hosting platforms and applica-
tion API for interacting between devices from applications running in the cloud. This
is a well-known model used for business systems which prefer a centralized application.
LWM2M joins this group as a device governance framework with a centralized approach.
IPSO, Thread, ThingSquare, IzoT, SEP 2.0, AllJoyn and IoTivity have approached IoT
application development from a device level and support a high level of peer-to-peer
operation. This approached serves their customers well in home automation and device
management. Within the local cloud category, Arrowhead sits alone. The unique ap-
proach of this framework is the support for integration of applications between secure
localized clouds. The Quality of Service, Security and scalability of industrial automation
has necessitated this approach.
The qualities of peer-to-peer communication, mesh network support, 6LoWPAN and
low power make the ThingSquare attractive for running edge of network in conjunction
with cloud platforms such as Xively, Cumulocity or ThingWorx.
Table 2 shows a few of the larger IoT framework backers against the frameworks in
this paper. Platforms are not shown here as they are usually supported by the platform
provider only.

Table 2: Frameworks organizations members and supporters.

Framework Arrowhead AllJoyn Thread IoTivity LWM2M
Google X
Microsoft X X
IBM
Intel X X
Cisco X X
GE X
AT&T X X
Samsung X X X
ARM X X
Motorola X
Qualcomm X X
LG X
Schneider Elec. X X
AVL X
STM X
Members 81 101 81 55 96

LWM2M has support from many larger organizations as it falls under the OMA
umbrella of specifications. Frameworks with support from large organizations mitigates
sudden end of support for a chosen framework. It can be seen that Microsoft and Intel,
while providing an IoT platform, are also members of the IoT framework development.
88 Paper A

The number of member organizations within each framework also indicates the confidence
industry has in each framework.
The centralized frameworks mentioned earlier offer message protocol flexibility and
will usually support MQTT, REST and sometimes CoAP and XMPP. Table 3 shows the
frameworks against common protocols. Under the ’other column’ are either proprietary
protocols or less common protocols such as DDS.

Table 3: Communications protocols supported by the studied frameworks and platforms

MQTT XMPP CoAP REST Other
IPSO Alliance X X
LWM2M X
Arrowhead X X X X X
SEP 2.0 X
AXCIOMA X X X X X
Thread Group X
AllJoyn X
ThingSquare X X
IzoT X X
ThingWorx X X X X X
Xively X X X X X
Cumulocity X
IBM X X
Microsoft X

To mention open source frameworks and the protocols they rely on is important to
developers looking for flexibility in choosing libraries, vendor platforms and interoper-
ability. Many of the frameworks are proprietary, but most support at least one open
source messaging protocol. IPSO Alliance, OMA-LWM2M, AllJoyn, IoTivity and Ar-
rowhead are open-source and based on open-source technologies. IBM and Microsoft are
proprietary and the Thread specification is only available for member companies. The
platforms sch as ThingWorx, ThingSquare, Xively and Cumulocity are proprietary and
so moving an application from one to another would be a costly exercise.
In table 4 the frameworks and protocols in this survey are categorized by layer. Most
of the data centric frameworks sit in the application layer, while many of the control
centric frameworks are in the messaging layer. In this case messaging layer is referring
to a layer above transport while still not offering a rich application layer support.
Some of the frameworks such as, IoTivity and AllJoyn support a dual stack implemen-
tation, supporting a reduced functionality stack for constrained devices. However this
is not always the case, such as Xively, Cumulocity and ThingWorx who do not support
constrained devices and rely on intermediary agents or gateways to integrate resource
constrained devices. Security aspects are important for hardware requirements. Crypto
6. Conclusion 89

Table 4: Frameworks and Platforms categorized by most representative layer

Layer Frameworks or Protocols
Application Arrowhead, IPSO Alliance, Xively,
Cumulocity, ThingWorx, Smart En-
ergy Profile 2.0, Microsoft, IBM,
ThingSquare, Industrial Internet
Consortium, AllJoyn, IoTivity, IzoT
Messaging Web-Sockets, XMPP, MQTT,
CoAP, HART, Thread, AllJoyn
Transport TCP, UDP, WirelessHART, SMS
Network IP, ZigBee

hardware support is required by IzoT, AllJoyn and IoTivity. Echelon’s IzoT platform
offers hardware components as part of their framework and also provides adaption layers
for non-IzoT devices. IBM supports its MQTT based framework with a dedicated server
appliance which runs the MQTT broker.
Rapid application development, (re-)configurability, scalability and deployment con-
siderations are important characteristics. It is difficult to make evaluation on such as-
pects, but it is worth mentioning frameworks with comparative strengths. IBM and Mi-
crosoft’s strong background in enterprise service bus means they have a good advantage
for scaling up as business needs grow. ThingWorx, Cumulocity and Xively demonstrate
strength in rapid application development and focus on value added work. Thread, Iotiv-
ity and AllJoyn tend focus on customers using commercial off the shelf devices and there-
fore simplify the deployment. Arrowhead’s strength is in its re-configurability, through
the use of dynamic orchestration of services and systems.
Further on governance and management of the devices, services and interfaces will
assist with rapid application development and maintenance. The cloud platforms such as
Cumulocity, ThingWorx and Xively offer great application governance and some device
management of an active device. However AllJoyn, IzoT platform and ThingSquare
offer good device management and lesser support for application governance. IBM and
Microsoft both have mature cloud application governance and management. Microsoft
has good device management through its embedded OS family and also it’s embedded .net
run time. Arrowhead provides application configuration and authorization governance
through its core services. Applications can discover services, download configurations and
authorize access. The primary purpose of LWM2M is the governance and management
of devices, at a scale for cellular operators. This tends to suggest it will have good
performance for large scale device networks.

6 Conclusion
As the market for IoT applications grows it industry has worked with academia to create
a standardized set of communications protocols. Next frameworks and platforms for the
90 Paper A

IoT are being developed by industrial consortia. This is in order to lay down a foundation
at the application layer which will enable deployments of large scale, either in instance
size or in instance number, IoT applications.
This survey has presented a number of commercially available frameworks and plat-
forms for developing industrial and consumer based IoT applications. The studied frame-
works have each approached IoT from the perspectives and priorities of their customer
needs. The priority was either on; centralizing distributed data sources for cloud-based
applications, referred to as a global cloud approach; or supporting integration of devices
for home(building)-automation, referred to as the peer-to-peer approach; or integrating
devices and clouds together for factory and industrial automation systems, referred to as
the local cloud approach.
A comparative analysis of the frameworks was conducted based on industry support,
use of standards based protocols, interoperability, security, hardware requirements, gov-
ernance and support for rapid application development. Based on this analysis academia
and industry can identify frameworks most suitable for their future projects and identify
gaps in the current frameworks.
Finally, for platforms and frameworks to succeed they must recognize and facilitate:

1. Enable devices, applications and systems to securely expose API’s for 3rd party
systems and to facilitate API management.

2. Enable systems to have protocol interoperability with other 3rd party API’s and
ensure they are extendable for new protocols.

3. Enabling constrained devices to participate into application networks. That is size,

bandwidth, power supply(battery) and processing power constraints.

4. Governance - Enabling management and governance of heterogeneous networks of

devices and applications.

The value of the whole is greater than the sum of its parts.

Acknowledgements
The Authors would like to thank AVL List GMBH for supporting this survey of com-
mercial frameworks for the Internet of Things.

References
[1] D. Evans, “The internet of things how the next evolution of the internet is changing
everything,” White Paper, Cisco, April 2011.

[2] L. Atzori, A. Iera, and G. Morabito, “The Internet of Things: A survey,” COM-
PUTER NETWORKS, vol. 54, no. 15, pp. 2787–2805, OCT 28 2010.
References 91

[3] D. Singh, G. Tripathi, and A. Jara, “A survey of internet-of-things: Future vision,

architecture, challenges and services,” in Internet of Things (WF-IoT), 2014 IEEE
World Forum on, March 2014, pp. 287–292.

[4] C. Perera, C. Liu, S. Jayawardena, and M. Chen, “A survey on internet of things

from industrial market perspective,” Access, IEEE, vol. 2, pp. 1660–1679, 2014.

[5] Xively solutions by application. Xively. [Online]. Available: https://xively.com/

solution/by opportunity/

[6] Thingworx industrial innovation platform - ptc. PTC. [Online]. Available:

https://www.ptc.com/en/products/iot/thingworx-platform/

[7] Technical information. IPSO Alliance. [Online]. Available: http://www.ipso-

alliance.org/technical-information

[8] P. Varga, F. Blomstedt, L. L. Ferreira, J. Eliasson, M. Johansson, and J. Delsing,

“Making system of systems interoperable - the core components of the arrowhead
technology framework (accepted for publication),” IEEE Internet of Things Journal,
August 2015.

[9] The ipso application framework draft-ipso-app-framework-04. IPSO Alliance.

[Online]. Available: http://www.ipso-alliance.org/wp-content/media/draft-ipso-
app-framework-04.pdf

[10] Home iotivity. IoTivity. [Online]. Available: https://www.iotivity.org/

[11] Allseen alliance wiki. AllSeen Alliance. [Online]. Available: https://wiki.

allseenalliance.org/

[12] Learn - allseen alliance. AllSeen Alliance. [Online]. Available: https://

allseenalliance.org/developers/learn

[13] H. Pennington, A. Carlsson, A. Larsson, S. Herzberg, S. McVittie, and D. Zeuthen.

D-bus specification. [Online]. Available: http://dbus.freedesktop.org/doc/dbus-
specification.html\#auth-protocol

[14] Thread group - home. Thread Group. [Online]. Available: http://www.threadgroup.

org

[15] Machine to machine (m2m) solution. Open Mobile Alliance. [Online]. Available:
http://openmobilealliance.org/about-oma/work-program/m2m-enablers/

[16] Smart energy profile 2 application protocol standard. ZigBee Alliance. [Online].
Available: http://splintered.net/z/Zigbee-smart-energy-profile-2.pdf

[17] An extendable component-based interoperable open model-driven architecture.

Remedy IT. [Online]. Available: http://www.remedy.nl/en/axcioma
92 Paper A

[18] Interfacing devices. Cumulocity. [Online]. Available: http://www.cumulocity.com

[19] Joy mining connected products success story. ThingWorx. [Online]. Avail-
able: http://www.thingworx.com/learning content/connected-products-success-
story-joy-mining-2/

[20] B. Dolin. Requirements for the industrial internet of things. Echelon. [Online].
Available: http://info.echelon.com/IIoT-Requirements-Whitepaper.html

[21] Thingsquare docs. Thingsquare. [Online]. Available: http://www.thingsquare.com/

docs/

[22] Transform business with intelligent gateway solutions for iot. Intel. [Online].
Available: http://www.intel.com/content/www/us/en/internet-of-things/gateway-
solutions.html

[23] Intel gateway solutions for the internet of things. Intel. [Online]. Available:
http://www.mcafee.com/us/resources/solution-briefs/sb-intel-gateway-iot.pdf

[24] Vxworks. Wind River. [Online]. Available: http://windriver.com/products/

platforms/network equipment/#content 3

[25] Intel iot gateway - security profiles. Intel. [Online]. Avail-

able: http://www.intel.com/content/www/us/en/embedded/design-tools/
evaluation-platforms/gateway-solutions/iot-security-profiles-white-paper.html

[26] Microsoft azure intelligent systems service. Microsoft. [Online]. Available:

https://connect.microsoft.com/ISS

[27] C. M. Torsten Grabs. (2012, March) Microsoft streaminsight - building

the internet of things. [Online]. Available: https://msdn.microsoft.com/en-us/
magazine/hh852591.aspx

[28] Intelligent systems: A new level of business intelligence. Microsoft. [Online]. Avail-
able: http://www.microsoft.com/windowsembedded/en-us/intelligent-systems.aspx

[29] .net micro framework. Microsoft. [Online]. Available: http://www.netmf.com/

[30] C. Dixon, R. Mahajan, S. Agarwal, A. J. Brush, B. Lee, S. Saroiu, and

P. Bahl, “An operating system for the home,” in Proceedings of the 9th USENIX
Conference on Networked Systems Design and Implementation, ser. NSDI’12.
Berkeley, CA, USA: USENIX Association, 2012, pp. 25–25. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2228298.2228332

[31] P. Oliphant. (2014, May) Alljoyn vs homeos for the connected home. [On-
line]. Available: http://www.connectedhomeworld.com/content/alljoyn-vs-homeos-
connected-home
 93

[32] “What is ibm bluemix,” IBM Corp., accessed: 2018-04-19. [Online]. Available:
http://www.ibm.com/developerworks/cloud/library/cl-bluemixfoundry

[33] V. Lampkin. (2012, March) What is mqtt and how

does it work with websphere mq? [Online]. Avail-
able: https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/
what is mqtt and how does it work with websphere mq?lang=en

[34] A. Trice. (2014, November) So, what is ibm mobilefirst? [Online]. Available:
http://www.tricedesigns.com/2014/11/19/so-what-is-ibm-mobilefirst/

[35] A. S. Arnaud Mehieu. (2014, November) Ibm messagesight. [Online].

Available: https://www-950.ibm.com/events/wwe/grp/grp004.nsf/vLookupPDFs/
IBM%20MessageSight/$file/IBM%20MessageSight.pdf

[36] Ibm messagesight - ibm messaging. IBM Corp. [Online]. Available: https:
//developer.ibm.com/messaging/messagesight/
94
 Paper B
Translation Error Handling for
Multi-Protocol SOA Systems

Authors:
Hasan Derhamy, Pal Varga, Jens Eliasson, Jerker Delsing, Pablo Punal Pereira

Reformatted version of paper originally published in:

Conference paper, IEEE Emerging Technologies and Factory Automation, Luxembourg,
2015.

© 2015, IEEE, Reprinted with permission.

95
96
Translation Error Handling for Multi-Protocol SOA
Systems

Hasan Derhamy, Pal Varga, Jens Eliasson, Jerker Delsing, Pablo Punal Pereira

Abstract

The IoT research area has evolved to incorporate a plethora of messaging protocol stan-
dards, both existing and new, emerging as preferred communications means. The variety
of protocols and technologies enable IoT to be used in many application scenarios. How-
ever, the use of incompatible communication protocols also creates vertical silos and
reduces interoperability between vendors and technology platform providers. In many
applications, it is important that maximum interoperability is enabled. This can be for
reasons such as efficiency, security, end-to-end communication requirements etc. In terms
of error handling each protocol has its own methods, but there is a gap for bridging the
errors across protocols. Centralized software bus and integrated protocol agents are used
for integrating different communications protocols.
However, the aforementioned approaches do not fit well in all Industrial IoT ap-
plication scenarios. This paper therefore investigates error handling challenges for a
multi-protocol SOA-based translator. A proof of concept implementation is presented
based on MQTT and CoAP. Experimental results show that multi-protocol error han-
dling is possible and furthermore a number of areas that need more investigation have
been identified.

1 Introduction
The Internet of Things (IoT) has assisted in breaking down application domain silos and
promoting horizontal integration between application domains. The Arrowhead frame-
work, presented by Blomstedt et al. in [1] is looking to improve interoperability and
integrability of services provided by networked embedded devices. Cisco has estimated
that there will be 50 billion devices connected to the Internet by 2020 [2]. This is a
staggering number of devices and managing the differing communication standards is
not trivial.
The IoT area has seen many existing and new communications protocols emerging as
preferred standards. The adoption of the varied communication protocols can be linked to
specific application vertical requirements and is likely to stay this way as the IoT further
develops. Thus in order for the Arrowhead framework to provide interoperability between
application verticals, methods and technologies for communication protocol translation
are required.

97
98 Paper B

Some of the IoT protocols used in Service Oriented Architecture (SOA) based ap-
plications and systems are; Representational State Transfer (REST) over HTTP, eXten-
sible Messaging and Presence Protocol (XMPP), Message Queue Telemetry Transport
(MQTT), Constrained Application Protocol (CoAP) and OLE for Process Control - Uni-
fied Architecture (OPC-UA). Each of these protocols offers benefits in particular applica-
tion requirements, such as low-power operation, verbose headers and semantics, connec-
tion oriented messaging, decoupling producer from consumer, discovery, bootstrapping,
real-time or reactiveness, and statelessness.
For the automaton domain OPC-UA is the predominant SOA protocol when commu-
nicating from the Distributed Control System (DCS) or Supervisory Control and Data
Acquisition (SCADA) level and upwards in ISA-95 architecture. With the expectations
of IoT devices to be used in such ISA-95 architectures, it’s clear that IoT SOA proto-
cols like CoAP, XMPP, MQTT, and REST will show up together with OPC-UA and
legacy technology in large automation systems. Large EU projects like Socrates and
IMC-AESOP have published several papers on such architectures and migration to such
architectures [3][4][5][6][7][8].
Today, there is a variety of commercial IoT platforms which support interaction be-
tween different communication protocols. They offer an API for either; translation agents
[9], [10] running embedded on the device or in gateways or a cloud based software bus
[11], [12] for each protocol. This indicates that there is a need to integrate different
communications protocols.
However, these platforms either confine applications to adapters integrated into their
solutions, or require all communications be routed through a central server. Both ap-
proaches reduce flexibility for application designers and integrators, introduce security
vulnerabilities with untrusted third-party clouds. This creates inefficiencies in the com-
munications path and bandwidth usage for localized applications. Enabling protocol
interoperability by the use of SOA will increase design flexibility, enable local applica-
tions and remove dependency on third-party translators. But Quality of Service (QoS),
end-to-end connectivity, robustness and error handling become challenges which need
to be addressed. A literature search did not reveal much research in error handling in
multi-protocol translation. This indicates the need for more research in this area.
This paper investigates the question of error handling in a multi-protocol translation
for SOA systems. While in a single protocol system, errors are propagated according
to protocol specification. In the case of multi-protocol systems error handling becomes
more complex. In designing a SOA-based translator error handling and considerations
becomes critical to robust communication. An error in one protocol must be translated
to be understood by other protocols. While a SOA-based translator must also address
other aspects such as QoS, control messaging, security and semantic translation, these
are not considered in this paper and are considered future work.
This paper is structured as follows: Section II provides background and related work,
followed by problem definition in Section III and proposed solution in Section IV. An
example application scenario and implementation details and results are presented in
Sections V and VI. Finally, conclusions are summarized in Section VII, with suggestions
2. Background and Related Work 99

for future work presented in Section VIII.

2 Background and Related Work

A SOA-based architecture presented by Karnouskos et al. in [8] shows a shift towards
SOA paradigm for Industrial IoT (IIoT). Development in the area of SOA is driven by
the need for collaboration within ultra-large Scale systems [13]. SOA has been used to
great effect in web based systems to create an ecosystem of collaborative parts.
The IoT is seeing growth in new application spaces and new application requirements
with an evolving ecosystem of platforms, frameworks, protocols and devices [14]. This
means system integrators are presented with the challenge of evolving their legacy sys-
tems, and technologies, to satisfy the new requirements and make use of new technologies.
By tightly coupling translation agents into the systems the cost of upgrading the system
is increased. Relying on centralized cloud based software bus also limits the ability to
leverage the native benefits of using new technologies.
Collina et al. in [15] have proposed an MQTT to REST bridge. This architecture
exposes MQTT topics as REST resources. This allows MQTT clients and REST clients
to interact through the new centralized QEST broker. There are two terms that appear in
this field: protocol translation and protocol conversion. There is no clear differentiation
between these terms, although ”translation” is used generally in computer networking
(especially and almost exclusively for network layer translation) - whereas ”conversion” is
used more widely in the industrial automation domain [16]. The traditional networking
ISO-OSI terminology for nodes dealing with translation are that switches, routers and
gateways work at the data link, network and transport layers, respectively [17]. The
most widely known functionality is Network Address Translation (NAT), although its
protocol translation version working between IPv4 and IPv6 (NAT-PT) was suggested
for historic status [18] due to a series of serious issues.
There are no general guidelines or standards for higher level translation - these are
seemingly all legacy solutions. Such translators do merely parameter mapping between
two protocols, although sometimes also deal with the issues of the transport layers. For
this study two protocols used in similar application spaces were selected. This investiga-
tion helped to refine challenges error handling and proposal of solutions to address some
of these challenges. CoAP and MQTT were selected as they are both intended to be
used in highly efficient industrial applications.

CoAP The CoAP protocol [19] has been developed by the IETF for use in extending
Internet capability down to resource constrained devices. It applies the request-response
communication pattern to a client-server network model. CoAP is targeting sleepy and
lossy networks in which supporting TCP becomes inefficient and power consuming [20].
It is based on UDP and provides an optional retry mechanism at the CoAP layer. It
has a RESTful API with the GET, PUT, POST and DELETE verbs supported with the
addition of the OBSERVE function. It creates a publisher-subscriber session between
a CoAP server and client, sending notifications either when resource state changes or
100 Paper B

periodically, on expiry of ’Max-Age’ [19]. Having this flexibility makes it an ideal choice
for machine-to-machine interaction.

MQTT The MQTT protocol has been developed for enabling efficient communication
between data sources and data sinks. It applies the publisher-subscriber pattern to a
client-server network model. It has recently been standardized by OASIS but has a long
history with IBM being used in sensor networks. Some of its features [21] are decoupling
data producer from data consumer through the centralized broker system; reduced header
size and event based publishing enable highly efficient communication; QoS levels with
message delivery; and, simple centralized security model with connection initiation by
clients enabling useful firewall and NAT traversal features.

3 Problem Definition
Dependent on how much the protocols overlap in the OSI layers there can be much
complexity in the translation process. Errors which can be detected and monitored need
to be handled in a manner which will enable adequate debugging and issue resolution,
either automated or by manual intervention.
In this section some of the challenges with handling errors of multi-protocol translation
are elaborated and specifically, the case of translation between MQTT and CoAP studied.
The error cases defined may not be exhaustive; however they represent some of the most
common and in some cases challenging errors.

Error cases Connection errors can occur when trying to establish new connections,
having a current connection lost for some reason, or inability to close a connection grace-
fully. These kinds of errors need to be detected and translated appropriately to ensure
efficient use of resources (at the end points as well as at the translator). Also infor-
mation about connection error events need to be made known for analysis on network
performance and identifying candidates for possible improvements.
Lossy communication errors are a real problem in wireless sensor networks (WSN),
which make up a good proportion of the future IIoT. The problem with lossy commu-
nication is made more complex with layers handling the issue at different layers of the
OSI stack. A higher level protocol may rely on a lower level layer to guarantee transport
while the target protocol may perform such transport checking itself. This means that
handling lossy communication at may need to go across layers or provide informational
error alert which will then rely on application layers to monitor and perform corrective
action if needed.
Response related delays and application introduced delays are two such categories of
delay related errors. Miss-matched timeouts at the communications or application layers
can lead to one sided timeouts. To handle one sided timeouts the channels need to be
re-synched, how will the translator deal with this? Application delays could be found
on resource constrained devices not being able to service a request or publish an update
within the time limit expected by the other party.
3. Problem Definition 101

Application layer packetization which relies on ordered delivery by underlying layers is

a special case of moving between ordered delivery protocols such as TCP and un-ordered
delivery such as UDP. Is this something which can be handled by the translator? Is this
something which is required by the translator?
Invalid messages arriving at the translator requires the translator to be able to detect
and take appropriate action. In request-response protocols an error message can be sent
to the origin. However in many publish-subscribe protocols it is not possible to send an
error back. The translator must be responsible for providing some level of confidence
to the end points about how much of the protocol dependencies are still valid and how
much cannot be support.
Errors produced as part of the protocol procedure such as authentication problems,
resource availability or others can generate legitimate error codes. These error codes
while represented on one side of the translator need to be passed to the other side. To
address this first the error codes of each protocol must be listed and then mapped based
on the cause. Often the mapping will not be symmetric with many error codes being
mapped to a lesser number, or even some error codes not being able to be mapped at all.
This is described in detail in Section IV.

Transient Error Cases Transient error cases can occur at end points but also in the
translator. This class of error cases relate to errors which can occur at any time and
will generally ’heal’ in a short period of time. They require special treatment in terms of
handling and translation, in terms of maintaining protocol behavior.
Transient errors due to resource usage occur when there is an increase in transla-
tion demand to such an extent that the translator is no longer capable of handling the
throughput. It must take remedial action in accordance with the protocols which are be-
ing translated. These actions are highly dependent on the source of the increased demand
and the capability for the translator to influence this demand. This can be illustrated
where two end-points have a contract for delivery of notifications on periodic updates.
If the size of the messages begins to increase while the rate of the messages remains at
a high frequency, the translation of these message packets may consume resources which
are not available. In this case the source of the data cannot be asked to reduce the
frequency as the contract is between the end points. In such a case the translator must
become a negotiating party and reduce the frequency or the size of the messages. In
some cases the protocol does not allow such freedom for negotiation and in fact requires
the end-point to drop the connection in such situations, as is the case for MQTT [21].
The second transient error case is from buffer problems. Buffer problems can occur
when resource usage increases without correct remedial actions. But in this case we are
referring to miss matches in buffer sizes between protocols and end-points. Whilst one
high performance end-point, such a REST based web application, may be able to handle
large verbose messages, once this is translated for a constrained end-point, such as CoAP,
an error will occur between the end-points and once again remedial action will need to
be taken and the event will need to be logged.
Lastly, transient errors can occur due to miss-matches in protocol or application
102 Paper B

sensitivity to jitter or delays. An application or protocol which has been designed to be

sensitive to a specific range of jitter or delay may find that this is not always possible to
be met either because of the jitter and delays introduced at the translator, or because
the other protocol cannot guarantee the same jitter and delay bands. This can impact
either the protocol behavior or the application behavior and thereby generate errors in
the end-to-end process.

4 Proposed Solution
There is much work to be done in order to address the challenges in the previous section.
As each challenge is tackled there is likely to be an increase in the solution complexity.
This section looks to address error code translation and discuss QoS and Error reporting.

4.1 Mapping Error codes

In this paper the error mapping was done a single pair of protocols. This is the basis
for the development of an error code mapper which would produce a generic interface
which individual protocols must use to define their error codes to the map. In MQTT,
protocol error codes are only reported from the broker to the client [21]. The protocol
does not specify error code reporting by the client to the broker. This is illustrated in
Figure 1. CoAP in turn also only reports errors from the server to the client [19]. This
unidirectional use of error responses means that in certain configurations the error codes
cannot be passed to the end-points. This is discussed further below.

Figure 1: MQTT and CoAP block diagrams. The bold lines show direction of error reporting

The MQTT protocol defines error codes for initial connection and subscription control
packets, other control packets do not have associated error codes [21]. The decoupled
nature of MQTT networks means that clients have much less visibility of errors occurring
in other clients.
Below are two tables with the error codes producible in MQTT and in CoAP. Table 1
shows the error cases which are generated by MQTT and mapped to CoAP. The mapping
in this case is used when a CoAP client is attempting to initiate a subscription to an
4. Proposed Solution 103

MQTT broker through the translator. This is illustrated in Figure 2.

Figure 2: CoAP to MQTT translator block diagram

In this case error codes generated by the MQTT broker can be translated and passed
to the CoAP client. For example the CoAP client will be awaiting the response to its
GET request and if the MQTT broker does not allow the connection or the subscription,
then this error can be passed to the CoAP client.

Table 1: Error code mapping from MQTT broker to CoAP client

MQTT Error case MQTT code CoAP code
Failure - Topic filter not accepted SUBACK 0x80 Error 4.04
Not Authorized CONNACK 0x05 Error 4.01
Bad username or password CONNACK 0x04 Error 4.01
Server not available CONNACK 0x03 Error 5.03
Identifier rejected CONNACK 0x02 Error 4.00
Unacceptable protocol version CONNACK 0x01 Error 4.06

In Table 2 the CoAP error codes are mapped to MQTT. These represent the case
when an MQTT client subscriber is attempting to retrieve data from a CoAP server.
This case is illustrated in Figure 3.

Figure 3: MQTT to CoAP translator block diagram

In this case there is no path for the error codes to be transferred to the subscribing
104 Paper B

MQTT client. This is due to the nature of the MQTT protocol; and so the CoAP error
codes do not have a mapping to MQTT clients.

Table 2: Error code mapping from CoAP server to MQTT client

CoAP Error case CoAP code MQTT
Bad Request Error 4.00
Unauthorized Error 4.01
Bad Option Error 4.02
Forbidden Error 4.03
Not Found Error 4.04
Method Not Allowed Error 4.05
Not Acceptable Error 4.06 Not supported
Request Entity Too Large Error 4.13
Unsupported Media Type Error 4.15
Internal Server Error Error 5.00
Not Implemented Error 5.01
Bad Gateway Error 5.02
Gateway Timeout Error 5.04
Proxying Not Supported Error 5.05

However, the error codes can be used by the translator and can be mapped to transla-
tor behaviors. That is, for translating between two different protocol pairs, there will be
different behavior by the translator to take corrective actions or logging. This could be
translating the error code as in Table 1, or other remedial actions as defined in the trans-
lator. For the use case implemented in this paper the translator actions are described in
Section VI.

4.2 Quality of Service aspects

Since the translator is in the path between the service producer and the service consumer
application systems, its performance affects the end-to-end QoS. The translator, as a
physical entity has resource limitations for memory and processing power.
Furthermore, it can serve very different application needs and very many of those.
Which means it will handle many queues, which fragment those limitations further:
allocating memory for the various queues, for each entity in the queue; and handling
the processing overhead due to the handle queuing mechanisms (i.e. scheduling). The
translator has similar types of QoS-related issues as a network-level processing node
(i.e. router); although these are somewhat enlarged. This is due to the differences in
information volume: translator needs to process application payload, whereas a router
merely processes the network layer header.
It is not only that the translator (that handles various types of service needs) should
handle QoS profiles, but these should describe further detailed metrics than those well-
4. Proposed Solution 105

known at the network level. Besides handling loss, delay, delay and utilization metrics,
their more specified versions [22] should be kept under control: one-way and two way
throughput and delay, as well as their variance. Availability as a QoS metric is hard to
address other than binary terms - either it is available, or not. Loss as a quality metric
gets another meaning here - loss in translation - where not the whole message, but its
parts get lost. Depending on the context and the parameters that weren’t able to be
mapped, may lead to QoS degradation - or it may have no noticeable effect.

4.3 Error reporting aspects

One of the most challenging aspects of distributed systems is error diagnosis. The IoT
promises massively distributed systems and with the introduction of cross protocol trans-
lation the error cases not only increase in number but also in complexity. Therefore error
logging and reporting is critical to the success of a translation system.
Systems can take an active approach towards error monitoring which will mean that
the error notification may have soft real-time requirements. While other systems may take
a reactive approach to error monitoring which will mean that they will require persistent
log of the error events leading up to the final event which caused the investigation. This
is true for all distributed systems with or without translation, but as the translation is
a third party to the two end points involved in most interactions it becomes pertinent
to discuss the implications. This means that the error stream reported by the translator
must maintain a link with the end points it is translating for.
Either one of the end-points being translated or a third application will need to
be able to securely and efficiently query the error log. This introduces challenges in
authenticating and authorizing the interested parties to access the logs and to then find
the relevant error records. If the error logs need to be exposed to a third-party which is
providing support, then how can they be authorized to access the required logs?

4.4 Application Scenario

In order to test the proposed method in a real world monitoring application, subtask
1.8 in the Arrowhead project was chosen. Arrowhead is a European R&D project with
the aim to develop SOA-based interoperable systems [1]. Arrowhead’s Task 1.8 is a
research and development activity aimed at delivering hardware and software for ball-
bearing monitoring of a wheel loader. Task 1.8 is a joint collaborative effort conducted by
Lulea University of Technology, SKF and Eistec AB. The translation scenario selected for
demonstrating the challenges of error handling within the framework of Arrowhead is a
CoAP based sensor network monitoring the condition of the wheel loader’s ball bearings
and providing this data as a service within the Arrowhead framework. See Figure 4 for
a layout of the network architecture of Arrowhead Task 1.8
An MQTT based service consumer is behind a firewall and initiates a session with
the broker consuming the sensor data from the CoAP based sensor. The MQTT service
consumer could be a head office system which does not allow incoming UDP packets or
a hand held device running a VPN which again does not allow incoming UDP packets.
106 Paper B

Figure 4: Arrowhead Task 1.8 network architecture

In this case CoAP is not suitable as a service consumer. A high level diagram of this
scenario can be seen in Figure 5.

Figure 5: Secure MQTT behind firewall translated to CoAP for communication

The translator must connect the MQTT broker and the CoAP server to allow data
flow. There are many error conditions possible, as stated in Section III. As a proof of
concept the authors have chosen to detect a sensor disconnect at the wheel loader. In an
industrial environment such as mining, road works or construction sites a disconnection
error is something which the translator must be able to handle. The interaction diagram
between the different components is shown in Figure 6.
Of particular interest in this scenario is the nature of CoAP running on UDP which
is connectionless and therefore requires an agreed timeout based approach to connection
loss. While on the MQTT side TCP is used and therefore a connection state is maintained
between the end points. Even so not all TCP disconnections can be identified and so if
the detection of a disconnection is desired, a heartbeat or a keep-alive timeout is also
5. Implementation and Results 107

Figure 6: Error condition interaction diagram

required.

5 Implementation and Results

In order to validate the error case assumptions and begin the process of identifying
limitations in error handling of the translation process an error translation scenario has
been implemented. This section will describe the implementation setup and the results
of running the error cases.
Eclipse Paho MQTT client library has been used for developing a simple visualization
of data and events. This was connected to a Mosquitto MQTT broker running on a
standard Windows computer on a loop back network. The wheel loader sensor was taken
from the Arrowhead task 1.8 pilot project and it uses Contiki OS 2.7 and Erbium. The
sensor connected to the translator through a Contiki border router and a BeagleBone
Black gateway. The translator was implemented in Java and uses a Californium [23]
CoAP client to initiate an observe on the state of the wheel loader sensor and using a
hub and spoke architecture passes the resource notifications to an MQTT Paho client
which then publishes the notification to the corresponding MQTT topic in the Mosquitto
broker. This setup is shown in Figure 7 below.

Figure 7: Implementation technologies mapped to system components

108 Paper B

The translator itself is not the core of this paper and so the implementation has
been kept to a simple transfer of payload from CoAP to MQTT. Semantics and other
protocol procedures have not been considered. Using a hub and spoke architecture for
implementation, results in a decoupled component based translator with only simple
object method calls being setup by the hub between the spokes. This can be seen in
Figure 8.

Figure 8: Translator architecture diagram

Running the experiment uncovered several error cases which were in addition to the
disconnect error case that was to be modeled. This unintended error case was very useful
as it shows a real world use case. The wheel loader sensor is a resource constrained
device running on a low power network and therefore notification timeout due to late
delivery or packet loss was common. In these cases the CoAP spoke would follow the
CoAP specification and on max-age expiry, would attempt to re-register the observation.
This was in almost all cases successful and would re-establish the periodic notifications.
However in the MQTT specification there is no mechanism within the protocol to pass
information regarding update timeout except by disconnection. The keep-alive timer was
controlled by the Paho library and so would keep the connection alive even when no data
was being sent. This means that a non-standard message would need to be sent from the
CoAP spoke to inform the MQTT client that the sensor has had a timeout. Processing
of this message would be at the discretion of the MQTT client application.
However, in the event of a disconnection error which does have protocol procedures
in both MQTT and CoAP there is still special behavior required. So the CoAP spoke
monitors the max-age of the last resource state update. If this max-age is exceeded then
a timeout is noted and the CoAP spoke will attempt to re-establish the observation, as
described earlier. However if the re-establishment is not successful then the CoAP spoke
cancels the observation and an error passes an error event to the translator hub. In a
normal setup an ungraceful disconnection detected by the MQTT system, would result
all subscribing clients being delivered a last will message, if it is available. However in
this case the MQTT system does not have visibility of the CoAP disconnection. It is
required to translate and notify the MQTT system of this disconnection event. The
proposed translator maps the CoAP disconnection to a last will message in the MQTT
side. This mapping takes place in the translation hub. Figure 9 shows the interaction
between internal components of the translator system.
In this way, the translator has made use of the protocol procedures of both sides
6. Conclusion 109

Figure 9: Internal object interaction of the translator behavior

to make sure both protocols are aware of the error event. For interoperable use of the
translator between systems the definition of both the timeout event and the disconnect
event signals is a must. For this implementation two event signals were defined and
encoded in XML. These tags are presented below.
<s n=”e v e n t ” sv=” d i s c o n n e c t ”/>
<s n=”e v e n t ” sv=”t i m e o u t ”/>
Implementation of the rudimentary good path payload translation between CoAP
and MQTT was relatively trivial. However implementing the mapping of disconnection
and timeout errors introduced a lot of complexity to the code. By refining the implemen-
tation the error handling and mapping were moved to the central hub. This reduced the
complexity of the translation effort immensely. Each translation spoke did not need to
have knowledge of the other. This means that once a spoke is developed it can be con-
nected through the hub to any other spoke. This reduces the effort required to develop
translation services between the ever changing array of protocols.
By decoupling the protocol specific handling to spokes and translation aspects to
the translation hub has meant that the solution is extensible for new spokes and allows
interesting potential for a multi-spoke translator. It has also meant that error cases can
be handled in a standard manner within the translation hub and new spoke development
need only use available hooks in the translation hub in order to pass error conditions.
The results were promising with key advantages to the use of a hub and spoke SOA
based translator. Its active participation in the network, its simplicity for handling errors
and potential for extension to being orchestrated and also into semantic translation are
the main advantages.

6 Conclusion
This paper has presented the challenges and solution for error handling in multi-protocol
translation scenarios for SOA systems. This work is motivated by the creation of new
110 Paper B

systems-of-systems that are composed of application domains with different communi-

cations requirements. Current protocol translation solutions use tightly-coupled soft-
ware components or integrated middleware that reduces flexibility and increases cost of
change. Moreover, utilizing centralized software bus for translation increases round trip
time, bandwidth usage and introduces further dependencies (i.e. on cloud platforms,
often operated by third parties). In both of these cases intermediary protocols are used
and this limits the benefits of the native communications protocol.
On the other hand, SOA-based translation systems provide the opportunity to de-
couple the translation components from the application development and also create
flexibility in deciding execution location of the translation service.
This paper discussed the challenges of error handling in the case of loosely coupled
SOA translators. Some of the investigated error cases are connection errors, lossy commu-
nication, application introduced delays and protocol error code mapping. Beside these,
the transient errors in the translator and at end-points need to be handled. This means
that not just message parameter mapping, but the protocol procedures of one side needs
to be reflected on the other side. Transient errors can occur, when resource requirements
in terms of memory and processing power do not scale as usage demand increases, a mis-
match in buffer requirements between a protocol pair, or a mismatch in jitter and delay
sensitivity. These transient errors require the translator to be capable of self-monitoring
and also negotiation capabilities with the protocol pairs.
The proposed solution uses translator behaviors, which are then mapped to a protocol
procedure or error code. The translator’s overall behavior depends on the protocols
being translated. This is realized in the implementation by the use of a hub and spoke
architecture with the hub containing the possible behaviors of each spoke.
The proof of concept implementation provided error handling for translation between
CoAP and MQTT. To accomplish this, it was required to pass error cases generated on
the CoAP side of the connection to be communicated to the MQTT side. It was decided
to define a set of xml tags and attributes, which would communicate the errors at the
application level. There were two such events defined for this proof of concept, they were
the disconnect event and the timeout event.

7 Future Work
In the future, the architecture of the multi-protocol translator needs to be defined and
refined. Use case extension to other SOA protocols such as XMPP and REST will also
be needed.
Challenges are seen in orchestration and co-ordination of the translator end-points,
managing resource requirements, providing security in terms of privacy, confidentiality
and authenticity, and proving performance and flexibility gains.
Performance metrics, evaluation and bench-marking will be needed in order to prove
the advantages of a multi-protocol SOA translator. Further development of the semantics
used to send error information and signals should be looked into.
Error logging and diagnosis has much work to be done. Logging encompasses a larger
8. Acknowledgment 111

scope than just error events should enable SOA based applications to create an end to
end stream of events. An API must be developed with the ability for machine query and
manual query of the logs.

8 Acknowledgment
The authors would like to express their gratitude towards the European Commission and
Artemis for funding, and our partners within the Arrowhead project.

References
[1] F. Blomstedt, L. L. Ferreira, M. Klisics, C. Chrysoulas, I. M. de Soria, B. Morin,
A. Zabasta, J. Eliasson, M. Johansson, and P. Varga, “The arrowhead approach for
soa application development and documentation,” in Industrial Electronics Society,
IECON 2014 - 40th Annual Conference of the IEEE, Oct 2014, pp. 2631–2637.

[2] D. Evans, “The internet of things how the next evolution of the internet is changing
everything,” White Paper, Cisco, April 2011.

[3] A. W. Colombo, T. Bangemann, S. Karnouskos, J. Delsing, P. Stluka, R. Harrison,

F. Jammes, and E. Jose L. Martinez Lastra, Industrial Cloud-Based Cyber-Physical
Systems - The IMC-AESOP Approach. Springer, 2013.

[4] J. Delsing, O. Carlsson, F. Arrigucci, T. Bangemann, C. Hübner, A. W. Colombo,

P. Nappey, B. Bony, S. Karnouskos, J. Nessaether et al., “Migration of scada/dcs sys-
tems to the soa cloud,” in Industrial Cloud-Based Cyber-Physical Systems. Springer
International Publishing, 2014, pp. 111–135.

[5] F. Jammes, B. Bony, P. Nappey, A. W. Colombo, J. Delsing, J. Eliasson,

R. Kyusakov, S. Karnouskos, P. Stluka, and M. Tillyµ, “Technologies for soa-based
distributed large scale process monitoring and control systems,” in IEEE IECON
2012. IEEE, 2012.

[6] F. Jammes, S. Karnouskos, B. Bony, P. Nappey, A. W. Colombo, J. Delsing, J. Elias-

son, R. Kyusakov, P. Stluka, M. Tilly et al., “Promising technologies for soa-based
industrial automation systems,” in Industrial Cloud-Based Cyber-Physical Systems.
Springer International Publishing, 2014, pp. 89–109.

[7] S. Karnouskos, A. W. Colombo, T. Bangemann, K. Manninen, R. Camp, M. Tilly,

M. Sikora, F. Jammes, J. Delsing, J. Eliasson et al., “The imc-aesop architecture
for cloud-based industrial cyber-physical systems,” in Industrial Cloud-Based Cyber-
Physical Systems. Springer International Publishing, 2014, pp. 49–88.
112 Paper B

[8] S. Karnouskos, A. W. Colombo, T. Bangemann, K. Manninen, R. Camp, M. Tilly,

P. Stluka, F. Jammes, J. Delsing, and J. Eliasson, “A soa-based architecture for em-
powering future collaborative cloud-based industrial automation,” in IEEE IECON
2012. IEEE, 2012, pp. 5770–5775.

[9] Interfacing devices. Cumulocity. [Online]. Available: http://www.cumulocity.com

[10] Home iotivity. IoTivity. [Online]. Available: https://www.iotivity.org/

[11] Thingworx industrial innovation platform - ptc. PTC. [Online]. Available:

https://www.ptc.com/en/products/iot/thingworx-platform/

[12] Learn & explore. Xively. [Online]. Available: https://xively.com/resources/\#

panel3

[13] P. Feiler, K. Sullivan, K. Wallnau, R. Gabriel, J. Goodenough, R. Linger,

T. Longstaff, R. Kazman, M. Klein, L. Northrop, and D. Schmidt, Ultra-Large-Scale
Systems: The Software Challenge of the Future. Software Engineering Institute,
Carnegie Mellon University, 2006.

[14] P. Suresh, J. V. Daniel, V. Parthasarathy, and R. H. Aswathy, “A state of the art

review on the internet of things (iot) history, technology and fields of deployment,”
in Science Engineering and Management Research (ICSEMR), 2014 International
Conference on, Nov 2014, pp. 1–8.

[15] M. Collina, G. E. Corazza, and A. Vanelli-Coralli, “Introducing the qest broker:

Scaling the iot by bridging mqtt and rest,” in 2012 IEEE 23rd International Sym-
posium on Personal, Indoor and Mobile Radio Communications - (PIMRC), Sept
2012, pp. 36–41.

[16] Z. Hao, L. Guohuan, W. Honghui, and S. Zhongkui, “Development for Protocol

Conversion Gateway of Industrial Field Bus,” in ADVANCED TECHNOLOGY IN
TEACHING - PROCEEDINGS OF THE 2009 3RD INTERNATIONAL CONFER-
ENCE ON TEACHING AND COMPUTATIONAL SCIENCE (WTCS 2009), VOL
2: EDUCATION, PSYCHOLOGY AND COMPUTER SCIENCE, ser. Advances in
Intelligent and Soft Computing, Wu, YW, Ed., vol. 117. HEIDELBERGER PLATZ
3, D-14197 BERLIN, GERMANY: SPRINGER-VERLAG BERLIN, 2012, Proceed-
ings Paper, pp. 211–216, 3rd International Conference on Teaching and Computa-
tional Science (WTCS 2009), Shenzhen, PEOPLES R CHINA, DEC 19-20, 2009.

[17] B. SINGH, DATA COMMUNICATIONS AND COMPUTER NETWORKS.

PHI Learning, 2011. [Online]. Available: https://books.google.se/books?id=
DNGFIF71eZgC

[18] C. Aoun and E. Davies, “Reasons to move the network address translator - protocol
translator (nat-pt) to historic status,” Internet Requests for Comments, RFC Editor,
RFC 4966, July 2007.
 113

[19] Z. Shelby, K. Hartke, and C. Bormann, “The Constrained Application Protocol

(CoAP),” RFC 7252 (Proposed Standard), Internet Engineering Task Force, Jun.
2014. [Online]. Available: http://www.ietf.org/rfc/rfc7252.txt

[20] C. Bormann, A. Castellani, and Z. Shelby, “Coap: An application protocol for

billions of tiny internet nodes,” Internet Computing, IEEE, vol. 16, no. 2, pp. 62–
67, March 2012.

[21] “Mqtt version 3.1.1,” OASIS, October 2014. [Online]. Available: http:
//docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html

[22] P. Varga and I. Moldovan, “Integration of service-level monitoring with fault man-
agement for end-to-end multi-provider ethernet services,” IEEE Transactions on
Network and Service Management, vol. 4, no. 1, pp. 28–38, June 2007.

[23] M. Kovatsch, S. Mayer, and B. Ostermaier, “Moving application logic from the
firmware to the cloud: Towards the thin server architecture for the internet of
things,” in Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS),
2012 Sixth International Conference on, July 2012, pp. 751–756.
114
 Paper C
IoT Interoperability - On-demand
and low latency Transparent
Multi-protocol Translator

Authors:
Hasan Derhamy, Jens Eliasson, Jerker Delsing

Reformatted version of paper originally published in:

Journal paper, IEEE Internet of Things Journal, 2016.

© 2016, IEEE, Reprinted with permission.

115
116
 IoT Interoperability - On-demand and low latency
Transparent Multi-protocol Translator

Hasan Derhamy, Jens Eliasson, Jerker Delsing

Abstract

In the Industrial Internet of Things there is a clear need for a high level of interoperabil-
ity between independently developed systems, often from different vendors. Traditional
methods of interoperability including protocol gateways and adapters, are often used at
the network layer. Recent work on application interoperability has emphasized the use of
middleware or protocol proxy/gateway. However, middleware tends to move the interop-
erability problem rather than solving it, and there are scalability issues with increasing
the number of proxies; re-configuration effort, and required bandwidth and processing
overheads.
This paper proposes a secure, on-demand and transparent protocol translator for
the Industrial Internet of Things. Targeting the challenge of interoperability between
IP-based communication protocols, the paper analyses current solutions and develops a
set of requirements to be met by IoT protocol interoperability. The proposed protocol
translator is not a middleware, it is a SOA-based participant, it is used on-demand when
needed, it does not introduce design time dependencies, it operates transparently, it
supports low-latency, and it is secured through the use of Arrowhead authorization and
authentication.

1 Introduction
The Internet of Things (IoT) is a large and heterogeneous collection of networks, devices,
developers, owners, users and stakeholders. Advances in low cost processors have been a
key enabler of intelligent automation devices. IoT takes the next step of networking these
devices, resulting in intelligent environments. With the heterogeneity of independent
stakeholders a plethora of protocols have been developed. Many of the protocols will
never be known as they are proprietary. But even within standardized protocols there
is a large variety to choose from. They are the result of evolving requirements and
technology. Leading to a highly dynamic ecosystem of co-existing protocols unable to
work with each other. Interoperability in such an ecosystem is a major challenge, and
yet it is a crucial aspect of successful IoT.
This challenge has motivated a large body of research in both academia and indus-
try . One such ambitious project is Arrowhead [2]. With 79 partners from industry
and academia, its grand challenges are ”enabling the interoperability and integrability
of services provided by almost any device”. Arrowhead envisions that Service-Oriented

117
118 Paper C

Architecture (SOA) is able to tackle these challenges of integration and interoperabil-

ity. In this effort, there are many pilot domains in the areas of Virtual energy market,
Energy mobility and Energy production and end-user services to Production and Smart
buildings and infrastructure. Communication protocol convergence has been seen within
the pilot domains. However differing requirements between pilot domains has shown a
selection of various protocols. They are HTTP, XMPP, MQTT, CoAP and OPC-UA.
This exemplifies both protocol convergence and divergence.
Interoperability in communications protocols is not new, research dates back to the
1990’s with regard to protocol gateways [3, 4] and adapters [5, 6] and more. These
protocol converters primarily concentrate on network layer protocols.
Middleware is an intuitive approach to creating interoperability. It is often a set
of software libraries or infrastructure mediating communication between two systems,
for example starlink [7], INDISS [8] or uMiddle [9]. However, middleware enforces and
influences application design decisions. Additionally, middleware moves, rather than
solves, the interoperability problem. Middleware is further discussed in the related work
in Section 2.
More recent work has looked at bridging IoT protocols using a protocol proxy [10, 11]
and using Enterprise Service Bus (ESB) for multi-protocol systems, such as MuleSoft
ESB [12], IBM MQ [13] and Artix [14].
A proxy is essentially a direct protocol-to-protocol converter. There are three forms a
proxy can take, the forward proxy acts on behalf of the client, the reverse proxy acts on
behalf of the server and the interception proxy is a man-in-the-middle. Protocol proxy
is limited in two ways, firstly, the number of proxies will increase very quickly with each
new protocol. Secondly, each IoT device or application requires re-/configuration for
making use of a proxy or changing a proxy.
An ESB is a form of centralized software bus, it is an integration middleware which
will normally support a variety of protocols [1]. It is based on using an intermediate
protocol [13] between service consumers and providers. Centralized systems such as ESB
introduce bandwidth and processing overheads. Every message between peers must be
routed through the ESB regardless of the need. Additionally, integration points between
independent ESBs is not clear or standardized, therefore requires custom configuration.
An obvious alternative which should be mentioned, is that of convergence. Is it
possible for IoT to converge on a single shared protocol? This is not un-precedented, the
Internet has seen convergence on the Internet Protocol (IP). But even so, above TCP
and UDP there are still numerous application protocols and un-countable proprietary
protocols. At this time, it is unlikely to see convergence on a single IoT communication
protocol. Section 2 further elaborates on related work.
It can be said that protocol interoperability has unsolved questions and with the
attention on IoT it has become a crucial challenge to resolve.
The IoT and in particular Industrial IoT (IIoT) adds new requirements for protocol
interoperability which complicates the challenge. It demands that the protocol bridge is
transparent, scalable, secure and verifiable. Additionally it must be capable of verbose
reporting and supporting Quality of Service (QoS). QoS Service Level Agreement (SLA)
1. Introduction 119

is often protocol dependent and when translating between protocols the SLA must be
preserved or relevant changes notified to consumer and provider systems. These require-
ments are described in Table 1.

Table 1: Summary of challenges in IoT protocol interoperability

Challenge Description
Transparency Transparency is that there should not be any configura-
tion introduced to application systems due to protocol
bridging. As the numbers of devices increase, the cost
of configuring and re-configuring inhibits change.
Scalability Scalability means the interoperability solution must
scale down to a single threaded embedded Linux board
and scale up to handle hundreds of active service
consumer-provider exchanges.
The proposed solution takes this into consideration, but
does not perform evaluation.
Secure Secure means that independently distributed applica-
tions are able to authenticate and authorize with the
interoperability solution so that it is a trusted-man-in-
the-middle.
This work proposes a solution, but does not implement,
or evaluate.
Verifiability Verifiability provides confidence the interoperability so-
lution is well-formed, but also enables automated gen-
eration for new protocols.
The proposed solution does not address this challenge.
Reporting Reporting is that the interoperability solution must re-
port errors, exceptions, utilization and performance.
When dealing with large networks of devices, edge cases
become common and only thorough reporting can root
cause analysis be performed.
The proposed solution addresses this challenge, but does
not implement.
QoS QoS is essential for IIoT applications. SLA negotiation
and monitoring is the very minimum toward robust IIoT
applications.
The challenge will be addressed in future contributions.

This paper is an effort toward addressing the research challenge of implementing an

IIoT suitable protocol translator satisfying the above mentioned requirements. The pro-
posed translator approaches the challenge from a SOA point of view. A benefit of the
120 Paper C

proposed translator is treating translation as an active function within the SOA-based

framework. The proposed translator provides on-demand protocol bridging and is a com-
posable service. It supports overall operation without introducing dependencies on the
application systems. Described in Section 3, the SOA-based Arrowhead framework intro-
duces the key ingredients for a SOA-based application. It is the dynamic orchestration
and authorization which the proposed translator design relies on for enabling secure,
on-demand, protocol translation injection.
The main contributions of this paper are;

1. Proposed protocol translator for SOA-based systems. The protocol translator is a

good way of achieving interoperability without imposing design dependencies on
application software (Section 4);

2. Software architecture of the protocol translator (Section 5). Based on hub-and-

spoke the architecture is both scalable and modular;

3. Performance evaluation against two main cases: the Californium HTTP-CoAP

Proxy and in the ideal scenario of single protocol (Section 7).

The next section will present the related work in detail before briefly introducing the
Arrowhead framework in Section 3.

2 Related Work
Middleware is a common approach to addressing interoperability, some examples are
uMiddle [9], starlink [7], INDISS [8] and UIC [15]. In UIC client systems are required
to implement parts of the middleware in order to interact with the middleware infras-
tructure and each other. On the other hand, uMiddle and INDISS require no changes
to the client systems they are bridging, so they are transparent to the client. As these
solutions have focused on networked homes, they are oriented toward protocols such as
Service Discovery Protocol and Simple Service Discovery Protocol. To use in wider IoT
areas, there are problems with security and scalability. Furthermore, the middleware so-
lutions have moved the interoperability challenge so that they bridge some protocols, but
would themselves require bridging between each other. In-fact both starlink and uMiddle
claim to address interoperability issues between existing middleware, while themselves
introducing another layer of middleware.
ESB is a form of middleware infrastructure used in SOA-based enterprise systems. It
is often used in highly controlled or static environments where service composition is the
primary challenge. It provides an intermediate protocol, for example as in [13], to which
other service based protocols are translated. For dynamic systems or systems consisting
of many independent device owners, these solutions incur heavy configuration and delay
costs.
Protocol translation has been the tradition of network layer protocols and research
has been on-going since very early networks pre-dating the Internet. A detailed survey
3. Arrowhead Framework 121

of the problem of protocol conversion was presented by Green in [16]. In this work Green
describes in great detail the challenges of protocol conversion and presents a structured
approach toward converter creation. Green’s work on protocol conversion carries concepts
and ideas which can be applied to current networks.
Kenneth et al. in [17] lay the ground work to use formal methods in protocol con-
version. This work utilized two formal methods; Conversion via projection and by finite
state machine, presented by Okumaru. This fundamental work has led to further, more
recent, work in the area. Sinha et al. in [18] worked toward a formal method of syn-
thesizing protocol converters for use in System on-a-chip(SoC). Utilizing CTL Module
Checking they were able to generate verifiable converters capable of buffering signals and
handle behavioral protocol mismatches.
Also, relevant to this work is protocol translator generation. Work in this area fol-
lows very closely on from formal methods, automated converter generation is presented
by Liu et al. in [19] and Bromberg et al. in [20]. They argue that manual generation
of translation schemes is error prone, costly and introduces implementation delays when
new protocols are introduced. Automatic generation of translators is a huge cost reduc-
tion, but requires specification in a Domain Specific Language (DSL). This issue is not
addressed in this paper.
Protocol proxying is a well known method of creating interoperable applications.
For example an HTTP-CoAP proxy mapping [21] was proposed even while the CoAP
protocol was still in early draft form. There have been several works undertaken to prove
the implementation and usability of HTTP-CoAP proxies including work by Lerche et
el. in [10] and Castellani et al. in [11]. In [10] the forward proxy required that the HTTP
client be configured to pass the CoAP URI in the query path of the proxy request. In
[11], in-addition to the forward proxy, a reverse proxy was also utilized, which requires
the CoAP server nodes to register their routes with the proxy. After this configuration,
the rest of the operation can be made transparently between the protocols. In order to
remove even this, [10] suggests that an interception proxy is usable, but not desired as
it requires further network configuration. The protocol proxy does not act as a formal
middleware, but in most configurations is an always on translator. Meaning that even
when translation is not required the message will transit through the proxy and incur a
certain delay.

3 Arrowhead Framework
The Arrowhead framework [22] provides the operational environment for the translator.
Arrowhead exploits existing open standards for communication and security. There are
three core functions it provides, service registry, orchestration, and authorization. These
are detailed in the next sub-sections. Directly related to the challenge of interoperability
is the SOA principle ”Standardized Service Contract” [23]. Arrowhead uses the document
structure shown in Figure 1. In the center Service A Contract defines Service A which is
exchanged between System P and System C. The four documents AS D, AI DD, AC P and
AS P are presented by Blomstedt et al. in [24] and are described in Table 2. As defined
122 Paper C

by the Object Management Group, the SoaML [25] specification has participant entities
which provide and consume services. Within the context of the Arrowhead framework a
system is the participant entity.

Figure 1: The Arrowhead service contract

Table 2: Arrowhead Service Contract Documentation

Document Description
SD Service Description - A pure functional description of the service.
IDD Interface Design Description - The concrete description of the in-
terface including access methods and information exchange flow.
CP Communication Profile - the concrete description of the messag-
ing protocol and/or how the logical transportation of information
occurs.
SP Semantic profile - The technology for encoding and the semantic
standard used.

Following is a brief introduction to the core components of the Arrowhead Framework:

3.1 Service Registry

The service registry is the first step in a SOA-based system and acts as the directory for
advertising and discovering services. Arrowhead uses the standardized DNS-SD service
registry and provides a REST over HTTP/CoAP interface for performing the functions
of publishing presence and discovery.

3.2 Authorization
Authorization is a core service which allows service providing systems to delegate respon-
sibility of access control to a centralized system. This allows constrained systems a high
4. Proposed SOA-based translator 123

level of security without introducing security processing overhead in the service exchange
[26].

3.3 Orchestration
Within the context of SOA-based software systems, orchestration is the process of find-
ing the optimal service provider for a target consumer. For example; a thermostat for
a room within a building will need to consume a temperature service provider. Orches-
tration should be able to identify a suitable temperature service provider, such that the
thermostat is able to function appropriately.

4 Proposed SOA-based translator

Protocol conversion can be done as direct protocol-to-protocol translation. It has the
minimum information loss in the conversion process. A pair of protocols will utilize
one translator. However in a system with more than 3 protocols, the number of direct
translations grows faster than the number of protocols, given by Formula 1 where n =
number of protocols.
n−1
X n(n − 1)
k= (1)
k=1
2

For example 4 protocols will require 6 translators and 5 protocols will require 10 and
so on. This is illustrated in Figure 2-a.
To reduce the number of translators, an intermediate protocol can be selected from
the existing protocols. All protocols will be translated to the intermediate protocol, and
the number of translators will be equal to n-1. But as shown in Figure 2-b a peering of
nodes with protocols A and E, will use two translators to communicate. This results in
increased information loss and delay.

Figure 2: Comparison of direct vs mediated translation

124 Paper C

To address this issue the proposed translator utilizes an intermediate format, not
protocol, shown in Figure 2-c. The intermediary format is not sent on the wire and can
capture all protocol specific information. The information loss is reduced to that of a
single translator, from intermediate format to target protocol. There is a lower parsing
requirement going from protocol to intermediate format and so the translation delay is
reduced to that of a single translator.
The proposed translator is located within local clouds of IoT Things. Figure 3 shows
a cloud servicing an industrial site, while another cloud has mobile things and a charging
station. In these cases, the translator is positioned along side the Arrowhead core services.
The translator will utilize information regarding the service contract (detailed in Table
2), to identify mismatches and what is required to resolve these mismatches. For the
protocol translator, only the Communication Profile is relevant, as this is what captures
all protocol related information. But the rest of the document structure will be utilized
as the translator is extended with translation between different semantics and security
technologies.

Figure 3: The proposed translator within IoT

Referring back to Arrowhead service contract, it can be said that the proposed trans-
lator resolves differences in the contracts between a service provider and consumer. This
interoperability scenario is shown in Figure 4. System P honors contract Service A and
System C honors Service A*. The difference between service contracts is in the commu-
nication protocol and possibly the interaction pattern. The translator must honor both
contract A and A* and bridge the two service contracts.
The translator is transparent to both systems in the service exchange which means
that the translation mechanism does not rely on any in-band information beyond than
the normal service contract. The translator therefore is provided out-of-band information
in order to correctly honor both Service A and Service B contracts. This out of band
4. Proposed SOA-based translator 125

Figure 4: An Arrowhead service contract with a protocol mismatch

information is detailed in Section 6. In Figure 5 the SOA-based Translation System is

shown with all of its external interfaces and also its transient translation interfaces. At
the top of the figure are the external interfaces to the Arrowhead core services. The
Translation Service is the discoverable interface for requesting instantiation of transient
translation interfaces. The Protocol A and Protocol B can be any combination of IP based
protocols. The Transient Services are not discoverable and are dedicated to bridging a
specific combination of service provider and consumer.

Figure 5: Translator SOA system interfaces diagram

The Translation Service interface receives information regarding the service contracts
to be honored and extracts the required protocol information. Part of the request also
includes the addressing information of the service provider instance. There is further
126 Paper C

detail in section 6 for this interface.

5 Translator Architecture
In this section the proposed translator has been implemented. It is an important contri-
bution as it serves as a proof of extendability and guide for refining the translator.
Each protocol is implemented as a service provider, server, and a service consumer,
client. The overview block diagram of the translator architecture follows in Figure 6. The
proposed translator has been implemented in Java. The translation system is composed
of a single translator service instance and the translation service initializes and managers
translation hubs. Each translator hub has two base spokes, which are initialized according
to the requested protocol configurations.

Figure 6: Translator Block diagram

The translator service and hub are aimed at being kept simple and lightweight. The
sophistication of the translator is in the spoke implementations. Almost any protocol
library could be used to implement the protocol spokes. Evolution and extension to the
spoke capability should not impact the core of the translator or other spoke implemen-
tations.

5.1 Intermediary format

An intermediary format enables inter-changeable usage of protocol spokes. This makes
the intermediary format critical for the extendability of the translator. The intermediary
format must support data, meta-data and state associated with a message. Addition-
ally the intermediary format must also hold behavioral aspects of the protocol. This
is because protocols define interaction patterns and these must also be translated. A
request-response or publish-subscribe pattern will require different approaches, for ex-
ample toward error handling [27].
The intermediary format is shown in Figure 7. Method is the CRUD operation; object
is the entity to be operated on; query is the parameters of a read operation; payload is
5. Translator Architecture 127

the body of the packets; payloadFormat is the format of the payload; exception is the
error code.

Figure 7: The intermediary format is made up of 7 items

5.2 Error Handling

Error handling is a special concern for application layer translators. Protocol translation
error handling has been investigated in [27]. They concluded that in order to bridge
differences in protocol error/exception handling, the notion of behaviors must be intro-
duced. A behavior is a protocol spoke specific routine which handles a protocol mismatch
by a combination protocol and non-protocol actions.

5.3 Quality of Service

At times Quality of Service (QoS) might be best-effort or it could be critical to correct
system operation. Although the translator is not intended to be used in hard real-time
applications, QoS is still possible to be monitored and reported. The translator Service
Level Agreement (SLA) is negotiated at the time of translator service request. Table 3
shows the QoS parameters which are of primary concern to the translator.

5.4 Security
IoT security offers many challenges in its own right. The protocol translator does not
need to interrogate service payloads, and so end-to-end encryption of the payload is
possible. The question of authenticity and authorization is addressed by handshaking
with the Arrowhead authorization system, as shown in operations b, e and g of Figure 9.
128 Paper C

Table 3: Quality of Service considerations for translators

QoS Parameters Description
Losses Represents the number of requests or responses lost, can either be
by the translator or in the network
Delays Must represent the maximum, minimum and median delays asso-
ciated with translated requests
One-way throughput The number of requests translated over a given period
Two-way throughput The number of request-response pairs translated over a given period
Variances Variance measurements for each of the parameters is desired to
monitor health of the translator
Utilization metrics Total requests, responses, payload sizes, up-time and maximum
concurrent requests

5.5 CoAP Spoke

The CoAP spokes have been implemented using the Californium CoAP library. The
CoAP server spoke consists of a Californium CoAP server with the RootResource over-
ridden. The CoAP client spoke uses a worker to asynchronously perform the CoAP
request and return the updated context to the calling spoke. There are no special be-
haviors implemented in these spokes.

5.6 HTTP Spoke

The HTTP server spoke has been implemented using the Apache HTTP libraries, it is
loosely based on the Californium proxy HTTP Stack. The server spoke sets up the HTTP
listener and on reception of an HTTP request forwards the request to the next spoke.
Similar to the CoAP spoke the HTTP client spoke uses a worker to asynchronously
perform the HTTP requests. There are also no behaviors implemented in these spokes.

5.7 MQTT Spoke

The MQTT spoke has been implemented using Eclipse Paho library. The primary differ-
ence in this spoke is the worker thread is now on the MqttServer spoke rather than the
client spoke. This is needed to achieve pro-active behavior. The worker thread is used
in the different configurations shown in Figure 8. In the case of 8-a the translator must
initiate a pull from the REST server and publish the result to the MQTT broker. While
in the case of 8-b the translator, must pro-actively subscribe to the MQTT broker, and
push the results to the REST server. In both cases the spoke orientation remains the
same, while the direction of data flow changes the interaction pattern of the translator.
In the cases of 8-c & 8-d the translator now has a MQTT client spoke and a REST
(CoAP/HTTP) server spoke. The translator can wait for a REST client to initiate a
request to the REST server spoke before either subscribing, 8-c, or publishing, 8-d, to the
MQTT broker.
6. Transparency 129

Figure 8: MQTT translator configurations - interaction pattern challenge

6 Transparency
Within the context of this paper, transparency is from the perspective of the application
systems. That is, application system designers should not be concerned with protocol
mismatches or how they are resolved.
The proposed transparency works in conjunction with Arrowhead orchestration de-
scribed Section 3.3. Figure 9 shows a sequence diagram of a translation. After Arrowhead
system start-up, in 9−a the translator receives a service request and sets up a transient
translation hub. In 9−b the hub creates two protocol spokes and retrieves temporary
credentials from the Arrowhead authorization system. Next in 9−c it sends the response
with the address information of the provider spoke. This could be an MQTT topic +
broker, XMPP chat room + server or CoAP/HTTP URL. 9−d initiates the service ex-
change. System A is authenticated in 9−e. The translated request is now sent to the
target system 9−f . Prior to servicing the request, the translator is authenticated and
authorized by the authorization system. Finally the service response is translated back
to the origin. As can be seen from this diagram, the orchestrated ”consuming” system
is not required to take any actions outside normal Arrowhead boot-strapping.
Following the principles of SOA-based design, the translator system is autonomous
and so not coupled to the orchestration system. The translator system monitors and
cleans up redundant translator hubs. Garbage collection of disused translators is handled
by an internal spoke watchdog which will notify inactivity after a timeout. The time is
configurable in the translator service request generated by the orchestrating system. The
translator may receive duplicate requests for the same translation pairing. De-duplication
130 Paper C

is handled by the translator system and rather than creating a new translation hub
instance, the existing one will be sent in response.

7 Testing
The delay introduced by the translator has been measured and evaluated against the
Californium proxy. A control test with a single protocol scenario was also run. These
three setups are seen in Figure 10. Test setup 10-a has an CoAP request generated from
Firefox Copper plug-in and sent directly to the CoAP sensor, of course via a 6LoWPAN
border router. Setup 10-b generated an HTTP request from Firefox HttpRequester plug-
in to the translator, which then generated the corresponding CoAP request to the CoAP
sensor. Setup 10-c followed on from setup 10-b except that it utilized the Californium
Proxy rather than the SOA Translator. Both translator and proxy were run on the same
BeagleBone Black (BBB) hardware.
The hardware is setup with the Mulle sensor node running a ContikiOS CoAP Server
offering two services; Firstly, the Wheel Loader service measures the vibration, rotation
and temperature of a ball bearing on a Volvo wheel loader, the service endpoint is ad-
dressed as coap://[sensor-ip]:5683/wheel loader. The payload structure can be seen in
Figure 11 and has a length of approximately 375 bytes. It is transfered in two CoAP
blocks and is transmitted as a confirmable response.
The Power service measures the sensor nodes power status. The payload can be seen
in Figure 12 and is approximately 202 bytes. It is also a confirmable response, and is
sent in a single CoAP block.
The BBB is running Debian Wheezy Linux distribution and is powered and tethered
to a Laptop via a USB slave. The USB Network Adapter (usb0) is used and has static
IP Addresses 192.168.7.2 and FDFD:55::80FE. IPv6 packet forwarding has been enabled.
The border router of the 6LoWPAN network is connected to the BBB through Contiki’s
tunslip6 program. Therefore there is full IP connectivity from the test pc to the sensor
node on the 6LoWPAN network. This hardware setup is shown in Figure 13.
There are three Putty sessions connected to the BBB for monitoring and control of
the applications. In order to monitor the delay introduced by the translator, timing
measurements are scoped to a single time domain on the BBB.
A network traffic monitor tcpdump is running on the BBB and is used to take time
stamps on the external usb0 network interface and the 6LoWPAN tun1 network interface.
The network time stamps provide two round trip times (RTT) which, when subtracted,
isolates the time spent between the BBB’s network interfaces.
In the case of the proposed translator, there are 4 additional time measurements which
help to analyze the translation delays. The timers t1 , t2 , t3 , t4 are Java nanosecond timers,
with millisecond resolution. All time measurements from Figure 13 are listed in Table 4.
The calculated times in the lower part of the table are: T1 is the RTT through usb0; T2
is the RTT through tun1; T3 is the duration that the packet is held within the BBB.
The tests were carried out 20 times per service request per scenario. The results
are averaged and can be seen tallied in Table 5. The CoAP only scenario shows the
7. Testing 131

Figure 9: Translator interaction sequence diagram

132 Paper C

Figure 10: Overview of test scenarios

<?xml v e r s i o n =”1.0” e n c o d i n g=”UTF−8”?>

<senml xmlns=”urn : i e t f : params : xml : ns : senml ”
bn=”urn : dev : mac : 0 0 2 4 b e f f f e 8 0 4 f f 1 ”
bt =”1425256833” v e r =”1”>
<e n=”p k t i d ” u=”count ” v=”43”/>
<e n=”rpm” u=”r /m” v=”−3”/>
<e n=”rpm av ” u=”r /m” v=”0”/>
<e n=”t o t r o u n d s ” u=”count ” v=”0”/>
<e n=” t o t r o u n d s r ” u=”count ” v=”0”/>
<e n=”bearingtemp ” u=”Cel ” v=”25.9”/ >
<e n=” r s s i ” u=”%” v=”30”/>
</senml>

Figure 11: Wheel loader request payload

<?xml v e r s i o n =”1.0” e n c o d i n g=”UTF−8”?>

<senml xmlns=”urn : i e t f : params : xml : ns : senml ”
bn=”urn : dev : mac : 0 0 2 4 b e f f f e 8 0 4 f f 1 ”
bt =”1425256855” v e r =”1” bu=”V”>
<e n=”Vbat” v=”4162”/>
<e n=”Vchr ” v=”5080”/>
</senml>

Figure 12: Power request payload

optimal performance with only very small IPv6 forwarding delay in the BBB. While the
Californium Proxy had the largest delays on the BBB, with averaged delay introduced
by the proxy between 146 ms and 177 ms. The translator showed better performance
with shorter delays of between 50 ms and 77 ms.
Shown in Figure 14 is a stacked line graph of the translator timing for the power
service request. The lower line is the time spent in the HTTP library, the middle line
7. Testing 133

Figure 13: Test time stamp instrumentation on the BeagleBone Black

Table 4: Test setup timing instrumentation

Translator Timing Instrumentation
t1 = request arrives at application spoke
t2 = request leaves the application spoke
t3 = response arrives at the application spoke
t4 = response leaves the application spoke
Network Time Stamps
tusb1 = request passes through WAN adapter
tusb2 = response passes through WAN adapter
ttun1 = request passes through 6LoWPAN adapter
ttun2 = response passes through 6LoWPAN adapter
Calculations
T1 = tusb2 - tusb1
T2 * = ttun2 - ttun1
T3 = (ttun1 − tusb1 ) + (tusb2 − ttun2 )
* Includes IEEE 802.15.4 transmission and sensor processing time

Table 5: Averaged test results (ms)

CoAP only Translator Californium
Wh Ldr Power Wh Ldr Power Wh Ldr Power
T1 246 125 354 179 425 269
T2 246 125 277 128 248 123
T3 0* 0* 77 50 177 146
* This delay represents the Linux IPv6 packet forwarding delay
134 Paper C

is the time spent in the translator application, and the top line is the time spent in the
CoAP library.

Figure 14: Proposed translator delay timing for the /Power request

Except a few outliers, there is little variability in the translator timing. The translator
hub and spokes are currently very simplistic, and most of the processing will be in the
protocol libraries, handling new requests and tracking responses. Shown in Figure 15 is a
similar graph for the wheel loader request. There is much greater variability in the timing
for this request. The CoAP library seems to be responsible for much of the variability.
This is likely to be related to the 2-block transmission and the CoAP library handling
this differently to the first case.
The CoAP library tends to consume a larger proportion of the processing time than
expected. This could be because the CoAP library uses many threads [28] and so perhaps
running on a single core processor introduces overhead in thread context switching.
The next two Figures 16 and 17 compare the RTT of the three scenarios. It can be
seen in the Power request graph (Figure 16) that the CoAP only transmission is very
consistent. The Californium Proxy varies over the requests while the translator has a
few outliers but otherwise stays within a narrow band of +/- 40ms.
In the case of the Wheel Loader service request the control time is increased from
just above 125ms to close to 250 ms. The Californium Proxy seems to settle down after
the first 7 to 10 requests. Removing this settling time will bring timing very comparable
to that of the translator.
It is likely that the difference in timing between the Californium Proxy and the pro-
posed translator is due to the complexity of the implementation. The proxy is constructed
7. Testing 135

Figure 15: Proposed translator delay timing for the /Wheel Loader request

Figure 16: Comparison of RTT for the /Power request

to be able to handle a request from any HTTP client and to then translate and forward to
any CoAP server. Also, the forwarding address of the proxy uses must be passed in-line
in the HTTP path. The proxy must manage many in-flight messages sent to and from
many end-points. Where as the translator has a predetermined server and is expected to
be used by only client. It does not need to process addressing information while trans-
136 Paper C

Figure 17: Comparison of RTT for the /Wheel Loader request

lating. Another difference is the proxy also handles caching which the translator does
not. Still the proxy is a direct protocol-to-protocol translator and so is optimized for that
translation. While the proposed translator is built to translate between any protocol to
any protocol, dependent only on a protocol spoke being available.

8 Conclusion
IoT interoperability is a challenge that is being addressed widely. The limitations with
current solutions is reliance on permanent middleware or highly configured networks/ap-
plications. This results in scalability and performance issues. They create operational
silos requiring specialized integration points. Moreover, the IoT domain imposes new
requirements to the application of interoperability solutions, as described in Table 1.
This paper addresses this challenge by proposing an on-demand and transparent
multi-protocol translator for SOA-based IoT applications. Operating alongside a SOA-
based orchestrator, the proposed translator is used on-demand for definite protocol mis-
match. The orchestrator composes the service exchange through the translator resulting
in zero application configuration.
The proposed translator architecture splits a protocol into two spokes, a service
provider spoke, and a service consumer spoke. The translator can handle many pro-
tocols as each protocol implements just two spokes. The spokes integrate through an
intermediate format. This allows any combination of protocol spokes. However translat-
ing from publish-subscribe to request-response patterns demands pro-active spoke imple-
mentation. The MQTT spokes actively pull and push messages, imitating the original
9. Future work 137

interaction pattern..
Testing showed that the proposed translator performed as well, if not better than a
common protocol proxy, the Californium proxy. The proposed translator introduced on
average a 50 ms delay for a single block CoAP request + response and 77 ms delay for
a two block CoAP transmission. This difference is caused by the CoAP library handling
multi-block messages. This result demonstrates that the intermediate format does not
introduce more delay than the direct protocol-to-protocol translation. Which means that
the proposed translator demonstrates satisfactory performance and a minimum increased
latency.
The paper also proposes security and reporting aspects of the translator, however
these are not yet implemented. The translator actively participates in security hand-
shakes resulting in fine grain access control and authentication. The translator utilizes
event reporting for active fault detection and support diagnostics. The translator follows
SOA principles and can operate both autonomously, or as a traditional gateway.

9 Future work
Future work involves developing one-to-many protocol fan-out, multiple translator or-
chestration and translating encoding and semantics.
The current hub only supports two spokes. An enhancement would be to support a
one to many fan out. This would enable multiple service consumers of different protocols
to access a single service provider.
The current translator operates as a single translator within a local cloud. It is
desirable to deploy multiple translators to a single local cloud to help with load balancing
and reducing communication latency by selecting a nearby translator.
As stated in interoperability requirements translating encoding, compression and se-
mantic differences must be addressed. Once this is achieved, only the abstract service
description needs to be common between two service contracts in order to achieve inter-
operability.
Finally, the proposed security, reporting and formal specification of the translator
needs to be implemented.

References
[1] H. Derhamy, J. Eliasson, J. Delsing, and P. Priller, “A survey of commercial frame-
works for the internet of things,” in 2015 IEEE 20th Conference on Emerging Tech-
nologies Factory Automation (ETFA), Sept 2015, pp. 1–8.

[2] Arrowhead, ahead of the future. Arrowhead. [Online]. Available: http:

//www.arrowhead.eu

[3] S. K. Noh and S. H. Lee, “An implementation of gateway system for heterogeneous
protocols over atm network,” in Communications, Computers and Signal Process-
138 Paper C

ing, 1997. 10 Years PACRIM 1987-1997 - Networking the Pacific Rim. 1997 IEEE
Pacific Rim Conference on, vol. 2, Aug 1997, pp. 535–538 vol.2.
[4] E. Benhamou and J. Estrin, “Multilevel internetworking gateways: Architecture and
applications,” Computer, vol. 16, no. 9, pp. 27–34, Sept 1983.
[5] K. L. Calvert and S. S. Lam, “Adaptors for protocol conversion,” in INFOCOM
’90, Ninth Annual Joint Conference of the IEEE Computer and Communication
Societies. The Multiple Facets of Integration. Proceedings, IEEE, Jun 1990, pp. 552–
560 vol.2.
[6] P. Dhar, R. C. Ganguly, S. Das, and D. Saha, “Network interconnection and protocol
conversion-a protocol complementation approach,” in TENCON ’92. ”Technology
Enabling Tomorrow : Computers, Communications and Automation towards the
21st Century.’ 1992 IEEE Region 10 International Conference., Nov 1992, pp. 116–
120 vol.1.
[7] Y. D. Bromberg, P. Grace, and L. Réveillère, “Starlink: Runtime interoperability
between heterogeneous middleware protocols,” in Distributed Computing Systems
(ICDCS), 2011 31st International Conference on, June 2011, pp. 446–455.
[8] Y.-D. Bromberg and V. Issarny, “Indiss: Interoperable discovery system
for networked services,” in Proceedings of the ACM/IFIP/USENIX 2005
International Conference on Middleware, ser. Middleware ’05. New York, NY,
USA: Springer-Verlag New York, Inc., 2005, pp. 164–183. [Online]. Available:
http://dl.acm.org/citation.cfm?id=1515890.1515899
[9] J. Nakazawa, H. Tokuda, W. K. Edwards, and U. Ramachandran, “A bridging frame-
work for universal interoperability in pervasive systems,” in Distributed Computing
Systems, 2006. ICDCS 2006. 26th IEEE International Conference on, 2006, pp. 3–3.
[10] C. Lerche, N. Laum, F. Golatowski, D. Timmermann, and C. Niedermeier, “Con-
necting the web with the web of things: lessons learned from implementing a coap-
http proxy,” in 2012 IEEE 9th International Conference on Mobile Ad-Hoc and
Sensor Systems (MASS 2012), vol. Supplement, Oct 2012, pp. 1–8.
[11] A. P. Castellani, T. Fossati, and S. Loreto, “Http-coap cross protocol proxy: an
implementation viewpoint,” in 2012 IEEE 9th International Conference on Mobile
Ad-Hoc and Sensor Systems (MASS 2012), vol. Supplement, Oct 2012, pp. 1–6.
[12] Mule esb, enterprise service bus, open source esb, mulesoft. MuleSoft. [Online].
Available: https://www.mulesoft.com/platform/soa/mule-esb-open-source-esb
[13] Ibm mq. IBM Corp. [Online]. Available: http://www-03.ibm.com/software/
products/en/ibm-mq
[14] Artix - micro focus. Micro Focus International plc. [Online]. Available:
https://www.microfocus.com/products/corba/artix.aspx
References 139

[15] M. Roman, F. Kon, and R. H. Campbell, “Design and implementation of runtime

reflection in communication middleware: the dynamictao case,” in Electronic Com-
merce and Web-based Applications/Middleware, 1999. Proceedings. 19th IEEE In-
ternational Conference on Distributed Computing Systems Workshops on, 1999, pp.
122–127.

[16] P. Green, “Protocol conversion,” IEEE Transactions on Communications, vol. 34,

no. 3, pp. 257–268, Mar 1986.

[17] K. L. Calvert and S. S. Lam, “Formal methods for protocol conversion,” IEEE
Journal on Selected Areas in Communications, vol. 8, no. 1, pp. 127–142, Jan 1990.

[18] R. Sinha, P. S. Roop, and S. Basu, “A module checking based converter synthesis ap-
proach for socs,” in VLSI Design, 2008. VLSID 2008. 21st International Conference
on, Jan 2008, pp. 492–501.

[19] C. Liu, B. H. Shen, S. Y. Oh, M. Gerla, J. Palsberg, C. Banner, and R. Butler,

“Agnostic protocol translation for cross-domain information sharing,” in Military
Communications Conference, MILCOM 2013 - 2013 IEEE, Nov 2013, pp. 1447–
1452.

[20] Y.-D. Bromberg, L. Réveillère, J. L. Lawall, and G. Muller, Middleware 2009:

ACM/IFIP/USENIX, 10th International Middleware Conference, Urbana, IL, USA,
November 30 – December 4, 2009. Proceedings. Berlin, Heidelberg: Springer Berlin
Heidelberg, 2009, ch. Automatic Generation of Network Protocol Gateways, pp.
21–41.

[21] A. Castellani, S. Loreto, A. Rahman, T. Fossati, and E. Dijk, “Guidelines

for http-coap mapping implementations,” Working Draft, IETF Secretariat,
Internet-Draft draft-ietf-core-http-mapping-09, April 2016, http://www.ietf.org/
internet-drafts/draft-ietf-core-http-mapping-09.txt. [Online]. Available: http:
//www.ietf.org/internet-drafts/draft-ietf-core-http-mapping-09.txt

[22] Arrowhead framework wiki. Arrowhead. [Online]. Available: https://forge.soa4d.

org/plugins/mediawiki/wiki/arrowhead-f/index.php/Main\ Page

[23] T. Erl, SOA Design Patterns, 1st ed. Upper Saddle River, NJ, USA: Prentice Hall
PTR, 2009.

[24] F. Blomstedt, L. L. Ferreira, M. Klisics, C. Chrysoulas, I. M. de Soria, B. Morin,

A. Zabasta, J. Eliasson, M. Johansson, and P. Varga, “The arrowhead approach for
soa application development and documentation,” in Industrial Electronics Society,
IECON 2014 - 40th Annual Conference of the IEEE, Oct 2014, pp. 2631–2637.

[25] “Service oriented architecture modeling language 1.0.1,” Specification, May 2012.
[Online]. Available: http://www.omg.org/spec/SoaML/1.0.1/
140

[26] P. P. Pereira, J. Eliasson, and J. Delsing, “An authentication and access con-
trol framework for coap-based internet of things,” in Industrial Electronics Society,
IECON 2014 - 40th Annual Conference of the IEEE, Oct 2014, pp. 5293–5299.

[27] H. Derhamy, J. Eliasson, J. Delsing, P. P. Pereira, and P. Varga, “Translation error

handling for multi-protocol soa systems,” in Emerging Technologies Factory Au-
tomation (ETFA), 2015 IEEE 20th Conference on, Sept 2015, pp. 1–8.

[28] M. Kovatsch, M. Lanter, and Z. Shelby, “Californium: Scalable cloud services for
the internet of things with coap,” in Internet of Things (IOT), 2014 International
Conference on the, Oct 2014, pp. 1–6.
 Paper D
Orchestration of Arrowhead
services using IEC 61499:
Distributed Automation Case
Study

Authors:
Hasan Derhamy, Dmitrii Drozdov, Sandeep Patil, Jan van Deventer, Jens Eliasson and
Valeriy Vyatkin

Reformatted version of paper originally published in:

Conference paper, IEEE Emerging Technologies and Factory Automation, Berlin, 2016.

© 2016, IEEE, Reprinted with permission.

141
142
 Orchestration of Arrowhead services using IEC
61499: Distributed Automation Case Study

Hasan Derhamy, Dmitrii Drozdov, Sandeep Patil, Jan van Deventer, Jens Eliasson and
Valeriy Vyatkin

Abstract

This paper presents a novel approach to automation of flexible manufacturing systems

with mechatronic intelligence and distributed control. The mechatronic intelligence layer
is implemented using a combination of wireless sensor/actuator networks with service-
oriented architecture, where services are located at the device level, as well as in local
and global Clouds following the Arrowhead framework.
The machine/floor level coordination is implemented using the distributed automation
architecture of IEC 61499, which is also used as a graphical tool for orchestration of
services.
The paper discusses the enablers developed in-order to combine IEC 61499 and Ar-
rowhead and the use is illustrated on a laboratory scale flexible factory example. By
integration of Industrial IoT with IEC 61499, we envision that large gains in terms of
engineering effort and system operation performance can be made.

1 Introduction
Modern concepts like Internet of Things, Cyber-physical systems and Industry 4.0 are
defining the shape of future industrial automation systems. These concepts use state
of the art technologies like low-power wireless communication, web services and low-
cost embedded devices. This, with increasing capabilities of computer networks and
embedded microcontroller devices, enables for example enhanced distributed monitoring
and control in factory automation applications.
As far as software development for distributed automation concerns, the currently
dominating paradigm of programmable logic controllers and their software framework
defined by the IEC 61131-3 standard are showing lots of limitations. There are several
approaches in software and systems engineering aiming at addressing the new challenges.
The IEC 61499 standard [1] has been developed as one such effort in the automation
community. It aims at a gradual extension of PLC capabilities and their convergence
with distributed systems. The use of service-oriented architecture in automation was
first proposed by Jammes and Smit [2] who outline opportunities and challenges in the
development of next-generation embedded devices, applications, and services, resulting
from their increasing intelligence. The work plots future directions for intelligent device

143
144 Paper D

networking based on service-oriented high-level protocols. According to [3] some features

of SOA attractive for automation are:

ˆ Loose coupling, since software modules provide services to other modules they are
designed in a relatively generic format. Communication between components is
asynchronous and only done when required.

ˆ Modularisation of software components. Control is not programmed for the entire

system, rather only for individual components resulting in natural control distri-
bution.

ˆ Common communication protocol, which is particularly important since service

providers are abstracted from the low level all the way to the high level, so that
implementation makes no differentiation of hardware devices or enterprise systems.

Potential gains of using SOA are claimed as cost reduction, potential to hire less skilled
labour, interoperability (cross-platform and cross-company) and implementation speed.
There is increasing number of use-cases demonstrating that functionality of complex
automation systems can be implemented in a distributed way. In [4, 5], it was shown
that functionality of material handling systems can be fully implemented via collaborative
effort of decentralised controllers embedded to basic devices, such as conveyor sections.
In [6, 7, 8, 9] the combination of IEC 61499 with SOA is comprehensively presented,
while [10] demonstrates the use of web-services in IEC 61499 applications.
This paper is addressing the challenges arising in application of a relatively ma-
ture SOA-based and automation-oriented platform Arrowhead for development of high-
integrity automation applications that include a good deal of distributed control in ad-
dition to the traditional data acquisition.
The results of this paper are threefold. Firstly, we describe main features of Arrow-
head framework and IEC 61499 architecture in the context of a use case that deals with
implementing distributed control of a manufacturing system. Using a combination of Ar-
rowhead services and IEC 61499 function blocks. Secondly, a framework for integration
of IEC 61499 with SOA-enabled IoT is described. Finally, test results from a prototype
implementation are presented.
The rest of this paper is structured as follows: Section 2 presents a brief overview of
the IEC 61499 standard. Section 3 provides an overview of the Arrowhead Framework.
Case study setup is discussed in section 4. Section 5 presents elements of the case study
automation implemented with the Arrowhead architecture, and Section 6 demonstrates
the use of IEC61499 for coordination and orchestration of Arrowhead services at the
system level. Finally, the paper is concluded in Section 6.

2 IEC 61499
The IEC 61499 [1, 11] is an international standard that introduces an open architecture
for distributed control systems, which is an important class of embedded systems with
2. IEC 61499 145

sd [Interaction] Arrow head Core Services w ith tickets [ Arrow head Core Services w ith tickets ]

«block» «block» «block» «block» «block»

Station1 : Station rs : Registry Service as : Authorization Service os : Orchestration Service aut : Application

1: POST: available services

2: GET: services
3: GET: services

4: services: address.port.path

5: services: address.port.path

loop
[0,*]
opt
6: GET: Ticket for service
[Ticket expired or about to expire]

7: Ticket

8: GET: service w ith ticket

9: GET: authorization w ith ticket

10: authorization w ith ticket lifetime

11: Ticket accepted w ith lifetime

12: GET: service

alt
[Valid ticket]
13: service

[else] 14: service request rejected

Cameo Systems Modeler, 1-1 C:\Users\deventer\iCloudDrive\SysML\AssemblyLine\AssemblyLine.mdzip Arrowhead Co

Figure 1: Interaction between application systems and Arrowhead core systems

a strong legacy background. The standard is called function block architecture after its
main design artefact that is an event driven (and event activated) function block. If one
146 Paper D

would abstract out unnecessary details, the standard introduces quite an elegant model
of distributed application that is a network of function blocks connected via control and
data flows. The control flow is modeled using the concept of event that is emitted from an
output of one function block and can be received at one or several inputs of other function
blocks. The most essential claim of the IEC 61499 architecture is about minimizing
developers’ efforts in deploying automation software to different distributed architectures
of hardware, which is attributed to the event-based communication mechanism.

3 Arrowhead
Arrowhead is an open source framework for industrial SOA-based applications. It has
been designed to not interfere in the basic control operation of industrial systems. The
grand challenges of the Arrowhead framework are to enable interoperability and integra-
bility of services produced and consumed by any device. To achieve this it defines three
core services which enable discovery, security and composition. Further support core ser-
vices are defined which enabled advanced system management, Quality of Service (QoS)
and service provisioning. In Figure 1 a SysML diagram models the interaction sequences
between application systems and the core systems. All Arrowhead SOA-based services
will follow a very similar line of interaction, only the systems hosting the services would
change depending on desired architecture.
Arrowhead defines systems as the participant entities of the SOA. The systems then
communicate via service contracts. Thus a service provider instance or consumer in-
stance must be associated with a distinct system. A service contract is defined by four
documents, Service Description, Interface Design Description, Communication Profile
and Semantic Profile. The details of these documents can be found in Arrowhead Wiki
[12]. Following the core functionality of the framework is described.

Orchestration The process of orchestration is essential in support of Service Re-

usability, Service Discoverability and Service Composability. Orchestration is the process
of matching optimal service providers and consumers. For industrial automation cases
such as the one described in this paper, the service matching is often one to one. This
is because for correct operation of the production line the optimal service pairing is the
only valid pairing. The Orchestration system has two services, an orchestration store
service holds the connection requirements and the orchestration engine service performs
advanced look-up and service provisioning.
The connection requirements, also referred to as orchestration rules, are gathered by
analysing engineering data such as SysML models.

Service Discovery Based on DNS-SD the service discovery core service has three func-
tions publish, un-publish and query. When a system comes ´online´ it must publish
all service providers which it hosts, and un-publish once the service is no longer available.
The query function is used by systems to locate services available for consumption.
4. Case study: Conveyor Loop with Distributed Control 147

Authorization Arrowhead authorization service uses x.509 certificates or tickets [13]

based on RADIUS, to authenticate systems and grant access rights to service consumers.
Clients such as sensor- and actuator nodes, other systems or users are required to get a
ticket from an authorization service. The tickets are then presented to the services which
perform a lookup to validate the and thus end peer. This approach also mandates that
encrypted communication is used such as DTLS, IPsec or VPN tunnels.

Translation As the distinction between domains blurs, interoperability becomes a key

challenge for integrators. The Arrowhead translator [14] is the beginning of a trans-
parency technology which makes services built on different technology stacks to seamlessly
interoperate. Initiated by a request from an Orchestrator, the translator will provision
a new end-point which reverse proxies the service provider. The translation is possible
between HTTP, CoAP, MQTT and OPC-UA.

4 Case study: Conveyor Loop with Distributed Con-

trol
A laboratory scale model of flexible manufacturing system Festo MPS FMS-50 (Figure
2) is used as a case study. We take a subset of the FMS-50 system that is a conveyor
loop, transporting trays (also called pallets), as shown in Figure 3. The conveyor and
the variable frequency drives are actuated by Arrowhead enabled services running on
embedded controllers. Systems A, B, C, D, E and F are running IEC61499 distributed
control and the main conveyor system is managed by the services. There are total of six
such entry exit points and we are using 4 of them.
Performed experiments show that the sensing and transmission delay usually is below
20 ms which makes the 6LoWPAN-based network architecture suitable for moderate
low-latency requirements and applications.

5 Application of Arrowhead architecture

The core Arrowhead systems used in this first phase are; the Service Registry, Orches-
tration and Translation systems. Use of the authorization system is to be implemented
in the coming phase. We start off with identifying the application services needed and
the existing core system usage.
In this case study the connection requirements are simple textual strings which de-
scribe which service provider type should be used.

5.1 Service identification

In order to identify the application services goal-service modelling [15] and existing asset
analysis methods were applied. The goal of the load/unload points was to inform the
station when a pallet is waiting for processing and to send the pallet onto the next
148 Paper D

Figure 2: Complete Festo MPS system.

station once all processing associated with the pallet is complete. Additionally, IEC 61499
application preferred a request-response pattern. This meant a poll based web service
structure would be followed, which identified two services defined as the palletAvailable
and processComplete services.

palletAvailable The palletAvailable service is responsible for monitoring the proxim-

ity sensors and execute decision logic to identify when a pallet is present, the barcode
of the pallet, and if it is laden with product. The service is implemented with CoAP
[16] and it is accessed through a CoAP resource coap://[ipv6]:5683/palletAvailable. The
response to a GET request is shown in Figure 4.

processComplete The process complete service is responsible for operating the actua-
tion mechanism and allows the pallet to move on to the next station. It is not responsible
for deciding when processing is complete, rather it receives this indication from the sta-
tion. This service is also implemented with CoAP using coap://[ipv6]:5683/palletAvailable
URL. However, this service is awaiting a POST from the service consumer to indicate
processing is complete.
5. Application of Arrowhead architecture 149

Figure 3: Case Study: A subset of the Festo MPS systems.

{
” bt ” : 0 ,
”e ” : [
{”n ” : ” b a r c o d e ” , ” v ” : 9 } ,
{”n ” : ” a v a i l a b l e ” , ” v ” : 0 } ,
{”n ” : ” l o a d e d ” , ” v ” : 0 } ,
{”n ” : ” m cnt ” , ” v ” : 0}
]
};

Figure 4: palletAvailable response payload

5.2 Service integration

Integrating SOA-based services with IEC 61499 has been achieved through localized
integration-oriented services. There are two services involved in integration, firstly the
orchestration service is used to identify a desired service provider among those available.
Secondly, the translation service is used to bridging protocol differences. The service
providers running on constrained networks make use of the CoAP protocol, as opposed
150 Paper D

to the IEC 61499 preferring HTTP. The translator is transparently injected so that inte-
grating the two systems is transparent to protocols. By using Arrowhead Orchestration
and Translation systems, the lead time for change is reduced and the integration process
is simplified. Association between systems is made through evaluation of semantic meta-
data describing the services provided or consumed by the systems. Figure 5 shows the
result of orchestration rule look-up, followed by service discovery to satisfy the rules and
then compiled as a response.

{
” t a r g e t ” : ” s t a t i o n −01” ,
” services ”: [
{
”name ” : ” palletAvailable ” ,
” address ”: ”[ fdfd : : f f ]:5683/ palletAvailable ”
},
{
”name ” : ” processingComplete ” ,
” address ”: ”[ fdfd : : f f ]:5683/ processingComplete ”
}
]
}

Figure 5: Orchestration Engine response

6 Application integration in IEC 61499

The overall application for automation of the entire Festo MPS system is presented in
Figure 6. Here the transportation subsystem is represented by the TransportArrowhead
function block, connected with the (yellow) communication lines to the function blocks,
representing other stations.
Figure 7 shows a single conveyor station (load/unload point) with adapter FBs com-
bined into the transport system. Each station will have identical group of function blocks.
Each loading/unloading point of the conveyor is represented by ConveyorStation func-
tion block, which gets service providers from orchestration during initialization stage,
and updates pallet(tray) data every time when the UPD event is triggered. Inside the
ConveyorStation FB there is a network of interfaces to three basic arrowhead services:
orchestration, palletAvailable and processingComplete (Figure 8). The interfaces are or-
ganized and use C# code to call services via http requests. When the INIT event is
triggered, the first FB - OrchestrationInfo requests orchestration engine with Orchestra-
tionAddress and station name and initializes PalletAvailable and ProcessingComplete
FBs with provider addresses received from orchestration engine.
From the FB application point of view, orchestration engine is a web service which can
6. Application integration in IEC 61499 151

Figure 6: Function block control application for the entire system.

Figure 7: IEC 61499 implementation of the transport system with arrowhead services.

be called by HTTP GET request to address http://[ipv4]/orchestrationengine/[station-

name] and responds with JSON object with a list of available services. Figure 5) shows
an example response.
PalletAvailable FB is an interface to palletAvailable arrowhead service, it requests
the service every time the UPD event is triggered and updates output data with received
information. Below the description of basic tray parameters:

Barcode is the tray number read from arrived tray (the parameter is valid only when
the tray is available)

Available is true when the tray is on the corresponding loading/unloading point

152 Paper D

TrayLoaded is the data from work piece sensor on loading/unloading point; true when
there is a work piece at corresponding point on the tray

m cnt is sequential message counter used for internal service purposes.

Figure 8: Orchestration of Arrowhead services encapsulated to IEC 61499 function blocks.

7 Conclusion
In this paper, work progress towards integration of IEC 61499 standard and Arrowhead
IoT services for flexible manufacturing applications is presented. Among usual benefits,
the approach described in this paper allows the future usage of formal verification [17] for
service oriented control applications. The use of Industrial Internet of Things technologies
enables multi-year lifetime when sensing digital inputs, and events normally has a delay
of tens of ms from when a digital signal changed state to when an event was received by
the gateway. Future work may include an integrated tool development for configuration
of distributed automation systems combining Arrowhead and IEC 61499.

Acknowledgment
This work was supported, in part, by the Arrowhead project funded by Artemis PPP
and Vinnova.
References 153

References
[1] “IEC 61499-1: Function blocks-part 1 architecture,” International Standard, First
Edition, Geneva, vol. 1, 2005.
[2] F. Jammes and H. Smit, “Service-oriented paradigms in industrial automation,”
Industrial informatics, IEEE transactions on, vol. 1, no. 1, pp. 62–70, 2005.
[3] A. Cannata, M. Gerosa, and M. Taisch, “Socrades: A framework for developing
intelligent systems in manufacturing,” in Industrial Engineering and Engineering
Management, 2008. IEEM 2008. IEEE International Conference on. IEEE, 2008,
pp. 1904–1908.
[4] J. Yan and V. Vyatkin, “Distributed execution and cyber-physical design of baggage
handling automation with iec 61499,” in Industrial Informatics (INDIN), 2011 9th
IEEE International Conference on. IEEE, 2011, pp. 573–578.
[5] ——, “Distributed software architecture enabling peer-to-peer communicating con-
trollers,” Industrial Informatics, IEEE Transactions on, vol. 9, no. 4, pp. 2200–2209,
2013.
[6] W. Dai, V. Vyatkin, J. H. Christensen, and V. N. Dubinin, “Bridging service-
oriented architecture and IEC 61499 for flexibility and interoperability,” Industrial
Informatics, IEEE Transactions on, vol. 11, no. 3, pp. 771–781, 2015.
[7] W. Dai, J. Peltola, V. Vyatkin, and C. Pang, “Service-oriented distributed control
software design for process automation systems,” in Systems, Man and Cybernetics
(SMC), 2014 IEEE International Conference on. IEEE, 2014, pp. 3637–3642.
[8] W. Dai, J. H. Christensen, V. Vyatkin, and V. Dubinin, “Function block imple-
mentation of service oriented architecture: Case study,” in Industrial Informatics
(INDIN), 2014 12th IEEE International Conference on. IEEE, 2014, pp. 112–117.
[9] W. Dai, V. Vyatkin, and J. H. Christensen, “The application of service-oriented
architectures in distributed automation systems,” in Robotics and Automation
(ICRA), 2014 IEEE International Conference on. IEEE, 2014, pp. 252–257.
[10] E. Demin, V. Dubinin, S. Patil, and V. Vyatkin, “Automation services orchestration
with function blocks: Web-service implementation and performance evaluation,” in
Workshop on Service Orientation in Holonic and Multi-Agent Manufacturing (SO-
HOMA’15), 2015.
[11] V. Vyatkin, “IEC 61499 as enabler of distributed and intelligent automation: State-
of-the-art review,” Industrial Informatics, IEEE Transactions on, vol. 7, no. 4, pp.
768–781, 2011.
[12] “Arrowhead framework wiki,” May. 2016. [Online]. Available: https://forge.soa4d.
org/plugins/mediawiki/wiki/arrowhead-f/index.php/MainPage
154

[13] P. Punal, J. Eliasson, and J. Delsing, An Authentication and Access Control Frame-
work for CoAP-based Internet of Things. IEEE, 2015, pp. 5293–5299.

[14] H. Derhamy, J. Eliasson, and J. Delsing, “Iot interoperability - on-demand and

low latency transparent multi-protocol translator,” IEEE Transactions on Services
Computing, 2016, submitted.

[15] “Service-oriented modeling and architecture,” May. 2016. [Online]. Available:

https://www.ibm.com/developerworks/library/ws-soa-design1/

[16] Z. Shelby, K. Hartke, and C. Bormann, “The Constrained Application Protocol

(CoAP),” RFC 7252 (Proposed Standard), Internet Engineering Task Force, Jun.
2014. [Online]. Available: http://www.ietf.org/rfc/rfc7252.txt

[17] S. Patil, D. Drozdov, V. Dubinin, and V. Vyatkin, “Cloud-based framework for

practical model-checking of industrial automation applications,” in Technological
Innovation for Cloud-Based Engineering Systems. Springer, 2015, pp. 73–81.
 Paper E
Service Oriented Architecture
Enabling The 4th Generation Of
District Heating

Authors:
Jan van Deventer, Hasan Derhamy, Khalid Atta and Jerker Delsing

Reformatted version of paper originally published in:

Conference paper, The 15th International Symposium on District Heating and Cooling
September 4-7, 2016, Seoul, Republic of Korea (South Korea).

© 2016, The 15th International Symposium on District Heating and Cooling, Reprinted
with permission.

155
156
 Service Oriented Architecture Enabling The 4th
Generation Of District Heating

Jan van Deventer, Hasan Derhamy, Khalid Atta and Jerker Delsing

Abstract

The 4th Generation of District Heating (4GDH) is a complex agglomeration of heat

providers, distributors, and consumers that must be automatically, continuously man-
aged and coordinated. It is a complex system of systems; a definition which we align to
Maier’s architecting principles for systems-of-systems as collaborative systems. Wrapped
in the idea of system of systems is the reality that the 4GDH systems’ descriptions and
specifications are not currently all known. Nonetheless, the transition into the 4GDH
is actual. We propose the use of two frameworks to secure a smooth metamorphosis
and assure systems’ operation, maintenance, and evolution. The two frameworks are the
Arrowhead Framework and the OPTi Framework. The first one enables system integra-
tion through Service Oriented Architecture (SOA) and the second one offers the overall
system optimization with respect to all stakeholders. This paper uses the Model Based
Systems Engineering (MBSE) tool SysML to model a district heating complex’s struc-
tures and behaviors from the concept level down to the sensors and actuators within a
district heating substation where we apply the SOA technology based on the open Ar-
rowhead Framework. We focus on the Arrowhead Framework’s core services, i.e. Service
Registry, Authorization and Orchestration to clearly describe the interactions between
the different service providers and consumers. Going back up from the sensors to the
systems, it is clear that SOA is the architecture that will empower the 4GDH.

1 Introduction
District Heating is a common sense concept that becomes progressively more evident as
one reflects on it. It is a centralize production of heat energy that is stored in a fluid and
distributed through a network to different buildings within a district. It is a resourceful
idea because the centralized production is more efficient than several distributed small
ones as long as the distribution losses are small [1]. The concept becomes even more
indisputable when the heat energy is a byproduct of an industrial process. For example,
a paper mill, which produces paper, can sell off its incidental heat. Another great example
is the Combined Heat and Power (CHP) that produces electricity from burning refuse
and simultaneously heats buildings and associated domestic hot water. This is a win-win
solution for all, including the environment. And yet, it can further be improved.
In its Energy Roadmap 2050, the European Union has set itself the impressive long-
term goal of reducing greenhouse gas emissions by 80-95% as compared to 1990 levels by

157
158 Paper E

2050. In its commitment to reach that goal, it has considered different scenarios and their
impacts. Connollly et al. assert that increased use of district heating in Europe supports
the goals of the roadmap at a lower cost [2]. This continued use of district heating is
accompanied with its normal evolution, which brings it to its fourth generation.
Lund et al. present an excellent review of the four generations of district heating
and of the European Union’s energy roadmap leading to the fourth generation [3]. They
describe the trends over the generations: increase in energy efficiency, decrease in tem-
perature of the supplied transport medium and an increasing complexity in stakeholders.
The first generations period was from about 1880 to 1930 with transport medium temper-
atures below 200°C. The second generation from 1930 to 1980 with temperatures greater
than 100°C and third from 1980 to 2020 with temperatures less than 100°C. The fourth
generation should span from 2020 to 2050 with temperatures less than 70°C. The com-
plexity increase includes an increase of different producers, which at some time might be
heat consumers, as well as daily variations of heat demand.
To address this complexity increase and aim for its harmonious operation, it becomes
essential to understand the district heating system. The 4GDH is made up of several
heat production systems, distribution systems, and consumer systems. Each of them with
their own set of sub-systems. The natural temptation is to designate the district heating
concept as a system of systems, but the Maier argues that it might be a misclassification
[4]. According to Maier’s tenets, an equivalent term to system of systems would be
“collaborative systems“ where the sub-systems fulfill valid purposes in their own rights
and continue to operate to fulfill those purposes if disassembled from the overall system.
Additionally, the sub-systems must be managed (at least in part) for their own purposes
rather than the purposes of the whole. Aligning to these views, serves not only the sub-
systems, but also 4GDH and forms the point of the herewith article. Especially when the
implementation specifications of the 4GDH do not yet exist. This leads Maier to state
that systems-of-systems are largely defined by their interface standards rather than their
structures. He points to the Internet as a good example of collaborative systems. It is
that same technology that is used here to define the interface standard of the proposed
solution. This enables new emerging behaviors to surface up as the 4GDH becomes
mature because the services within the architecture are loosely coupled and therefore
late binding.
This article presents a systems architecture for the 4GDH knowing that the systems
specifications will not all be complete for a long time. This plan of action relies on
two frameworks and their related interfaces: the Arrowhead Framework and the OPTi
Framework. The OPTi Framework offers a simulation platform for different components
of the DH systems that can includes the components of the 4GDH, which enables the
optimization of the 4GDH’s dynamics. The Arrowhead Framework enables system inte-
gration and collaboration through Service Oriented Architecture (SOA). We use Model
Based System Engineering (MBSE) to convey how this is implemented in a structural
and behavioral sense.
The structure of the article presents the two frameworks before turning to models
of district heating enhanced with SOA. The models are based on uses the MBSE tool
2. The Frameworks 159

SysML. The models describe district heating’s structures at the concept level down to
the sensors and actuators within a district heating substation where we apply the SOA
technology based on the open source Arrowhead Framework. The models additionally
describe the behavior of the system with focus on service discovery, authorization, and
orchestration at the lowest level to clearly demonstrate the mechanisms involved prior
to scaling back up to the whole system of systems in which the OPTi Framework is an
integral part.

2 The Frameworks
If systems are expected to be independent, and yet be collaborative to form a system
of systems, they must somehow find a benefit to co-operate. The OPTi Framework
addresses the yearning for collaboration, and the Arrowhead Framework addresses the
service oriented architecture to make the collaboration possible. Coincidentally, the
OPTi Framework can provide services within the Arrowhead Framework. This is possible
because they both have well defined interfaces.

2.1 The OPTi Framework

The OPTi Framework is the product of the European Union Horizon 2020 project OPTi,
which aims to deliver the tools to optimize district heating and cooling with full con-
sideration of the end customers [5]. In other words, the framework is a suite of tools
that offers solutions in which all stakeholders benefit. This key concept is what drives
the systems to be inclined to collaborate together. The benefits include saving natural
resources, assuring comfort, increasing profits and cost reductions.
One of these tools is the scalable simulation tool OPTi Sim, c.f. figure 1. It consid-
ers production, distribution, consumption, end consumers, and economics. It is also a
system of systems as each sub-simulation is independent and communicates through a
co-simulation router. The communication interface follows a clear standard: the open
interface standard FMI (Functional Mock-up Interface) [6]. It is an independent stan-
dard that supports both model exchange and co-simulation of dynamic models using a
combination of xml-files and compiled C-code. This permits each simulation tool to be
developed and refined independently.
One example being addressed is the daily peak loads as demand for domestic hot water
increases in the morning before work and at dinner time. The planning of production
and distribution has to be done; and a good optimization should use also the buildings
thermal capacitance to save resources while striving for end customer comfort. With an
increase of heat suppliers, the complexity of the task becomes even more challenging.
Collaborative simulation for planning will be evermore essential for the 4GDH.
Having clarified why the systems would want to collaborate, it becomes interesting
to understand how they could interact with each other.
160 Paper E

Figure 1: A block diagram depicting OPTi Sim with its different simulation engines.

2.2 The Arrowhead Framework

The open source Arrowhead Framework is the product of another European Union project
[7]. The project’s vision has been to enable collaborative automation by networked em-
bedded devices. Its grand challenges were to enable the interoperability and interchange-
ability of services provided by almost any device. This has been done by offering services
established on the Internet Protocol Suite, which is a proven technology. It reflects the
paradigm of a person, i.e., service consumer, using a web browser to address a service
provider, e.g., a search engine or bank, to obtain information. The different software
modules at the service providers and consumers can be updated at anytime without af-
fecting the others as they are loosely coupled. What is clearly defined are the interface
protocols themselves. Adhering to them simplifies development and insures quality.
The Arrowhead Framework proposes an assortment of services, in the form of software
modules, of which three are core services. The core services are the Service Registry,
Authorization and Orchestration. We describe how they interact within a district heating
substation in the next section. Of the many other service modules, worth mentioning are
the Historian, the Gate Keeper, and the Translator, although we do not involve them
here [8][9][10]. The Historian is a data logger, while the Gate Keeper is a secure interface
to the outside Internet world. The Translator is a service provider that intervenes when
3. Structure and Behavioral Models 161

different component suppliers have chosen different Internet protocols, which could hinder
collaboration due to dialects [11].
To elucidate how these core services are used in district heating, we employ systems
engineering models

3 Structure and Behavioral Models

A short definition of Systems Engineering is: “Systems engineering is a methodical,
disciplined approach for the design, realization, technical management, operations, and
retirement of a system“ [12]. As systems become more complex, consistent approaches
and tools are needed to manage the complexity as well as to communicate with all stake-
holders. One such approach is to use models, which is referred to as Model Based Systems
Engineering. Associated to this, one finds the modeling tool SySML, System Modeling
Language. It is an offshoot of the Unified Modeling Language (UML) that was devel-
oped during the early 1990’s and later adopted as a standard by the Object Management
Group [13]. SysML uses different diagrams to represent different aspects of a system.
The classification of its taxonomy of diagrams forms three groups: structure, behavior,
and requirement diagrams. We shall here use structural and behavioral diagrams.

3.1 DH structural models

We begin by looking of the structure of a third generation district heating in Sweden
as depicted in figure 4, which is a SysML bock definition diagram (bdd). We find the
three expected sub-blocks of production, distribution, and consumption. One block that
is usually not mentioned is the system management part, although it is just as essential.
When models are clearly drawn, they propel the discussion of what was not initially
considered. The coordination of district heating, including the billing of end customers,
does exist but is often omitted. Through modeling, it is also those omissions that we are
trying to capture without overwhelming the stakeholders. It is system management that
we can find OPTi Sim.
One advantage of MBSE is to go deeper into a block to further one’s understanding
when this is of interest. In the present case, we can dive into the consumption block and
further into the single family house, whose bdd is shown in figure 3. Of interest here is
the standard district heating substation, which takes a majority of the diagram.
To improve communication between the different modules within the DH substation,
we added tiny web servers on several of them [14][15]. Figure 4 shows a graphical model
of a substation where the wireless servers are depicted with an “S“. The energy meter,
the valve on the primary circuit to the space heating heat exchanger, the temperature
sensor of the supply line to the radiators, the space heating circulation pump, and the
outdoor temperature sensor each have their own web servers. Similar to a search engine
or a bank web server, these little web servers offer services. For example, the outdoor web
server measures temperature and offers a temperature service. Those tiny web servers
are dubbed Mulles [16]. Figure 4 also has an access point or gateway, which is composed
162 Paper E

Figure 2: A block definition diagram (bdd) of district heating.

Figure 3: A block definition diagram of a single family house.

of a BeagleBone computer and a Mulle, as illustrated by the bdd in figure 5. The

figure additionally shows the software modules of interest: the three core services, the
gatekeeper service, and a district heating application. The district heating application
could be the one that controls the valve on the primary circuit to the space heating heat
exchangers based on outdoor temperature and supply temperature to the radiators. It
could also be another application such as a building thermal analysis. One advantage with
SOA is that these modules can be developed after infrastructure deployment; this being
an essential key to 4GDH. This exemplifies the ideas of loosely coupled modules and late
binding. Being software modules, the services can exist on any servers in contingency to
a gateway malfunction thereby avoiding a single point of failure, which typifies to system
fault tolerance concepts. Having an idea about the structure of the system of systems
down to a single wireless sensor server or service provider, a demonstration of how the
system behaves is essential to begin to accept the SOA idea.
Among SysML’s behavior diagrams are the sequence diagrams with their “swim lanes“
as depicted in figure 6. We begin with a simple service discovery sequence diagram to
demonstrate how SOA functions. We then present a sequence diagram with all the three
3. Structure and Behavioral Models 163

core services in a local cloud. Figure 6 shows the outdoor temperature server presenting
its temperature service to the service registry. It does that by POSTing, using the World
Wide Web’s representational state transfer (REST) style, the following message:

{
”name ” : ” temperature −em219 ” ,
” type ” : ” temp−j s o n −coap . udp ” ,
” h o s t ” : ” [ f d f d : : d f 5 : 8 c6a : 5 ca2 : 4 4 a6 ] ” ,
” port ” : 5683 ,
” properties ”: {
” property ”: [
{
”name ” : ” v e r s i o n ” ,
” value ”: ”1.0”
},
{
”name ” : ” path ” ,
” v a l u e ” : ”/ t e m p e r a t u r e ”
}
]
}
}

With this posting, the service provider system tells its name, its chosen communica-
tion semantic, and the path to the service it offers. Additional services would be listed
in the property array. By default or from an installation update, the service provider
knows to register its services at the gateway host (host address and port).

Figure 4: A district heating substation with a wireless sensor network, where each ”S” is a tiny
wireless server.
164 Paper E

Returning to figure 6, the service registry then builds a database of available services
from all possible providers. Arrowhead’s service registry is based on DNS-SD, which is
an extension to DNS. DNS, domain name system, part of the Internet Protocol Suite’s
application layer, provides the address at which a service is hosted. The service discovery
extension, DNS-SD, provides the ability to discover services, which an application might
want to use.

Figure 5: The gateway composed of a BeagleBone, a Mulle, along with the core services and an
application.

A service consumer, e.g., the DH application, then can ask the service registry for
the desired service and receive back the address of the service provider with the path
to the service of interest. The service consumer directly contacts the service provider
whenever it wants the needed information, also referred to as “pulling“. The answer
from the service provider is:
{
e : [{
”n ” : ” urn : dev : mac : 0 0 2 4 b e f f f e 8 0 4 f f 1 ” ,
” t ”:1425256855 ,
”u ” : ” Cel ” ,
”v ” : 2 3 . 5
}]
}
with its name, the timestamp of measurement, along with the unit and measured
value. Alternatively, after being requested by a consumer, the service provider could
“push“ an update at specific time interval or upon some agreed event. The Arrowhead
Framework does not restrict pull and push behaviors.
With service discovery recounted, several natural questions surface up and include
the following. What is the latency in the control loop? How is security assured both for
privacy protection and against tampering? How are service selections sorted out in the
case of multiple providers and how is provider drop out handled? How is the connection
to the Internet handled?
The Arrowhead project has addressed such questions by first defining local clouds.
The service consumers do not interact with the service providers through a cloud far away,
3. Structure and Behavioral Models 165

Figure 6: Sequence diagram of a service registry and discovery.

rather through a local cloud, e.g., the gateway. This assures the low latency necessary in
control application and offers some security from the outside world. Additional security
is handled by the authorization service, which could use certificates or tickets. Figure
7 shows the sequence diagram using the Arrowhead Framework core services. If the
service provider is power constrained, it would prefer to receive an authorization ticket
with a limited lifetime to communicate with a given consumer to avoid wasting power
by obtaining an authorization at each request. The logic is that the consumer obtains
a ticket from the authorization service, which it passes on to the service provider. The
service provider then checks if it is valid with the authorization service and if so, then
will communicate with the consumer for the lifetime of the ticket.
The third core service proposed by the Arrowhead project is the Orchestration service.
Its purpose is to give some “intelligence“ in the selection of service provider or recommend
an alternative provider when a specific one has dropped out. If the application itself is
enhanced with this ability, it does not need the orchestration, but this would mean that
the application is quite specific and not general, leaving the choice to a system architect.
166 Paper E

Figure 7: Sequence diagram with all three Arrowhead Framework core services.

For example, if the outdoor sensor is offline, the orchestration service could infer the
outside temperature from the heat meter’s primary supply temperature or go beyond the
local cloud.
The Arrowhead Framework has a collection of services such as the Gatekeeper to
interact safely with the Internet or the Historian to log data. With service providers like
the Gatekeeper, the Arrowhead Framework permits collaboration between local clouds to
build a system of systems where the OPTi Framework entices collaboration as optimized
performance benefit all stakeholders. The Gateway can then join other clouds, e.g.,
a district heating cloud, to provide and consume services in the same manner as is
done in the local cloud. This notion ensures scalability of the systems; a concept the
4. Conclusion 167

Internet has already proven. It also addresses possible failure points, e.g., if some parts
of the communication network would be unreachable, the OPTi framework could, as a
service, provide estimated values of the current state within the DH network. SOA, being
based on the same architecture as the World Wide Web, empowers the transition in and
evolution of the 4GDH with security and ease of maintenance.

4 Conclusion
Using MBSE and SysML, we have modeled a district heating’s structure and zoomed
down into a substation. The substation incorporated very small wireless sensor and ac-
tuator nodes that were web servers offering services. Using SysML’s sequence diagrams,
we have illustrated the message exchange between service providers and consumers in
partnership with the Arrowhead Framework core services. The advantages with the Ar-
rowhead Framework include properties such as loosely coupled modules (i.e. two SOA
systems do not need to know about each other at design time to allow a runtime data
exchange), late binding (i.e. the exchange of data between two systems is established at
runtime), autonomy, pull and push behavior (i.e. data can be requested or sent with-
out request upon predetermined conditions), several standardized SOA protocols, data
structures, information semantics, and data encryption. As we zoom out from the district
heating substation to the distribution network and production plants, we use the OPTi
Framework to address scalable data management, macro and micro simulations, which
can be applied to manage and optimize the 4GDH while we point to new collaborative
and emerging behaviors of a system of systems. We contend that the use of these two
frameworks support the transition into, as well as the growth of, the 4GDH.

5 Acknowledgment
The OPTi project has received funding from the Horizon 2020 Research Programme of
the European Commission under the grant number 649796. The Arrowhead project is a
part of Artemis Innovation Pilot Project. ARTEMIS stands for “Advanced Research and
Technology for Embedded Intelligence and Systems“- and is the European Technology
Platform for Embedded Computing Systems.

6 References
References
[1] S. Frederiksen and S. Werner, District Heating and Cooling. Studentlitteratur AB,
2013. [Online]. Available: https://books.google.se/books?id=vH5zngEACAAJ

[2] D. Connolly, H. Lund, B. Mathiesen, S. Werner, B. Möller, U. Persson,

T. Boermans, D. Trier, P. Ostergaard, and S. Nielsen, “Heat roadmap europe:
168 Paper E

Combining district heating with heat savings to decarbonise the {EU} energy
system,” Energy Policy, vol. 65, pp. 475 – 489, 2014. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0301421513010574

[3] H. Lund, S. Werner, R. Wiltshire, S. Svendsen, J. E. Thorsen, F. Hvelplund,

and B. V. Mathiesen, “4th generation district heating (4gdh): Integrating smart
thermal grids into future sustainable energy systems,” Energy, vol. 68, pp. 1
– 11, 2014. [Online]. Available: http://www.sciencedirect.com/science/article/pii/
S0360544214002369

[4] M. W. Maier, “Architecting principles for systems-of-systems,” Systems

Engineering, vol. 1, no. 4, pp. 267–284, 1998. [Online]. Available: http:
//dx.doi.org/10.1002/(SICI)1520-6858(1998)1:4h267::AID-SYS3i3.0.CO;2-D

[5] A. Gylling. Optimization of district heating & cooling systems. [Online]. Available:
http://www.opti2020.eu

[6] “Functional mock-up interface for model exchange and co-simulation

v2.0,” Modelica Association, Linkoping, Tech. Rep. [Online].
Available: https://svn.modelica.org/fmi/branches/public/specifications/v2.0/
FMIforModelExchangeandCoSimulationv2.0.pdf

[7] “Arrowhead - Enable collaborative automation by networked embedded devices.”

Feb. 2013. [Online]. Available: http://www.arrowhead.eu/

[8] P. Punal, J. Eliasson, and J. Delsing, An Authentication and Access Control Frame-
work for CoAP-based Internet of Things. IEEE, 2015, pp. 5293–5299.

[9] P. Varga and C. Hegedus, “Service interaction through gateways for inter-cloud col-
laboration within the arrowhead framework,” 5th IEEE WirelessVitae, Hyderabad,
India, 2015.

[10] P. Varga, F. Blomstedt, L. L. Ferreira, J. Eliasson, M. Johansson, and J. Delsing,

“Making system of systems interoperable - the core components of the arrowhead
technology framework (accepted for publication),” IEEE Internet of Things Journal,
August 2015.

[11] H. Derhamy, J. Eliasson, J. Delsing, P. P. Pereira, and P. Varga, “Translation error

handling for multi-protocol soa systems,” in Emerging Technologies Factory Au-
tomation (ETFA), 2015 IEEE 20th Conference on, Sept 2015, pp. 1–8.

[12] “Systems Engineering Handbook: NASA/SP-2007-6105 Rev1,” National Aero-

nautics and Space Administration, Tech. Rep., Dec. 2007. [Online]. Available:
http://hdl.handle.net/2060/20080008301

[13] (2015, March) Object management group. [Online]. Available: http://www.omg.

org/gettingstarted/gettingstartedindex.htm
 169

[14] J. van Deventer, J. Gustafsson, J. Eliasson, J. Delsing, and H. Makitaavola, “Inde-

pendence and interdependence of systems in district heating,” in Systems Confer-
ence, 2010 4th Annual IEEE, April 2010, pp. 267–271.

[15] J. Gustafsson, “Wireless sensor network architectures as a foundation for efficient

district heating,” Ph.D. dissertation, 2011.

[16] “Eistec AB,” Feb. 2013. [Online]. Available: http://www.eistec.se/

170
 Paper F
Protocol Interoperability of OPC
UA in Service Oriented
Architectures

Authors:
Hasan Derhamy, Jesper Rönnholm, Jerker Delsing, Jens Eliasson, and Jan van Deventer

Reformatted version of paper originally published in:

Conference paper, IEEE Industrial Informatics, Emden, Germany, 2017

© 2017, IEEE, Reprinted with permission.

171
172
 Protocol Interoperability of OPC UA in Service
Oriented Architectures

Hasan Derhamy, Jesper Rönnholm, Jerker Delsing, Jens Eliasson, and Jan van Deventer

Abstract

Industrial Internet of Things covers all aspects of networked intelligent manufacturing

systems. This means covering a wide array of application domains and user requirements.
In such scenarios it is not feasible to define a single protocol for all situations. Hence, a
multi-protocol approach is required. OPC UA has strong backing from Industry 4.0 as
the protocol for the Industrial Internet of Things. Interoperability of OPC UA has been
investigated in the context of migration from legacy and with protocols such as DPWS.
Additionally HTTP and CoAP have been investigated as possible transport mediums.
However, OPC UA interoperability has not been investigated within a multi-protocol
settings and no generic protocol translation exists. This paper proposes an OPC UA
translator following the service translator model proposed in the Arrowhead project.
Utilizing a mapping to intermediate format, it can be used along side CoAP, HTTP and
MQTT protocols.

1 Introduction
Internet of Things (IoT) communication protocols cover highly varied application do-
mains. Trying to select a single communications protocol reduces flexibility and removes
possibility for leveraging domain specific benefits of a certain protocol. Therefore, to be
able to operate in a multi-protocol ecosystem a form of translation is required. Rather
than a dedicated middleware or network proxy, Derhamy et al. propose an ”on-demand”
translation service. Following Service Oriented Architecture (SOA), this translation ser-
vice is injected between a service exchange when a mismatch is detected by an Orches-
trating entity.
Object linking and embedding for Process Control - Unified Architecture (OPC UA)
is an industrial communication framework that is highly promoted for integrating dis-
tributed systems. While it has many promising features which we describe in section 2,
there are still situations where other communication protocols have benefits. Primarily
in constrained environments protocols such as Constrained Application Protocol (CoAP)
is well suited for battery powered and processor constrained applications. But also, as
communications protocols change and improve, solutions must adapt to the new tech-
nology. Hence, a successful interoperability solution must work with existing and future
protocols.

173
174 Paper F

There is existing work towards migrating applications from existing protocols and
frameworks to OPC UA and on integrating OPC UA with other protocols. Some of
these works are presented in the next section.

1.1 Related work

There exist a number of solutions (proposed or implemented) with the aim of increasing
factory floor interoperability. A common theme is vertical integration, where solutions
for achieving interoperability have involved implementing OPC UA and Device Profile for
Web Services (DPWS) to enable high-level access of field-level data [1], [2]. Candido et al.
in [1] analyze the two protocols, OPC UA and DPWS, for effectiveness in access device
level (field level) data. They concluded that neither protocol could completely satisfy
all requirements for Industrial IoT. However the combination offered complementary
benefits.
In [2], Sauter et al. describe a protocol gateway used to connect high level IP based
networks to field-bus device level data. While maintaining ability to operate with any
field bus, they restrict the IP interfaces to a smaller subset available. The gateway is
responsible for more than protocol translation as it does logging and historical storage
also. They do not however describe what is required to setup such a gateway. Manual
configuration and design time consideration of the gateway is described as a necessity for
the use of a gateway architecture.
Another solution proposed for enabling interoperability with OPC UA is described in
[3]. It again uses a gateway approach with a complex event processor bridging DPWS and
OPC UA networks. The gateway uses a middleware to subscribe to events originating
from an OPC UA server and from DPWS enabled devices. The events are then translated
so that a complex event processor can handle events from both protocols. Here, the
middleware must be pre-configured to subscribe to the desired events and the events
must be translated for processing.
Arguments for direct modification of OPC UA to better adapt it to the industrial IoT
have also been raised by Grüner et al. in [4] and [5]. Specifically, this has involved a series
of adaptations to the OPC UA binary protocol, enabling stateless service requests and
reducing communication overhead - thereby making it more RESTful, and more friendly
to resource-constrained devices. Similarly CoAP has been proposed as a transport option
for the OPC UA stack [6]. Here, CoAP is the transport messaging supporting the OPC
UA services [7]. Therefore, CoAP applications using that solution must implement OPC
UA services.
Releases of the initial OPC UA specification included an SOAP/XML/HTTP API.
Since release 1.03 this has been deprecated. This was due to a lack of industrial adoption
of WS Secure Conversation.
There are already a few proprietary solutions that strive for interoperability involving
OPC UA. An example of this is Kepware’s KEPServerEX [8], which expose its data as
legacy OPC DA data (tags) accessible through OPC UA services. Interoperability is
provided by a gateway [9], which also allows third-party access through REST- and
2. OPC Unified Architecture 175

MQTT-client APIs.
HyperUA [10] is another proprietary solution for accessing OPC-UA servers from web
clients. An HTTP server provides a gateway for web clients to address HyperUA nodes,
references, monitored items, servers and subscriptions. The gateway handles sessions to
the OPC UA servers and translates Hyper nodes to OPC-UA nodes. In order to interact
with an OPC UA server, clients must implement the HyperUA API.
The primary problem with interoperability solutions such as gateways and protocol
adapters is that they require custom configuration per site, and do not scale well with
each additional protocol added to the mix. As the complexity of a centralized gateway
grows, the solution becomes more brittle and resists change.

1.2 Contribution
OPC UA applications have not been used alongside standard web applications without
knowledge of OPC UA. As stated in the related works previous efforts with HTTP and
CoAP have been to use them as a transport for regular OPC UA communications.
This paper proposes an OPC UA translator that works with standard IoT protocols
such as HTTP, CoAP and MQTT. Enabling access to OPC UA nodes from non-OPC
UA based IoT applications. The paper presents:

1. Mapping between a subset of the OPC UA services and an intermediate format

which can then be translated to any other IoT protocol.

2. Mapping between OPC UA address space and generic IoT protocols.

3. Implementation demonstrated upon an Industrial use case for condition monitoring.

4. Challenges for complete OPC UA service translation.

The paper, in Section 2 introduces the OPC UA protocol and high level aspects
relevant for translation. Following this, Sections 3 and 5 present the proposed solution
and the use case. Finally, a discussion and conclusion are in Sections 5 and 6.

2 OPC Unified Architecture

OPC UA is a protocol for communication in industrial automation, developed by the
OPC Foundation and standardized in IEC-62541 [11]. It defines services through which
OPC UA clients interact with information models maintained by the server. The base
component of the information model is an object-oriented entity called node. A node is
identified through a nodeid which consists of a namespace index and a node name. Edges
between nodes are references which create semantic relationships. References enable
modeling of both hierarchical tree and horizontal mesh graphs. Every node conforms to
a nodeclass which specifies the attribute set of the given node. There are two groups of
nodeclasses: type classes and access classes. By navigating type information and node
176 Paper F

relationships, an OPC UA client can access node and reference data to gather its own
understanding of any OPC UA information model.
Interactions between an OPC UA client and server is standardized through a set
of services provided by the server. The services allow access to and management of
nodes: 1) management of node and information model; 2) read and write data to nodes,
both query and subscription based; and 3) establishment of communication channels to
perform further requests. A server can choose to support only a subset of the full service
set, this is determined by using a server profile. OPC UA can be used as a data oriented
historian. Clients store data within a defined information model. Upper layer functions
can then access the data stored in an asynchronous manner.
In the next section the interoperability solution is proposed and the fundamental
architecture described.

3 Proposed Translation Solution

The proposed interoperability solution consists of a multi-protocol translator, which is
injected into a service exchange in an on-demand basis. The primary aspects of the
translator have been described in previous work [12] and so only briefly mentioned in
this paper. The main contribution of this paper is proposing a method of mapping
OPC UA to an intermediate format, which can then be mapped to other standard IoT
protocols such as CoAP, HTTP and MQTT.
Firstly, interactions are defined by OPC UA services. Standard Create, Read, Update,
and Delete (CRUD) operations can be used for translating a subset of these services.
Next, the address space in OPC UA is defined by nodes. A URI can be used to uniquely
address a node. The mapping has been designed to try to minimize leakage of the OPC
UA protocol into the translated protocol.
The next sub-section will describe the base architecture on which the translator is
based.

3.1 Base architecture

The base architecture of the translator as presented by Derhamy et al. in [12], consists
of a translation hub that is composed of two protocol spokes. The protocol spokes are
based on freely available libraries and handle protocol specific behaviors. The protocol
spokes communicate by an intermediate format, which is protocol agnostic. The interme-
diate format is intended to preserve the properties of the incoming protocol and reduce
information loss when translated to the target, outward, protocol. Figure 1 shows this
in a block diagram. In the proposed solution, the OPC UA protocol spoke must handle
all OPC UA translation and preserve the properties as much as possible.
The details of the protocol spoke implementation can be found in Sub-section 3.5.
3. Proposed Translation Solution 177

Figure 1: Block diagram of the base translator architecture

3.2 Service translation

As introduced in Section 2, a subset of the OPC UA services are only used to establish the
communications channel, the session and secure channel. These services are completely
handled by the OPC UA protocol spoke. This means that they are not exposed to the
intermediate format.
The translator’s intermediate format uses the RESTful methods to communicate
CRUD, namely, GET, POST, PUT, and DELETE. These methods are mapped to the
OPC UA services as shown in Table 1. This is a mapping from intermediate format to
OPC UA services.

Table 1: OPC UA service mapping to intermediate format

Operation RESTful OPC UA Service
Create PUT AddNodes
Read GET Read, Browse
Update POST Write, AddReferences, DeleteReferences
Delete DELETE DeleteNodes

When a Create request is passed to the OPC UA spoke, the node id is stored in the
URI and the information for creation is stored in the payload. Only the AddNodes service
is mapped to the Create operation. This is where the OPC UA specific information leaks
into the non-OPC UA protocol.
When a Read request arrives at the OPC UA spoke, it could be either a read ser-
vice request or a browse service request. These services are differentiated based on the
presence of an attribute index in the URI path. If no attribute index is present, the
translator spoke performs a Browse on the node.
An Update corresponds to 3 services that modify an existing node: Write, AddRefer-
ences and DeleteReferences. References are not addressable and belong to a source node.
Hence, the translator takes the node id in the URI as the source node, and requires
further information in the payload. Both deleting and adding references are treated in
178 Paper F

the same manner, except that the payload information can differentiate the operation.
In order to perform a write operation on a node, an attribute index must be supplied
in the URI. The translator addresses the attribute of the node directly. The payload of
a write operation is treated as a serialized string, hence it can be JSON, XML, CBOR,
EXI, or etc. This is the most common approach for IoT protocols such as CoAP, HTTP,
MQTT and XMPP.
A Delete operation is translated as a DeleteNodes service request. Removing the node
corresponding to the node id stored in the URI. The request will also delete all references
of the target node.

3.3 Address space translation

OPC UA address space uses servers, name space indexes, node names and attribute
indexes to provide a fine level of granularity to read and write data. These can be broken
down into URI format for IoT protocols to construct. In Figure 2 the URL structure is
shown with the OPC UA server endpoint defined by the IP address, port and base path.
The name space index (ns-idx) and node name (name) make up the complete node-id
which uniquely identifies a node on an OPC UA server.

<IP >:<port >/<path >/<ns-idx >/<name >/<attr -idx >/<arr -idx >

Figure 2: The general URL structure to address an OPC UA node. Attribute-index can be
added for attribute-granularity, and array-index can be added for accessing single elements in
arrays stored under attributes.

To reduce the non-OPC UA path parametrization when translating to OPC UA, the
URI format uses string for the node name format. The standard node-set, in the OPC
Foundation name space, use numeric node names, so for this name space the node names
are interpreted as numeric. Hence, nodes with name space index 0 will have their name
interpreted as a numeric type.
A reference is identified in a triple: the source node, target node and reference type.
The proposed translator addresses references according to a source node described in the
URI. The target node and reference type is provided in the payload.

3.4 Request and Response formats

The translator aims at having least interference in the payload structure or meaning.
This is not always possible due to the node structure and service interfaces of OPC UA.
The request and response formats are required to follow certain parameters. A non-OPC
UA interface must be prepared to provide structured information for management of
nodes and references. However, there is no requirement on a write operation, aside from
being limited to a serialized string. While the examples are in JSON encoding, XML
3. Proposed Translation Solution 179

is just as applicable. JSON has certain advantages over XML, in this case the reduced
overhead in the encoding and readability.
When writing to a single node, update operation, the request payload format is shown
in Figure 3. The node can be accessed down to the variable attribute, which is addressed
in the URI. The value is the only parameter in the payload and can be passed straight
through to the OPC UA domain without modification.

POST /<namespaceindex >/<nodename >/<attributeindex >/

Payload: JSON Object or XML Document

Figure 3: Write request format for a single node attribute value

When performing a read, the same URI format is used. During a read operation, the
payload is not modified, it is serialized as a JSON object or XML document and given to
the intermediate format. In the case of browse operation the attribute index is omitted
from the URI. Payload was not modified because, ideally, the protocol translator should
not get involved in semantic translation. A semantic translator would be responsible and
could work in tandem with the protocol translator. However full semantic translation is
outside the scope of this work.
On the other hand node and reference creation requires additional information in the
payload. The structure of a payload is shown in Figure 4.

PUT /<namespaceindex >/<nodename >

{"node": {
"parent": "<parentnode -path >",
"reference": "<referencetypenode -path >",
"browsename": "<browsename -string >",
"nodeclass": "<nodeclass -index >",
"type": "<typenode -path >",
}}

Figure 4: JSON object format for adding a node.

In order to enable node and reference instantiation a formulation for node- and refer-
ence representation was provided for parametrization. Figure 5 shows the add reference
payload format, the source node is defined in the URI structure and the reference be-
longs to this node. The ”forward” parameter specify in what direction non-symmetrical
reference will point their semantics. This was one of the areas where OPC UA API must
have representation within the payload structure.
Figure 6 has the format for deleting a reference. It follows the same format as the
add reference. The difference is in object name, ”deleteReference”, and the lack of the
nodeclass parameter, which is only needed when adding references. The deleteReference
180 Paper F

POST /<namespaceindex >/<nodename >

{" addReference ": {
"type": "<referencetype -node -path >",
"target": "<targetnode -path >",
"nodeclass": "<target -nodeclass -index >",
"forward": "<boolean >"
}}

Figure 5: JSON object format for adding a reference.

object is how the translator differentiates between the service calls.

POST /<namespaceindex >/<nodename >

{" deleteReference ": {
"type": "<referencetypenode -path >",
"target": "<targetnode -path >",
"forward": "<boolean >"
}}

Figure 6: JSON object format for deleting a reference.

In the next section the proof of concept implementation is presented.

3.5 Proof of concept implementation

The proposed translator has been implemented using Java and applied to the use case
presented in Section 5. The protocol spoke uses the OPC Foundation Java client library
to perform client specific functions, including creating the session/secure channel and
executing the service request to the target OPC UA server. The OPC UA protocol
spoke is instantiated by the translation hub and internally routed to a corresponding
protocol spoke for the other protocol. The session and secure channel is setup on spoke
instantiation. The current prototype uses anonymous certificates to connect to the OPC
UA server.
Requests from non-OPC UA protocol spokes are processed into the intermediate
format and passed to the OPC UA protocol spoke. The protocol spoke parses the URI
path and payload body to extract OPC UA node address and any service parameters.
These parameters are used to select the OPC UA service to be used to communicate
with the server and to construct the OPC UA object payload. The OPC UA client then
performs a synchronous service invocation to the OPC-UA server. The response from the
OPC-UA server is presented to the protocol spoke as OPC-UA parameter objects. These
objects are re-serialized to a JSON string and passed as a response in the intermediate
format.
The proof of concept has been tested between a CoAP client and an OPC UA server.
4. Application Scenario 181

This use case is presented in next section.

4 Application Scenario
The proposed solution can be used as a middleware solution with permanent presence
in the communication path. Another approach is to provide the translator as a service
which dynamically can be invoked based on need. In the here described application sce-
nario, the translator is a service. The SOA-based Arrowhead Framework [13] enables
dynamic provisioning and composition at run time. Leveraging these features the pro-
posed translator can be invoked dynamically based on the current orchestration of service
consumption. In this way the Arrowhead Framework supports multi-protocol System of
Systems. This may become a key enabler to Industrial Internet of Things.
At the core of the Arrowhead framework are three mandatory core services, these are:

1. ServiceRegistry - provides a store for publishing service provider presence and per-
forming look-up.

2. Service Authorisation - provides authorisation for service consumption.

3. Service Orchestration - provides match making and advanced service look-up.

Arrowhead also defines support core services. These services are used only when needed.
The Multi-protocol translator is one of these support core services. The proposed solution
extends the existing translator with the OPC UA protocol.
In the context of the proposed solution the Orchestration is responsible for, firstly,
identifying a protocol miss-match between a potential service provider and consumer.
Secondly, it issues provisioning requests to the translator, to which the translator in-
stantiates two endpoint spokes which match the protocols of each service provider and
consumer respectively. The detail is described in Section 3.1
The Arrowhead framework documentation architecture defined by Blomstedt et al.
in [14] describes three levels: services, systems and systems-of-systems. Of interest to
translation; the service layer documentation captures 1) abstract service description,
2) interface design, and 3) protocol and semantic profiles. These artifacts are used to
identify the protocol miss-match between matching services.
A pilot scenario from the Arrowhead project has been selected to demonstrate the
proposed solution. The scenario involves monitoring the vibrations and heat produced
by ball bearings on a Volvo wheel loader. Volvo wheel loaders are expensive machinery
which must be maintained in order to extend lifetime. By monitoring the ball bearing it is
possible to measure wear and schedule preventative maintenance. In addition, monitoring
the rotations of the wheel, it is possible to ascertain the performance of the ball bearing
and find early end of life.
In order to extract information at such a granular level from the industrial machinery,
IoT technologies offer a promising choice. IoT based ball bearings is more than simply
connecting the Wheel loader to the Internet. Industrial environments require a solution
182 Paper F

which is robust to communication interruption, long service life and able to scale up to
many thousands of devices within a single solution. This means, that the solution needs
low-power operational modes and intelligent, automated, re-configuration.
Constrained Application Protocol (CoAP) is a preferred protocol for low-power en-
vironments, offering support for RESTful interface design [15]. In this use case a CoAP
based ball bearing sensor is running a client as a service consumer. It seeks-out a ser-
vice provider where it is able to push sensory data. Arrowhead orchestration is used to
find such a service provider which matches the needs of the ball bearing sensor. The
Orchestration has been loaded with a description of the requirement for the sensor. The
requirement is such:
1. If available, use the head-office historian,
2. else, use a local historian to temporarily store data.
The Systems of Systems (SoS) Description has a head office historian and a local historian.
If Internet connection is available, then the head office historian is used, otherwise the
local historian is used. When the Orchestration service is used, it will return either the
local or the remote historian. This decision is based on availability, the remote service
provider is dependent on an Internet connection. However, the head office historian must
integrate with many more systems than the simply the ball bearing sensor. It has been
implemented with OPC UA, perhaps according to company policy. Figure 7 shows the
SoS with an Internet connection and Figure 8 without an Internet connection.

Figure 7: System of System diagram when Internet connectivity with VPN is possible

In this case we have an interoperability issue between the ball bearing and the remote
historian. It would be possible to implement a local gateway which caches the data and
forwards to the remote historian. However, this gateway would require implementation,
testing and configuration effort. Furthermore, the local historian in this use case supports
CoAP. So, in the local connection there is no interoperability issues. The proposed
solution is used here as an ”on-demand” protocol bridge when communicating with the
head office, while saving resources when direct communication is possible between sensor
and local historian.
4. Application Scenario 183

Figure 8: System of System diagram when no Internet connectivity is possible

The application scenario is setup as shown in Figure 9. The sensor is implemented on

an embedded processor, the Mulle [16]. It connects to an Arrowhead local cloud running
on a BeagleBone Black (BBB). The OPC UA server is running an a general purpose PC
with an OpenVPN connection to the BBB.

Figure 9: Demo technology usage and setup

The interactions between sensor, Arrowhead systems is following pre-defined Arrow-

head patterns [13]. The OPC UA historian information model is shown in Figure 10.
The interface is to a single OPC UA node for each ball bearing. The node is of type
variable and stores a string representation of the JSON object sent from the ball bearing.
The sensor uses a CoAP POST operation to push data to the historian service
provider. As discussed in Section 3 this is translated to an OPC UA node write op-
eration. The translator maps OPC UA write operations to a fully addressed node-id
and attribute index, therefore no additional OPC-UA semantics leaks into the payload.
This is shown in Figure 11. Here, rotation, temperature and min/max/rms vibration
data is stored as a SenML JSON string. The name space index is 5, the node name is
wheelsensordata0x and the attribute is 13. The node name changes for each ball bearing.
This orchestrated historian enables end-to-end communication, with resilience to In-
ternet disruption, and flexibility between communication protocols. The proposed im-
plementation allows service based systems to push their data to an OPC UA server,
regardless of their protocol usage. In the following section, the proposed solution will be
discussed.
184 Paper F

Figure 10: OPC UA historian information model

POST /5/ wheelsensordata0x /13/ CoAP

Host: 192.168.7.1:48010
{
"e":[
{"n":" rotation", "v":1, "u":"count"},
{"n":" temperature", "v":60, "u":"Cel"},
{"n":" vibration_max", "v":4, "u":"m/s 2 "}
{"n":" vibration_min", "v":0, "u":"m/s 2 "}
{"n":" vibration_rms", "v":3, "u":"m/s 2 "}
],
"bn":"urn:dev:mac :0024 befffe804ff1"
}

Figure 11: CoAP payload example and structure, three nodes being updated in a single CoAP
message

5 Discussion
The translator has attempted to reduce leakage of OPC UA semantics across to native
web protocols. As with a translation between MQTT and HTTP where a path in HTTP
represents a topic in MQTT, a path in HTTP represents a node in OPC UA. However
the path structure must conform to a particular structure. By utilizing common methods
between HTTP and CoAP with OPC UA, the OPC UA protocol spoke does not require
additional semantic information within the non-OPC UA interface definition.
In this translation, the core features of the OPC UA information model have been pre-
served. Namely, it is possible to address individual nodes and to browse nodes. Therefore
it is possible to explore the address space and to read and write to nodes.
However, OPC UA supports 37 different services while the proposed solution has only
mapped 7 services. The supported service set provides a bare minimum interoperability
for non-OPC UA systems to interact with OPC UA servers. In this way payload interface
6. Conclusion 185

requirements made on the non-OPC UA systems is kept to a minimum. In fact for read
and write, only the addressing scheme of the service interface is impacted This impact
is also kept to a RESTful style, thereby would not be unnatural to any REST API
developer. It could be possible to include the full set of OPC UA services within the
URL, as path variables or URL query parameters. This would however break RESTful
design approach by including operation information in the address space. It would also
apply further interface design considerations on the non-OPC UA systems.
A service, in SOA concept, provides a functionality. This is beyond the interaction
services defined in OPC-UA. A call made to a service is expected to result in some value
added result. Perhaps the concept of the method object in OPC UA could map well to
a functional service. So an OPC UA server would provide services such as ”sound fire
alarm”, ”turn off lights” or ”store wheel loader data”. It is these services which should
be registered in a generic service discovery methodology such as DNS-SD would allow all
systems (OPC UA or not) to discover services provided by an OPC UA server.

6 Conclusion
Communication in Industry 4.0 requires flexible interconnections between integration
layers and information layers. The OPC UA communication framework shows potential
for such interconnections. Convergence to a single protocol is possible, but it is likely
that a multi-protocol communication network will exist for some time.
The here proposed translation service enables interoperability between OPC-UA and
other SOA protocols supporting integration of legacy automation system with upcoming
IoT devices. Such integration is supported by integration platforms like the Arrowhead
Framework. Frameworks such as Arrowhead enable dynamic provisioning and composi-
tion of the solution into the service exchange communication path. The proposed solution
allows for OPC UA translation mapping to be defined once, and then used for CoAP,
HTTP and MQTT protocols.
The translation preserves management, browsing and read/write of OPC UA nodes.
Because of the complexity of the OPC UA service interface, the translation does not
cover all functions of OPC UA. The mapping from OPC UA address space, requires that
nodes are addressed in the URL path or topic name in a specific manner. The name
space index and node name in the path are required to address the node. Combining
this with standard CRUD methods, a subset of OPC UA services can be invoke from a
generic non-OPC UA service interface.

7 Future work
Future work involves working translation back from OPC UA to HTTP/CoAP/MQTT.
This would enable OPC UA clients access to non-OPC UA services.
Incorporating OPC UA server discovery into a generic service discovery is also re-
quired for full look-up and runtime binding routines to be performed.
186 Paper F

Investigating the possibility of expanding the supported OPC UA service set while
maintaining a minimum requirement on service interfaces implemented with non-OPC
UA protocols.
An interesting new protocol for IoT is gRPC. Developed by Google and used within
their microservices infrastructure, it would be interesting to investigate its suitability for
interoperable use with OPC UA. It would enable cloud based service interaction with
automation services on the factory floor.

Acknowledgment
This work is supported by the EU ARTEMIS JU funding, within project ARTEMIS/0001/2012,
JU grant nr. 332987 (Arrowhead).

References
[1] G. Cândido, F. Jammes, J. B. de Oliveira, and A. W. Colombo, “Soa at device level
in the industrial domain: Assessment of opc ua and dpws specifications,” in 2010 8th
IEEE International Conference on Industrial Informatics, July 2010, pp. 598–603.
[2] T. Sauter and M. Lobashov, “How to access factory floor information using internet
technologies and gateways,” IEEE Transactions on Industrial Informatics, vol. 7,
no. 4, pp. 699–712, Nov 2011.
[3] M. J. A. G. Izaguirre, A. Lobov, and J. L. M. Lastra, “Opc-ua and dpws interop-
erability for factory floor monitoring using complex event processing,” in 2011 9th
IEEE International Conference on Industrial Informatics, July 2011, pp. 205–211.
[4] S. Grüner, J. Pfrommer, and F. Palm, “A restful extension of opc ua,” in 2015 IEEE
World Conference on Factory Communication Systems (WFCS), May 2015, pp. 1–4.
[5] ——, “Restful industrial communication with opc ua,” IEEE Transactions on In-
dustrial Informatics, vol. 12, no. 5, pp. 1832–1841, Oct 2016.
[6] P. Wang, C. Pu, and H. Wang. (2017) Opc ua message transmission method over
coap 01. [Online]. Available: https://tools.ietf.org/html/draft-wang-core-opcua-
transmission-01
[7] ——. (2016) Requirement analysis for opc ua over coap. [Online]. Available:
https://tools.ietf.org/html/draft-wang-core-opcua-transmition-requirements-00
[8] Kepware. (2017, May) Kepserverex. [Online]. Available: https://www.kepware.
com/en-us/products/kepserverex/documents/kepserverex-manual.pdf
[9] ——. (2017, May) Kepware iot gateway. [Online]. Avail-
able: https://www.kepware.com/en-us/products/kepserverex/advanced-plug-ins/
iot-gateway/documents/iot-gateway-manual.pdf
 187

[10] Projexsys. (2017) Hyperua. [Online]. Available: http://projexsys.com/hyperua/

[11] IEC-62541, OPC UA Specification. OPC UA Specification Release 1.03, 2015.

[12] H. Derhamy, J. Eliasson, and J. Delsing, “Iot interoperability - on-demand and

low latency transparent multi-protocol translator,” IEEE Transactions on Services
Computing, 2016, submitted.

[13] J. Delsing, IoT Automation Arrowhead Framework. CRC Press, 2017.

[14] F. Blomstedt, L. L. Ferreira, M. Klisics, C. Chrysoulas, I. M. de Soria, B. Morin,

A. Zabasta, J. Eliasson, M. Johansson, and P. Varga, “The arrowhead approach for
soa application development and documentation,” in IECON 2014 - 40th Annual
Conference of the IEEE Industrial Electronics Society, Oct 2014, pp. 2631–2637.

[15] Z. Shelby, K. Hartke, and C. Bormann, “The Constrained Application Protocol

(CoAP),” RFC 7252 (Proposed Standard), Internet Engineering Task Force, Jun.
2014. [Online]. Available: http://www.ietf.org/rfc/rfc7252.txt

[16] “Eistec AB,” Feb. 2013. [Online]. Available: http://www.eistec.se/

188
 Paper G
In-network Processing for
Context-Aware SOA-based
Manufacturing Systems

Authors:
Hasan Derhamy, Jens Eliasson, Jerker Delsing, and Jan van Deventer

Reformatted version of paper originally published in:

Conference paper, IEEE Industrial Electronics Conference, Beijing, China, 2017

© 2017, IEEE, Reprinted with permission.

189
190
 In-network Processing for Context-Aware
SOA-based Manufacturing Systems

Hasan Derhamy, Jens Eliasson, Jerker Delsing, and Jan van Deventer

Abstract

To achieve flexible manufacturing, increasingly large amounts of data are being gener-
ated, stored, analyzed, archived and eventually fed back into the product life cycle. But
where is this data stored and how is it transported? Current methods rely on centralized
or federated databases to manage the data storage. This approach has several challenges,
such as collection bottlenecks, secure retrieval, single point of failure and data-scheme
fragility as data heterogeneity increases. Additionally, manufacturers are finding the
need to open their networks for service based equipment suppliers. This means previous
security assumptions regarding network encryption and information access-control must
be re-evaluated.
Proposed here is a method of in-network processing that gathers information only
where and when it is needed. Systems build context at runtime by creating dynamic
queries which make service composition. The service composition processes raw data and
presents it as information to the calling system. This reduces the movement of data/in-
formation and removes single point collection bottlenecks. Furthermore, fine grained
access control and shared trust can be granted between untrusted systems. The pro-
posed methods are demonstrated on a lab setup of an industrial use case.

1 Introduction
Industry is demanding more Flexible, Efficient and Sustainable (FES) methods for man-
ufacturing [1]. Factory digitization was seen as the next step forward. ISA 95 assisted
factories in moving data between the factory floor and enterprise systems. However, due
to pure vertical integration [2], information flow and communication in the horizontal
and diagonal directions was not considered. In this pursuit, the Industry 4.0 initiative
has generated the Reference Architectural Model for Industry 4.0 (RAMI 4.0) and the
I4.0 component model [3]. They envisage communication between smart equipment and
products that are wrapped in administrative shells. The FES gains require a change to
many security assumptions and security objectives. This is to enable communication,
co-operation and information flow between multi-stakeholder Systems of Systems. Infor-
mation is a key enabler to finding efficiency gains and improving sustainable processes.
Raw data is gathered from IT systems and factory automation systems and stored
in databases. Raw data has limited value and must be processed before its value can
be realized. However processing the data into useful information can be costly and

191
192 Paper G

time consuming. Furthermore, even once improvements are identified, the software and
automation systems are often brittle and resist this change.
Flexible manufacturing means that a production line is able to adapt rapidly to
changes in the production process. This means that a customized order of lot size 1 is
a unique product/variant. So a single production line must handle multiple products or
variants at once. Managing the order/specification information and then disseminating
it to the right automation system at the right time requires significant synchronization.

1.1 Gap

Current solutions to data extraction, analysis and dissemination either suffer from cen-
tralized bottlenecks and single points of failure or fixed/rigid middleware. An information
exchange solution must conform the requirements:

Table 1: The requirements for information exchange in next generation industrial manufactur-
ing.
Requirement Description
Dynamic rout- Systems a moving, changing
ing of data be- and adapting, so static routing
tween systems of data is not flexible.
On-demand Mass collection/processing of
access to data is not efficient and se-
information cure. On-demand means in-
formation made available only
when and where it is needed.
Controlled Granular access rights and
distribution ownership of data/information
of data and must be supported.
information

Existing solutions using centralized data collection and processing or enforced mid-
dleware lack flexibility and are brittle to change.
The proposed solution is dynamic in nature, supports privacy-by-design and fine-
grained access control through supporting systems. The proposed solution presented
in Sections 3 and 4 enables context-aware industrial systems by allowing information
query from SOA-based networks without reliance on centralized data collection. This is
implemented in a lab setup of an industrial use case, condition monitoring of a wheel
loader ball bearing.
2. Background 193

1.2 Contribution
Here we propose an information extraction architecture that provides in-network pro-
cessing for context-aware systems. It builds on SOA-based principles. To achieve this
there are three main challenges:

1. Language to express interest in certain information

2. Orchestration method to parse the interest and identify the correct composition

3. Services available for processing raw data into presentable information

This paper tackles the first and third challenge. The second challenge is to be tackled
in future work.
The first challenge requires a method of expressing interest for information. An
interest must be able to express the required type of information (indicating the source
of the data) and the functional transformations of data. We propose to use Cypher [4],
an SQL inspired, graph query language. Section 2.3 details the expression of interest
Dynamic services are proposed to assist with standard data transformations. This
includes thresholding trigger, data aggregation, Filter and etc. Dynamic services are
service instances which are provisioned based on composition instructions. Section 4
describes the details.
Orchestration for SOA-based systems builds on concepts introduced by Arrowhead
Framework [5]. This paper assumes that composition facility is available.
The following sections will introduce the background to the importance of information
exchange and its application and the related work.

2 Background
2.1 Context-awareness
Context-aware computing starting in the 1990s focused primarily on providing location
dependent information to mobile users. According to Abowd and Dey [6] context can be
defined as:
“Context is any information that can be used to characterize the situation of an
entity. An entity is a person, place, or object that is considered relevant to the interaction
between a user and an application, including the user and applications themselves”.
At its essence context-aware computing involves systems modifying their behavior
based on changes to context over time. It is primarily used to decide what information
to present to a user. For example a shopping list system may remind a user about the
shopping list as they travel toward or past a supermarket. In this case, the system
monitored the users context.
However context-aware computing when applied to Industrial Internet can enable
advanced machine-to-machine behavior changes. Location is one aspect for a mobile
system within a factory floor. However, a full list of relevant context might incorporate:
194 Paper G

1. Production recipe,

2. Available machinery,

3. Available material

4. Operator synchronization - safety, validation,

5. Specifications,

6. Environment - temperature, humidity,

7. Location.

A context-aware industrial system interacts with other systems without user inter-
action. In this machine-to-machine environment, context moves in both vertical and
horizontal manners. Overall production objectives and methods are based on planning
and production recipes. While devices and product operation will generate context mov-
ing upward.
Context-aware systems decide the order of process execution based on available ma-
chinery. They do not begin dangerous processes while safe operation is not possible. A
customer specification decides the quality and customizes a product recipe. Environ-
mental factors affect quality, a context aware system do not begin a process before the
environmental requirements are satisfied.
Many automatic control systems such as ABS braking react to input and could there-
fore be considered context-aware systems. These embedded systems fit the pervasive
ubiquitous notion. They are single purpose tailored systems. However, these systems
collect data, process it, and make decisions within a single system. Industrial context-
aware systems cannot collect all raw data and process it themselves. This would result
in much duplicate collection and processing of data. Rather they gather context based
on responses from other systems in the service network.

2.2 Related work

Traditional approaches utilize centralized strategies with push based “command center”
synchronization. In this case a number of enterprise databases connected by a software
service bus are used to collect and disseminate factory information. If there are any
issues with the service bus, an entire production lines can suffer reduced performance if
not complete breakdown. This risk means that data access by non-production related
activities are restricted and minimized. This further reduces access to the raw data for
processing by quality or production engineering, looking to measure improvements or
find issues.
Context-aware systems are systems that utilize knowledge about their environment
and self driven objectives in their decision making. Flexible manufacturing processes use
smart products/equipment that are able to question their environment and build a life
cycle understanding. It can then make decisions about: 1) what is the next step in its
2. Background 195

process; 2) when it can execute the next step; 3) raise alerts regarding quality assurance;
4) make use of surrounding services; In this way, a flexible manufacturing line will have
many context-aware systems that make varied decisions as the context changes. Complex
operations emerge from simple actions. Once again, the issue of information management
raises questions of scalability, security and changeability.
In-network processing as described by Tannenbaum [7] is an approach of treating the
computer network as a means of processing raw data and aggregating or presenting the
information to the requester. TinyDB [8] is an example of in-network processing applied
to wireless sensor networks. But it can also be applied to general computing. Any
network can act as a database that can be queried for information that may not exist
until the query is processed. One of the benefits of in-network processing are to avoid
network and memory overheads of centralizing data and information. Furthermore, by
aggregating and preparing results, it can help reduce the complexity systems consuming
data from multiple sources.
Another similar approach to in-network processing is Information centric networking
(ICN). It is a proposal to move from host based addressing to information name address-
ing [9]. Regardless of where the information may exist, a system sends out an “interest”
for a particular information. This “interest” is then spread throughout the information
centric network until the information is found and returned to the requester. An ex-
tension to ICN is Service Centric Networking (SCN) [10]. SCN extends ICN to include
in-network functions that process existing data/information into new information. This
is a recursive operation where information = f (information).
These three approaches provide interesting methods of collecting, processing and dis-
tributing information. However, they all tackle information management at the network
layer, requiring significant changes to the network infrastructure. This is not feasible
with current trends in technology.
However, if it is possible to achieve in-network processing at the application layer,
then network infrastructure would not inhibit adoption of ICN. Building on the concepts
such as:

1. information requests are expressed as an “interest”

2. information and its source are decoupled

3. information modeled as a recursive function of information

Service Oriented Architecture (SOA) is an architectural approach that decouples systems

based on service interfaces [11]. There are clear lines of abstraction between the service
provider and consumer. The systems are bound at run-time, meaning that many systems
can be composed and re-composed dynamically at run-time. These principles of SOA can
be used to create networks of applications which find data, process it into information
and present it to the seeker.
196 Paper G

2.3 Cypher query language

Open Cypher [4] is an SQL inspired graph query language first developed by Neo4J [12].
It is intended as an intuitive way of exploring graphs and executing graph algorithms such
as shortest path. It supports full property graphs which allow properties to be associated
with edges. It’s notation uses a form of ASCII art to describe nodes and edges. A simple
graph matching can be done as shown in Figure 1.

MATCH (nodea:nodeTA)-[edgeb:edgeTB]->(nodec:nodeTC)
RETURN nodea

Figure 1: A simple graph query matching two nodes related by an edge. Returned is a reference
to the nodea entity. The nodea entity must match the relationship edgeb to nodec to satisfy the
query.

Cypher also supports “WHERE” clause, and also property conditions within the
“MATCH” clause. Additionally multiple “MATCH” clauses can be used within a single
query in order to build the query.

2.4 Arrowhead Framework

The Arrowhead Framework (AF) [13] is a supporting technology for the solution. The
AF is a SOA-based System of Systems (SoS) integration framework. The AF defines a
service as provided by a system. Systems are then composed in SoS. A single system
can provide and consume multiple services. Interactions between systems can only occur
through these services. The AF provides the composition aspect of the solution. The
expression builds a graph of systems and services. Each system is represented by a node,
and each service as an edge between two systems. Systems can only interact if they
have matching service provider and consumer interfaces. Matching service provider and
consumer interfaces are described in the AF documentation structure [14].

3 Proposed solution - Expression of Interest

The proposed solution uses a declarative SQL like graph query to express interest. The
query language is based on the Open Cypher graph query language. Cypher is selected
as it has a natural manner of expressing service compositions based on System of System
(SoS) approaches. Before describing the detail of the solution two alternatives are de-
scribed here. In [15] a custom algebra was defined that can describe a graph composition
based on service input/output matching. The benefit of this is that the graph can be di-
rectly applied to a graph of systems present in the ecosystem. Also, security and Quality
of Service (QoS) aspects can be represented directly in the graph. However, expressing
configuration of services (transformation of data) is not immediately clear.
4. Proposed solution - Dynamic Service Provisioning 197

In [8] an SQL styled declarative expression is used to describe the desired data. The
SQL expression has been enhanced to allow data processing on returned data. However
from the expression the mapping to services is unclear. The expressions are usually
spread to all nodes and a middleware is used to process the query.
Here Cypher allows the both declarative queries, while maintaining a SOA, com-
position based, expression of interest for information. The Cypher query language is
introduced in order to fully define its usage in the proposed solution.

3.1 Expression
An information query must define the source of the data, the processes, and the return
format. In AF a service represents a function, including the parameter format, the pro-
cessing and the return format. The system providing the service will either get raw data
internally or process data from another system. Therefore, using Cypher to describe the
system and service graph, all three aspects are defined. A simple example of a graph
expression can be seen in Figure 2 Here the source of the information is a temperature
sensor system (the far right system) and the processing is a threshold check. An inter-
mediate system provides the thresholding service upon data retrieved the temperature
service.

MATCH ()-[ts:ThresholdService]->()
-[:TemperatureService]->({l:chamber1})
WHERE ts.value > 60
RETURN ts.address

Figure 2: A simple Cypher query to express interest in the event of a chamber exceeding a
threshold temperature.

In this case, the application system need not perform thresholding internally and
thus has reduced complexity. This may seem a trivial simplification, however, it means
that its internal logic is not dependent on the data processing. It is possible to change
the processing of data to a band-pass threshold without modification to the application
system. A change like that could be due to a quality enhancement being identified by
keeping temperature bounds within band.

4 Proposed solution - Dynamic Service Provisioning

So far the proposed solution has presented a way to describe interest in information. To
process this information from raw data, there are certain functions required. For example,
rather than raw temperature data, only values exceeding a threshold are required, or in
the case of an IoT enabled wheel loader rather than data from individual wheel nodes,
aggregated data is required. Once the information interest has been expressed it is
198 Paper G

parsed into a suitable composition of System of Systems (SoS). Parts of this composition
are data processing systems such as:

1. Thresholding - reference triggers

2. Averaging - over time, over space

3. Filtering - low pass, high pass or band pass

4. Aggregating - list, array

5. Math - Summation, Subtraction, Multiplication, Division

6. Logic - OR, AND, XOR, NOT

7. Converting - converting measured units, i.e. km to miles.

8. etc

In this section a form of Dynamic Service Provisioning (DSP) is proposed. DSP allows
for creation of SoS based on services which a created only when needed. This allows the
ecosystem of services to dynamically grow and shrink as required by the context-aware
systems. For example in Figure 3 the threshold system compares the temperature of a
chamber to a reference level and pushes a notification when an event is triggered.

Figure 3: A basic service composition which creates context about the chamber temperature. If
the chamber temperature crosses 60°C an event is pushed to a consuming system. The service
labeled “a” contains the expression of interest as defined in Section 3. The service labeled “b”
is the provisioning interface. Services “c” and “d” are the data exchange links which are part
of normal operation.

The systems in Figure 3 have several interfaces. Service “b” between the Orches-
tration system and the Threshold system contains 1) customization parameters and 2)
4. Proposed solution - Dynamic Service Provisioning 199

target system addressing information. This is the systems composition interface. When
a request has been received, if the threshold system is able to comply with the request, it
will provision a sub-system which contains the interfaces for service “c” and “d”. This is
called dynamic system, which creates new services based on provisioning requests. The
services are created for use in a particular system composition. Each of the subsystems
is capable of handling different customization parameters. Furthermore, the individual
subsystems are uniquely identified and have independent security controls and authen-
tication. This provides many management advantages such as Quality of Service and
logging. Figure 4 shows this as an architectural block diagram of a dynamic system
structure.

Figure 4: The block diagram of a dynamic system, able to provision new threshold interfaces
based on request. Each sub-system is independent in terms of operation, but shares a single
execution instance.

The dynamic provisioning interface will vary from system to system depending on
the customization requirement. The variation of customization requirements is large and
growing as new processes are added. SenML is a markup language used for reading and
configuring sensor devices. It is lightweight and its loose definitions do not limit semantic
meaning or limit its extension to new sensor information. Here we propose to use SenML
as a means to send the customization information to the system. By using SenML it is
possible to have a simple, reusable, library for serializing different parameters. As shown
in Figure 5 the customization parameters for the threshold system are captured fully.
Adapting this for another system is not difficult as the JSON parameter object “n” is
updated with the new parameter.
In response to such a request the processing system will give access details of the the
newly provisioned service provider. Figure 6 shows an example of a response payload.
The dynamic service instance makes use of the Arrowhead framework authorization
pattern. This means that access control by the data source allows ownership of the
data to be maintained. Where mutual commercial agreement between two systems is not
available, the dynamic system could act as a third party, so long there exists a commercial
agreement between both systems and the dynamic system. This also holds for building a
200 Paper G

POST coap://ip:port/threshold/provision
{
bn:"query.system.origin",
"e": [{"n":" reference", "v":"60"} ,
{"n":" source", "v":"coap ://192.168.7.2:5683/ value "}]
}

Figure 5: A JSON example request to provision a new interface for the threshold system. Here,
SenML/JSON content type is used. “bn” is a basename and is the system who originated the
query. “e” refers to the element array and each item of the element array are configuration pa-
rameters related for the specific processing system type. In this case CoAP is the communication
protocol, but this could also be HTTP or MQTT

{
"con":"coap ://192.168.7.2:62158" ,
"res":"</name >;rt=threshold;ct="
}

Figure 6: A JSON example response to a provision request.

trust bridge where direct trust cannot be established, a Friend-of-a-friend. Furthermore,

in some cases, the data holds more information than required by the context-aware
system. In such cases, the context aware system need only have access rights to sub
information and not the complete data. Hence, a dynamic system which has access
rights to the full data, can process it and reduce the information content to only that
which the context-aware system has rights to. This enables privacy preserving features
of the information extraction architecture.
These advanced features are enabled because of the independence of the dynamic
systems and their provisioned sub-systems. It is also possible for multiple instances of
a dynamic systems run by different operators to exist within a single network. There-
fore, the paths available from the composition graph can be varied depending on the
information and security requirement.

5 Wheel loader ball bearing monitoring

A pilot scenario from the Arrowhead project involves monitoring the vibrations, temper-
ature and rotations of wheel loader ball bearings. Wheel loaders must be maintained in
order to extend lifetime. Measuring wear, looking for anomalies and scheduling preven-
tative maintenance are key objectives.
On a wheel loader with 4 wheels, there are 4 independent systems monitoring each
wheel. Each system provides a service which when queried will return the ball bearing
data presented in Figure 7.
The monitoring application could be designed to “seek out” each of the 4 sensor
systems and begin query them for their data (publish/subscribe is also possible). This
5. Wheel loader ball bearing monitoring 201

{
"e":[
{"n":" rotation", "v":1, "u":"count"},
{"n":" temperature", "v":60, "u":"Cel"},
{"n":" vibration_max", "v":4, "u":"m/s 2 "}
{"n":" vibration_min", "v":0, "u":"m/s 2 "}
{"n":" vibration_rms", "v":3, "u":"m/s 2 "}
],
"bn":"urn:dev:mac :0024 befffe804ff1"
}

Figure 7: The wheel loader ball bearing sensor data returned when queried.

would need to be repeated for each wheel loader in service. This complexity is a by-
product of the original mission of the monitoring application: 1) to schedule preventative
maintenance and, 2) to notify if early end-of-life has occurred.
Using the proposed methods, it is possible to reduce application complexity. By
allowing the SOA-based System of Systems to do the processing the application only
need handle relevant information. In this case, the expression of interest for information
is shown in Figure 8.

MATCH ()-[ns:NotificationService]->()
-[ts:ThresholdService]->()
-[:WheelNodeService]->({l:machineABC})
WHERE ts.vibration_rms > 3.3
RETURN ns.address

Figure 8: The expression of interesting using open Cypher query language and the proposed
SOA-based description. Where “ns” is the notificationService, “ts” is the threshold service, “l”
is the machine which the wheel loader node is on.

In this case first the ball bearing data must be collected and threshold tested, then
the data must be aggregated for all wheels on a malfunction wheel loader, before being
sent back to the monitoring application. This requires a dynamic threshold system and
aggregation system to be provisioned in order to present the ball bearing sensor data.
This composition is shown in Figure 9.
This means that the information has been completely processed by standardized and
reusable systems. It means that the monitoring application simply awaits notification by
the composition and then schedules the maintenance.

5.1 Implementation
The objective of this implementation is to test that interface definitions and the proposal
is sound. The base programming language is Java for applications systems. Using Java
202 Paper G

Figure 9: A diagram of the system composition which creates the information for the context-
aware monitoring application. The blocks connected with dashed line are a single module, with
the blocks above being persistent, while the blocks below being temporary.

allowed for development on a windows machine and subsequent deployment to Linux

distribution. Where Java is not suitable, i.e. embedded devices, implementation uses the
C programming language.
This scenario has been implemented using a Beagle Bone Black (BBB) [16] to run the
Arrowhead core systems, and the ball bearing sensor systems on the Mulle [17] boards.
Using ContikiOS [18] and CoAP [19] server on each node, the services register with the
Arrowhead service registry in the BBB. The monitoring application creates an interest
for information request to the Arrowhead Orchestration engine, which performs a lookup
on current ball bearing sensors and composes the threshold and aggregation systems.
The monitoring application in this case was a stub application which simply recorded
that the wheel loader required servicing or not.

6 Conclusion
Industry is looking to gain flexibility in growing and evolving their manufacturing sys-
tems. Applying Service Oriented Architecture (SOA) to System of Systems (SoS) engi-
neering allows flexible networked embedded systems to participate in multilayer enter-
prise systems. In particular interest of reusable systems is that of information processing.
There has been a trend to move away from centralized systems. This is due to the weak-
nesses of; 1) single-point-of-failure, 2) bottlenecks, 3) resistance to change and 4) costly
maintenance.
References 203

This work has presented a novel approach for information extraction for context-
aware systems in Industry 4.0. Context-aware computing when applied to SoS results in
intelligent systems which can make decisions based changes to location, recipe, quality,
availability, safety, and environment. This reduces, and in some cases removes, the
requirement on centralized infrastructure for data processing and storage. It also helps to
maintain soft real-time operation in the overall production line by enabling edge analytics.
Centralized systems are used to manage overall objectives and to compose SoS. However,
once the SoS is composed and the systems have the required knowledge, they can operate
without further guidance from centralized systems.
The proposed solution was showcased on a demonstrator use case of a wheel loader
ball bearing monitoring application.
This work will be continued to further refine and formalize the composition method-
ology. Building on the expression of interest for information and the dynamic systems
proposed here, the composition engine must analyze the available composition graph to
find the optimal shortest path.
In addition to the information requirement, the security overlay must be considered
for the composition. This includes strict access-control and trust rules and more malleable
trust which can be established dynamically.

Acknowledgement
The authors would like to express our gratitude towards our partners within the Far-Edge
project and the on going work within the Arrowhead Framework open project.

References
[1] H. Lasi, P. Fettke, H.-G. Kemper, T. Feld, and M. Hoffmann, “Industry 4.0,”
Business & Information Systems Engineering, vol. 6, no. 4, pp. 239–242, Aug 2014.
[Online]. Available: https://doi.org/10.1007/s12599-014-0334-4

[2] B. Scholten, The road to integration : a guide to applying the ISA-95 standard in
manufacturing. Research Triangle Park NC : ISA, 2007.

[3] “Reference Architectural Model Industrie 4.0 (RAMI4.0) - An Introduction,”

April 2018. [Online]. Available: https://www.plattform-i40.de/I40/Redaktion/EN/
Downloads/Publikation/rami40-an-introduction.pdf

[4] “OpenCypher,” May 2017. [Online]. Available: http://www.opencypher.org/

[5] “Arrowhead framework wiki,” May. 2016. [Online]. Available: https://forge.soa4d.

org/plugins/mediawiki/wiki/arrowhead-f/index.php/MainPage
204

[6] G. D. Abowd, A. K. Dey, P. J. Brown, N. Davies, M. Smith, and P. Steggles,

“Towards a better understanding of context and context-awareness,” in International
Symposium on Handheld and Ubiquitous Computing. Springer, 1999, pp. 304–307.

[7] A. S. Tanenbaum and M. v. Steen, Distributed Systems: Principles and Paradigms

(2Nd Edition). Upper Saddle River, NJ, USA: Prentice-Hall, Inc., 2006.

[8] S. R. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong, “Tinydb:

An acquisitional query processing system for sensor networks,” ACM Trans.
Database Syst., vol. 30, no. 1, pp. 122–173, Mar. 2005. [Online]. Available:
http://doi.acm.org/10.1145/1061318.1061322

[9] M. A. Yaqub, S. H. Ahmed, S. H. Bouk, and D. Kim, Information-Centric Networks

(ICN). Singapore: Springer Singapore, 2016, pp. 19–33.

[10] T. Braun, V. Hilt, M. Hofmann, I. Rimac, M. Steiner, and M. Varvello, “Service-

centric networking,” in 2011 IEEE International Conference on Communications
Workshops (ICC), June 2011, pp. 1–6.

[11] T. Erl, SOA Design Patterns, 1st ed. Upper Saddle River, NJ, USA: Prentice Hall
PTR, 2009.

[12] “Neo4j Graph Database,” May 2017. [Online]. Available: https://neo4j.com/

[13] “Arrowhead - Enable collaborative automation by networked embedded devices.”

Feb. 2013. [Online]. Available: http://www.arrowhead.eu/

[14] F. Blomstedt, L. L. Ferreira, M. Klisics, C. Chrysoulas, I. M. de Soria, B. Morin,

A. Zabasta, J. Eliasson, M. Johansson, and P. Varga, “The arrowhead approach for
soa application development and documentation,” in Industrial Electronics Society,
IECON 2014 - 40th Annual Conference of the IEEE, Oct 2014, pp. 2631–2637.

[15] S. V. Hashemian and F. Mavaddat, “A graph-based framework for composition of

stateless web services,” in 2006 European Conference on Web Services (ECOWS’06),
Dec 2006, pp. 75–86.

[16] “Beagle Bone Black,” May 2017. [Online]. Available: https://beagleboard.org/black

[17] “Eistec AB,” Feb. 2013. [Online]. Available: http://www.eistec.se/

[18] A. Dunkels, B. Gronvall, and T. Voigt, “Contiki - a lightweight and flexible operating
system for tiny networked sensors,” in Local Computer Networks, 2004. 29th Annual
IEEE International Conference on, Nov 2004, pp. 455–462.

[19] C. Bormann, A. Castellani, and Z. Shelby, “Coap: An application protocol for

billions of tiny internet nodes,” Internet Computing, IEEE, vol. 16, no. 2, pp. 62–
67, March 2012.
 Paper H
Workflow Management for Edge
Driven Manufacturing Systems

Authors:
Hasan Derhamy, Mattias Andersson, Jens Eliasson and Jerker Delsing

Reformatted version of paper accepted for publication in:

Conference paper, IEEE Industrial Cyber-Physical Systems, St Petersburg, Russia, 2018

© 2018, IEEE, Reprinted with permission.

205
206
 Workflow Management for Edge Driven
Manufacturing Systems

Hasan Derhamy, Mattias Andersson, Jens Eliasson and Jerker Delsing

Abstract

The fourth industrial revolution is one of digitization. As manufacturers build more

digital equipment into their process harnessing the potential becomes a challenge. ISA
95 based manufacturers rely on top down decision making, with redundant flow of in-
formation in order to organize Workflows on the factory floor. With advances in edge
computing and smart objects, more decision are made at lower levels of the infrastructure
hierarchy.
However, how can the MES tracking and execution functions be decentralized while
maintain an organized collaboration of smart objects? This paper proposes a Cyber
Physical Service Oriented System of Systems operating on edge computing. There are
three systems proposed here; Smart Product, Workflow Manager and Workflow Executor.
These systems operate within Arrowhead Local Clouds and create a fully autonomous set
of system to complete a production order. Planning, Supply chain and Quality of Service
monitoring are out of scope and remain as centralized activities. The Smart Product is
the core Cyber Physical System that must be context-aware in order to pass the correct
Workflow information to the Workflow System responsible for current activities.

1 Introduction
The fourth industrial revolution is distinct from the third industrial revolution in: ve-
locity, scope, and systems impact [1]. This is because the fourth industrial revolution
involves the fusion of technology from physical, digital and biological domains. This
means new products will be developed and manufactured and traditional products will
be manufactured in new ways. Two key challenges facing traditional manufacturers in
the fourth industrial revolution involve scale and cost of digitization. Issues of scale are
not limited to just the volume of devices or inputs and outputs. But also the scale of
complexity is also increasing. Flexibility and adaptability are some of the key drivers
of this complexity. Applying traditional centralized approaches suffers from the require-
ment for availability and security. As the dynamics on the factory floor increases the
infrastructure demands increase, thereby increasing costs of backup systems.
New approaches such as the Reference Architecture Model for Industry 4.0 (RAMI
4.0) decentralize responsibility to individual components [2]. Figure 1 shows the RAMI
4.0 diagram. It shows that some enterprise logic exists even at the device level. However,

207
208 Paper H

how much logic should be included into decentralized systems? How to coordinate all
the logic?

Figure 1: The RAMI 4.0 layers vs hierarchy vs life cycle diagram

Edge computing paradigm has been suggested to help alleviate the backhaul traffic of
large volumes of sensory traffic. However, edge computing when applied to a manufactur-
ing scenario can help to reduce the decision making path. If RAMI 4.0 is applied to edge
computing then a pattern for answering the question of coordinating and decentralizing
logic can be made. The Far-Edge project is developing a platform that enables RAMI 4.0
based application development. The platform is building on the Arrowhead Framework,
utilizing Service Oriented Architecture (SOA) thinking to divide responsibilities.
This paper proposes edge automation services that coordinate and enable productive
collaboration between Cyber Physical Systems (CPS). The edge automation services
consist of functionality required to plan, sequence and interconnect CPS systems. It uses
the Arrowhead Framework core support systems as its basis and adds new systems, in
particular for sequencing of the production process.

1.1 Edge automation objective

According to the ISA-95 standard, the Manufacturing Execution System (MES) layer of
the factory is responsible for Workflow and recipe control. Production orders generated
by Enterprise Resource Planning (ERP) and Product Lifecycle Management (PLM) tools
will create customized workflows according to the product specification and manufactur-
ing requirements.
Edge Automation Services (EAS) role is to automate the execution of a production
order at the edge tier. Meaning that the edge tier (edge gateway and smart objects) are
able to complete the production order (or part of it) without reliance on the centralized
1. Introduction 209

MES layer. Alternatively, it means that the MES layer has been decentralized across the
edge tier, with the edge automation services as the enabler.
It stands to benefit to define some of the core operations of the MES layer; Planning,
Supply chain, Inventory, Quality of Service management, Tracking and Execution.
This paper focuses on the tracking and execution aspects of the MES. Tracking is made
through the intelligent product unit that holds the planned workflow and the current state
of activities. Execution is conducted through the Workflow manager, Workflow executor
and smart objects such as intelligent equipment.
Once the MES data is available at the edge tier, EAS are able to complete the required
production steps with very little or no connectivity to any cloud based systems. Moreover,
Production Orders are also updated by EAS as a consequence of production steps being
completed. A working copy of the production order is interacted with at the edge, this is
a ”disconnected” approach. The work copy is then propagated to the master copy once
connectivity is available.
Overall, edge automation services are expected to have the following advantages over
a traditional centralized architectures:

1. Shopfloor more tolerant to network and central system faults.

2. Shopfloor more modular and flexible, due to individual workstation (workcell) being
autonomous.

3. Automation more reactive to shopfloor events, due to the shorter path between
business logic and sensing/actuation hardware.

There are two situations where these advantages become very apparent. Where a
workstation must be operated within an environment with no factory wide MES is avail-
able. This ”autonomous” workstation only requires that the smart product it will operate
on has an interface to communicate the workflow at the target station. This information
that the smart product contains can be loaded to the product prior to shipping from the
origin factory. Or can be transfered through some secure means to the workstation.
The second situation is that because the workflow does not specify execution details
(this is the responsibility of the workflow executor orchestration), ”plug-in-produce” is
supported for equipment installation to the work station. Meaning that at runtime an
”intelligent nutrunner” can be plugged into the workstation and utilized immediately.

Cyber Physical Systems

Edge Automation Services are directly related to Cyber Physical Systems. CPS are
defined as networked embedded systems [3]. Embedded systems are computer systems
that interact, sense and actuate, on their environment. The information carried by
CPS and the services provided by CPS mean that MES functions that are traditionally
centralized can now be decentralized.
This paper proposes a method of co-ordinating the efforts of CPSs within the frame-
work of edge MES functionality.
210 Paper H

2 Related work
Workflow, also known as business process, management has been a topic of research since
the start of industrialization [4]. A workflow, or business process, is the combination of
activities and information toward the creation of new value [5]. In [6], many reusable
workflow patterns are documented and existing workflow frameworks are evaluated for
support of the pattern. Workflow patterns are split into four core categories; Control,
Resource, Data and Exception Handling workflows.
In [7], Sardis et al. describe a distributed workflow and monitoring system. They
propose an architecture that combines rule based alarm detection, fault planning through
pre-planned behaviors, decision support and workflow monitoring. They suggest that a
distributed architecture such as Multi-Agent Systems (MAS) or SOA be used for real-
ization of the workflow manager and monitoring. However, they do not propose how the
workflow system should be realized.
Quintanilla et al. propose a holonic approach toward workflow execution using SOA
[8]. In this work, they propose 3 holons, the Product (PH), Order (OH) and Resource
holons (RH). The PH is responsible for exploring workflow solutions, meaning that it
performs planning and scheduling. The OH is responsible for execution of the workflow
tasks specified by the PH. While the RH provides the manufacturing services utilized
by the OH to complete the workflow. Petri-nets are used model the workflow and to
evaluate alternatives.
REST and hypermedia has been proposed for work flow automation. For example in
[9], Balis proposes a hypermedia driven workflow for data-driven scientific experiments.
In this context a data driven workflow is implemented using an Hypermedia As The
Engine Of Application State (HATEOAS) approach. Data and Tasks are addressed as
resources and a client ”executor” is responsible for following the hyper media links. The
links have embedded an appropriate state machine. Clients and resources must have
agreement on link relation meanings and resource types. The client executor knows the
overall work flow state machine required to be executed. Starting by preparing data, the
client moves through servers according to the link and relations returned by the servers.
This work closely resembles the work by Quintanilla et al. in [8]. There are clear
differences with seperation of concerns and centralization of the Order holon. Manufac-
turing systems that require segregated work centers require decentralized task execution.
Here, the proposed approach has a decentralized, stateless Workflow manager that imple-
ments a blackboard architecture pattern. The blackboard is available to all equipment
systems operating in the workcenter. Meaning that, concurrent workflow description
available can be acted upon by individual systems as needed.

3 Arrowhead Framework
The Arrowhead Framework is based on Service Oriented Architecture (SOA) [10]. It
defines Systems as the containers of Services that are exchanged between systems and
clients. The concept of local clouds is defined as a means of controlling Quality of Service
4. Work flow handling 211

(QoS), increasing security and segregating functionality. Within each local cloud there
are several mandatory core systems and support core systems. These services provide
functionality for discovery, access control, and interconnect management. Additionally
they provide interoperability, storage and configuration services. In order to describe the
plant layout and capability locations the Plant Description [11] support core system is
defined. Figure 3 shows the main building blocks of the Arrowhead Framework.

Figure 2: The Arrowhead Framework block diagram of constituent systems.

The Arrowhead Framework is an active open source project that is being contributed
to be European Union funded projects; Productive 4.0 and Far-Edge. The relevant parts
of the Arrowhead Framework are described in the proposed solution.

4 Work flow handling

The production order is the fundamental data required by the edge automation services.
It is carried by CPS based smart products or a digital twin (where CPS is not available).
Figure 4 shows a block diagram of a smart product in a factory with edge automation
services.
Therefore the edge automation services have an interaction point in the smart product
unit. Also shown in the Figure 4 the smart equipment at each work cell make up I4.0
components that interact with the edge automation services.
212 Paper H

Figure 3: High level perspective of Industry 4.0 enabled smart product and work cells

4.1 Production Order

The ISA95 describes a model for the connection between ERP and MES. But in this
work a simpler model that uses a Product to Process approach has been taken, i.e. how
to exchange information between a product and the process. Additionally a batch-size
one approach is taken, where every product unit has a unique recipe that is configured
directly from the specification ordered of the customer.
The production order consists of the production specification (customer requirement)
and the product recipe (production requirement). The recipe contains the Bill Of Mate-
rials (BoM), List of Operations and Equipment Requirements. The recipe does not bind
the operations to either equipment instances or to work stations. Shown in Figure 4.1
the Entity Relation Diagram (ERD) of the production order can be seen.
As described earlier, CPS-based products or their digital twins are containers of the
production order. The synchronization between digital twin and physical product is
made directly. It is not the focus of this paper and so is assumed to be available. The
specification holds product variant information of the production order. The List of
Operations is grouped into ordered work cells, meaning that a flow of instructions is
defined and loosely grouped.

4.2 Production Orders to the factory edge

PLM and ERP tools generate production orders based on a product specification. The
product specifications are created by engineers working with customers to identify re-
quirements and solutions. The PLM/ERP tools maintain a direct link between customers,
requirements, specifications, and the production order.
The product specification defines the production capabilities required to manufacture
the product. The production process step of the the production order presented in
5. Edge Automation Services 213

Figure 4: High level Entity Relation Diagram of a production order

Figure 4.1 is mapped to a work station (cell). The mapping is decided based on known
capabilities available at the work station. This can be done dynamically meaning that
scheduling and routing must be performed by local systems. The planning function is not
addressed in this paper. It can be assumed that the plan is statically created and that
the product unit is the container. Factory, work center, station and device capabilities
must be known when creating the production order.

5 Edge Automation Services

The proposed solution involves SOA-based systems running in the edge tier. In this
section the individual systems which are used to achieve the automation objectives.

5.1 Arrowhead local cloud

The Arrowhead Framework local cloud is running on an edge gateway. It has all pre-
requisite components for Service Oriented applications to run on the edge tier. It is
expected that individual edge gateways may have customized local clouds according to
requirement. With support core systems only present and operating when required. Full
authorization security is supported at the edge automation layer, meaning that edge
gateway is protected by the Arrowhead security mechanisms. Readers can also refer to
the Arrowhead Wiki for further documentation and notes [12].

5.2 Workflow Manager

The Workflow Manager is a SOA-based system developed for the Far-Edge platform [13]
and operating within the Arrowhead local cloud paradigm. It is a form of book keeper
214 Paper H

that is responsible for managing progress of the production process/recipe and reporting
back to the Factory Control System (FCS).
Here the FCS is a Volvo term, however it can also be seen as a function. The func-
tion it provides is for overall coordination of the factory. The FCS is not responsible for
coordinating equipment, rather it is an information source for products and equipment
to refer when requiring information outside their direct scope. Here it can be seen that
classical MES vs. RAMI4.0 come into contact. The Control logic: Quality, Process, Lo-
gistics, etc. are placed inside the smart product/virtual twin rather than as a centralized
service. However, the FCS remains as a migration point where not all information can
be located within the I4.0 Component.

Interfaces

The Workflow Manager has 3 interfaces used to synchronize with smart product units
(CPS/Digital Twin) and 2 interfaces toward smart equipment. The Workflow Manager
consumes services provided by the product unit, making ”pull” requests for production
recipe information in a RESTful manner.

1. RetrieveProcessStep

2. StoreProcessStepResult

3. StoreOperationResult

The Workflow Manager only need identify itself (workstation id) to the product unit,
as the recipe has operations allocated at a workstation level. It is assumed that this ID
is set when the local cloud instance (edge gateway) is being setup. It must be completed
before the Workflow Manager is able to take part in the edge automation activities. It
is possible that these service interfaces will evolve to allow further parametrization as
required. Toward the edge, the Workflow Manager is a service provider and will listen
for service invocation requests in a RESTful manner.

1. RetrieveOperation

2. StoreOperationResult

The workflow manager services are designed such that they can be used by both Push
and Pull communications patterns, such as MQTT or CoAP/HTTP. CoAP has been
selected for the prototype implementation due to convenience of application libraries
and simple integration with application code. Additionally CoAP, has advantages for
RESTful communication with a push notification channel available. The usage of the
interfaces are described in the next sections.
5. Edge Automation Services 215

Operation

A Workflow Manager will remain passive awaiting notification that a product unit has
arrived at the work station. On arrival, of a new product to the work station, the
Workflow Manager will query the smart product (or its digital equivalent) for the recipe
related to the the work station. The production order will be checked to make sure any
dependencies between work station flows have been met. Only relevant steps of the recipe
will then be passed to the work station (Workflow Manager).
With the new recipe loaded, the Workflow will begin with edge systems pulling work
instructions. Once work instructions are completed, the results are sent back to the
Workflow Manager. Overall completion of the workflow is decided by the Workflow
Manager once all results are compiled. The final results are then sent back to the product
unit with confirmation. A simplified version of this is shown diagrammatically in Figure
5.2. It illustrates a typical interaction between the Edge Automation Services. Here the
third item ”Workstation equipment” groups together any individual smart objects that
can be coordinated through the Workflow Manager.
The Workflow Manager holds the digital instruction list as a form of state machine.
This state machine is passed to an executor system within the Workstation. The executor
system is customized and configured for a particular Workstation. The customizations
are both in terms of configuration parameters and capability interactions. The capability
interactions are defined by Orchestration rules, described under orchestration section.
Figure 5.2 shows a block diagram of this interaction. The Workflow Manager passes the
instruction list as a hypermedia enabled state machine. Using IETF RFC 5980 and 6690,
namely, Web Linking and Core Link Format, it is possible to describe the state machine
as a set of related resources.

5.3 Plant Description

The Arrowhead Plant Description system is responsible for maintaining a functional and
topological map of the systems available in the local cloud. This means that the plant
description has knowledge regarding the edge nodes connected to the edge gateway. It is
kept up to date through the Arrowhead Device and System Registries operating within
the Field Abstraction component. The information stored in the plant description is used
for engineering the edge tier gateway and nodes [11].

5.4 Orchestration System

The Arrowhead Orchestration System enables central control of system coordination.
It holds information on how systems should be composed into Systems of Systems. So
when the workflow executor needs to make use of equipment of a certain capability it
will request connection information from the Orchestration System and thereby ”locate”
the correct service instance representing the equipment.
216 Paper H

Figure 5: The Workflow Manager interacts with its peers

5.5 Service Registry

The Arrowhead Service Registry holds information on active services available at the
edge gateway. This information is used to ”locate” service provider instances. Edge
nodes will register (advertise) their capabilities through services registered here. Edge
automation services such as the executor system will find the end-point address of the
capability required to complete the operations in the process step.

5.6 Authorization System

Edge nodes and edge automation services are protected from un-authorized access through
the Arrowhead Authorization system. This system is providing the PIP and PDP ac-
cording to the NGAC terminology. A system providing services will enforce the policy
decision made by the Arrowhead Authorization system.
6. Use case implementation 217

Figure 6: The Executor is a HATEOAS enabled application state machine.

6 Use case implementation

The prototype implementation of the edge automation services presented in this paper
is validated in an industrial use case. In this section a high level description of the use
case is described. It is intended to run a small scale pilot of the use case in a parallel
production environment (using non-production data).
The use case involves two key validation criteria: Plug and Produce Equipment and
Autonomous Workstation. In the first case, an Industrial Nutrunner is enhanced with
an administrative shell. It is shown that the smart Nutrunner can be moved between
work stations and its capabilities made use of without manual configuration of the work
station.
In the second case the entire workstation is self contained so that it can be ”moved” to
an external site and still work. This means that it has operational independence from the
factory control system. A production order may be delivered to it digitally or manually
in a secure format.
The business benefits of the use case can be sumarized as:

1. Simplify or eliminate configuration and engineering of equipment.

2. Reduce IT integration costs and reliance on IT competence resources.

3. Flexibility of resource location.

4. Enabling continuous improvement through reduction in effort to realize change.

Figure 6 shows the system block diagram of the use case solution. In the figure the
FCS Adapter is a temporary system that serves as a migration path. It serves the same
function as the Product Unit Hub described in the original platform.
218 Paper H

Figure 7: Block diagram of RAMI 4.0 enabled work station utilizing Edge Automation Services
from the Far-Edge platform and the Arrowhead Framework.

The Workflow manager and executor systems can be seen in Figures 6 and 6. These
systems create the link between the CPS Smart Product and Smart Equipment. The
Service links are discovered through the Arrowhead Service Registry and Orchestration
Systems.

Figure 8: I4.0 enabled Workflow Manager System with administrative shell and services. Up
stream services are on the left side and down stream services are to the right.

In this prototype implementation, an Atlas Copco P4000 Nutrunner is interacted

with through an administration shell implemented as a SOA-based System. It is a ser-
vice provider that registers its services with the Arrowhead Service Registry. The I4.0
7. Future work 219

Figure 9: I4.0 enabled Workflow execution system with administrative shell and services. Up
stream service connecting to the Workflow manager are to the left and down stream services
connecting to the smart equipment and objects are to the right.

Nutrunner is shown in Figure 6.

Figure 10: I4.0 enabled Nutrunner with administrative shell and services. DoJob and NotifyRe-
sult are standard services for generic equipment requests.

A multi-protocol approach is taken to demonstrate the interoperability between differ-

ent systems. This means that the Nutrunner uses MQTT for its service provision, while
CoAP is used by the Workflow Manager and HTTP is utilized by the Core Systems.

7 Future work
Still to be added is the configuration integration and the plant description updates. Being
able to generate the Orchestration rules through the plant description and then utilizing
a workflow through the Workflow Manager and Executor systems. Fully utilizing the
Next Generation Access Control mechanism in the Arrowhead Authorization system.

8 Conclusion
Demonstrated in this paper is an approach to decentralizing the ISA 95 architecture
utilizing the RAMI 4.0 approach on edge computing and SOA-based system design.
A Workflow Manage is proposed as the integration point of MES data between smart
products and smart equipment. Reducing the demands on centralized infrastructure
whilst maintaining centralized resource planning of the overall production. The proposed
solution is demonstrated on an industrial use case.
Summarizing the Arrowhead concepts can be described as:
220 Paper H

1. Plant Description is used for planning and scheduling.

2. Work flow Manager is used for coordination and sequencing.

3. Orchestration is used for realizing the Plant Description.

In this regard, the paper has also contributing to creating an seperation of concerns
between the Arrowhead Plant Description, Orchestration and Work flow management.

Acknowledgment
The authors would like to thank the Far-Edge and Productive 4.0 projects for funding
and sharing thoughts through discussion and debate.

References
[1] “The Fourth Industrial Revolution - What it means and how to respond. World Eco-
nomic Forum,” Jan 2018. [Online]. Available: https://www.weforum.org/agenda/
2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/

[2] “Reference Architectural Model Industrie 4.0 (RAMI4.0) - An Introduction,”

April 2018. [Online]. Available: https://www.plattform-i40.de/I40/Redaktion/EN/
Downloads/Publikation/rami40-an-introduction.pdf

[3] E. A. Lee, “Cyber physical systems: Design challenges,” in 2008 11th IEEE In-
ternational Symposium on Object and Component-Oriented Real-Time Distributed
Computing (ISORC), May 2008, pp. 363–369.

[4] W. M. van der Aalst and K. M. van Hee, Workflow Management: Models, Methods,
and Systems. The MIT Press, 2000.

[5] W. M. P. van der Aalst, A. H. M. ter Hofstede, and M. Weske, “Business pro-
cess management: A survey,” in Proceedings of the 1st International Conference on
Business Process Management, volume 2678 of LNCS. Springer-Verlag, 2003, pp.
1–12.

[6] (2018) Workflow patterns. [Online]. Available: http://workflowpatterns.com/

[7] E. Sardis, V. Anagnostopoulos, A. Voulodimos, and T. Varvarigou, “On surveillance

and workflow control system design,” in 2010 IEEE International Conference on
Progress in Informatics and Computing, vol. 1, Dec 2010, pp. 267–271.

[8] F. G. Quintanilla, O. Cardin, and P. Castagna, Product Specification for Flexible

Workflow Orchestrations in Service Oriented Holonic Manufacturing Systems.
Cham: Springer International Publishing, 2014, pp. 177–193. [Online]. Available:
https://doi.org/10.1007/978-3-319-04735-5 12
 221

[9] B. Balis, “Hypermedia workflow: A new approach to data-driven scientific work-

flows,” in 2012 SC Companion: High Performance Computing, Networking Storage
and Analysis, Nov 2012, pp. 100–107.

[10] J. Delsing, IoT Automation Arrowhead Framework. CRC Press, 2017.

[11] O. Carlsson, C. Hegedus, J. Delsing, and P. Varga, “Organizing iot systems-of-

systems from standardized engineering data,” in IECON 2016 - 42nd Annual Con-
ference of the IEEE Industrial Electronics Society, Oct 2016, pp. 5277–5282.

[12] “Arrowhead framework wiki,” May. 2016. [Online]. Available: https://forge.soa4d.

org/plugins/mediawiki/wiki/arrowhead-f/index.php/MainPage

[13] “Far-Edge Project,” Jan 2018. [Online]. Available: http://www.faredge.eu/

222
 Paper I
System of Systems Composition
based on Decentralized Service
Oriented Architecture

Authors:
Hasan Derhamy, Jens Eliasson and Jerker Delsing

Reformatted version of paper submitted to:

Journal paper, IEEE Internet of Things Journal, 2018

© 2018, IEEE, Reprinted with permission.

223
224
 System of Systems Composition based on
Decentralized Service Oriented Architecture

Hasan Derhamy, Jens Eliasson and Jerker Delsing

Abstract

As society has progressed through periods of evolution and revolution, technology

has played a key role as an enabler. In the same manner that mechanical machines of
the 1800’s drove the industrial revolution, now digitalized machines are driving another
industrial revolution. The Industrial Internet of Things works toward achieving flexi-
ble manufacturing, with increasingly large amounts of data generated, stored, analysed,
archived and eventually fed back into the product life cycle. But where is this data
stored and how is it transported? Can centralized or federated databases continue to
manage the data storage? Centralization of data has several challenges, such as collec-
tion bottlenecks, secure retrieval, single point of failure and data scheme fragility as data
heterogeneity increases. As manufacturers increase the digital foot print on the factory
floor, harnessing the Industrial Internet of Things becomes a challenge.
This paper proposes a method of information distribution based on the principle
of data at its source. Data gains context at runtime through the creation of dynamic
queries that are used to make system compositions. These System of Systems compo-
sitions can retrieve, process and present data as information to the consuming system.
Processing can be aggregation, signal processing, or format manipulation. The proposal
starts by creating a graph model of the Arrowhead Framework. Then building on the
graph model, an approach to specify, validating and forming Systems of System is pro-
posed. The proposed graph model allows for unambiguous description of Systems and
their interrelations, including security relations. In addition, the graph model is also a
prerequisite to be able to describe and then form Systems of Systems. The proposed
composer operates on edge computing hardware and gives production floor ability to ex-
tract information without impacting the overall operation of the factory. The proposed
solution shortens the lead time for data to be distributed.

1 Introduction
Initiatives in the manufacturing industry are looking to boost productivity by leveraging
advances in connectivity. The increase in connected devices is due, in large part, to
increases in low cost networking and computing. This has meant that there is a significant
increase in the volume and flow of information. All these developments have occurred
within a landscape of ISA 95 [1] deployments. ISA 95 supports vertical integration

225
226 Paper I

and information flow. However, this also created bottlenecks from many perspectives.
Comparing centralized and decentralized information flow:

Storage Centralized storage exhibits database bottlenecks, meaning that storing and
retrieving data is usually restricted to ensure that the database is not over loaded with
requests. Local storage handles requests within its local scope, therefore less requests
would be present. Also, if a decentralized database does experience reduced performance
due to over loading, the area of impact is local and does not reach to wider applications.

Communication Centralized storage must have strong networks. The network be-
comes a bottleneck as traffic increases, meaning close monitoring and costly backbone
infrastructure is required. Redundant networks become needed to reduce down time in
case of network failure. Performance degradation on the network also has a wide impact.
On the other hand with local storage, network performance degradation has a lesser
impact on normal operations.

Technology Centralized systems are sensitive to disruptions and so change process

must ensure normal operation. Also, technology selection is limited to those that have
been tested for high performance operation. With a localized storage system, the sensi-
tivity to disruption is reduced and the options to work around the disruption increases.
Also, technology selection can evolve more readily, with a more heterogeneous set of
technologies better suited for specific requirements.

Engineering Centralized systems will have a specific set of competencies such as data
warehousing, network administration, server maintenance, etc. These specialists are
required to handle scaling up infrastructure and tools. While localized systems do not
require specialized knowledge for scale. They simply do not reach such demanding levels
and so competent general engineering skills are sufficient.
Leading on from ISA 95, Industry 4.0 initiative has proposed the Reference Architec-
tural Model for Industry (RAMI) 4.0 [2] and I4.0 Component model [3]. The I4.0 compo-
nent model captures the notion an administration shell abstracting digitized equipment
and products with high levels of connectivity. A key diagram of RAMI 4.0 is a 3-
dimensional pyramid/cube. It represents that I4.0 components at different ”hierarchies”
must go through an individual ”life-cycle” and participate at each ”layer”. Or from a
layer point of view, a single ”layer” cannot be confined to a single level of the ”hierarchy”.
Rather a single ”layer” is spread across many levels of the ”hierarchy” and present at
each stage of the components ”life-cycle”. This is shown in Figure 1.
A central philosophy of the RAMI 4.0 approach is that connectivity is no longer purely
vertical. I4.0 components can communicate with one and other vertically, horizontally,
or diagonally. Due to layers being spread across the hierarchy, decision making is no
longer centralized. If information flow is flexible in this manner, then certain decision
making can be made on the factory floor rather than in an MES cloud. Meaning that
work centres, cells and stations could have a much higher level of autonomy. Reducing
1. Introduction 227

Figure 1: The RAMI4.0 3 perspective cube. Figure adapted from [2]

the reliance on centralized decision support will also reduce the software and networking
dependence in between physical work cells. Meaning if one work cell malfunctions, then
others may not suffer any performance degradation.
In addition, internal development teams may not be able to handle the complexity
and size of the automation solutions. Involving multiple vendors in this development
indicates that the networked systems may not all belong to a single owner. Equipment
as a Service may be a viable manner to distribute life cycle costs (design, development,
maintenance) of such systems. This means that security assumptions regarding privacy,
confidentiality and trust must be reviewed. Heavy integration costs and information
blockages must be avoided between multi-vendor systems. This means that each system
is able to establish trusted and confidential connections for exchanging data.
The final aspect and consideration, is that software technology evolves at a rapid
pace, much more quickly than traditional mechanical or electronic technologies. Software
development practices tend toward adoption of Agile [4] practices. This entails short
development cycles with small releases and fast deployment. The solution therefore must
support incremental change with minimized impact due to disruption of operations. In
addition software technology evolution means that the approach toward and treatment
of software technology must be one of embracing change. A successful solution must have
a clear path for change and ensure that it is not bound to a single technology that will
be expensive to change.
Service Oriented Architecture is one such architectural approach that tackles technol-
ogy abstraction through abstract messaging interfaces. However, IIoT systems are not
pure software, they involve hardware and this hardware will create physical dependencies
between services offered from the hardware. Hence, utilizing System Theory with ser-
vices being offered and utilized. Modular and independent systems, which can be pure
228 Paper I

software or embedded software, are modelled as black box with technology independent
service interfaces between one and other for communication and collaboration.

2 Challenges for Industrial Information Distribution

Industry want to have immediate access to data without high availability network infras-
tructure, rigid technology and specialist competencies. This means that domain special-
ists on the shop floor are able to access local data, changing data sources/routing without
IT specialists direct involvement. Both the tools for data access and the sources of data
can evolve independently. Both purely centralized and peer-to-peer solutions require de-
manding network support or sophisticated technology - resulting in specialist support.
In both cases making changes to the system will usually require IT competencies.
Therefore, a successful solution for decentralized information extraction would require
the following characteristics:

ˆ Communication protocol agnostic (supporting both push & pull)

ˆ Executable and human readable data flow specification

ˆ Supports fine grained access control

ˆ Not dependent upon a centralized data store

ˆ Data sources, processors and sinks are loosely coupled

The proposed solution is designed to satisfy the above requirements. The solution
proposes a software system that is able to process information queries and setup com-
munication routing to produce and move that required information. To achieve this, a
graph model is proposed. The graph model is required to support reasoning about the
System of Systems. The queries is also based on a standard graph query language Open
Cypher. The next section will introduce some related work before the presenting the
proposed solution in Section 4.

3 Related Works and Technologies

In this section some work related to the challenges is presented and some technologies
used within the solution are presented.

3.1 Mashups
The notion of mashups comes from mixing and combining the presentations of different
data sources into a single view. For example, Twitter and Facebook feeds can be viewed
on one page together with Instagram updates. This page would be a mashup of three
different social media sources. Another example Trendsmap [5] is a web application that
3. Related Works and Technologies 229

utilizes Twitter and Google Maps APIs to show Twitter activity by location. Meaning
that users are able to view what the trending Twitter topics are in particular cities. The
same can be applied for sensory data from multiple sensors or equipment KPIs along
with the current production orders.
Mashup approaches such as the one offered by ThingWorx [6] are a great example of
run-time based creation of information flow. ThingWorx is a business oriented platform
for connecting IoT devices with different protocols and routing data between devices.
Access to IoT data and so the visualization of related data from different sources is a
primary driver. The Web of Things have developed WoTKit [7] for building mashups.
The WoTKit targets web mashups and meaning that only web servers and clients are
used. However, these solutions both relay on centralization of either the data or the
distribution the data. Node-red [8] is Javascript based graphical environment that can
mash together different services into a composition. Its structure is similar to typed pipe
and filter diagrams. The IFTTT (“if this then that”) platform [9] is an IoT mashup
cloud that supports applets and services that can connect heterogeneous things together.
Using chained conditionals it is a strong commercial mashup platform.

3.2 The Arrowhead Framework

The Arrowhead Framework (AF) [10] is a supporting technology for the proposed solu-
tion. The AF is a Service Oriented Architecture (SOA)-based System of Systems (SoS)
integration framework. Within Arrowhead a System can provide and consume services
while systems themselves are hosted upon devices. This allows hierarchical meta-data,
split between devices, systems and services. Reusable services abstract the physical and
digital implementations. This means that applications can be specified using purely
functional models.
Arrowhead Systems are composed into SoS using orchestrations. A single system
can provide and consume multiple services and can interact with multiple other systems.
Interactions between systems can only occur through these services. For communication
between systems there must be matching service provider and consumer interfaces. The
AF documentation structure [11] defines how to specify a service interface.

3.3 Next Generation Access Control

Next Generation Access Control (NGAC) is a attribute based method of authorizing
user operations on objects [12]. By utilizing attributes NGAC enables fine grained access
control based on context. A user that is allowed to operate on an object in one location
may not have access from a different location, simply due to the user attribute association.
Alternatively, if the object is moved to a new location (its attributes change) then it may
reject operations from all users. Access control rules can then be expressed based on
context. Achieving such complex awareness becomes unmanageable with static rules.
Therefore NGAC uses a graph approach to describe the rules.
NGAC defines objects, users, attributes and operations as nodes within a graph. A
policy must then define the required attributes by a user to reach an operation and an
230 Paper I

object to reach the same operation. When evaluating the policy, the evaluation engine
must have reliable information regarding the current attributes of the object and the
user. Therefore a secure information extraction solution must work with context aware
security such as NGAC.

3.4 In-Network Processing

In-network processing a method treating the network as a database. As described by
Tannenbaum [13] the network will process raw data and aggregate or route the informa-
tion to the calling application. Using IEEE 802.15.4 wireless networks, TinyDB [14] can
perform in-network processing. TinyDB receives declarative queries that are passed down
the network mesh. Nodes that are capable of handling the request will process it and
pass the result back up the route. Each parent node of the mesh will aggregate the child
node responses. Meaning that there is a natural formation for query delivery and results
aggregation. In-network processing can also be applied to general computer networking.
Any network can act as a database that can be queried for information that may not exist
until the query is processed. One of the benefits of in-network processing are to avoid
network and memory overheads of centralizing data and information. Furthermore, by
aggregating and preparing results, it can help reduce the complexity systems consuming
data from multiple sources.

3.5 Information Centric Networking

Information Centric Networking (ICN) is a comprehensive approach toward in-network
processing. ICN proposes to move away from host addressing and to simply perform
named information queries [15]. This is highly suitable for many Web examples such
as video streaming and file sharing. Interest is expressed for information and regardless
of where the information exists the network will seek out the information and return
it to the caller. One of the issues with native ICN is in deployment, ICN overlays are
proposed so that ICN and TCP/IP can co-exist [16]. Service Centric Networking (SCN)
[17] extends ICN to include processing functions within the network infrastructure. This
recursive operation, information = f (information), turns existing information into new
information.

4 Proposed solution
The solution proposes a software System, called the System Composer, that is responsible
for receiving the information query, processing the query against the system, service and
access control graphs, formulating an information flow path and issuing the orchestration
rules for the SoS. The system that originated the information query receives instruction in
the form of an address that it can message to retrieve the result of the SoS. Alternatively
an MQTT topic could be provide publish/subscribe channel for the system to receive
information from. This is the simple interface required of the System Composer for the
5. A Graph analysis of Arrowhead Framework 231

proposed solution. The System Composer service interface is discoverable through a ser-
vice registry and its access rights is secured through an NGAC enabled Authorization
System. Internally the System Composer relies on a graph of the local scope systems,
service instances and types and access rights. This graph is based on the proposed graph
model presented in Section 5. It is built through interaction with Arrowhead; Device,
System and Service Registries; With Service Inventory and Authorization System. The
System Composer is stateless in terms that non of its memory is persistent. The graph is
rebuilt on startup and maintained by active communication with the mentioned Arrow-
head Framework core support systems. The information query is expressed in a graph
query using Open Cypher. The complete solution is presented in the following sections,
first a graph model of an SoS enabled SOA Framework, the Arrowhead Framework, is
presented.

5 A Graph analysis of Arrowhead Framework

The Arrowhead Framework has three basic entities Devices, Systems and Services. De-
vices are physical entities with processing and networking capability. Systems are soft-
ware entities that are hosted on devices and which provide and consume services. Services
are goal oriented interfaces which form the basis for collaboration between systems. Ser-
vices are then split into two entities;

ˆ The Service Description defines the goal of the Service, while

ˆ The Interface Design Description defines the Interaction Pattern, Communications

Protocol, Data structure and semantics of the Service.

As shown in Figure 2 these four entities form the base vertices of the Graph.

Figure 2: Building blocks of the Arrowhead Framework

The Arrowhead Framework Uses these primary blocks to builds Systems of Systems
(SoS). To build an SoS, the primary entities must be related to one another. These
relations are:

ˆ A System is Hosted By a Device

ˆ A Service Description is Provided By a System (Provider)

232 Paper I

ˆ A Service Interface is Offered By a System (Provider)

ˆ A System (Consumer) Requires a Service Description

ˆ A System (Consumer) Supports a Service Interface

ˆ A Service Description is Implemented By a Service Interface

With this set of edges, it is possible to construct a simple path between systems.
For example the path can be functional using Required and Provided By edges, or a
communications path based on Supported and Provided edges. A sub-graph with a
functional path is illustrated in Figure 3. Here it can be seen that the path makes up a
bipartite graph. There are only edges between the two sets of System nodes and Service
Definitions. There are no edges within either set.

Figure 3: A first System of Systems: A functional path between two system

A functional specification of the System of Systems can be formed by looking only at

the function graph. This can be formalized by:

G={V,E}

Where

V={A,B}

Where A = Set of Systems

and B = Set of Service Definition

| B| = | A| − 1

E=B*2
5. A Graph analysis of Arrowhead Framework 233

The communications graph is built from a set of participant Systems and Service
Interfaces. To build this graph the Service Definition is used as an anchor to the Service
Interfaces. In this way it is possible to move from the functional sub-graph to the
communications sub-graph.
This sub-graph can be seen in Figure 4.

Figure 4: A first System of Systems: A bipartite communication graph with two participant
Systems.

The communications graph is a concrete interaction path between participant Sys-

tems. A formal description of the communications path can be seen in Figure 5. It can
be used to validate the interaction path.
Based on the graph definitions above it can be deduced that a graph is disconnected
when ever the number edges is less than | B| then one or more required Service Definitions
are missing and the SoS is incomplete. If the SoS is incomplete then the SoS must be
re-engineered. Either requirements can be modified for example remove unattainable
requirements. Or adding new systems and enhancing existing systems with additional
services.
Dynamic bridging is may be an alternative to full re-engineering. For example, where
a missing edge is detected within the communications graph, this indicates that a service
interface mismatch exists. In this case a translator can be injected to the graph. Dynamic
systems are discussed in detail in Section 7.1. Once a communications graph has been
established. The security authorization path must be checked.

5.1 NGAC graph entities

The connection to the Arrowhead entities. Access control is a core element to secure SoS.
Access control means restricted System interaction (Service Exchange) to only authentic
and authorized peers. Next Generation Access Control (NGAC) [12] is a standard that
utilizes graph based attribute based access control (ABAC). The NGAC entities can be
seen in Figure 6. To use NGAC with Arrowhead, either an equivalence or a relation
must be formed between the two models. Here, it is proposed that a relation mapping is
formed:
234 Paper I

G={V,E}

Where

V={A,B}

Where A = Set of Systems

and B = Set of Service Interfaces (with matching shared Service Definitions)

| B| = | A| − 1

| E| = | B| ∗ 2

or

| E| >= (| A| − 1) ∗ 2

therefore

| V | = | A| ∗ 2 − 1

Figure 5: A complete communications graph is formalized.

Figure 6: The NGAC graph entities/nodes. User as ‘usr’, Attribute as ‘attr’, Object as ‘obj’
and Operation as ‘op’.

ˆ An User entity -aliases- a consuming System

ˆ An Object entity -represents- a providing System

ˆ An Operation entity is -defined- by a Service Interface

Figure 7 shows the graph relations between the Arrowhead Framework nodes and the
NGAC nodes. Utilizing the system nodes and the service interface node as anchors the
related User, Object and Operation nodes can be found. Next a path must be traversed
between the two systems through only the NGAC graph nodes.
A complete path, in the NGAC graph, between the systems means that the required
attribute permissions exist for the consuming system to invoke the operation upon the
6. Building Systems of Systems using graph queries 235

Figure 7: NGAC: An authorization path between two system

providing system. With these three primary graphs it is possible to build validated and
secured Systems of Systems. The next section will discuss the method of utilizing the
graph models.

6 Building Systems of Systems using graph queries

To build a SoS there are multiple steps. The fundamental steps are; 1) Specification, 2)
Validation, 3) run-time binding. The proposed solution can be used in each step.

Specification In order to specify a System of System, an engineer must be able to

express the expected Systems and their expected interconnections. This can be expressed
naturally using the already defined vertices and edges. Using set theory, it is also possible
to generate the mathematical representation of the SoS from the graph.

Validation Utilizing the mathematical representation, it is possible to create a list of

components which are expected to be present. This list can be used to provide accurate
requirements to engineers and developers. It can also be used to check if a specification
can be met by currently available systems.

Run-time Orchestration When the SoS specification is run against a live graph, the
service exchanges between systems are identified. These identified service exchanges are
what generate the Orchestration rules.

6.1 Building the Graph

The data required to build the graph is distributed amongst different data sources. To
perform the graph queries, the data must be gathered together and sorted into a graph
database. The data is collected from:
The data stored in each of these systems must be retrieved through service interfaces.
In the case outlined in this paper, the interfaces are defined by the Arrowhead Framework.
These interfaces could be defined according to any preferred approach.
236 Paper I

Table 1: Caption text here

Data Store Description
Device Registry The Device Registry contains
records of each device and as-
sociated meta-data, such as
physical location and operat-
ing conditions.
System Registry The System Registry contains
records of each system and as-
sociated meta-data, such as
the physical device it is hosted
on and what are the systems
provide capabilities and con-
sume requirements.
Service Registry The Service Registry contains
records of each service avail-
able (live) for consumption and
associated meta-data, such as
the responsible system, a ref-
erence to the service type and
operating capabilities.
Service Inventory The Service Inventory contains
records of each service defini-
tion and possible service inter-
face combinations.
Authorization System The Authorization System
contains the access control
Policy Information at a role or
attribute level.

A full graph of the local cloud can be maintained by synchronizing the different data
stores. The graph data does not change rapidly as there is limitations on how quickly a
Cyber Physical System can be relocated. Depending on the protocol choice this could
require polling the data stores for changes. However, to avoid this, the data stores can
notify the graph that a change has occurred, allowing the pull from data sources to be
event driven. At no stage does the graph make changes to the data stores. Therefore
there is no write synchronization required.

As described in the next section, the queries made to the graph are only read based,
changes such as injection of local bridges are either not reflected in the data stores, or
are changed through the normal Device, System, Service registration.
6. Building Systems of Systems using graph queries 237

Figure 8: System composer interaction with Arrowhead Framework core systems

6.2 Querying the Graph

The whole value of the graph model is the ability query the connected data. Before
diving into the mechanisms, the objective of the query can be listed as:

1. To find a functional path connecting a set of Systems

2. To find a communications path connecting a set of Systems

3. To find an authority path connecting a set of Systems

4. To identify a System based on its relations

Function Path

To find the function path a combination of known Systems and Service interfaces must be
anchored (known). The information which makes up this graph is formed using System
Registry and Service Inventory data. Given a start System, an ordered set of services
and an end System, a functional path can be built using the following query:

MATCH p = (a1 : System{name : “a”})

− [:RequireskOffered by*]− >
(a2 : System{name : “e”}) RETURN p (1)

The query will return all nodes and relations which create the path. The nodes will
consist of a set of Systems and a set of Service Definitions. These two sets are then used
to find the communications path.
238 Paper I

Communications Path
To find the communications path, each of the Service Definition and System pairs must
form into a triple with the interface definition. A consuming system supports an interface
definition and the interface definition is offered by a providing system. Hence, Figure 9
shows the result of query in Equation 2 that is the consuming triple.

MATCH
(a : System{name : “a”}) − [:Supports]− > (inst),
(inst) − [:Implements]− > (b:Type{name : “b”})
RETURN inst; (2)

Figure 9: Interface Matching for a Consumer system and service

Figure 10 shows the results of the provider side query shown in Equation 3.

MATCH
(a : System{name : “c”}) < −[:Provided by] − (inst),
(inst) − [:Implements]− > (b:Type{name:“b”})
RETURN inst; (3)

A connected communication path will match to a single interface definition that means
the two systems utilize interoperable services. This is shown in Figure 4 which has a
connected functional path between the two participant Systems. Where a communication
path is not found the resulting graph will look like Figure 11. In this situation, the System
of Systems can be adapted to created a connected communication path.
This case is covered in the system composer implementation section that uses the
translator to inject a local bridge and create a communications path.
7. Implementation of a System of System Composer 239

Figure 10: Interface Matching for a Provider system and service

Figure 11: A broken communications path is shown here. The two systems support and provide
different interface designs for the same service definition.

Authority Path
Using the Arrowhead and NGAC relationships, it is possible to shift from the functional
domain to the security domain. An attribute path must be found between the providing
system and consuming system which passes through the operations which represent the
service instances. The attribute designation is made through a secure management portal.
The requirement for attribute policy is also made through a secure management portal.
Hence, this part of the graph is certainly read only. A break in the security path will
mean that there is no authorization for service exchange between the participant systems.
The graph query to find the security path must first identify the nodes which represent
the users, objects and operations. Using the definitions from Figure 7 each interface
definition is mapped to a single NGAC operation. A System can have at most one
NGAC Object and one NGAC User.
A path must be found between the User that is the Alias of the Consuming System,
the Operation defined by the Interface Definition, and the Object represented by
the Providing System. Figure 7 shows a generic example of the authorization path.

7 Implementation of a System of System Composer

It is now possible to use the Graph Model and SOA-based data stores to design a solution
that in (soft) real-time will compose the distributed functionality across the Systems of
240 Paper I

Systems.
The System of System Composer is a tool which can be utilized by technical and non-
technical users to compose new Systems combinations. It uses the processes described
to build a graph of the device, system, service and security landscape.
To navigate the graph an anchor needs to be located. This is the starting system and
is usually found through direct reference to a unique system name, or through a unique
combination of meta-data connections.
The System Composer is itself a participant in an SoS. Figure 8 draws the SoS required
to generate the graphs. The graphs are kept up to date through polling and notifications
between the Systems in Figure 8.
Utilizing Open Cypher, the responses from the data stores are stored in a Neo4J
instance using the following queries:
A merge operation is used to avoid creating duplicate nodes and edges. Device data
can have many nodes connected to it. They represent connected meta-data. For example
a device is connected to a location. This can be physical location or logical location. No
edge label is required for meta-data nodes.

MERGE
(dev:Device{name:’A8076’}),
(loc:Location{name:’A2301’}),
(dev) − − > (loc),

System data will include an association to a device which is hosting the system, the
services provided and consumed from the system and any connected meta-data associated
with the system. A sample Cypher command to store the minimum System data:

MERGE
(sys:System{name:’Nutrunner’}),
(jobServiceOffering:ServiceType{name:’EquipmentJob’}),
(notifier:ServiceType{name:’EquipmentNotification’}),
(ins:ServiceInstance{name:’showworkorder-presenter’}),
(jobServiceOf f ering) − [:OFFERED BY]− > (sys),
(notif ier) − [:OFFERED BY]− > (sys),
7. Implementation of a System of System Composer 241

To store the security attribute data and policy data:

MERGE
(ser:ServiceType{name:’showworkorder’}),
(sys:System{name:’presenter’}),
CREATE
(ins:ServiceInstance{name:’showwork’}),
(sys) − [: OF F ERS]− > (ins),
(ins) − [: OF ]− > (ser);

To store the Service data (resulting from a query to the Service Inventory):

MERGE
(sd:ServiceDefinition{name:’EquipmentJob’}),
(idd:ServiceType{name:’EquipmentJobCoAP’}),
(sd) − [:IMPLEMENTED BY]− > (idd)

Once the data has been collected and the graph has been built, the graph can be
presented to a user to graphically draw the SoS composition.
The service interconnections are primarily and in some cases exclusively used to spec-
ify the SoS. This is achieved by querying the graph with a MATCH query like below:

When only validating the possibility of an SoS intermediate and end systems can be
left anonymous. Otherwise, while generating the SoS, the path query should return the
intermediate and end systems. It is possible that alternative paths exist, meaning that
there are alternative service providers which can satisfy the specification. In the next
stage, concrete SoS compositions can be generated by looking at the communications
path. To do this, shared service interfaces that implement the service definitions must
be found between the intermediate systems. The queries defined earlier are used to build
the communications path:
So long as the consumer service interface ”consumerInt” and the provider service
interface ”providerInt” are the same node, then this means that the communication path
is valid. Building the communications path means performing this query on each pairing
of Systems found from the functional path query.
The communications path is complete once the bipartite graph connecting systems
and service interfaces has been formed. In case that the communications bipartite graph
242 Paper I

MATCH
(sys:System{name:”}),
(sd-1:ServiceDefinition{name:”}),
(sd-2:ServiceDefinition{name:”}),
(sd-3:ServiceDefinition{name:”}),
(sys) − [:Requires]− >
(sd-1) − [:Offered By]− >
(sys-1:System) − [:Requires]− >
(sd-2) − [:Offered By]− >
(sys-2:System) − [:Requires]− >
(sd-3) − [:Offered By]− > (sysEnd:System)
RETURN
sys-1, sys-2, sysEnd

Figure 12: Query to generate an SoS functional path

is disconnected it can be assumed that there are at least two systems whom do not share
a common service interface for the same service definition.

Once a valid communications path has been found the System composer checks for
a valid authorization route between participant systems using those service interfaces.
Recalling the NGAC mapping utilized in Section 6, each service interface is mapped to
an NGAC operation, and each system is mapped to a user and/or object depending on
if it will consumer or provide the operations. The Authorization path can be built using
the query in Figure 14. The providing system, consuming system and operation nodes
are inputs to this query.

This query must be executed for each System pairing within the SoS. If the path
is successfully returned then a valid authorization can exist between the participant
systems.

The System Composer now is able to create and validate Functional SoS, check for
communications path, and valid authorization permissions.
7. Implementation of a System of System Composer 243

MATCH
(sysCon)-[:Supports]->(conInterface),
(conInterface)<-[:Implemented By]-(sd),
MATCH
(sysPro)<-[:Provided By]-(proInterface),
(proInterface)<-[:Implemented By]-(sd)
WHERE
(conInterface = proInterface)
RETURN
conInterface, proInterface

Figure 13: A query to identify the matching service interface, given a common service definition
and two participant systems.

M AT CH
(sysP rov) − [: Represented By]− > (obj : Object),
(sysCons) − [: Has Alias]− > (usr : U ser),
p = (usr) − [: ∗]− > (opr) − [: ∗]− > (obj)
RET U RN p

Figure 14: Build and check for authorization path for a given set of communications paths

7.1 Dynamic Systems and their composition

The System Composer is able to compose Systems to create new behaviours. So far
all Systems have been static, meaning that they have fixed sets of service interfaces
(functionality). Only the combination of usage of these systems has been dynamic.
Dynamic systems are defined as Systems that are able to provision and consume new
service interfaces at run-time. On a continuum, a system on one side can be highly static
or up to fully dynamic - likely using machine learning. Rigidity is both functional and
operational. Proposed here are systems that are able to provision new service interface
instances of a fixed type but with independent configurations and life-cycles.
Dynamic systems as introduced in [18], provide a provisioning service interface. Re-
244 Paper I

quests to this service result in a new function being created within the host System.
The new function has service interfaces as required to match the specific requirements.
For example, a ”consumer” service interface for initiating ”pull” based interaction or a
”provider” service interface to ”push” based interaction.
These dynamic systems are able to create local bridges, connecting disconnected
graphs. However, to allow interoperability between dynamic systems from different de-
velopers the interface for provisioning must be standardized. In [18] the dynamic systems
were provisioned using SenML. Hypermedia As The Engine Of Application State (HA-
TEOAS) is a method of creating dynamic APIs for web based applications. IETF Stan-
dards RFC 5988 [19] and 6690 [20] introduce web linking as a way to achieve HATEOAS
applications. In addition IETF draft ”draft-ietf-core-interfaces-10” [21] (core-interfaces)
builds on RFC 6690 to propose a set of standard interfaces for interacting with URI
resources. Here we refine the provisioning interface of [18] to utilize the core-interfaces
specification. The specification handles link-format and SenML for parametric values.
The Translation System proposed in [22] is an example of a dynamic system. The
Translation System is reliant on protocol information and link information to build the
protocol translator. The dynamic provisioning request is shown in Figure 15.

[
{
”name ” : ” dynamic ” ,
” value ”:” t r a n s a l t o r ”
},
{
”name ” : ” s e r v i c e ” ,
” v a l u e ”:” < coap : / / { i p : p o r t }/ s e r v i c e i n s t >;
r t =[ s e r v i c e t y p e ] ;
i f =[ i n t e r f a c e t y p e ] ;
c t =[ c o n t e n t t y p e ] ;
r e l=p r o v i d e r ”
},
{
”name ” : ” s e r v i c e ” ,
” v a l u e ”:” < mqtt : / / { i p : p o r t }>;
r t =[ s e r v i c e t y p e ] ;
i f =[ i n t e r f a c e t y p e ] ;
c t =[ c o n t e n t t y p e ] ;
r e l=consumer ”
}
]

Figure 15: A run-time request to the Translation System to provision a new protocol translator.
With this information, the translator is able to provision two interfaces that will interoperate
with the specified interfaces.
7. Implementation of a System of System Composer 245

Once the protocol translator and its interfaces have been provisioned the response is
sent. It contains link information, linking to the newly constructed service interfaces.
The translator’s response is provided in Figure 16.

[
{
”name ” : ” r e s p o n s e ” ,
” value ”:” t r a n s l a t o r ”
},
{
”name ” : ” s e r v i c e ” ,
” v a l u e ”:” < mqtt : / / { i p : p o r t }/ t o p i c >”;
r t =[ s e r v i c e t y p e ] ;
i f =[ i n t e r f a c e t y p e ] ;
c t =[ c o n t e n t t y p e ] ;
r e l=p r o v i d e r
}
]

Figure 16: The response from the Translation System once a new protocol translator instance
has been provisioned. Only the new provider interface is shown, this should be compatible with
the specified consumer interface to be satisfied.

Dynamic Systems are able to bridge two Service providers that require information
exchange. Because service providers cannot pro-actively seek out service consumers, then
two Providers or two consumers cannot communicate and so must have a bridging. In
this case rather than a translator a dynamic ”proxy” can mediate the two Systems. The
dynamic systems being HATEOAS based systems themselves will return links to the
dynamic resources from the base URI of the service. The IETF core-interfaces draft
defines Sensor, Actuator and collection interfaces. However, in the case of configuring or
provisioning dynamic systems more complex configuration parameters are required. Here
we propose to add a new interface that will enable transport of provisioning information
for dynamic systems. The interface which is based on the example above, includes SenML
with Link-Format embedded as an element. Link-Format items can point to external
definitions or service endpoints. The ”rel” element is used to understand the meaning of
the link.

A general definition and mapping of Link-Format attributes to describe services (core

and application) in the Arrowhead Framework is shown in Figure 7.1. The items within
the inner square brackets represent variable values, whiles the rel values must be kept
as ”provider” or ”consumer”. All attributes (rt,if,ct,rel) must be present in the interface
Here the servicetype refers to service type definition held in the service inventory.
246 Paper I

<coap : / / { i p : p o r t }/ s e r v i c e i n s t a n c e >;
r t =[ s e r v i c e t y p e ] ;
i f =[ i n t e r f a c e t y p e ] ;
c t =[ c o n t e n t t y p e ] ;
r e l=p r o v i d e r ,

Figure 17: Core link format used to describe service interfaces.

8 Demonstration of proposed solution

The user case demonstrating the proposed solution is a typical industrial manufacturing
environment. During the manufacturing of large industrial vehicles, there are safety
critical nuts that must be tightened according to a quality specification. The nuts must
be tightened to a specific torque value and then rotated through a fixed angle after that.
The target values along with actual values are archived for legal purposes. In case of
accident with the vehicle this data can be reviewed to remove manufacturing problems
as source of the accident. A station in the hierarchy level must satisfy all the layers.
This means present at a station, are business layer down to asset layer. Applications
operating at different layers will then make use of the services at different layers.
In this use case, a production engineer working on the shop floor will compose a
new SoS to extract process information. Flexible process information extraction is a key
enabler to production optimization. Doing information extraction at the edge reduces
loading on centralized infrastructure. However, edge based applications that do not
collect mass information, as centralized systems do, are relatively more difficult to adapt
to changes in information requirements. As proposed in [18], Derhamy et al. propose an
information extraction based on graph model and queries. Extending this concept, this
work has applied the System Composer to an information extraction and communication
problem.
An operator on the production line has noticed a quality assured nut tightening is
reducing station efficiency due to ergonomic reasons. Operators at the station are not
able to access the tightening nut with ease and this reduces speed. In consultation with
production engineers it is agreed to collect operational information of the Nutrunner at
the station. Each time the electronic Nutrunner is used it publishes notifications with its
specification and actual data. Each of these notifications are individually time stamped.
In order to measure operational efficiency of the specific equipment, the notifications
must be captured, stored and processed (i.e. KPI information presented to operators
and engineers).
There are a number of ways that the nut runner KPI problem could be tackled:
1) A custom application could be written that subscribes to the Nutrunner notifications
and stores the results. The application would then provide a set of fixed operations upon
the data that would perform the required processing.
2) Alternatively, an SoS can be composed to capture the Nutrunner data in a re-usable
storage system. Then - on-demand - the data can be passed through re-usable filter
8. Demonstration of proposed solution 247

systems to perform the required processing.

Option one requires an Information Technology (IT) engineering expertise to pro-
duce a customized application. The IT engineer must consult with domain engineers
(production engineers) to understand the requirements. Then implement the application
according to the requirements. It is possible to utilize re-usable design methods to opti-
mize the development time. However, if there are any change to processing requirements,
this will likely require the IT engineers expertise to re-implement or make the change.
This means increased lead times and costs for implementing the information extraction.
There are however advantages that there may be very little needed in terms of existing
infrastructure. Where as with option two, IT engineering expertise may not be required.
Domain engineers are able to interact with a tool on the factory floor, expressing their
information (KPI) requirement. The IT competency is required to setup the system, but
the generation of KPI information is completely in the hands of the domain engineers.
This section describes the option two implementation. Firstly, the domain engineer
uses the System Composer to specify the information extraction requirement. Figure 18
shows this specification in the Open Cypher language.

MATCH
(sysCon)-[:Supports]->(conInterface),
(conInterface)<-[:Implemented By]-(sd),
MATCH
(sysPro)<-[:Provided By]-(proInterface),
(proInterface)<-[:Implemented By]-(sd)
WHERE
(conInterface = proInterface)
RETURN
conInterface, proInterface

Figure 18: The information capture, storage and processing requirement specification.

The new System composition is routing new data from the Nutrunner system through
a time calculator and storing the results in a Historian System. The GUI system is able
to get the latest up-to-date KPI from the Historian at any time.
The System Composer takes this specification and builds functional, communications
and authorization graphs. Once the different paths are validated, the validated commu-
248 Paper I

nications path is turned into composition rules that can be used by the Orchestration
engine. The new SoS has not impacted operation of the station. It is running along
side normal SoS compositions that achieve the station objectives. Collecting data in
this manner can occur over any period of time. After a suitable length of time, the
information can be used to report an inefficient process.
The suggested process change is to move the task using the Nutrunner to the next
station. Existing tasks at the next station involve orienting the product in a manner
which provides more ergonomic access for Nutrunner operation. Hence, human operators
are able to complete the task more quickly and efficiently. The SoS Composition can be
moved to the next station. Allowing an identical data collection and processing and
comparison of the change. This is all done at the edge, meaning that local scope changes
to SoS compositions do not impact centralized infrastructure.

9 Conclusion
This paper has presented a method to dynamically extract information from its source
within an IIoT network. Based on the usage of SOA and SoS theory, it is possible to 1)
visualize the IIoT as graph elements, 2) express an information query in the form of data
sources, transformations and sinks and 3) orchestrate communications paths and dynamic
bridges. The proposed graph model is used to capture the SOA and SoS elements. With
separation between functional and communications specifications allowing for multi-level
reasoning on requirements. Where a graph path is disconnected between collaborating
systems a local bridge can be injected, thereby connecting the systems. The local bridge
could be a translator (at the communications path), or a data manipulator (i.e. filter
or trigger) (at the functional path). The graph model goes further to map the SoS and
SOA architecture to the NGAC model. Thus enabling the proposed System Composer
to evaluate access control policies to ensure that SoS will be attainable under current
conditions. The System Composer does not hold the NGAC policy information, but is
allowed to query the policy information and retrieve the rules. The System Composer
retrieves the System context information and thus has the knowledge required to reason
upon likely hood of access being granted between systems. The System Composer can
be interacted with by domain engineers on the factory floor. Enabling them to modify
or create KPI algorithms at run-time without support of IT specialists. The results
show dynamic manipulation or creation of an SoS during run-time by domain engineers.
Because of local information access, centralized datastores (warehouses) are avoided.

Acknowledgement
The authors would like to thank the Far-Edge and Productive projects for support in
conducting this research.
References 249

References
[1] B. Scholten, The road to integration : a guide to applying the ISA-95 standard in
manufacturing. Research Triangle Park NC : ISA, 2007.
[2] “Reference Architectural Model Industrie 4.0 (RAMI4.0) - An Introduction,”
April 2018. [Online]. Available: https://www.plattform-i40.de/I40/Redaktion/EN/
Downloads/Publikation/rami40-an-introduction.pdf
[3] “Structure of the Administration Shell,” April 2018. [Online]. Avail-
able: https://www.plattform-i40.de/I40/Redaktion/EN/Downloads/Publikation/
structure-of-the-administration-shell.pdf
[4] A. Cockburn and J. Highsmith, “Agile software development, the people factor,”
Computer, vol. 34, no. 11, pp. 131–133, Nov 2001.
[5] “Real-time twitter trending hashtags and topics.” [Online]. Available: https:
//www.trendsmap.com/
[6] Thingworx industrial innovation platform - ptc. PTC. [Online]. Available:
https://www.ptc.com/en/products/iot/thingworx-platform/
[7] M. Blackstock and R. Lea, “Iot mashups with the wotkit,” in 2012 3rd IEEE Inter-
national Conference on the Internet of Things, Oct 2012, pp. 159–166.
[8] “Node-red.” [Online]. Available: https://nodered.org/
[9] “Ifttt helps your apps and devices work together.” [Online]. Available:
https://ifttt.com/
[10] J. Delsing, Ed., Arrowhead Framework: IoT Automation, Devices, and Maintenance.
CRC Press, 12 2016. [Online]. Available: http://amazon.com/o/ASIN/1498756751/
[11] F. Blomstedt, L. L. Ferreira, M. Klisics, C. Chrysoulas, I. M. de Soria, B. Morin,
A. Zabasta, J. Eliasson, M. Johansson, and P. Varga, “The arrowhead approach for
soa application development and documentation,” in IECON 2014 - 40th Annual
Conference of the IEEE Industrial Electronics Society, Oct 2014, pp. 2631–2637.
[12] V. Hu, D. F. Ferraiolo, D. R. Kuhn, R. N. Kacker, and Y. Lei, “Implementing and
managing policy rules in attribute based access control,” in 2015 IEEE International
Conference on Information Reuse and Integration, Aug 2015, pp. 518–525.
[13] A. S. Tanenbaum and M. v. Steen, Distributed Systems: Principles and Paradigms
(2Nd Edition). Upper Saddle River, NJ, USA: Prentice-Hall, Inc., 2006.
[14] S. R. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong, “Tinydb:
An acquisitional query processing system for sensor networks,” ACM Trans.
Database Syst., vol. 30, no. 1, pp. 122–173, Mar. 2005. [Online]. Available:
http://doi.acm.org/10.1145/1061318.1061322
250

[15] M. A. Yaqub, S. H. Ahmed, S. H. Bouk, and D. Kim, Information-Centric Networks

(ICN). Singapore: Springer Singapore, 2016, pp. 19–33.

[16] G. Xylomenos, C. N. Ververidis, V. A. Siris, N. Fotiou, C. Tsilopoulos, X. Vasilakos,

K. V. Katsaros, and G. C. Polyzos, “A survey of information-centric networking
research,” IEEE Communications Surveys Tutorials, vol. 16, no. 2, pp. 1024–1049,
Second 2014.

[17] T. Braun, V. Hilt, M. Hofmann, I. Rimac, M. Steiner, and M. Varvello, “Service-

centric networking,” in 2011 IEEE International Conference on Communications
Workshops (ICC), June 2011, pp. 1–6.

[18] H. Derhamy, J. Eliasson, J. Delsing, and J. van Deventer, “In-network processing

for context-aware soa-based manufacturing systems,” in IECON 2017 - 43rd Annual
Conference of the IEEE Industrial Electronics Society, Oct 2017, pp. 3460–3465.

[19] M. Nottingham, “Web linking,” Internet Requests for Comments, RFC Editor, RFC
5988, October 2010, http://www.rfc-editor.org/rfc/rfc5988.txt. [Online]. Available:
http://www.rfc-editor.org/rfc/rfc5988.txt

[20] Z. Shelby, “Constrained restful environments (core) link format,” Internet Requests
for Comments, RFC Editor, RFC 6690, August 2012, http://www.rfc-editor.org/
rfc/rfc6690.txt. [Online]. Available: http://www.rfc-editor.org/rfc/rfc6690.txt

[21] Z. Shelby, M. Vial, M. Koster, C. Groves, J. Zhu, and B. Silverajan,

“Reusable interface definitions for constrained restful environments,” Working
Draft, IETF Secretariat, Internet-Draft draft-ietf-core-interfaces-11, March
2018, http://www.ietf.org/internet-drafts/draft-ietf-core-interfaces-11.txt. [Online].
Available: http://www.ietf.org/internet-drafts/draft-ietf-core-interfaces-11.txt

[22] H. Derhamy, J. Eliasson, and J. Delsing, “Iot interoperability - on-demand and

low latency transparent multi-protocol translator,” IEEE Transactions on Services
Computing, 2016, submitted.
 Paper J
Software Architectural Style for the
Industrial Internet of Things

Authors:
Hasan Derhamy, Jens Eliasson and Jerker Delsing

Reformatted version of paper submitted to:

Journal paper, IEEE Internet of Things Journal, 2018

© 2018, IEEE, Reprinted with permission.

251
252
 Software Architectural Style for the Industrial
Internet of Things

Hasan Derhamy, Jens Eliasson and Jerker Delsing

Abstract

Technology has played a key role as an enabler of societal evolution and revolution.
Mechanical machines of the 1800’s gave way to electromechanical machines, then to
software controlled machines and now to fully digitized and networked machines. Rec-
ognizing the fourth industrial revolution, the Industry 4.0 initiative was founded to help
bring about and drive the desired change. Computing ubiquity, interconnectedness, intel-
ligence and delegation drive Industrial Internet of Things applications complexity. The
result is fragmented software intensive nodes that often lack a strong architectural ba-
sis. Within IIoT, software solutions are distributed over heterogeneous communications
links, computing platforms and business domains. These platforms are likely physically
separated over large distances, or could be highly mobile with large variations in security
requirements. The scale of complexity has been unseen previously in the software in-
dustry. A lack of architectural rationale will lead to brittle IIoT applications that resist
evolution.
This paper proposes an architectural style for the Industrial Internet of the Things. It
develop a set of principles that guide architects and designers to decompose and create re-
silient, secure, scalable and evolutionary IIoT applications. Several views are introduced
with reference to concerns derived from the Reference Architecture Model for Industry
(RAMI) 4.0. The style and principles are not created in vacuum and primary source of
inspiration is referenced to Systems of Systems theory and Service Oriented Architecture.
The architectural style creates clear traceability from business and domain requirements
through to device integration.

1 Introduction
The world of computer systems, including software and hardware, are in a constant state
of evolution. Software is taking a growing proportion of the responsibility for overall
system functionality. It has become a crucial tool of almost all industries from bakeries
and arts to manufacturing facilities and banking. As systems gain more software aspects,
managing software complexity and heterogeneity becomes a key challenge [1]. In the late
80s and early 90s, it was already well recognized that software architecture is a key
to manage the complexity. This early work laid the foundation for the area of software
architecture [2]. It is the study of the form and structure of software abstractions intended

253
254 Paper J

to satisfy a set of functional and non-functional requirements. The field of software

architecture has grown and been applied to many different computing challenges.
The Industrial Internet of Things (IIoT) is a logical consequence of widely present
computing ubiquity and interconnectedness. With simple beginnings in 1982, Carnegie
Melon University researchers connected a vending machine to the Internet allowing staff
to query the state of Coke in the machine (empty, warm, cold) before walking to buy
a soda [3]. In this sense, the vending machine was the first Internet of Things (IoT)
device. It was actively sensing its environment, storing and processing information and
had connectivity to its stakeholders. The current trend of IIoT (digitalization) seen
around the world is an evolution of this simple scenario.
Technological and business opportunities and challenges within IIoT have given rise
to a “zoo” of software solutions. In this context “zoo” implies variations of size, domain,
methods, forms (web app, smart app, libraries, tools), solutions and more.
Computing domains in industry are commonly grouped into operational and informa-
tion technology (OT, IT). They are shaped by different requirements and environments:
one focused on control (automating mechanical processes), real-time requirements, relia-
bility and long life cycle etc.; the other focused on data management, process efficiency,
business process management and frequent updates etc. This impacted the development
of hardware, communications and software. The hardware platforms acquired different
properties. In OT domain, deterministic operation is built into the platform that is also
tightly integrated to the physical world (e.g. A2D/D2A and DIO ports). In IT domain,
platforms are high performance processors with large memory and storage. Similarly,
communication networks diverged in their properties. In OT domain, communication
commonly is deterministic, requires hard wired addressing and secured by dedicated or
restricted access. In IT domain, Internet stack is the prevailing communication stan-
dard, with global addressing scheme, high bandwidth links and cryptographic security.
Software development specialized to address demands of each domain. In OT domain,
software emphasises deterministic modelling abstractions, machine-to-machine interac-
tions and robustness. In IT domain, software realizes high level abstractions, reconfig-
urable functions and human oriented interfaces. This resulted in differences in software
properties, capabilities, scale, tools, technologies and competencies etc. i.e. architectural
approaches. Software within the two domains are put together to achieve overall business
goals. The IT and OT have divided ISA 95 [4] pyramid and taken responsibility for the
different levels (Figure 1). OT is at the manufacturing control layer (level 1 and 2), re-
sponsible for sensing, manipulating, monitoring and controlling the production processes
etc. IT is at the manufacturing operations, business planning and logistics layers (level
3 and 4), responsible for plant scheduling, inventory management, logistics, workflow,
recipe control etc.
The IT domain is well suited to handling highly complex applications. There are al-
ready many standard patterns and approaches handling such complexity; such as, event,
data, or service based architectures. Testing and development methods are also more
mature, with Agile [5] and Continuous Delivery techniques originating in IT/Web soft-
ware development. However, the larger the spatial distance from the physical world the
1. Introduction 255

Figure 1: ISA 95 pyramid with Information and Operational Technology domains shown.

computing moves, the more challenging deterministic and real-time operation becomes.
Alternatively, OT can help with achieving determinism and real-time operation require-
ments. Modern Micro-Controllers (MCU) can handle a greater share of the processing
power requirements. This means that some IT logic is moved to the OT domain. How-
ever, there is still a strong reliance on a centralized IT infrastructure, creating a single
point of failure and bottleneck. Additionally, traditional OT methods are not yet as
capable of handling large scale complexity when compared with IT software architecture.
Thus, these solutions rely on vertical integration with clear separation between IT and
OT. It is becoming more clear that this separation of concern, due to technology domain,
rather than functionality could be reduced or broken. A third option is to combine IT
and OT domains - becoming IIoT [6]. This pushes away the pure vertical integration
and forces a holistic view of the software architecture on each node. Key enabling tech-
nologies and methods for this change is in Edge Computing [7][8] and Cyber Physical
Systems [9].
What is software architecture and how does it impact implementation design? Un-
constrained software can satisfy most user requirements in an almost unlimited number
of ways [10]. Software is made through a series of design decisions until implementation
in code. Each design decision made adds some constraint to the solution and each con-
straint cascades to downstream design decisions. Early decisions, such as architectural,
are fundamental; they have a higher and longer lasting impact on the overall solution.
Hence, software architecture should only constrain critical (load bearing) parts of the
solution [2]. Thereby allowing as much flexibility for solution variation as possible. This
practice is in-line with development methods such as Agile [5].
Software design has major implications on migration toward Industry 4.0 [11]. When
designing new solutions legacy systems, infrastructure and procedures must be taken into
account. For industrial automation this is acute because some equipment life-cycles can
span well over a decade. Software architecture must be designed to accommodate legacy
technology. On the other hand, software architecture must also minimize constraints
on how future (unknown) technologies can be integrated. Therefore, the software ar-
256 Paper J

chitecture will evolve while the components may have limited ability to change. As the
architecture adapts for changing requirements and technologies, its architectural style
acts as a anchor through this evolution. An architectural style is the language or man-
ner in which the concrete architecture is described. It utilizes abstract concepts that
communicate intent and rationale of the software architecture. What would a suitable
architectural style be for engineering IIoT applications?
The paper is structured as follows: Section 2 discuss software architecture practice.
Section 3 gives overview of advanced architectural styles. Section 4 and 5 discusses
principles of system theory and Systems of Systems (SoS). These principles and earlier
discussed architectural principles give basis to the proposed software architectural style.
Section 6 discusses challenges of software engineering in IIoT. These are addressed the
proposed software architectural principles presented in Section 7. Section 8 presents the
proposed decomposition method that guided by the architectural principles and views
that are presented in Section 9. This section presents the proposed architectural style.
Section 10 concludes the paper with final remarks and outlook to future work.

2 Software Architecture Practice

The practice of software architecture was defined to manage the increasing complexity
of software applications. Software design challenges evolved from algorithm design to
include communication structures, information storage and retrieval and interactions
with physical sensory and actuation capabilities. This section discusses what software
architecture practice is, its styles and how it is used to manage complexity.
Software architecture is used to model high level design that satisfies stakeholder re-
quirements. The architecture gives guidance for application design and development. It
is also used as a framework for technical and managerial decision making [2]. The archi-
tecture is not only used for design and implementation decisions, but also cost estimation,
requirements validation and design analysis. Software architecture is a fairly recent re-
search area with fundamental scientific papers published in the late 80’s and early 90’s.
As computing technology (hardware and networking), software practices and user ap-
plication spaces have evolved, so too have architectural practices. Recent research into
software architecture has focused on blue prints and patterns [12]. These reference archi-
tectures specify component entities and methods to tackle spatial distribution, streaming
and discreet information flow, and close coupling to physical world.
At its essence software architecture follows the analogy of construction architecture
(as in civil engineering). Perry and Wolf [2] introduce it as the presentation of different
views for the purpose of communication. Similarly, construction projects have structural,
plumbing, electrical and aesthetics views. These views highlight key aspects (i.e. load
bearing walls) of the design and are utilized by different stakeholders in the validation
and implementation of the project. Perry and Wolf [2] propose a three view approach;
process, data, and connector views. The process view describes the operational con-
struction of transformation activities, the data view describes the information and how
it changes during the life of the application and finally, the connector view describes the
2. Software Architecture Practice 257

interconnection between transformation activities.

Capturing early design decisions and rational guides the overall architectural style. An
overall architectural style is established dependant on how the architecture has described
these early design decisions and rationale. The concept of style again draws a parallel in
the construction world. For example, an architect may suggest the home design to follow
a Victorian style. This constrains and guides the subsequent design decisions, such as
flooring, lighting selections, may etc. Once a style has been built into the home, it will
influence future development or extension to the home. Similarly, once an architectural
style has been selected for an IIoT application it will influence future development or
extension of the application.
In a relatively short period of time, architectural concepts matured. Kruchten intro-
duced the 4+1 view model of software architecture [13]. This model was later adopted by
the Rational Unified Process (RUP). As shown in Figure 2, the ”4+1” views are logical,
development, process and physical view, plus a scenario view that extends across each of
the former views.

Figure 2: The 4+1 architecture with views mapped to the RAMI 4.0 software layers. The
business, communication and integration views are not represented in this architecture. On the
other hand, RAMI 4.0 does not have a development view.

The logical view is an entity model of the design. For example this could be an object
based decomposition. Here, application data and information models could be captured,
however, because the focus is on end-users or domain engineers, it may lack the needed
depth for software development. The development view is a detailed break down of the
structural aspects of the organization of the code. This assists with division of work
between developers and dependency matching of development resources. The process
view is an execution view of the design, looking into concurrency and synchronization
aspects, capturing their transformations and relationships. The physical view shows
258 Paper J

the distribution of the software over the executing hardware. Capturing certain non-
functional aspects of the design and guiding the deployment requirements and planning.
Interestingly, Krutchen did not utilize separate connection view and data view as
Perry and Wolf did. The 4+1 architecture does however pin-point that each view targets
a different audience, this is an important element of architecture practice. Users are
interested in functionality presented in the logical view. Developers would split workload
and design code repositories based on the development view. Technical and business
integrators tune performance and scalability using the process view. System engineers
plan topology, communications and deployments based on the physical view. Kruchten
further elaborates that each view has its own set of elements, patterns, rationale and
constraints. Within each view different architectural styles can be used. The 4+1 style is
a meta style that allows freedom to adapt the architecture to a minimal set of constraints
for the intended usage. Scenarios are used as a shared validation vision - to make sure
that requirements are satisfied by each of the architectural views.
The World Wide Web (WWW) which morphed into a key technology over the 90’s
lacked a clear definition of its architectural style. Without a clear style, changes to the
WWW architecture lacked consistent stylistic principle and were be based on short term
solutions to immediate issues. In his dissertation, “Architectural Styles and the Design
of Network-based Software Architectures” [14], Fielding defines an architectural style
for the WWW and web application development. This study addressed this issue and
provided guidance on adapting and evolving WWW protocols and techniques. The the-
sis introduces software architecture as an abstraction of run-time elements. It describes
that an architecture is not only a static structural description of a system, but rather a
description of a dynamic running system. The resulting Representational State Transfer
(REST) architectural style has become a common approach to web application develop-
ment. REST combines a number of architectural styles, including Client-stateless-Server
with caching and proxying; layered; mobile code; and uniform interface. Each style uti-
lized by REST introduces a constraint to the freedom of design decisions, thus, providing
a foundation for the rationale of the design. The REST style definition can be used to de-
termine which application architectures conform to the web and those that do not. Thus,
a web application that is developed by several different teams, maintains architectural
continuity and avoids embrittlement.

3 Advanced Architectural Styles

After looking at some of the founding and fundamental architectural styles used, we now
look into some advanced styles. These styles tend to create hybrid style of many other
styles. Their principles have a broad coverage and are intended to be applied to software
engineering in general, rather than a specific usage. In addition, they may take into
consideration development methodologies, such as agile based methods.
3. Advanced Architectural Styles 259

3.1 Service Oriented Architecture

Service Oriented Architecture is the use of software interfaces called services to create
distributed computing and facilitate remote system interaction and data exchange. SOA
grew from the use distributed objects and OOP for distributed software systems. It has
been highly successful as an enterprise architecture much of its success can be attributed
to the flexibility and scalability of the architecture. There is much literature on this
subject but the work from Erl in [15] is a reference authority. It defines eight principles of
service oriented design. These principles are the characteristics which should be followed
by developers and met by the software in order for the application be classified as a SOA-
based application. These are defined in the enumeration below with a short description
of each.
Principles of Service Oriented Design:

1. Standardised service contract Are used for services to express their capabilities
and method of interaction.

2. Service loose coupling Means to reduce dependencies between service provider

and consumer. Allowing independent implementation and evolution of provider and
consumer systems.

3. Service abstraction Refers to including only essential data in a service contract

and hiding the implementation detail. This abstraction enables independent evolution of
the implementation from the service contract.

4. Service reusability Requires that logic is broken down into agnostic functionality,
which can be reused within different contexts. This principle is dependent on the service
modelling approach used.

5. Service autonomy Means that service should control their own logic and environ-
ment. It is essential for reliable and consistent behaviour. The scope of the autonomy is
dependent on the service capability.

6. Service statelessness As opposed to autonomy, statelessness advocates moving

state information to an external store such as a database or hardware. Too much state
information will reduce service scalability as the number of consumers increases. Stateless
services are also more robust to restarts, as state information is not lost.

7. Service discoverability Requires that services, in addition to the service con-

tract, have a meta description allowing evaluation of fitness for use. Service reuse and
composition is founded on strong service discoverability.
260 Paper J

8. Service composability Services are reused in different compositions depending on

application requirements. This requires some reconfigurability of the services.
A service is the definition of a communication contract which includes the abstract
functional description (the purpose of the service), its interaction pattern (how to utilize
the service), and its structure and semantic (how to interpret the payload and make it
meaningful).

3.2 Microservices
The microservices architectural style is an evolution of SOA. It is generated through
experience of architects and engineers overcoming the pitfalls in the SOA style. The
argument for moving from monolithic applications to a Microservices application are the
same as for moving to SOA. As stated by Newman in [16], Microservices are software
applications with a single purpose. Newman goes on to argue that SOA lacks clarity
on service design. In particular guidance on service size and application decomposition
are two areas of ambiguity. In addition, Microservices also raise the notion of smart
endpoints and “dumb” pipes [16]. This is in response to heavy middleware such as ESB
that are more intelligent than the services.

The principles of the Microservices can be summarized as:

1. Single responsibility A core principles of microservices is the service size selection.

The single responsibility principle guides architects to not bunch too many capabilities
into a single service. It is Martin who proposes the single responsibility principle [17],
as a means of decomposing software systems into modules. Avoid mixing requirements
from different responsibility domains. Microservices apply this very literally and reduce
the service provided to a single responsibility service. So, as domain requirements evolve
Microservices can also evolve without requiring testing against adjacent (unchanged) do-
main requirements. The single responsibility principle promotes evolutionary design.
This is because as new responsibilities appear or existing ones disappear, microservices
can be added, deleted or changed with no (or little) impact on other services.

2. Smart endpoints and “dumb” pipes With implementation of “dumb” pipes it

means reducing or removing dependence on intermediary infrastructure. This principle
is very much inspired by the successful approach used by the Web. Specific layers in the
pipes are only responsible for particular functions such as addressing, packet/datagram
routing and security. Otherwise, the pipes only act as the interconnection fabric. Smart
endpoints also entail design for failure. This means that smart endpoints are developed
with resilience to service call failure and that active monitoring of smart endpoints, data
stores and caches can catch operational or business relevant metrics
3. Advanced Architectural Styles 261

3.3 Web 2.0

The first Web 2.0 [18] conference was held in 2004. The Web 2.0 is an evolution of
the WWW, acknowledging a significant shift in the direction of software applications.
O’Rielly and Battelle introduced 7 principles of the Web 2.0 [19] as:
1. Web as a platform
2. Harnessing Collective Intelligence
3. Data as the next Intel Inside
4. End of software release cycle-the ”perpetual beta”
5. Lightweight programming models
6. Software Above the level of a Single Device
7. Rich user experience
These principles are in many ways reflect a smaller set of changes significant to soft-
ware development. The web as a platform directs software vendors away from building
applications as platforms for locking consumers in with. Harnessing collective intelligence
is acknowledging the success of utilizing user input and participation over publishing. The
Web 2.0 users produce huge amounts of data. Data driven applications then are sources
of revenue. The perpetual beta is linked to the Web as a Platform as software appli-
cations are on-going services rather than packaged artefacts. Lightweight programming
models boost the perpetual beta through quick development/release cycles. Software
built with a single computer in mind does not take advantage of the Web as a Platform
and perpetual beta. A Rich user experience is defining, if user collaboration is desired.
Additionally, user experience becomes the user lock-in and service differentiation.
These principles lead to 8 design patterns for the Web 2.0:
1. The Long Tail
2. Data is the Next Intel Inside
3. Users Add Value
4. Network Effects by Default
5. Some Rights Reserved
6. The Perpetual Beta
7. Cooperate, Don’t Control
8. Software Above the Level of a Single Device
As can be noted, some of these design patterns are directly brought from the princi-
ples. Showing that the principles must first be elaborated before patterns can be defined.
The Long Tail means to not focus only on the central applications of the Web, but on the
users of the web. Target applications to the larger masses on the web. Network effects
by default promotes making privacy settings open by default. Web users must opt-out of
the data sharing. Some rights reserved is indicating the need to make applications easy
to adopt. The pattern Cooperate, Don’t control, means to utilize and build on shared
web services and re-use of data services.
Some of these principles and design patterns are highly relevant to web software
engineers, whilst others are specific to the Web. The future of software development
262 Paper J

is bringing about a loosely coupled set of software applications that can, each, be in
perpetual beta, whilst being developed by ad-hoc combinations of software developers,
in a cooperative manner.

3.4 Mashups and Service Mashups

Mashups as a concept are the combination of web resources including data and API
resources, to create new web applications. It is generally for user interface creation,
information sharing and aggregation and content publishing [20]. It is a design pattern
born of the Web 2.0 principles. For example Trendsmap [21] is a web application that
utilizes the Twitter and Googlemaps APIs to show Twitter activity by location. Users are
able to view what the trending Twitter topics are in particular cities. Service mashups are
similarly user generated combinations of web services to create new web applications. IoT
platforms such as ThingWorx [22] utilize mashups for user interface design and service
orchestration. A service mashup can pipe data between services allowing user driven
processing, filtering, aggregation and presentation. Node-red [23] is Javascript based
graphical environment that can mash together different services into a composition. Its
structure is similar to typed pipe and filter diagrams. The IFTTT (“if this then that”)
platform [24] is an IoT mashup cloud that supports applets and services that can connect
heterogeneous things together. Using chained conditionals it is a strong commercial
mashup platform.

3.5 Multi Agent Systems

Multi-agent systems (MAS) are sought to address challenges of heterogeneous, dynamic
systems with distributed nature and requiring distributed decision making [25, 26].
MAS is a network of communicating agents. Agents communicate their preferences,
negotiate sub-goals and manage their intentions to realize individual or overall system
goals [27, 25].
An agent is a computer system. It is placed in an environment and can autonomously
act in this environment. The actions of an agent are directed to achieve delegated goals.
It operates independently (autonomously) without direct intervention. An agent has
ability to act in a designer’s/user’s best interest while interacting with other humans,
machines or agents. While autonomy is the key principle, an intelligent agent can also
exhibit reactivity, pro-activeness and social ability.
Reactivity. Agents are able to perceive and react to changes in their environment (in
a timely fashion). This implies both soft and hard real time requirements. This might
impact decision making process (as it can be time consuming and require processing
power), constraining agent’s intelligent behaviour.
Pro-activeness. Agents can take initiative to achieve their goals. In other words,
agents can employ goal directed behaviour.
Social ability. Interaction, negotiation and collaboration underpins the social ability.
MAS then represents a network of self-interested systems cooperating and competing
to accomplish their goals. The theory of agents and MAS is a step in the
4. Systems 263

“... steady move away from machine-oriented views of programming to-

ward concepts and metaphors that more closely reflect the way in which we
ourselves understand the world.”[26]

MAS is often used to implement Game Theory algorithms [28]. MAS applications,
though, are vast from economics and social sciences to industrial automation and power
systems.
Agent architectures tend to focus on agents’ decision-making process such as Proce-
dural Reasoning Systems and Practical Reasoning [26]. MAS is more about interactions
between agents, e.g. FIPA specification etc.

4 Systems
As defined by the US Military DoD JP 1 02 a System is:

“A functionally, physically, and/or behaviorally related group of regularly

interacting or interdependent elements; that group of elements forming a
unified whole”[29]

While ISO/IEC 42010 defines a system as:

“A collection of components organized to accomplish a specific function

or set of functions”[30]

These two definitions provide a basis for understanding what makes up a system. In
this thesis, a system is a key building block for understanding an architecture. Here we
say, a system captures a fully autonomous entity with an independent set of functions,
life-cycle and purpose (objective). A system may or may not communicate and collab-
orate with other systems. However, in our case, many of the systems will need to work
collaboratively in order to achieve their own goals. Systems thinking means that when
designing a system, all aspects of the life of the system must be considered. This means
that all software and hardware aspects; such as logical, functional, structural security,
mechanical housing, spatial location, electronic and human interaction are captured un-
der a single entity. Spatial and temporal boundaries have been the intuitive delineation
lines between systems. However, as the definitions above indicate, functional boundaries
must also be considered. Add to this, managerial (change/configuration management),
political and economic boundaries, will capture strong lines of separation between sys-
tems.

5 Systems of Systems
Moving onto System of Systems extends this theory with goals and objectives that require
participation of multiple component systems to be achieved. In an SoS, the primary
modelling unit remains a single system with self-centred goals. The SoS introduces a new
264 Paper J

set of goals that are only achievable through the collaboration of participating systems.
An SoS is itself a system because it satisfies the properties of a system. However, a system
does not satisfy the properties of an SoS, therefore a (pure) system is not an SoS [31]. This
is primarily because the management and life-cycle aspects of an SoS are quite different
to those of a system. Compared with systems an SoS is much more un-predictable life
cycle that is prone to change. Any assumptions regarding the component systems of
the SoS must be moderated with fall-backs and mitigation plans. Management and
development of an SoS is also rather different to that of a system. Control over change,
budget and overall direction of a component system may not be under the control of the
SoS team. Hence, there is only limited influence over the systems team, and so careful
synchronization and analysis of change is required.

5.1 SoS Design principles

Work has been done to understand the design principles of SoS architecture. In [32],
Maier describes principles of heuristic design for SoS. They are:

1. Stable Intermediate Forms While designing SoS, incomplete forms of the SoS
should be designed and put into action. As stated, an SoS is made up of component
systems. Utilizing a sub-set of these component systems to create a partially functioning
but stable SoS will act as a proof of concept and a platform for early learning. This
principle matches very well with concepts of Agile development and iterative design.

2. Policy Triage The SoS engineering team does not have control over the system
development or modes of operation. They must triage the situation and choose carefully
where to exert influence on systems teams and where/when to adapt the SoS to the system
choice. This is captured as over-control - leading to failure due to lack of authority, or
under-control - leading to failure due to elimination of an integrated SoS.

3. Leverage at the Interfaces The component systems of an SoS are often designed
and managed independently of each other. Furthermore, the SoS designers have limited
influence over internal architecture of the systems. This leads to SoS design having a much
greater emphasis on interfaces between systems rather than the design of component
systems.

4. Ensuring Cooperation An SoS is based on independently operating systems.

There is no guarantee that these systems should cooperate. There is also a challenge of
influence upon each of the component systems. Therefore, the incentives for cooperation
must be managed. In addition, barriers (costs) for cooperation must be reduced. Again,
the Internet is a prime example, with light weight communications protocols, combined
with great value of interconnectedness.
5. Systems of Systems 265

5.2 SoS Types

SoS can exhibit emergent behaviour [33]. In some cases it is desirable and other it can
be damaging. Depending on the class of SoS, emergence may be controlled (or planned
for). In Maier’s work [32], he describes a taxonomy of three SoS types:

1. Virtual Virtual SoS lack centralized management and purpose. Behaviour is not
created with intention, rather it emerges from the resulting SoS. The Internet is an
example of a virtual SoS; There is not enforcement of Internet standards and there is no
centrally agreed reason for the Internet. The Internet Engineering Task Force (IETF)
[34] must utilize principle 4 ensuring cooperation to create the Internet standards.

2. Voluntary Voluntary SoS can also be referred to Collaborative SoS. Involve cen-
trally agreed purpose of an SoS, but do not subscribe required adherence to the SoS. SoS
management does not have any coercive power on individual systems. This is perhaps
the most challenging and common form of SoS within the context of IIoT.

3. Directed In the case of directed SoS, purpose and management is fully centralized
and has significant control over component systems. The total SoS design lends itself
closely to Monolithic assumptions regarding behaviour control.

5.3 Summary
In this section the Industrial Internet of Things and some software architecture ap-
proaches were discussed. Architectural styles like SOA, MAS, Web2.0 and microservices
all have contributing principles toward a software design and development for the IIoT.
However, there are certain challenges of IIoT that can be handled by one style and not the
other. Pure SOA usage in a physical environment lacks the descriptive need for modelling
physical entities with multiple, inter-dependent services. Pure MAS has high resource
demands from nodes that are constrained in battery life and communications capability.
Web2.0 focuses predominantly on web based commerce and interaction. There is little
attention to low bandwidth or niche requirements of industrial use cases. The principles
of Microservices build up many advantages to SOA and can be useful to IIoT. They
however, lack the same expressiveness required for understanding the physical nature of
dependent services in IIoT nodes.
A consistent argument for a new architectural style is presented, based on defining
a set of principle that are able to cope with the challenges of IIoT software architecture
design. SoS is used along with SOA as the basis for modelling interacting IIoT nodes.
Functionality is spread amongst the nodes and applications emerge from the resulting
SoS. SoS is a flexible notion that lends itself well to environment with highly varying
and heterogeneous capabilities. For example, a company or society can be modelled
as a SoS with humans making up the component systems. An employee will have a
set of personal objectives and capabilities, for example, objective of earning an income
266 Paper J

and capability of developing software. Employees in a company work in a collaborative

centrally administered organisation. Products and services are the result of the emergent
value of the collaborating employees.

6 Challenges of Software Engineering in IIoT

The complexity of software is increasing as embedded/control systems meet with more
abstract computer science. This section summarizes the challenges this complexity poses
to current and future engineers. The architecture of the software must help with tackling
the challenges outlined below.

1. Intersection of concerns IIoT applications cross several engineering domains.

This collision of disciplines and thinkings requires careful consideration regarding where
functionality should be placed. Depending on the method of separation of concerns,
this challenge can be less significant. A traditional vertical layered architecture enables
independence of technical domains. Whereas a cross cutting architecture (slices) will de-
mand multidisciplinary development teams. Either way, this challenge will remain with
the question who should be concerned with a particular functionality? This challenge is
closely linked to the functional decomposition but also includes non-functional concerns.

2. Functional decomposition A large part of the architecture modelling is the func-

tional decomposition - what capabilities are required and who provides them. IIoT
applications are fundamentally spatially distributed and often have limited computing
power or communications links. Thus, a capability may need to be split into two smaller
sized, or two capabilities be combined into one. For example, if the communications
link cannot support streaming, then a Fast Fourier Transform (FFT) may be required
on the sensor. In this way, functional decomposition has impacted IIoT device processor
selection. Vice-versa, the available processing power may mean that FFT is not possible
at the sensor, and so an intermediary node may perform the FFT. This functional flexi-
bility shows that the architecture must enable functionality with freedom of movement.
Functional interface design also impacts the ease of movement of deployed functions.

3. System interdependence Managing dependencies between systems is a growing

challenge as intelligent software and functionality is spread out between production pro-
cesses. If the chain functional dependence between a set of systems breaks, then all up
stream systems could be interrupted. System life-cycle is another issue with system inter-
dependence: Change management that requires synchronization amongst interdependent
systems increases deployment effort (cost of change).
6. Challenges of Software Engineering in IIoT 267

4. Data modelling The intersection of different domains exacerbates the data mod-
elling issue for IIoT. Architects may need to utilize semantic web with highly typed
representations of data, or plain text /binary data over a low power wireless link. With
evolving techniques and technology this challenge is always moving. This challenge is in
semantics of data, usage of data, storage of data and serialization of data.

5. Communications bottlenecks Highly connected machines cause scalability issues

in the communications lines. A centralized messaging mechanism suffers from single point
of failure and bandwidth limitations. Point-to-point communications can be exhausting
for constrained devices (constraints being in bandwidth, energy, processing power or
memory). In case of degraded performance, routers and network devices have high im-
pact on IIoT applications.

6. User interactions Lines of separation between computers and humans are not
clear cut. User interact with machines with augmented reality, robotics and etc. So-
lutions cannot target a single display and information must flow between augmented
reality and traditional displays seamlessly. Virtual commissioning and human-machine
co-working also require fluid movement of information.

7. Systems of Systems design and definition Development of individual equip-

ment requires engineering knowledge. However, combining these systems into business
value may not need deep engineering knowledge. In fact domain knowledge is required to
extract value from a SoS. Therefore this challenge highlights the user (domain engineers)
ability to drive change in a SoS function.

8. Legacy equipment integration This challenge is especially acute for an industry

that has traditionally seen equipment life-cycles of up to a decade. These legacy systems
impact functional decomposition, security vulnerabilities and migration strategies.

9. Data security A big challenge with so much networked data being made avail-
able and with increasing operational dependence on access to this data. Also, due to
networks of multi-vendor applications open traditionally secure environments to sloppy
development or malicious code. The lines of attack for social engineering increases. This
challenge deals with: access control, forged credentials, social engineering and networks
of multi-vendor applications
268 Paper J

10. Protocol heterogeneity This is caused due to intersection of domains with

differing communications requirements. It is evident through the number of protocols
proliferated within industry that convergence onto a single protocol capable of handling
all communications requirements is not possible.

11. Determinism Deterministic behaviour is something that OT cannot ignore. Op-

erator safety is dependent on this. Included in this challenge is real-time communications.
These factors encompass many non-functional requirements and shape the functional de-
composition.

12. Development practices OT engineers are accustomed to high cost well planned
stage-gate projects. While, IT developers prefer iterative time-boxed projects. Although
there still remain differences between IT and OT, the software life-cycle is still becoming
shorter.
IIoT solutions must support evolutionary development and have fast mean time to
failure. Life-cycles of IIoT systems can vary greatly. At any given time, systems will
be at different stages in their life-cycle. The challenge for architects is to delineate
between systems so that sporadic disruptions do not interrupt overall operation of the
IIoT application.
The challenges listed here are not application requirements. They are general issues
that architects must be aware of and will influence architectural considerations even
before requirements are considered. In order to address these challenges from a software
architecture perspective, first a set of architectural principles must be defined.

7 Principles for Software Architecture design in IIoT

Architectural principles are fundamental to the reasoning of early design decisions. A
selected architectural style must satisfy the principles proposed in this section. In turn,
the concrete software architecture must conform with the architectural style. Each step
toward a concrete architecture constraints the final solution or the available means to
the ends. In Figure 3 the relation between architecture abstraction concepts is shown.
Although, the number of constraints introduced by the architecture principles are small,
they have long lasting impact. Common solutions to problems can be captured in ar-
chitecture patterns or reference architectures [35]. As opposed to a style that does not
specify component instances, an pattern will have instances of components that will re-
solve a specific problem or set of problems. For example, Bell documented common SOA
patterns in [36]. Each concrete architecture should be able to track its reasoning through
the layers up to an architectural principle.
We begin elaborating the principles by adapting existing architectural principles. SOA
has several principles that we use as a starting point. Here, the principles are defined in a
7. Principles for Software Architecture design in IIoT 269

Figure 3: Software architecture abstraction layers. Small changes in the principles at the core
echoes through all the layers.

manner independent of the architectural style (SOA). So for example, services, systems,
components and agents are referred to as decomposition elements.

1. Loose coupling (Adapted from SOA) Firstly, each decomposition element is inde-
pendent of the internal design of other elements. Meaning that, for example, logic, intel-
ligence, implementation technology or in-fact (internal) composition can change without
impacting other elements. This helps to improve maintainability and changeability of
the elements.

2. Searchable (Adapted from SOA) Loosely coupled decomposition elements can

discover each other. The location of both the physical process and the address designa-
tion can be moving and need not be static. This supports a means of scaling through
transparent caching or proxies. Deployment and (re-)adjustment are two major factors
this principle supports.

3. Run-time binding (Adapted from SOA) Functionality provided by different de-

composition elements are linked at runtime. This is usually achieved through messaging
interfaces of remote loosely coupled functon. Compile time binding means that any
change to the location or function of the underlying library forces a re-compile of the
whole system. Of course software libraries are important aspect, however, applications
with large library dependencies increase the cost of change.

4. Autonomy Decomposition elements independently make decisions regarding life-

cycle and capability offerings and dependencies. Closely linked to the principle of loose
coupling, autonomy requires that the decomposition elements exhibit resilience to exter-
nal failures. Usage, spatial and life-cycle are the main areas of autonomy.

5. Specialization This principle suggests that each element should have a limited
number of concerns. The decomposition elements can be responsible for varying amounts
of functional or business logic. An extreme of this principle would be as with microservices
270 Paper J

and limit each element to a single responsibility. This principle must be balanced with
other principles such as autonomy.

6. Data at its source This principle guides the architecture to avoid centralized
data stores where possible. Data is a commodity with value. Access must be controlled
and in some cases charged. This principle undertakes to direct privacy by design as a
characteristic of the architecture prior to any development. There are mechanisms to
allow cache and proxy, whilst allowing data source origin having certain controls over the
data.

7. First person perspective This principle guides the architecture to ”empathise”

with the decomposition element being described. The architect must think from the
point of view of the decomposition element. Not as simply as limited components of
the software application. The decomposition elements have their own agency. Apply-
ing this principle supports achieving autonomy, privacy by design, loose coupling and
specialization.
In Table 1 these seven principles are mapped to the challenges they address. In Figure
4 the challenges and clustered around respective design principles. This illustrates the
implicit influence and connection of these challenges. The design decisions driven by the
principle will address the respective challenge and to some degree the related challenges
in the cluster.

Table 1: Mapping of principles to the challenges they address

Challenge Principles
Intersection of Concerns Autonomy, First person perspective
Functional Decomposition Autonomy, Specialization,
First person perspective
Systems Interdependence Loose coupling, Autonomy, Specialization
Data Modelling Autonomy, Data at its source
First person perspective
Bottlenecks Data at its source
User Interactions Loose coupling, Data at its source
SoS Design/Definition Loose coupling, Searchable, Run-time binding
Legacy Equipment Loose coupling, Autonomy
Data Security Data at its source, First person perspective
Protocol heterogeneity Autonomy, Run-time binding
Real-time Communications Autonomy, Specialization
Development Practices All Principles

This section has presented the principles for design of a software architecture in the
context of IIoT application development. The practice of architecture in the IIoT requires
8. The Principled Decomposition 271

Figure 4: IIoT Software challenges clustered on software architecture principles

holistic consideration of the decomposition elements. The principles direct architects to

think about IIoT applications at different levels of abstraction. It encourages architects
to investigate methods of reducing security vulnerabilities without relying on encryption
or authentication. Primarily, the principles lead architects to think of the decomposition
elements with respect and care. Recognizing that the software will have a journey, build
a relationship with developers and leave an impact on its users.

8 The Principled Decomposition

Here, architecture decomposition is defined as the process of splitting and joining software
capabilities into implementable groupings and abstract logical groups.
Architectures such as SOA and MAS address decomposition aspects outside the scope
of the style. SOA uses architecture patterns [37] to guide application decomposition. In
MAS, design methodologies are used to guide the decomposition process, for example,
system oriented methodologies, such as Tropos [38] and Prometheus [39], propose agent
task and role decomposition. The approach proposed here, guides architects on how to
apply the principles to decomposition in a generic manner. Therefore without introducing
specific patterns, architects can begin to shape the overall architecture. To this end, it is
assumed that the requirements of the IIoT solution have been gathered and are presented
as a set of feature requirements. These features document both functional and non-
functional requirements, although they are more focused on functional requirements.
These features are used by the architect to decompose the application/solution. The
objective of the architect is to map these features into elements, connectors and modules.
Along with the rationale, these architectural components will be presented in different
views, dependant on the intended target stakeholders. Each of these steps should be
recorded and kept.
272 Paper J

Autonomy A high level solution can be modelled as a black box with all capabilities
required to satisfy the requirements - with no detail on how it achieves this. This is insuf-
ficient and a developer will come up with an architecture based on limited information.
Studying the feature life-cycle, usage and spatial separations, this black box is broken
down into groups of capabilities. This first step of functional decomposition resulted in
a number of black boxes representing groups of capabilities that have similar demands
of autonomy. The next step is to look into specialization.

Specialization The architecture is now black boxes decomposed based on; life-cycle,
spatial distribution and decision making autonomy. Each black box of capabilities could
have quite different objectives or implementation requirements (i.e. legacy equipment).
Through each block box, split the capabilities based on the level of relatedness. Creating
specialization is at the architects discretion, and there is no definite way to determine
relatedness. However, where multiple processes are involved, the lines may be drawn
based on equipment scope boundaries. This will result in a higher granularity of black
boxes that are more meaningful to developers.

Data at its source The black box groupings of capabilities can now be related by data
usage and storage. The architect should guide (constrain) the solution to utilize data at
its source. This is done by introducing direct(ed) relationships between the capability
black boxes. Where a it is not possible to maintain data at the source, then additional
(non-functional) cache/proxy/archive capabilities should be added. The architecture now
has hierarchical groupings of capabilities decomposed by autonomy and specialization and
connected through directed data relationships.

First person perspective The black boxes can now be viewed as systems. Each
system has a set of capabilities that it is responsible to fulfil. By putting themselves into
the perspective of the system the architect questions how changes in the environment may
impact capabilities. Looking outward at other systems, what are my functional and data
dependencies? Do I have any fail safe contingencies or are there any mitigating factors
for dependency failure or loss? How will others make use of the capabilities I am offering?
Looking inward, what is my life cycle? What mechanisms do I have for optimizing my
life time? How can I trust other systems and how can I prove who I am? What are
my security vulnerabilities or strengths? By applying the first person perspective, the
architect will have captured capabilities as concrete service interfaces. This principle
has the added benefit that an architecture that rationalizes such considerations can be
utilized by engineering teams to identify and better guide implementation technology
choices. In addition, this step must address the security constraints, keeping data safe
and safeguarding against malicious interactions.

Loose coupling, Searchable, Run-time binding From an architectural perspective

these three principles ultimately guide the interface implementation architecture. Ability
9. The Architectural Style 273

for SoS composition is ultimately down to these principles. The architect must the
decomposed systems to ensure they take action according to these principles.

9 The Architectural Style

In the previous section the functional artefacts of the architecture are now decomposed.
Each individual system could utilize a different internal architecture; such as agent archi-
tecture, object modelling, pipe and filter, layered architectures or any other. Additionally,
the intended high level collaboration between systems needs to be elaborated. In this
section the architectural style will be developed through elaborating the views and the
decomposition at each view. Because decomposition is done by utilizing architectural
principles, the style will naturally exhibit these principles. Before examining the style we
start with a short recapping of what an architectural style is. Referring back to Figure 3,
architectural style fits between principles and reference architectures and patterns. There
is no concrete delineation between a style and a pattern/reference. It is possible for a
reference architecture to be more abstract than a style. But over all an architectural style
should only contain the expressions required in order to create the form, structure and
decomposition of an architecture. This includes; elements, connectors and views that are
used in creation of a concrete architecture. A style provides context and a fundamental
framework for analysing the rationale behind architecture decisions. While a pattern or
reference architecture intends to describe reusable solutions to common problems, a style
is a way of thinking or an approach to solution design. Therefore, an architectural style
does not specify any component instances. On the other hand reference architectures
often describe precise component details or predefined capability separations. The ra-
tionale behind a reference architecture or pattern may not always be communicated or
become lost due to “copy paste” reuse.
The proposed architecture style is illustrated in Figure 5. It shows the discussed
challenges, principles and decomposition. It builds upon such theories and architectures
as systems, SoS and SOA. The Arrowhead Framework has applied SoS and SOA to great
effect with industrial pilots [40]. SOA has additionally been used for decades in IT and
is a new trend for OT in industry (research has been on-going). The decomposition
method has already been presented. They style then consists of views and elements.
The elements are built up from SoS and SOA notation and experience from working on
the Arrowhead Framework. Each view defines a form of the architecture. Drawing on
RAMI 4.0 gives a starting point for understanding the stakeholder views.These views are
described individually in the following sub-sections. The proposed architectural style is
not born in vacuum and builds on existing architectural styles, such as layering, n-tier,
client-server and pipes and filters. Depending on the specific application demands, the
proposed architecture style can also be combined with other architectural styles.
An on-going example will be used to illustrate the proposed architectural style. The
example is from a pilot use case of the Far-Edge H2020 project. It involves two primary
cases; plug and produce and autonomous work station. For a high level introduction to
the use case: 1) When moving equipment between workstations there is need to create
274 Paper J

Figure 5: The proposed architectural style for designing IIoT applications.

automated configuration of the equipment and work stations. Such that domain specialist
resources can make factory floor changes with out assistance from IT resources. 2) There
is a need to make the work station able to operate with little or no factory wide IT
infrastructure. It is autonomous and decoupled from factory wide IT applications such
as MES.

9.1 Views
The Figure 5 illustrates the views as they reflect the interests and concerns of all stake-
holders. A view must capture important design decisions relevant for the stakeholders
(including the architect). Views are a critical method of creating focus on relevant
concerns. The RAMI 4.0 layers define 6 areas of concern; business, functional, data,
communication, integration and asset. Roughly speaking, IT has focused on the upper 3
layers. This is in contrast to OT that focuses on the lower 3 or 4 layers, often providing
a thin service layer for interaction with IT infrastructure. Industrial middleware [41] has
positioned itself somewhere within the middle 4 layers, generally abstracting the physical
assets and remaining flexible to business processes. Suggested here, is to capture the
different RAMI 4.0 layers as application network views. Each layer captures the point
of view distribution across relevant RAMI4.0 hierarchical components. The views in this
9. The Architectural Style 275

architecture style can be as simple as a catalogue of concerns.

As already mentioned, systems and SoS make up the core modelling elements. The
concept of systems map well to capturing the concerns of each view. IIoT elements
are modelled as systems, such that, physical and logical independence is provided for
each element. Battery, processor, RAM, flash, networking (mesh vs tree), actuation and
sensing, are at the discretion of the domain architects and developers of each individual
system. This provides the architects of each IIoT system with the capability requirement
from business, function, information, communication, integration and asset concerns.

Business view
The business view captures concerns regarding business value
generation, regulatory compliance and domain or organization
norms.

The business view must capture the networked distribution of domain logic across
the IIoT device hierarchy. This requires a catalogue of requirements that capture value
generating activities, regulation compliance activities and any organizational norms. For
each requirement a matching capability must be identified so that a set of business ca-
pabilities are defined that will full fill all business requirements. Finally, the business
capabilities should be mapped to IIoT device hierarchy groupings. At this stage there is
no direct mapping to systems, only to the system classes (i.e. product, field device, work
centre, enterprise and etc.). This means that requirements and capabilities at the busi-
ness layer are visually allocated a specific layer of the IIoT hierarchy. The decomposition
of the capabilities with the subsequent mapping to hierarchy have applied the decompo-
sition technique defined in Section 8. This means that visualization and decomposition
of business logic leads to understanding infrastructure requirements and functional re-
quirements. Thus it is the starting point for the functional view, which must support the
business operations and objectives.
During creation of this view/form, the architect must ask themselves:

1. What are the business considerations each component group are responsible for?
2. What are the dependencies between constraints (therefore dependency between
groups)?
3. Are there any direct and indirect impact of the constraints on the overall design of
each group?

For example, a regulatory requirement may require execution of certain business op-
erations. Figure 6 captures the capability mapping required to provide regulatory com-
pliance to safety nut tightening. That is, the safety nut, controlling the wheel alignment,
must be tightened to a specific torque and follow through angle. This value must be
recorded and kept on record by the truck manufacturer. This view then shows how the
responsibility for performing the logical operations are distributed amongst the differing
hierarchical components to achieve the objective. With this view, the architect is able
to validate the business rule for wheel alignment safety nut tightening.
276 Paper J

Figure 6: The business view is made up of simple black boxes representing each layer of the IIoT
device hierarchy with the allocated business capabilities decomposed from the business concerns.

Functional view
The functional view captures concerns regarding system identi-
fication and capability assignment.

The business view captured the capabilities required to deliver value, stay within
regulations and any organizational norms. It has also allocated those capabilities to
the IIoT device hierarchy. Next the business capabilities are turned into functional
requirements. Domain knowledge is required to turn business capabilities into functional
requirements. In addition user requirements, such as interaction requirements, must
be captured. These requirements are all captured in a catalogue that will enable the
business capabilities and user interaction needs. For each catalogued requirement, a
functional capability must be defined, such that it satisfies the business capability at the
required IIoT device hierarchy level. The functional capabilities must be decomposed
according to the method in Section 8. Decomposition of the functional capabilities will
result in capability groupings that can be referred to as systems. Hence, the component
systems of the architecture have been identified, mapped to the IIoT device hierarchy
and their capability offerings/dependencies defined. The functional view has captured
the distribution and relationship of how objectives across the networked systems are
satisfied. Each system provides capabilities as services that can collaborate to complete
business objective. The systems provide and consume these services so the functional
dependencies between systems are identified as service interfaces. This is the view that
will often be used when communicating the solution back toward the customer and/or
system users.
While creating this view, the architect should be asking questions such as:

1. What functionality needs to be implemented to support the business capabilities?

2. How will users interact or work together with the systems to deliver the business
capabilities?
3. How will systems interact with one another to deliver the business capabilities?
9. The Architectural Style 277

4. Are there any systems with too many dependencies (i.e. high impact if failure
occurs)?
Following on from the earlier business rule regarding safety nut tightening, the systems
and services required to satisfy the business capability are shown in Figure 7. Of course,
these functional capabilities can also be reused for other business capabilities.

Figure 7: The functional view consists of the component systems and capability services with
dependency identification.

Information view
The Information view captures the concerns of what form data
takes and meaning it has.

The functional capabilities and dependencies have created service exchange paths
between systems. It is the business capabilities and user interaction requirements that
have motivated the systems and service exchanges. The data that is generated by the
execution of a capability or required in order to perform a capability has not been defined
yet. The Information view captures what data is required/generated, what the meaning
of the data is and what format it takes.
While creating this view, the architect should be asking questions such as:
1. What does the capability do? How does its execution effect the meaning of the
information?
2. Which capability produces the information?
3. Which capability requires the information?
4. What are the dependencies between information?
278 Paper J

The proposed architectural style does not dictate the elements or methods used in
the view. Any ontological language could be used to describe the information or an
external ontology could be referred to here. It is also possible to perform conceptual
data modelling here using techniques such as entity-relationship modelling [42]. In Figure
8 an example Entity Relationship Diagram (ERD) is shown. It defines the conceptual
model of the information required to make use of the nut tightening capability. It also
defines the information generated by execution of the nut tightening capability. The
generated information is shown to relate to the product entity. This view has defined
what information is used by the functional capabilities and where the source and sinks of
the information are. Using this model developers can understand exactly the information
that must be stored for the safety nut tightening of the wheel alignment.

Figure 8: This Entity Relationship Diagram shows the information used and generated by the
Nutrunner as it executes the nut tightening capability. It also relates the results back to product
being worked upon.

Communication view
The Communications view captures concerns of system interac-
tion patterns and methods.
Until now, the individual systems have been identified. Their capabilities and service
interfaces defined. The format and meaning of information exchanged between systems
specified. But how does the information move between systems? In which direction is
service exchange initiated? The communications view captures how information is passed
between the systems.
While creating this view, the architect should be asking questions such as:
1. How does information move between systems?
9. The Architectural Style 279

2. In which direction is service exchange initiated?

3. What are the suitable interaction patterns?
4. How secure does the information need to be during transit?
5. Are there any real time communications required?
6. How are communication QoS considerations handled?

To move information between systems it must be serialized, compressed, encrypted,

sent on the “wire”, decrypted, decompressed and de-serialized. On top of this, interaction
patterns must be selected. Common ones being request/response, publish/subscribe
or repository (shared memory). Interaction patterns and methods do not need to be
homogeneous across the distributed architecture. Communication requirements can differ
considerably across the distributed system. For example, a constrained IIoT device may
require a specialized protocol with small library foot print or packet size. This view will
show the communications network between the systems with allocation of capabilities
to a bus. For example, CoAP bus and HTTP bus would be represented as separately
although they use the same IP network. Network segregation with protocol gateways or
translators should be shown in this view. Figure 9 shows the functional view with the
communication lines added.

Figure 9: The Communication view specifies the communications patterns and methods as a
shared bus between systems.

Integration view
The Integration view captures concerns of interfacing between
the virtual and real worlds.

Thus far, the individual systems interconnected with each other through communica-
tions buses with known information formats and capabilities. Due to the nature of IIoT
applications, many of the systems will have sensors and actuators to perform real world
280 Paper J

interactions. The integration view captures what hardware interfaces are required by the
systems in order to interact with the physical world.
While creating this view, the architect should be asking questions such as:
1. Do the systems require A2D or D2A conversion?
2. Are there any legacy interfaces to be supported (i.e. serial interfaces)?
3. What support for digital inputs/outputs are required?
4. Does the system need an integrated sensor?
Not all questions need to be answered. The architecture should show the important
constraints or provide a selection of possible constraints. This can give valuable freedom
to designers for integration selection, so this decision is at the discretion of the architect.
For example, Figure 10 shows the touch screen interface to the display system and the
legacy nutrunner equipment with a serial interface. The databases shown here could be
represented in the system view as an internal component of the system, or in this case
as an outside store with a direct interface.

Figure 10: The Integration view highlights important real world or legacy equipment integration
points.

Asset view
The Asset view captures concerns of physical world devices.

System requirements for integration and communication have been formed. The Asset
view is where architects will identify important devices for running the systems on. In
addition legacy devices are shown here with the legacy communications bus defined in
the integration view. Additionally, the asset view should show networking appliances
and network interfaces such as Ethernet, Wi-Fi, IEEE 802.15.4 or etc.
While creating this view, the architect should be asking questions such as:
1. Is the IIoT device battery powered or mains powered?
2. Is it possible for the IIoT device to utilize power harvesting?
3. How secure must the system be?
4. How much cryptographic processing will be required?
9. The Architectural Style 281

5. Will the system need to store a lot of information?

The device requirements such as battery power, cryptography hardware acceleration,

processing power and volatile and non-volatile memory should be specified. Again, in this
view the architect should concentrate on important constraints in order to allow designers
flexibility to select optimal decisions based on first hand implementation information.
The safety nut tightening scenario is shown in Figure 11. Not all hardware are shown,
only those that require special attention. In this case a high level of detail is capture for
the devices shown.

Figure 11: The Asset view is communicates hardware specifics - this view can be very detailed,
or leave more flexibility to designers.

System view
The System view captures concerns of a single system with all
the software layers present.

Thus far the architecture style has focused on network views of the IIoT application.
This is no accident, as IIoT applications are highly distributed. For each of the IIoT sys-
tems defined so far this view must be developed. The system view is an implementation
view that can be utilized for estimation and development. Therefore, it will be utilized by
282 Paper J

software and hardware engineers in building the system. It must capture the system con-
cerns related to: asset, integration, communication, information, function and business.
This view will capture the system architecture, it could utilize a object oriented, layered,
pipe and filter, event driven or agent architectures etc. It it could be some suitable com-
bination. For example, a suggestion could be use a hybrid of two styles; 1) a layered
style captures the asset, integration and communication concerns and 2) an object style
captures the information, function and business concerns. Hardware Abstraction Layers
(HAL) create a platform upon which high layer objects can be manipulated. The layered
style indicates clear dependencies and responsibilities. Object oriented modelling pro-
vides flexible definitions and changing interactions between components. This is shown
in Figure 12.

Figure 12: The System view is utilized by the architect to communicate a model of the system
that can guide implementation.

10 Conclusion
The IIoT is a sphere of technological applications that encompasses a vast variety of
requirements and solutions. It is a cross disciplinary domain requiring knowledge and/or
awareness of materials, physics, electronics, embedded software, communications/net-
working, security, enterprise software and data storage/analytics. Therefore, architect-
ing an IIoT solution cannot rely on pure computer science nor on engineering. The
architect must rationalize design decisions based a principled approach. These princi-
ples are autonomy, specialization, data at its source and first class components. These
are combined with traditional SOA principles of loose coupling, searchability and run-
time binding. Through these 6 principles architects are able to decompose application
capabilities into interacting systems of appropriate size. Thus, the principles form the
guide line for an architectural style that utilizes SoS, SOA and the RAMI 4.0 model
to guide the capability decomposition process. First person, autonomous components
tend to show resilience to faults and share less internal state. Component specialization
supports evolutionary environment and reduce their impact on other components when
changed. Data at the source exhibit desire to retain ownership of data and encourage
References 283

less pooling of data, thereby increasing security. Furthermore, data at the source reduces
the semantic overhead of describing full information context when storing data.

10.1 Future Work

The proposed architecture style results in application objectives being turned into SoS
objectives utilizing SOA to collaborate. Future work should look into timeliness of these
applications. Due to the messaging overhead of SOA, applications will generally experi-
ence additional communications delays. It would be interesting how applying first person
perspective may affect the design of a system so that it is less sensitive to delays in peer
systems. Timing and delay is a relative measure and a matter of perspective. Meaning
that, due to system design, (moderate) communication delays may not be an issue.
The proposed architecture style has major implications on the design thought process.
No longer treating the system under development as a thing to be controlled and utilized.
For constrained IIoT devices Real Time Operating Systems (RTOS) are designed to
support application development. It would be interesting to understand how the concepts
at an RTOS level may change as developers define logic in human related terms. An
RTOS that natively supports human concepts such as plans, objectives, knowledge and
reactions. The intention with this future work is to result in an RTOS that would no
longer look at functions, threads or abstraction layers. Rather a nervous system with
receptors, pathways and reasoning.

Acknowledgement
The authors would like to thank all partners of the Far-Edge, Arrowhead and Productive
for constructive discussions regarding application architecture development.

References
[1] M. Hölzl, A. Rauschmayer, and M. Wirsing, “Software-intensive systems and new
computing paradigms,” M. Wirsing, J.-P. Banâtre, M. Hölzl, and A. Rauschmayer,
Eds. Berlin, Heidelberg: Springer-Verlag, 2008, ch. Engineering of Software-
Intensive Systems: State of the Art and Research Challenges, pp. 1–44. [Online].
Available: https://doi.org/10.1007/978-3-540-89437-7 1

[2] A. W. D.E. Perry, “Foundations for the study of software architecture,”

ACM SIGSOFT, pp. 40–52, October 1992. [Online]. Available: http:
//delivery.acm.org/10.1145/150000/141884/p40-perry.pdf?key1=141884&key2=
2495967421&coll=GUIDE&dl=GUIDE&CFID=44366193&CFTOKEN=42763085

[3] “The ’only’ coke machine on the internet,” Carnegie Melon University. [Online].
Available: https://www.cs.cmu.edu/∼coke/history long.txt
284 Paper J

[4] B. Scholten, The road to integration : a guide to applying the ISA-95 standard in
manufacturing. Research Triangle Park NC : ISA, 2007.

[5] A. Cockburn and J. Highsmith, “Agile software development, the people factor,”
Computer, vol. 34, no. 11, pp. 131–133, Nov 2001.

[6] J. Delsing, J. Eliasson, J. van Deventer, H. Derhamy, and P. Varga, “Enabling iot
automation using local clouds,” in 2016 IEEE 3rd World Forum on Internet of
Things (WF-IoT), Dec 2016, pp. 502–507.

[7] P. Garcia Lopez, A. Montresor, D. Epema, A. Datta, T. Higashino, A. Iamnitchi,

M. Barcellos, P. Felber, and E. Riviere, “Edge-centric computing: Vision and
challenges,” SIGCOMM Comput. Commun. Rev., vol. 45, no. 5, pp. 37–42, Sep.
2015. [Online]. Available: http://doi.acm.org/10.1145/2831347.2831354

[8] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing and its role in the
internet of things,” in Proc SIGCOMM, 2012.

[9] E. A. Lee, “Cyber physical systems: Design challenges,” in 2008 11th IEEE In-
ternational Symposium on Object and Component-Oriented Real-Time Distributed
Computing (ISORC), May 2008, pp. 363–369.

[10] F. P. Brooks, Jr., “The mythical man-month,” SIGPLAN Not., vol. 10, no. 6, pp.
193–, Apr. 1975. [Online]. Available: http://doi.acm.org/10.1145/390016.808439

[11] J. Delsing, O. Carlsson, F. Arrigucci, T. Bangemann, C. Hübner, A. W. Colombo,

P. Nappey, B. Bony, S. Karnouskos, J. Nessaether et al., “Migration of scada/dcs sys-
tems to the soa cloud,” in Industrial Cloud-Based Cyber-Physical Systems. Springer
International Publishing, 2014, pp. 111–135.

[12] H. P. Breivold, “A survey and analysis of reference architectures for the internet-of-
things,” ICSEA 2017, p. 143, 2017.

[13] P. B. Kruchten, “The 4+1 view model of architecture,” IEEE Software, vol. 12,
no. 6, pp. 42–50, Nov 1995.

[14] R. T. Fielding, “REST: architectural styles and the design of network-based

software architectures,” Doctoral dissertation, University of California, Irvine, 2000.
[Online]. Available: http://www.ics.uci.edu/∼fielding/pubs/dissertation/top.htm

[15] T. Erl, SOA Principles of Service Design (The Prentice Hall Service-Oriented Com-
puting Series from Thomas Erl). Upper Saddle River, NJ, USA: Prentice Hall PTR,
2007.

[16] S. Newman, Building Microservices: Designing Fine-Grained Systems, 1st ed.

O’Reilly Media, February 2015.
References 285

[17] “The single responsibility principle - 8th light.” [Online]. Available: https:
//8thlight.com/blog/uncle-bob/2014/05/08/SingleReponsibilityPrinciple.html

[18] T. O’Reilly and J. Battelle, “Web squared: Web 2.0 five years on,” Tech. Rep., 2009.

[19] T. O’Reilly, What is Web 2.0? Design Patterns and Business Models for the Next
Generation of Software. O’Reilly Media, September 2009.

[20] D. Merrill, “Mashups: The new breed of web app,” Tech. Rep., 2006.

[21] “Real-time twitter trending hashtags and topics.” [Online]. Available: https:
//www.trendsmap.com/

[22] Thingworx industrial innovation platform - ptc. PTC. [Online]. Available:

https://www.ptc.com/en/products/iot/thingworx-platform/

[23] “Node-red.” [Online]. Available: https://nodered.org/

[24] “Ifttt helps your apps and devices work together.” [Online]. Available:
https://ifttt.com/

[25] S. Bussmann, N. R. Jennings, and M. Wooldridge, Multiagent systems for manufac-

turing control: A design methodology. Germany: Springer-Verlag, 2004.

[26] M. Wooldridge, Introduction to Multiagent Systems. New York, NY, USA: John
Wiley & Sons, Inc., 2001.

[27] S. D. J. McArthur, E. M. Davidson, V. M. Catterson, A. L. Dimeas, N. D. Hatziar-

gyriou, F. Ponci, and T. Funabashi, “Multi-agent systems for power engineering
applications - part i: Concepts, approaches, and technical challenges,” IEEE Trans-
actions on Power Systems, vol. 22, pp. 1743–1752, 2007.

[28] K. L.-B. Yoav Shoham, Multiagent Systems - Algorithmic, Game-Theoretic and Log-
ical Foundations. Cambridge University Press, 2009.

[29] “Department of defense dictionary of military and associated terms,” Washington,

DC, USA, Tech. Rep., 2015.

[30] ISO/IEC/(IEEE), “ISO/IEC 42010 (IEEE Std) 1471-2000 : Systems and Software
engineering - Recomended practice for architectural description of software-intensive
systems,” 2007.

[31] “Systems engineering guide for systems of systems,” Washington, DC, USA, Tech.
Rep., 2008.

[32] M. W. Maier, “Architecting principles for systems-of-systems,” INCOSE

International Symposium, vol. 6, no. 1, pp. 565–573. [Online]. Available:
https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2334-5837.1996.tb02054.x
286 Paper J

[33] J. S. Osmundson, T. V. Huynh, and G. O. Langford, “Emergent behavior in

systems of systems,” INCOSE International Symposium, vol. 18, no. 1, pp.
1557–1568. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/j.
2334-5837.2008.tb00900.x

[34] “Internet engineering task force.” [Online]. Available: https://www.ietf.org/

[35] P. Reed, “Reference architecture: The best of best practices,” Tech. Rep., 2002.

[36] M. Bell, SOA Modeling Patterns for Service Oriented Discovery and Analysis. Wiley
Publishing, 2010.

[37] T. Erl, SOA Design Patterns, 1st ed. Upper Saddle River, NJ, USA: Prentice Hall
PTR, 2009.

[38] P. Bresciani, A. Perini, P. Giorgini, F. Giunchiglia, and J. Mylopoulos, “Tropos:

An agent-oriented software development methodology,” Autonomous Agents and
Multi-Agent Systems, vol. 8, no. 3, pp. 203–236, May 2004. [Online]. Available:
https://doi.org/10.1023/B:AGNT.0000018806.20944.ef

[39] L. Padgham and M. Winikoff, Prometheus: A practical agent-oriented methodology,

I. Group, Ed., 2005.

[40] J. Delsing, Ed., Arrowhead Framework: IoT Automation, Devices, and Maintenance.
CRC Press, 12 2016. [Online]. Available: http://amazon.com/o/ASIN/1498756751/

[41] F. Gosewehr, J. Wermann, W. Borsych, and A. W. Colombo, “Specification and

design of an industrial manufacturing middleware,” in 2017 IEEE 15th International
Conference on Industrial Informatics (INDIN), July 2017, pp. 1160–1166.

[42] Z. M. Sikora, Entity Relationship Modelling. London: Macmillan Education UK,

1997, pp. 46–55. [Online]. Available: https://doi.org/10.1007/978-1-349-14693-2 5