Download the Full Version of textbook for Fast Typing at textbookfull.

com

Distributed Applications and Interoperable Systems

17th IFIP WG 6 1 International Conference DAIS
2017 Held as Part of the 12th International
Federated Conference on Distributed Computing
Techniques DisCoTec 2017 Neuchâtel Switzerland
https://textbookfull.com/product/distributed-applications-
June 19 22 2 1st Edition Lydia Y. Chen
and-interoperable-systems-17th-ifip-wg-6-1-international-
conference-dais-2017-held-as-part-of-the-12th-international-
federated-conference-on-distributed-computing-techniques-d/

OR CLICK BUTTON

DOWNLOAD NOW

Download More textbook Instantly Today - Get Yours Now at textbookfull.com

 Lydia Y. Chen
Hans P. Reiser (Eds.)
LNCS 10320

Distributed Applications
and Interoperable Systems
17th IFIP WG 6.1 International Conference, DAIS 2017
Held as Part of the 12th International Federated Conference
on Distributed Computing Techniques, DisCoTec 2017
Neuchâtel, Switzerland, June 19–22, 2017, Proceedings

123
Lecture Notes in Computer Science 10320
Commenced Publication in 1973
Founding and Former Series Editors:
Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board
David Hutchison
Lancaster University, Lancaster, UK
Takeo Kanade
Carnegie Mellon University, Pittsburgh, PA, USA
Josef Kittler
University of Surrey, Guildford, UK
Jon M. Kleinberg
Cornell University, Ithaca, NY, USA
Friedemann Mattern
ETH Zurich, Zurich, Switzerland
John C. Mitchell
Stanford University, Stanford, CA, USA
Moni Naor
Weizmann Institute of Science, Rehovot, Israel
C. Pandu Rangan
Indian Institute of Technology, Madras, India
Bernhard Steffen
TU Dortmund University, Dortmund, Germany
Demetri Terzopoulos
University of California, Los Angeles, CA, USA
Doug Tygar
University of California, Berkeley, CA, USA
Gerhard Weikum
Max Planck Institute for Informatics, Saarbrücken, Germany
More information about this series at http://www.springer.com/series/7411
Lydia Y. Chen Hans P. Reiser (Eds.)
•

Distributed Applications
and Interoperable Systems
17th IFIP WG 6.1 International Conference, DAIS 2017
Held as Part of the 12th International Federated Conference
on Distributed Computing Techniques, DisCoTec 2017
Neuchâtel, Switzerland, June 19–22, 2017
Proceedings

123
Editors
Lydia Y. Chen Hans P. Reiser
IBM Research Zurich Lab University of Passau
Zurich Passau
Switzerland Germany

ISSN 0302-9743 ISSN 1611-3349 (electronic)

Lecture Notes in Computer Science
ISBN 978-3-319-59664-8 ISBN 978-3-319-59665-5 (eBook)
DOI 10.1007/978-3-319-59665-5

Library of Congress Control Number: 2017941504

LNCS Sublibrary: SL5 – Computer Communication Networks and Telecommunications

© IFIP International Federation for Information Processing 2017

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book are
believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors
give a warranty, express or implied, with respect to the material contained herein or for any errors or
omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.

Printed on acid-free paper

This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
 Foreword

The 12th International Federated Conference on Distributed Computing Techniques

(DisCoTec) took place in Neuchâtel, Switzerland, during June 19–22, 2017. It was
organized by the Institute of Computer Science of the University of Neuchâtel.
The DisCoTec series is one of the major events sponsored by the International
Federation for Information Processing (IFIP). It comprises three conferences:
– COORDINATION, the IFIP WG6.1 International Conference on Coordination
Models and Languages
– DAIS, the IFIP WG6.1 International Conference on Distributed Applications and
Interoperable Systems
– FORTE, the IFIP WG6.1 International Conference on Formal Techniques for
Distributed Objects, Components and Systems
Together, these conferences cover a broad spectrum of distributed computing sub-
jects, ranging from theoretical foundations and formal description techniques to sys-
tems research issues.
Each day of the federated event began with a plenary speaker nominated by one
of the conferences. The three invited speakers were Prof. Giovanna Di Marzo Seru-
gendo (UniGE, Switzerland), Dr. Marko Vukolić (IBM Research, Switzerland), and
Dr. Rupak Majumdar (MPI, Germany).
Associated with the federated event were also three satellite events that took place
during June 21–22, 2017:
– The 10th Workshop on Interaction and Concurrency Experience (ICE)
– The 4th Workshop on Security in Highly Connected IT Systems (SHCIS)
– The EBSIS-sponsored session on Dependability and Interoperability with
Event-Based Systems (DIEBS)
Sincere thanks go to the chairs and members of the Program and Steering Com-
mittees of the aforementioned conferences and workshops for their highly appreciated
efforts. The organization of DisCoTec 2017 was only possible thanks to the dedicated
work of the Organizing Committee, including Ivan Lanese (publicity chair), Romain
Rouvoy (workshop chair), Peter Kropf (finance chair), and Aurélien Havet (webmas-
ter), as well as all the students and colleagues who volunteered their time to
help. Finally, many thanks go to IFIP WG6.1 for sponsoring this event, Springer’s
Lecture Notes in Computer Science for their support and sponsorship, and EasyChair
for providing the reviewing infrastructure.

April 2017 Pascal Felber

Valerio Schiavoni
 Preface

This volume contains the proceedings of DAIS 2017, the 17th IFIP International
Conference on Distributed Applications and Interoperable Systems, sponsored by the
IFIP (International Federation for Information Processing) and organized by the IFIP
Working Group 6.1.
DAIS was held during June 19–22, 2017, in Neuchatel, Switzerland, as part of
DisCoTec, the 12th International Federated Conference on Distributed Computing
Techniques, together with FORTE (the 37th IFIP International Conference on Formal
Techniques for Distributed Objects, Components and Systems) and COORDINATION
(the 19th IFIP International Conference on Coordination Models and Languages).
There were 23 submissions for DAIS. Each submission was reviewed by at least three,
and on average 3.7, Program Committee members. The committee decided to accept 11
full papers, two practical experience reports, and two short papers.
The accepted papers represent a compelling sample of the state of the art in the area
of distributed applications, services, and systems. There was great emphasis on data
storage and security this year. The proceedings include contributions on optimizing
distributed applications and systems (SQL streaming processing, and P2P) as well as
novel techniques to store data (data deduplication, block placement, and executable
choregraphies). The focus of the security area ranges from practical evaluation of
cryptographic schemes, specialized hardware like Intel SGX, to emerging blockchain
access control. In the area of distributed services, there are contributions on building
collaborative services and packaging micro-services are included, and the techniques to
process distributed graph.
The conference was made possible by the work and cooperation of many people
working in several committees and organizations that are listed in these proceedings. In
particular, we thank the Program Committee members for their commitment and
thorough reviews and for their active participation in the discussion phase, and all the
external reviewers for their help in evaluating submissions. We would also like to thank
Maco Vukolic, our invited keynote speaker. Finally, we also thank the DisCoTec
general chair, Pascal Felber, organization chair, Valerio Schiavoni, and the DAIS
Steering Committee chair, Rui Oliveira, for their constant availability, support, and
guidance.

April 2017 Lydia Y. Chen

Hans P. Reiser
 Organization

Program Committee Chairs

Lydia Y. Chen IBM Research Zurich Lab, Switzerland
Hans P. Reiser University of Passau, Germany

Program Committee
Luciana Arantes Université Pierre et Marie Curie-Paris 6, France
Carlos Baquero HASLab, INESC TEC and Universidade do Minho,
Portugal
Sonia Ben Mokhtar LIRIS CNRS, France
Alysson Bessani University of Lisbon, Portugal
Robert Birke IBM Zurich Research Laboratory, Switzerland
Andrea Bondavalli University of Florence, Italy
Sara Bouchenak INSA Lyon, France
Nikolaos Chrysos Foundation for Research and Technology (FORTH),
Greece
Miguel Correia INESC-ID, Instituto Superior Técnico, Universidade
de Lisboa, Portugal
Wolfgang De Meuter Vrije Universiteit Brussel, Belgium
Jim Dowling Swedish Institute of Computer Science, Sweden
Frank Eliassen University of Oslo, Norway
David Eyers University of Otago, New Zealand
Kurt Geihs Universität Kassel, Germany
Karl M. Goeschka Vienna University of Technology, Austria
Franz J. Hauck Ulm University, Germany
K.R. Jayaram IBM Research, USA
Mark Jelasity University of Szeged, Hungary
Vana Kalogeraki Athens University of Economics and Business, Greece
Evangelia Kalyvianaki City University London, UK
Ruediger Kapitza TU Braunschweig, Germany
Attila Kertesz University of Szeged, Hungary
Benny Mandler IBM Haifa Research, Israel
Miguel Matos INESC TEC and Universidade do Minho, Portugal
Rene Meier Lucerne University of Applied Sciences, Switzerland
Alberto Montresor University of Trento, Italy
Kiran-Kumar Harvard University, USA
Muniswamy-Reddy
Juan Perez Universidad del Rosario, Columbia
Peter Pietzuch Imperial College London, UK
X Organization

Altair Santin Pontifical Catholic University of Paraná, Brazil

Spyros Voulgaris VU University Amsterdam, The Netherlands

Steering Committee
Alysson Bessani Universidade de Lisboa, Portugal
Sara Bouchenak INSA Lyon, France
Jim Dowling KTH Royal Institute of Technology, Sweden
Frank Eliassen University of Oslo, Norway
Pascal Felber Université de Neuchâtel, Switzerland
Karl Goeschka Vienna University of Technology, Austria
Rüdiger Kapitza Technical University of Braunschweig, Germany
Kostas Magoutis FORTH-ICS, Greece
Rui Oliveira Universidade do Minho, Portugal
Peter Pietzuch Imperial College, UK
Romain Rouvoy University of Lille 1, France
François Taiani Université de Rennes 1, France
 Contents

Running System Efficiently (Distributed System)

Similarity Aware Shuffling for the Distributed Execution of SQL

Window Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Fábio Coelho, Miguel Matos, José Pereira, and Rui Oliveira

DIsCO: DynamIc Data COmpression in Distributed Stream

Processing Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Nikos Zacheilas and Vana Kalogeraki

Distributed Random Process for a Large-Scale Peer-to-Peer Lottery. . . . . . . . 34

Stéphane Grumbach and Robert Riemann

Storing Data Smartly (Data storage)

DDFlasks: Deduplicated Very Large Scale Data Store. . . . . . . . . . . . . . . . . 51

Francisco Maia, João Paulo, Fábio Coelho, Francisco Neves,
José Pereira, and Rui Oliveira

Block Placement Strategies for Fault-Resilient Distributed Tuple Spaces:

An Experimental Study (Practical Experience Report) . . . . . . . . . . . . . . . . . 67
Roberta Barbi, Vitaly Buravlev, Claudio Antares Mezzina,
and Valerio Schiavoni

Private Data System Enabling Self-Sovereign Storage Managed

by Executable Choreographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Sinică Alboaie and Doina Cosovan

Roaming in Graph (Graph Processing)

Scalable Anti-KNN: Decentralized Computation of k-Furthest-Neighbor

Graphs with HyFN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Simon Bouget, Yérom-David Bromberg, François Taïani,
and Anthony Ventresque

Lifting Low-Level Workflow Changes Through User-Defined

Graph-Rule-Based Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Alexander Jahl, Harun Baraki, Huu Tam Tran, Ramaprasad Kuppili,
and Kurt Geihs
XII Contents

Building Collaborative Services (Services)

Packaging Microservices (Work in Progress) . . . . . . . . . . . . . . . . . . . . . . . 131

Fabrizio Montesi and Dan Sebastian Thrane

formic: Building Collaborative Applications

with Operational Transformation (Work in Progress) . . . . . . . . . . . . . . . . . . 138
Tim Jungnickel and Ronny Bräunlich

Filament: A Cohort Construction Service for Decentralized Collaborative

Editing Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Ariyattu C. Resmi and François Taiani

Making Things Safe (Security)

Benchmarking Cryptographic Schemes for Securing Public Cloud Storages

(Practical Experience Report) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Stefan Contiu, Emmanuel Leblond, and Laurent Réveillère

Secure Cloud Micro Services Using Intel SGX . . . . . . . . . . . . . . . . . . . . . . 177

Stefan Brenner, Tobias Hundt, Giovanni Mazzeo,
and Rüdiger Kapitza

Adaptive Cheat Detection in Decentralized Volunteer Computing

with Untrusted Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Nils Kopal, Matthäus Wander, Christopher Konze,
and Henner Heck

Blockchain Based Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

Damiano Di Francesco Maesa, Paolo Mori,
and Laura Ricci

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Running System Efficiently (Distributed
System)
 Similarity Aware Shuffling for the Distributed
Execution of SQL Window Functions

Fábio Coelho1(B) , Miguel Matos2 , José Pereira1 , and Rui Oliveira1

1
INESC TEC, Universidade do Minho, Braga, Portugal
[email protected], {jop,rco}@di.uminho.pt
2
INESC-ID/IST, Lisboa, Portugal
[email protected]

Abstract. Window functions are extremely useful and have become

increasingly popular, allowing ranking, cumulative sums and other ana-
lytic aggregations to be computed over a highly flexible and configurable
sliding window. This powerful expressiveness comes naturally at the
expense of heavy computational requirements which, so far, have been
addressed through optimizations around centralized approaches by works
both from the industry and academia. Distribution and parallelization
has the potential to improve performance, but introduces several chal-
lenges associated with data distribution that may harm data locality. In
this paper, we show how data similarity can be employed across par-
titions during the distributed execution of these operators to improve
data co-locality between instances of a Distributed Query Engine and
the associated data storage nodes. Our contribution can attain network
gains in the average of 3 times and it is expected to scale as the number
of instances increase. In the scenario with 8 nodes, we were to able attain
bandwidth and time savings of 7.3 times and 2.61 times respectively.

1 Introduction

Nowadays, the scalability of database engines is paramount, specifically when

it is targeted at large scale analytical processing. Systems must be able to sup-
port several computing nodes, enabling component scalability to possibly reach
hundreds or thousands of nodes. However, reaching such scale introduces sev-
eral challenges associated with data and request distribution and balance. Cloud
computing infrastructures offer a nearly transparent environment where compu-
tation is available as virtually infinite computing nodes. However, commercial
relational database engines (RDBMS) do not conform to such paradigm, typi-
cally offering a monolithic structure. Legacy-type servers are usually considered
for running RDBMSs, limiting system scalability from the purchase moment or
until they become economically unacceptable.
Window Functions (WF) define a sub-set of analytical operations that enable
the formulation of analytical queries over a derived view of a given relation R.
They are also known as OLAP Analytical Functions and are part of the SQL:2003

c IFIP International Federation for Information Processing 2017
Published by Springer International Publishing AG 2017. All Rights Reserved
L.Y. Chen and H.P. Reiser (Eds.): DAIS 2017, LNCS 10320, pp. 3–18, 2017.
DOI: 10.1007/978-3-319-59665-5 1
4 F. Coelho et al.

standard. All major database systems like Oracle [7], IBM DB2 [14], Microsoft
SQL Server [6], SAP Hana [21], Cloudera Impala [17] or Postgresql [20] have the
ability to execute a sub-set of the available WFs.
WF are widely used by analysts as they offer a highly configurable environ-
ment together with a straightforward syntax. In fact, SQL WF are used in at
least 10% of the queries in TPC-DS [22] benchmark, a benchmark suite aimed
to evaluate data warehouse systems. Despite their relevance, parallel implemen-
tations and optimizations considering this operator are almost non existing in
the literature. While [4,18,23] are notable exceptions, these works are targeted
at many-core CPU centralized architectures that are substantially different from
distributed architectures.
The nature of current centralized architectures do not typically take into
account data distribution. This eases their processing models, but prevents them
to scale beyond the limitations of the hardware that hosts them. The massively
parallel nature that distribution approaches enable requires, however, to care-
fully address data distribution. Having the right grasp on data placement allows
to improve data movement, but requires additional mechanisms to maximize
network efficiency.
In this paper we focus on WF, particularly exploring opportunities for their
distributed execution. We propose a technique that exploits similarity between
partitions as a metric that can be used to judiciously improve the affinity of
data and computing nodes, consequently minimizing the data movement between
computing nodes.
Contributions: First, we demonstrate that it is possible to improve data
forwarding by using partition similarity to chose the forwarding mechanism
between Distributed Query Engine (DQE) workers. Second we present an exper-
imental evaluation that confirms the merit of our approach. Roadmap: The
remainder of this paper is organized as follows: Sect. 2 introduces WF. Section 3
introduces Distributed WF, describing their query execution plans and cost mod-
els. Section 4 presents our similarity technique, improving affinity between data
and computing nodes. Section 5 evaluates our proposal. Section 6 presents related
work and Sect. 7 concludes our work.

2 Window Functions
WF started to be largely adopted by database vendors from the 2011 revision of
the SQL standard. These are powerful analytical operators that enable complex
calculations such as moving, cumulative or ranking aggregations to be computed
over data. WF are expressed in SQL semantics by the keyword OVER as shown
in Fig. 1. In the next Sections we will analyze each part of the query.

Fig. 1. Example of SQL query with WF.

 Similarity Aware Shuffling for the Distributed Execution of SQL 5

Like other analytical operators, WF are required to reflect several concepts,

namely: the processing order, the partitioning of results or the notion of the cur-
rent row being computed. These design constraints are clearly translated from
the syntax as seen in the previous example, and configure two main consider-
ations as foundation for the WF environment. Firstly, WF are computed after
most of the remaining clauses in the query (e.g., such as JOIN, WHERE, GROUP
BY or HAVING), but immediately before any required final ordering (e.g., ORDER
BY). Secondly, the analytical operator to be computed with the WF environment
will create an output attribute that reflects, but does not modify or filter the
input data present in the source relation. Therefore, the result-set will present
the same cardinality of rows as in the source relation, but will have an additional
attribute mapping the result.

2.1 Partitioning, Ordering and Framing

The WF environment can be decomposed into three stages, as depicted in Fig. 2,

defining the processing order: the partitioning (1), ordering (2) and framing
(3) stages. Each stage is defined by specific clauses namely: the PARTITION BY
and the ORDER BY that respectively create logical partitions of distinct data
elements and afterwards develop an intra-partition sorting. The logical partitions
are regulated by the mandatory argument of the PARTITION BY clause, defining
the column attribute or expression that controls the partitioning. The partition
clause resembles the behavior of the GROUP BY clause, but does not collapse all
group members into a single row.

Fig. 2. Stages of the Window operator: partitioning (1), ordering (2) and framing (3).

The intra-partition ordering follows the partitioning stage and is also regu-
lated by the mandatory column attribute or expression considered as argument
for the ORDER BY clause. The ordering stage is very important for a set of non-
cumulative analytical functions, that are the focus of our contribution, but also
as it is the costliest operation in the environment [4].
Finally, the framing stage builds on the provided ordering, taking into
account the current row being considered to introduce the concept of window or
frame. The frame is built from a group of adjacent rows surrounding the current
6 F. Coelho et al.

row and changes as the current row moves towards the end of the partition.
The framing is set by either the ROWS BETWEEN or the RANGE BETWEEN clauses.
The former considers n rows before and after the current row, while the latter
restricts the window by creating a range of admissible values and, the current
row is considered if the stored values fit in the provided range1 .
The WF environment allows to combine different clauses, enabling the inclu-
sion or exclusion of each clause type. For instance, it is possible to declare a WF
with just a partitioning or ordering clause. If no partitioning clause is declared,
the entire relation is considered as a single partition. If no ordering clause is
declared, then the natural ordering of the relations key, or partitioning clause
(if present) is considered. Moreover, each available analytical function may or
not change the computation logic. Due to space constraints we do not character-
ize all the possible configurations of the WF environment. The interested reader
should consult the 2003 and 2011 revisions of the ANSI SQL standard for further
information [1].

2.2 Cumulative and Ranking Analytical Functions

The analytical set of functions currently available in most Query Engines
(QE) can be classified into Cumulative or Ranking. Cumulative analytical
functions or aggregates, are a group of functions that are not order-bound.
That is, when they are computed within a WF, an ORDER BY clause is not
required. The sum(x), avg(x) or count() are just some examples of this cat-
egory of functions. Figure 3(a) depicts the result of computing a WF struc-
tured as “select analytical function() OVER (PARTITION BY A ORDER BY
D) FROM table”, but immediately before applying the requiring analytical func-
tion to a given relation. Figure 3(b) depicts the result of computing the previous
WF with the sum(D) function. The result of a cumulative function is the same
for all the members belonging in the same partition.

Fig. 3. WF query as: select analytical function() OVER (PARTITION BY A ORDER

BY D) FROM table. (a) WF where the partition by clause generated 1 partitions. (b)
Cumulative (sum) analytical function over WF in (a). (c) Ranking (rank) analytical
function over WF in (a)
1
Typically, the use of this clause is restricted to numeric types.
 Similarity Aware Shuffling for the Distributed Execution of SQL 7

Ranking analytical functions, on the other hand, are order-bound. That is,
the function requires the data to be ordered according to some criteria in order
to output a deterministic result, and thus, the ordering clause is always required.
The rank(), dense rank() or ntile() are just some examples of this category of
functions. Figure 3(c) outputs the result of computing the previous WF with the
rank() function, outputting a different result for each row in the partition.
The ordering requirement for the latter category of functions implies data
co-locality in order to minimize the number of sorting steps needed to achieve
intra-partition ordering [4]. In the remainder of this paper, we consider a WF
computing a ranking analytical function with a single partition and ordering
clause and no framing clause, since the rank function implicitly defines framing
constraints.

3 Distributed Window Functions

RDBMS are built from several components, namely the QE and the Query Opti-
mizer (QO). The former translates SQL syntax into a set of single operators. The
latter considers several statistical techniques to improve the query execution
plan of a query. In a nutshell, QEs split the execution of a query in two separate
stages: the query planning and the query execution. During the first stage, the
QE decides how the query is executed during the second stage, and which oper-
ators are used in such a query plan. This builds a complex multi-optimization
problem that has to be executed in polynomial time.
The QO uses hints about data in the form of statistical approximations,
allowing the query engine to optimize query execution based on the approxima-
tion cost of each individual operator in a given data set. When scaling from a
single QE to a DQE, data partitioning techniques are necessary in order to dis-
tribute data among instances. The number of available computing nodes config-
ures the installed Degree of Parallelism (DOP). However, non-cumulative ana-
lytical algorithms are order-bound, thus requiring that logical data partitions
are co-located (i.e., they should live in the same storage node). If elements of a
given logical partition are spread in a group of nodes it becomes impossible to
sort each logical partition in just one step. The sorting in each data partition
would induce a partial sorting that is not deterministic and that would prevent
inter-partition parallelism. The QOs therefore need to adapt their cost models
to reflect the data movement required in order to ensure co-locality of partitions
during execution time of the operator.

3.1 Distributed Query Engines

The DQE takes advantage of data distribution in order to scale query execu-
tion. The present architecture is provided by a Highly Scalable Transactional
PaaS [16]. Each node in the system is split in two layers, the DQE itself and
the storage layer, holding the data partitions to be manipulated by a given
DQE instance. Particularly, the considered DQE is based on the Apache derby
8 F. Coelho et al.

Fig. 4. Distributed Query Plan for Ranking WF. Round boxes represent individual
stages of the WF environment. Arrows represent data flow in a process or over the
network. PB and OB respectively represent Partition By and Order By attributes. H
describes a statistical histogram. Numbers represent process execution order.

project [2] and the storage layer is provided by Apache HBase [11], working over
the Hadoop Distributed File Systems (HDFS) [13].
The DQE instances are able to accept client query requests through a JDBC
connection and generate the distributed query plan. This plan is then shared
with all participating DQE instances. The data distribution in each Storage
Node is typically accomplished by means of an Hash function, considering a
single or a collection of attributes as key. The distribution of keys lies within the
inner characteristics of the considered hash function, usually producing uniform
distributions that evenly place tuples across all available storage nodes. Poorly
chosen hash functions may result in data skew and should be tailored to each
specific workload providing adjusted table splitting [8].
Figure 4 presents the simplified distributed query plan for a ranking analyt-
ical function. The following stage numbers resemble the ones depicted in Fig. 4.
With data partitioned in several nodes, each one will scan (1) its local partition.
The partial results found in each node derive from the data partitioning required
to distributed data. Data movement is then required in order to ensure that each
logical partition created by the partitioning clause will reside in a single node for
computation. This is achieved by the shuffle mechanism (2). Afterwards, data
is sorted according to the partitioning and ordering clauses (3), and results are
submitted to the rank function (4). At this stage, each computing node holds
partial results from each logical partition. The results from each logical partition
are then reunited in a single location (5) before being delivered to the client (6).
 Similarity Aware Shuffling for the Distributed Execution of SQL 9

Ranking aggregation algorithms are dependent on having full disclosure of the

entire logical partitions. If the first shuffling stage (2) is not performed, the par-
tial results in each partition will produce incorrect results. That is, if members of
a logical partition are processed in the same node where they are stored (there-
fore in distinct DQE instances), the partial aggregation results produced will not
be able to sort the entire logical partition. Thus, when the partial results of log-
ical partitions are merged in the final result-set (6), they will need to be entirely
recomputed. By considering the first shuffle stage (2), the results produced by
stage (3) in each logical partition are globally correct since, independence from
logical partitions ensures inter-partition parallelism, allowing computation to be
distributed through several computing locations. The partition strategy consid-
ered depends on the mandatory argument of the partitioning clause. It is thus
impractical to adjust the table splitting of the workload to a specific partitioning
clause, since the ideal configuration may change with each query. Moreover, the
environment allows the use of expressions as the arguments of the inner clauses,
posing an extra hurdle to this abstraction.

3.2 Data Shuffling

Data movement during the execution of a WF query is required, ensuring that all
the elements of each logical partition are in the same location. In order to judi-
ciously forward data while minimizing at the same time the transfer cost, in [5]
we introduced a mechanism that works together with the data transfer mecha-
nism, a shuffler, promoting co-location of logical partitions. This is achieved by
considering an histogram, characterizing the universe of elements present in each
partition. Briefly, the histogram should hold the cardinality of each different ele-
ment in each different column qualifier. The histograms referring to each node
are then combined into a global histogram. The introduction of this mechanism
along with the shuffler, allowed to forward data to the specific node that should
process a given partition.
Consider Fig. 5 where a table similar to the table in Fig. 3(a) was split
in two partitions on the storage layer. This initial partitioning is defined
by hashing the value of the nodes ol w qualifier and performing the arith-
metic modulo between the hash result and the number of computing instances
(Hash(value in ol w) % #N odes). Guided by the query in Fig. 1, the results
were then ordered according to the qualifier ol d (the partitioning clause). Both
nodes of the storage layer hold elements from the available three partitions in
ol d (p1, p2, p3). According to the previously introduced, ol d partitions (p2)
and (p3) in instance DQE w1 will be relocated to instance DQE w2 and, ol d
partition (p1) will be relocated from instance DQE w2 to instance DQE w1.
On the one hand, hash forwarding a single row at a time prevents batching
several rows in a single request. On the other hand, due to the asynchronous
nature of DQEs, latency is usually not the bottleneck and thus, data movement
can be delayed until network usage can be maximized [12]. This enables the
use of batching in order to improve network usage. A batch payload is formed
by grouping rows that need to be forwarded to a common destination and it is
10 F. Coelho et al.

Fig. 5. Shuffling instances partitioned by ol w. In WF context, they were partitioned

by ol d and Ordered by ol num. The DQE instances will use the network to combine
partitions during execution time. Instance w1 will hold partitions ol d = 1, instance
w2 will hold partitions ol d = 2 and ol d = 3, respectively.

regulated by a buffer within the shuffling mechanism, whose size and delivery
timeout are configurable. Nevertheless, the use of this mechanism can prove to be
a misfit in cases where workloads do not benefit from grouping data (i.e., logical
partitions with reduced number of rows). Therefore, not having to delay data
transmission reduces execution time. To understand up to what level a given
logical partition may or not benefit from batching, we considered a correlation
mechanism to guide such decision, identifying the logical partitions that are good
candidates for forwarding data in batch.

4 Similarity

QOs found in modern QEs use several statistical mechanisms to explore data
features in order to improve query execution performance. Without them, inde-
pendence assumptions between attributes are preserved, which commonly leads
to under or over provisioned query plans, which is particularly undesirable in
DQEs. As in real-world data, correlations between relation attributes are the
rule and not the exception, the array of correlation or other algebraic extraction
mechanisms in the literature is vast, namely [3,9,19]. Correlations can also be
used in DQEs to improve how data distribution is handled. When logical data
partitions need to be relocated in order to improve co-locality, the correlation
between qualifiers in different locations of the storage layer can be explored to
minimize the required data movement.
In this paper, we introduce a similarity measure to quantify to what level
the partitions of a given attribute held by different storage nodes are alike.
Data partitions with high similarity are good candidates to be shuffled within
a batch payload. This is so as a high similarity implies a high common number
of partitions. On the other hand, data partitions with low similarity are better
candidates to be immediately shuffled for their destination. This is so as they
share a low number of common partitions. This is efficiently achieved through
Algorithm 1. The similarity measure quantifies in a universe between 0 (not
 Similarity Aware Shuffling for the Distributed Execution of SQL 11

Algorithm 1. Similarity Aware Shuffling Mechanism

1: P (r) =< r0 , r1 , r2 , rn >← partition
2: ri ← current row
3: pbk ← partition by key
4: w id ← worker id
5: H ← histogram
6: t ← similarity threshold
7: procedure Similarity(attr A, attr B)
8: Sim ← unique(attr A∩attr B)
unique(attr A∪attr B)

9: procedure BatchShuffling(P (r), dest)

10: send P (r) to dest
11: procedure HashShuffling(ri , dest)
12: send ri to dest
13: function Shuffler
14: dest ← H(ri .pbk)
15: if w id = dest then
16: Sim ← similarity(w id pbk,dest pbk)
17: if Sim > t then
18: BatchShuffling(P (r), dest)
19: else
20: HashShuffling(ri , dest)

similar) and 1 (similar) how similar two attributes are, by considering the number
of unique values in each attribute to compute the metric. The data required to
compute this metric is already provided by the histogram introduced in previous
work [5], bypassing the need to collect additional statistical data. This structure
is characterized by a small memory footprint (few KB) and the update period
dictated by the DQE administrator. This algorithm will be considered during
the first shuffling stage (stage (2) of Fig. 5). It will consider each logical partition
(P (r)), the previously introduced Histogram (H) and a configurable similarity
threshold. Three auxiliary procedures are considered. The SIMILARITY procedure
computes the similarity measure from the set of unique values in the qualifiers
considered as arguments. The BATCHSHUFFLING procedure marshals all the rows
of partition P (r) and sends it to the destination worker dest. The HASHSHUFFLING
procedure marshals a single row ri and sends it to destination dest.
When the shuffler action is required, it consults the Histogram H to verify
what is the optimal destination (DQE instance) from row ri . When the des-
tination is a remote instance (line 15), the shuffling mechanism computes the
similarity measure between the local (attrA ) and destination (attrB ) qualifiers
(line 16). The partition P (r) is marshaled to the appointed destination when
the observed similarity is above threshold t (line 18) (BATCHSHUFFLING), or each
row ri is otherwise sent to destination (line 20) (HASHSHUFFLING). The parame-
ter t sets a threshold above which rows are forwarded in batch to the destination
instance. This parameter defaults to 0.5 meaning that if not modified, rows are
batch forwarded if the origin contains at least half the number of unique partition
values of the destination.
12 F. Coelho et al.

5 Evaluation
We validated that by batch shuffling tuples between DQE instances we would
save bandwidth, improving execution time of the shuffling stage. We considered
a synthetic data set and shuffled rows between distinct DQE instances. The data
set used was extracted from the TPC-DS [22], a benchmark suite tailored for
data analytics. We extracted a single relation (web sales) which is composed
of 35 distinct attributes, configuring TPC-DS with a scale factor of 50 GB. This
resulted in a relation with 9.4 GB corresponding to 36 million rows.
The outcome of the mechanism we propose is directly related with the data
distribution considered. In order to bound the outcome of our contribution in
terms of the lower and upper performance bounds, we statistically analyzed the
considered relation. The lower bound is set by not using the similarity mecha-
nism. The upper bound is set by considering the relation attributes that would
favor data distribution. This was achieved by identifying the placement key
attribute, but also a candidate attribute to be the partitioning clause or shuf-
fling key (PBK) of the WF. The placement key attribute will define the data
distribution in each DQE Storage Node through the use of an Hash function,
and the PBK will define the runtime partitioning within the WF environment.
The results are depicted in Fig. 6. The top plot presents the number of par-
titions in each single attribute in the considered relation. That is, the number
of unique values in each attribute. The bottom plot depicts the average cardi-
nality of each partition. That is, the average number of elements in each group
of unique values in each attribute. The ideal candidate attribute to become the
relation placement key is the attribute that displays the highest partition num-
ber and at the same time holds the smallest cardinality, ensuring an even data
distribution and reduced data skew. Observing both plots leads us to consider
attribute with index 17 (ws order number), displaying the highest number of

Fig. 6. Number of partitions per attribute (top) and the average number of elements
per partition/attribute (bottom). The horizontal axis represents the attribute index.
The vertical axis quantifies each measure in logarithmic scale. The attribute considered
for placement key (PK) is shown in black and the candidates for WF Partition By key
(PBK) are shown in dark gray.
 Similarity Aware Shuffling for the Distributed Execution of SQL 13

Fig. 7. Similarity between attributes in two data nodes. Horizontal axis represents the
attribute index. Vertical axis represents the Similarity measure in logarithmic scale.

partitions, each one with a single element. On the other hand, the candidate
attributes to be selected as WF PBK are the attributes that would hold at the
same time a high number of partitions and high partition cardinality. These are
good PBK candidates since they will induce a number of logical partitions that
is above the configured DOP. The observation of the plots leads to identify as
candidates the attribute indexes depicted in dark gray, from which we select
attribute 0 (ws sold date sk) as PBK.
After the election for the PK and PBK keys, we conducted a second experi-
ment to verify the computed similarity measure. Figure 7 depicts the results of
applying the metric in two scenarios. In both cases, we consider our scenario to
be built from several DQE instances and corresponding Storage Nodes. On all
experiments, we considered only the communication layer of the DQE where our
contribution is, thus avoiding the SQL parsing and optimization stages. Each
data partition was computed by applying an Hash function with the elected PK
dividing the data into as many partitions as configured DQE instances. We first
considered the configuration with 2 instances A and B. In the experiment in the
top plot we computed the similarity measure between the PBK of location A and
each distinct attribute in location B. It is possible to observe that attribute 0 in
location B presents the highest similarity, followed by attribute two. These are
also the only attributes that are above the set up threshold of 0.5 denoted by
the horizontal line. The remaining attributes have a residual similarity measure.
The bottom plot depicts a different configuration where attribute 15 was ran-
domly chosen among all non candidate attributes. The similarity measure in this
attribute is lower than our threshold, even though it seems to be equal given
the logarithmic scale required to observe the remainder attributes. Therefore,
the results achieved during the first configuration would induce the shuffler to
use batching mechanisms to forward partitions among DQE instances, instead
of hash forwarding. The latter would culminate in sending a single row at a time.
In order to verify the impact of our contribution regarding network usage, we
conducted an experiment to assess the magnitude of the network savings pro-
moted. Namely, we considered configurations with 2, 4 and 8 DQE and Storage
14 F. Coelho et al.

instances. The computing nodes were only set up with the communication layer
responsible for the shuffling in the WF environment. Each node is comprised of
commodity hardware, with an Intel i3-2100-3.1 GHz 64 bit CPU with 2 physi-
cal cores (4 virtual), 8 GB of RAM memory and one SATA II (3.0 Gbit/s) hard
drive, running Ubuntu 12.04 LTS as the operating system and interconnected by
a switched Gigabit Ethernet network. During execution, each computing node
acts as a DQE instance shuffler, forwarding data to the remainder instances.
In a distributed deployment, the DQE instance will be co-located with other
services (e.g., storage node) which will typically restrict the available memory
to the DQE instance.
We evaluated two configurations where the first represents a baseline compar-
ison, forwarding all data by hash shuffling, and a second where data is forwarded
according to our similarity mechanism.

Fig. 8. Bandwidth (outbound) registered during shuffling between instances.

The results depicted in Fig. 8 are twofold. The similarity measure registered
both a decrease in bandwidth and it also promoted a shorter execution period
for the shuffling technique. This is the result of pairing the batch shuffling mech-
anism together with the proposed similarity measure. The savings induced come
at a residual cost, since the statistical information is not collected for the single
purpose of this improvement, nor it has to be updated in each query execution.
The similarity measure technique only proved effective from the configuration
with 4 instances onward, since it is only from that configuration that both band-
width and execution time are lower than the baseline. For the configuration with
only two nodes, the baseline technique proved to be better by both shortening
the shuffling time and registered bandwidth. However, in the configurations with
4 and 8 nodes, the similarity measure was able to reduce the bandwidth and
execution time when compared with the baseline approach. As the number of
partitions in the system increase, each single partition becomes responsible for
a shorter set of data, promoting bandwidth savings up to 7.30 times for the 8
node configuration.
The previous experiment evaluated the shuffling mechanism by considering
an attribute with ideal similarity measure and partitioning on the storage layer.
 Similarity Aware Shuffling for the Distributed Execution of SQL 15

Table 1. Total Bandwidth (sent) and execution time registered for each configuration.

2 nodes 4 nodes 8 nodes

Baseline (MB) 1,132.45 4,172.59 7,237.56
Similarity (MB) 2,365.34 1,695.24 991.72
Bandwidth Gain (x) −0.48 2.46 7.30
Shuffle time
Baseline (sec) 149 172 170
Similarity (sec) 226 114 65
Speed up (x) −0.48 1.55 2.61

In order to demonstrate the impact of selecting an attribute that do not favor

an uniform distribution of data among data partitions, we conducted a second
experiment that considered an attribute with poor partitioning properties (i.e.,
reduced number of partitions). The results consider the same component config-
uration, but selected attribute 15 (ws warehouse sk) for the partitioning. When
selecting an attribute that lacks the desirable distribution, the logical partitions
will present an imbalance, thus promoting a low similarity measure. Therefore,
the shuffling mechanism will not be able to maximize network usage and will
end up having to consider the HASHSHUFFLING mechanism to forward data. The
results are not thoroughly presented due to space constraints. However, we point
out that they are in line with the considered baseline results presented in Table 1,
registering a bandwidth variance of ±4%. Moreover, even though we do not con-
sider it, the use of compression techniques may further increase the observed
savings.

6 Related Work
Window Functions were introduced in the 2003 SQL standard. Despite its rele-
vance, parallel implementations and optimizations considering this operator are
almost non existing. Works such as [4] or [23] fit in the first category, respectively
tackling optimization challenges related with having multiple window functions
in the same query, and showing that it is possible to use them as a way to avoid
sub-queries and lowering quadratic complexity. However, such approaches do
not offer parallel implementations of this operator. A vast array of correlation
mechanisms have been so far deeply studied in the literature. Nonetheless, most
of the conducted studies focus on efficient ways to discover and exploit soft and
hard correlations [15], allowing to find different types of functional dependen-
cies. Works like [18] introduced mechanisms to improve the performance of the
WF environment when many-core architectures are used. Distinct approaches
and algorithm improvements are introduced, enabling to parallelize the distinct
stages of the operator.
When addressing WFs, a common misconception generally brings a compar-
ison between SQL WF (in which our contribution focuses) and CEP windowing.
16 F. Coelho et al.

Differences are both semantical and syntactical. On the one hand, the CEP
environment is characterized by an incoming and infinite stream of events. From
there, a configurable, but constant sample (e.g., window) builds a sketch [10]
where aggregations are derived. On the other hand, SQL WF are computed over
finite sets built from SQL relations. While the former windows are fixed and the
data moves through, in the latter, the data is fixed and the window performs
the movement. Moreover each approach considers distinct SQL keywords (e.g.,
OVER, RETAIN) and subsequent syntax.

7 Conclusion

WF with ranking analytical functions are required to have full disclosure of a

given logical partition. Data partitioning is required to enable systems to scale,
but harms data locality, which poses added difficulties when trying to parallelize
these functions.
In this paper we motivate and validate how similarity between partitions can
be used to promote efficient data forwarding among instances of a DQE. We
introduced an algorithm to choose whether to batch or to hash forward rows
between such instances by understanding how the similarity measure between
distinct partitions of a DQE can be used towards the effectiveness of the WF
environment.
The WF environment changes how analytical functions are computed, requir-
ing specific implementation details for each functions. We therefore plan to lever-
age such parallelization opportunities to other analytical functions.

Acknowledgments. The research leading to these results was part-funded by

(1) the European Union’s Horizon 2020 - The EU Framework Programme for
Research and Innovation 2014–2020, under grant agreement No. 732051; (1) Project
TEC4Growth - Pervasive Intelligence, Enhancers and Proofs of Concept with Industrial
Impact/NORTE-01-0145-FEDER-000020 is financed by the North Portugal Regional
Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership
Agreement, and through the European Regional Development Fund (ERDF) and by
(1) the ERDF – European Regional Development Fund through the Operational Pro-
gramme for Competitiveness and Internationalisation - COMPETE 2020 Programme
within project POCI-01-0145-FEDER-006961, and by National Funds through the FCT
– Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and
Technology) as part of project UID/EEA/50014/2013.

References
1. ANSI: Information technology - database languages - SQL multimedia and
application packages. Technical report, ANSI (2003). http://webstore.ansi.org/
RecordDetail.aspx?sku=ISO%2fIEC+13249-2%3a2003
2. Apache: The apache derby project. Technical report, Apache Foundation (2016).
https://db.apache.org/derby/derby charter.html
 Similarity Aware Shuffling for the Distributed Execution of SQL 17

3. Brown, P.G., Hass, P.J.: BHUNT: automatic discovery of fuzzy algebraic con-
straints in relational data. In: Proceedings of the 29th International Conference on
Very Large Data Bases, vol. 29, pp. 668–679. VLDB Endowment (2003)
4. Cao, Y., Chan, C.Y., Li, J., Tan, K.L.: Optimization of analytic window functions.
Proc. VLDB Endowment 5(11), 1244–1255 (2012)
5. Coelho, F., Pereira, J., Vilaça, R., Oliveira, R.: Holistic shuffler for the paral-
lel processing of SQL window functions. In: Jelasity, M., Kalyvianaki, E. (eds.)
DAIS 2016. LNCS, vol. 9687, pp. 75–81. Springer, Cham (2016). doi:10.1007/
978-3-319-39577-7 6
6. Microsoft Corporation: Transact-SQL. Technical report, Microsoft Corporation
(2013). https://msdn.microsoft.com/library/ms189461(SQL.130).aspx
7. Oracle Corporation: SQL analysis and reporting. Technical report, Oracle Cor-
poration (2015). http://docs.oracle.com/database/121/DWHSG/analysis.htm#
DWHSG8659
8. Cruz, F., Maia, F., Oliveira, R., Vilaça, R.: Workload-aware table splitting for
NoSQL. In: Proceedings of the 29th Annual ACM Symposium on Applied Com-
puting, SAC 2014, pp. 399–404. ACM, New York (2014). http://doi.acm.org/10.
1145/2554850.2555027
9. Fan, W., Geerts, F., Jia, X., Kementsietsidis, A.: Conditional functional depen-
dencies for capturing data inconsistencies. ACM Trans. Database Syst. (TODS)
33(2), 6 (2008)
10. Garofalakis, M., Keren, D., Samoladas, V.: Sketch-based geometric monitoring
of distributed stream queries. Proc. VLDB Endowment 6(10), 937–948 (2013).
http://dx.doi.org/10.14778/2536206.2536220
11. George, L.: HBase: The Definitive Guide: Random Access to Your Planet-Size
Data. O’Reilly Media, Inc., USA (2011)
12. Gonçalves, R.C., Pereira, J., Jiménez-Peris, R.: An RDMA middleware for
asynchronous multi-stage shuffling in analytical processing. In: Jelasity, M.,
Kalyvianaki, E. (eds.) DAIS 2016. LNCS, vol. 9687, pp. 61–74. Springer, Cham
(2016). doi:10.1007/978-3-319-39577-7 5
13. Hadoop Apache: Hadoop (2009)
14. IBM: OLAP specification. Technical report, IBM (2013). http://www.ibm.com/
support/knowledgecenter/SSEPGG 10.5.0/com.ibm.db2.luw.sql.ref.doc/doc/
r0023461.html
15. Ilyas, I.F., Markl, V., Haas, P., Brown, P., Aboulnaga, A.: CORDS: automatic
discovery of correlations and soft functional dependencies. In: Proceedings of the
2004 ACM SIGMOD International Conference on Management of Data, pp. 647–
658. ACM (2004)
16. Jimenez-Peris, R., Patiño-Martinez, M., Magoutis, K., Bilas, A., Brondino, I.:
Cumulonimbo: a highly-scalable transaction processing platform as a service.
ERCIM News 89(null), 34–35 (2012)
17. Kornacker, M., Behm, A., Bittorf, V., Bobrovytsky, T., Ching, C., Choi, A.,
Erickson, J., Grund, M., Hecht, D., Jacobs, M., et al.: Impala: a modern, open-
source SQL engine for hadoop. In: CIDR, vol. 1, p. 9 (2015)
18. Leis, V., Kundhikanjana, K., Kemper, A., Neumann, T.: Efficient processing of
window functions in analytical SQL queries. Proc. VLDB Endowment 8(10), 1058–
1069 (2015). http://dx.doi.org/10.14778/2794367.2794375
19. Liu, H., Xiao, D., Didwania, P., Eltabakh, M.Y.: Exploiting soft and hard correla-
tions in big data query optimization. Proc. VLDB Endowment 9(12), 1005–1016
(2016). http://dx.doi.org/10.14778/2994509.2994519
18 F. Coelho et al.

20. Postgresql: Advanced features - window functions. Technical report, Postgresql

(2015). https://www.postgresql.org/docs/9.4/static/tutorial-window.html
21. SAP: SAP HANA SQL reference (2014). https://help.sap.com/hana/SAP HANA
SQL and System Views Reference en.pdf?original fqdn=help.sap.de
22. Transaction Processing Performance Council: TPC Benchmark DS (2012). http://
www.tpc.org/tpcds/spec/tpcds 1.1.0.pdf
23. Zuzarte, C., Pirahesh, H., Ma, W., Cheng, Q., Liu, L., Wong, K.: Winmagic: sub-
query elimination using window aggregation. In: Proceedings of the 2003 ACM
SIGMOD International Conference on Management of Data, pp. 652–656. ACM
(2003)
 DIsCO: DynamIc Data COmpression
in Distributed Stream Processing Systems

Nikos Zacheilas(B) and Vana Kalogeraki

Athens University of Economics and Business, Athens, Greece

{zacheilas,vana}@aueb.gr

Abstract. Supporting high throughput in Distributed Stream Process-

ing Systems (DSPSs) has been an important goal in recent years. Cur-
rent works either focus on automatically increasing the system resources
whenever the current setup is inadequate or apply load shedding tech-
niques discarding some of the incoming data. However, both approaches
have significant shortcomings as they require on the fly application recon-
figuration where the application needs to be stopped and re-uploaded in
the cluster with the new configurations, and can lead to significant infor-
mation loss. One approach that has not yet been considered for improv-
ing the throughput of DSPSs is exploiting compression algorithms to
minimize the communication overhead between components especially in
cases where we have large-sized data like live CCTV camera reports. This
work is the first that provides a novel framework, built on top of Apache
Storm, which enables dynamic compression of incoming streaming data.
Our approach uses a profiling algorithm to automatically determine the
compression algorithm that should be applied and supports both loss-
less and lossy compression techniques. Furthermore, we propose a novel
algorithm for determining when profiling should be applied. Finally, our
detailed experimental evaluation with commonly used stream processing
applications, indicates a clear improvement on the applications’ through-
put when our proposed techniques are applied.

1 Introduction
In recent years we observe a growing need for supporting complex real-time
processing of “big data”. Many systems need to process large volumes of live
data to detect events of interest in real-time. For example, in a traffic moni-
toring application it is necessary to inform the city’s authorities for events like
traffic congestion or accidents [19] as they occur. Similarly, healthcare appli-
cations [11] receive input from multiple sensors to detect unusual behavior in
the patients’ conditions. In order to be able to analyze such a high volume of
data, novel distributed systems such as Storm [14], Spark [21] and Flink [4] have
been proposed that enable us to perform scalable and low latency complex event
detection.
One important challenge in such systems is to support high throughput dur-
ing the applications’ execution despite changes in the data size or the input rate.

c IFIP International Federation for Information Processing 2017
Published by Springer International Publishing AG 2017. All Rights Reserved
L.Y. Chen and H.P. Reiser (Eds.): DAIS 2017, LNCS 10320, pp. 19–33, 2017.
DOI: 10.1007/978-3-319-59665-5 2
Exploring the Variety of Random
Documents with Different Content
 By Permission of Woman’s Land Army
Training Camp of Woman’s Land Army
The former report, covering the conditions at the end of our first
year of war, showed 100,000 women working in private munition
plants and Government owned arsenals, another 100,000 in trades
necessary for the prosecution of the war, such as work in airplane
factories, in chemical plants, in those making electrical appliances
and in metal trades making bolts, screws and other small parts
necessary for the building of many war essentials. More than
600,000 women were engaged in the manufacture of things
necessary for the soldier’s equipment and 800,000 more in
industries necessary to feed and clothe him. All these numbers were
greatly augmented during the seven following months until the close
of the war.
Training classes and entering schools were established in scores
of plants for the training of unskilled women workers. Practically all
the employers of women bore testimony to the efficiency with which
they worked. In order to protect their welfare the United States
Department of Labor organized a Woman in Industry Service which,
by means of a council of representatives from all the Federal
agencies for the prosecution of the war in which women were
employed, established standards and policies for the controlling of
wages and industrial conditions in plants employing women.
More than 100,000 women entered the service of the Railroad
Administration, where they undertook capably many forms of
unskilled labor and held many varieties of positions requiring
knowledge and experience, from bookkeeping to office
superintendency, while many thousands more filled places left
vacant by men on surface, elevated and subway car lines.
It is impossible even to estimate the number of women who
engaged in the production of food for the purpose of aiding the war.
They cultivated war gardens from end to end of the country; in the
South young women of social station, because of the lack of the
usual labor, helped to gather cotton and other crops; in the
Northwest women volunteered their help in the harvesting season
and in some localities they formed half or more of the workers who
shocked the grain in the fields; in other regions they picked berries
and gathered fruit; they went from cities and towns to country
districts to help the farmers’ wives; they took an active part,
individually and through clubs, in the increase of poultry, hog and
dairy production; in state after state they registered for farm work;
and they organized the Woman’s Land Army which gave much and
efficient aid in many parts of the country.
The Woman’s Land Army of America, numbering 15,000
members, was composed of women who had previously done little
or no farm work and who enlisted in it primarily for the sake of doing
something of consequence to help win the war. It was organized in
seventeen states, the state organizations uniting under the national
organization and each one forming and training its own farm units. In
one state, New York, there were forty of these land units, each
established at a camp under a woman supervisor. They lived at the
camp, boarding themselves, and were carried in their own auto-truck
to and fro between the camp and the farms where they worked by
the eight hour day. They were carefully selected from volunteers for
the work on the basis of physical qualification and probable morale
and among their numbers were represented teachers, college girls,
art students, telephone operators, stenographers, women of leisure.
They planted, plowed and hoed, aided in the harvesting, drove
horses and tractors, gathered fruit, did dairy work, cared for poultry
and stock and proved themselves equal to all the usual work of
truck, dairy and general farming. There were, altogether, one
hundred and twenty-seven units, ranging from twenty to one hundred
and fifty members each. Farmers who employed them found them
capable and efficient and their labor proved to be a welcome factor
in solving the problem of increasing farm production when farm help
had been seriously depleted by the draft and the munition factories.
So successful was the Woman’s Land Army during the first year of
its existence that in the autumn of 1918 an enthusiastic campaign
was started for increasing its numbers the following year and plans
were laid for courses of training during the winter.
In the conservation of food women everywhere coöperated with
the Government in many ways. They enthusiastically supported the
requests of the Food Administration, their organizations sent out food
experts, dieticians, conservation instructors through country districts,
into villages and towns and among the women of the poorer quarters
in cities to give free instruction in the economical but efficient use of
foods and in the best ways of canning, preserving and dehydrating
fruits and vegetables.
In the financing of the war the women of the country gave
noteworthy help. The National Woman’s Liberty Loan Committee
was organized by the Secretary of the Treasury in May, 1917, as an
independent bureau of the Treasury Department, the first and thus
far the only executive committee of women in the Government of the
United States. It was created too late to give much assistance in the
first Liberty Loan, but it was active in all the succeeding ones and
was thoroughly organized all over the country, for the greater part by
states, with county organizations under the state or the district. It had
3,200 county chairmen and under these, reaching out into every
community, 49,500 associate chairmen, while 800,000 women were
engaged in its work. They organized meetings, engaged speakers
and secured booths and workers for the sale of bonds, but the
greater part of the work of the organization was done by canvassing
from house to house.
This they did in cities, towns, villages, country districts, on foot, on
horse-back, by carriage. They did not stop for rain, or sun, or wind,
for dust, or mud. If it was planting time and all the horses of the farm
were in use, the chairman of a rural committee walked miles upon
miles to cover her territory. In two or three counties of the southern
mountain region famous for their bloody feuds women rode on
horseback up and down the mountain sides day after day
canvassing for the Liberty Loans and carrying the counties over the
top triumphantly with subscriptions above their quotas early in the
course of each campaign. In these counties so many men had
enlisted in the army before the draft went into effect that the burden
of taking care of the loans fell to women.
In state after state the Woman’s Committee raised from one-third
to one-half the quotas of the entire state and in the three Liberty
Loans in which it worked it sold $3,500,000,000 worth of bonds. It
was equally active in the campaigns for the sale of War Savings
Stamps and its aid proved so important that in several of the Federal
Reserve Districts it was asked to take over the entire work.
The importance of the aid American women gave to the Red
Cross was beyond computation and was so varied in kind and
enormous in quantity that anything more than the merest outline of it
is impossible. Volunteer women workers, nearly all of them doing the
work at odd moments in addition to their home or other duties,
knitted and sewed so busily that they made nearly 300,000,000
articles, valued at $60,000,000, for the Red Cross, to be used in
training camps, by our fighting forces, in hospitals at home and
abroad and by the refugees and sufferers in the war ridden countries
of Europe.
Many thousands of women worked in canteens, poured coffee, tea
and chocolate and carried baskets of cakes and cigarettes for the
refreshment of soldiers as their troop trains stopped at stations on
their way to and from cantonments or poured into and out of ports of
embarkation. More than a million and a half of the soldiers of
America as they boarded their transports had their last touch of
home at the hands of Red Cross women who, no matter what the
hour of day or night, were ready at the piers with buns and cigarettes
and cans of steaming hot drinks.
Many other thousands enlisted for the Red Cross Home Defense
work and in its offices or as home visitors gave advice, aid, comfort
to the families of soldiers and sailors, helped them to meet their
problems, material, financial, spiritual, and procured for them, when
necessary, professional advice and assistance, thus aiding morale at
the front by upholding that of the family at home. Other thousands of
women wearing the Red Cross insignia worked in the hospitals
overseas and in convalescent homes on both sides of the ocean. No
less than 8,000,000 women, and probably more, were actively
working for the Red Cross throughout the war, organizing, directing
and aiding the work of its chapters and making hospital bandages,
sweaters and other knitted articles, clothing for refugees, and
repairing soldiers’ garments.
More than 16,000 trained nurses enlisted in war service and
worked in hospitals at home and overseas and 10,000 more had
enrolled for service at the end of hostilities. The organization of the
American Women’s Hospitals of the Red Cross recruited, organized
and sent to France several units, each consisting of ten women
physicians and as many aids, with the necessary hospital
equipment.
Several hundred women entered the navy as yeomen and gave
capable and efficient service. Others joined the Signal Corps of the
army, 233 of these going to France, where their work as telephone
and telegraph operators received high praise from army officers.
In work for the welfare of the fighting forces the women of every
part of the country took a very prominent part. The War Camp
Community Service, described in “Big-Brothering the Fighting
Forces,” was carried on largely by their efforts. Organizations of
women of many kinds drew together women of similar occupations
for welfare work or brought together those of the greatest variety for
the same ends. The Stage Women’s War Relief, composed of
actresses, made and sent abroad or to hospitals at home great
quantities of comfort kits, knitted articles, bandages, hospital
supplies, dainties to tempt the appetite of convalescents, clothing for
refugees, cigarettes and tobacco. The members of the Young
Women’s Christian Association were to be found in active work for
the war in nearly all the camps and cantonments of the United
States, and also in France, and even in the frozen north of Russia,
where in several cities their Hostess Houses and canteens offered
cheer and comfort to soldiers and sailors.
The Association established a War Work Council which devised
and carried out methods by which it could aid in the prosecution of
the war. Its Hostess Houses in camps and cantonments were links
between the men in training and the life they had put behind them,
where their relatives and friends could meet them in pleasant
surroundings. The type of the Hostess House was created for the Y.
W. C. A. by a woman architect at the beginning of the war and was
planned for the special needs which the Association foresaw. It
combined the features of restaurant, reading and lounging rooms,
and sleeping rooms for relatives who might have to stay overnight in
the camp, while its semblance was that of a pleasant country club.
The Hostess Houses were the scenes of many war weddings, of
occasional christenings, of first meetings between returning happy
soldier or sailor fathers and their children born in their absence, and
they were sometimes a welcome refuge for mother or wife, sister or
sweetheart, summoned to the camp by the fatal illness of a loved
one.
The Association had a total of almost one hundred and fifty
Hostess Houses in this country, in the camps and cantonments for
both white and colored troops, in which were over four hundred
workers. In France it carried on fifteen of these or similar houses for
American women directly engaged in war work, such as those in the
Signal Corps, and for women connected with the British auxiliary
organizations, twenty-one for nurses in base hospitals and eighteen
for French women working in munition factories, offices, stores and
for the American army. The Y. W. C. A. gave much assistance also in
the providing of emergency housing for women engaged in work for
the war in this country, while its endeavors for the improving of
morale and the inculcating of American ideals among foreign born
and colored women and girls aided in rousing their patriotic spirit. It
operated War Service Industrial Clubs with cafeterias and recreation
halls and a variety of entertainments and classes for study in centers
of war industry where women were employed.
A Woman’s Division was instituted by the Young Men’s Christian
Association at the end of our first war year and during the next seven
months its work grew to important proportions. Carefully chosen for
the service, the women were given just before they sailed a week of
intensive training for their duties on the other side. Instruction in
hygiene taught them how to keep themselves fit under conditions
that would call for all their strength; their knowledge of French was
freshened; they had lectures on the kind of cooking needed for
canteen work and talks on the geography, history, customs and
characteristics of France, in order to give them a degree of
sympathetic understanding of the people among whom they would
have to work; they were encouraged to practice any sort of special
facility for the entertainment of groups of men which they might
possess; and they were expected to be accomplished dancers
before they were enlisted. On the other side they worked in canteens
and were especially useful in the recreation centers described in
“Big-Brothering the Fighting Forces,” of which twenty-six were
organized in different parts of France. In these recreation camps, or
“leave areas,” in the “Y” centers in Paris and other French cities, in
canteens in camps and behind the front lines, the Red Triangle
women made and poured coffee and chocolate and tea, distributed
candy, cakes, gum, cigarettes and tobacco, provided Christmas
boxes, sang, danced, recited, played games and did whatever the
moment demanded for the welfare and the entertainment of the
American fighting men. The women practically created the service of
the “leave areas,” which was something entirely new in warfare.
They went with the canteens to the front lines, advanced with the
Army of Occupation through Luxemburg and Alsace, and settled
down with it in Germany. They worked also with the American forces
in England and Scotland, Russia and Italy. After the armistice, when
many of the men secretaries of the Y. M. C. A. began to return to
their neglected business in the United States, the women took over
more and more of the canteen and other work. When hostilities
ended, a thousand women were engaged in Red Triangle work
overseas and so important was their service that in response to the
call for them that number was doubled during the next three months,
and the Association was then still recruiting, training and sending
them to France.
Three organizations enlisted women as automobile drivers for war
service,—the Motor and Ambulance Corps of the American Red
Cross, the Motor Corps of America and the Motor Corps of the
National League for Woman’s Service. Together they had an
estimated membership of several thousand women, most of whom
were women of leisure who owned their own cars and were glad to
give for the country’s needs their own time and work and the service
of their automobiles. Before being received in either of the
organizations they had to undergo a course of intensive training
averaging six weeks and including revolver shooting, first aid
treatment, surgery clinics as a test and training for the nerves, clinics
for the handling of the insane because mentally unbalanced soldiers
had to be transferred by ambulance from transport to hospital,
military drill twice a week and a course in mechanics. A member of a
woman’s motor corps had to know how her car was built and be able
to take it apart, if necessary, and put it together again and if it balked
to discover what was the matter and apply the needed remedy. The
Motor Corps women served both at home and overseas and they
drove trucks, ambulances and cars. Their service was ready for any
war organization that needed them, their vehicles plied between
transports and hospitals, carried convalescent soldiers out for an
airing, were on duty at cantonments and camps and answered many
similar calls. Their rules demanded at least nine hours per day on
duty, but actual service often stretched to fifteen or twenty hours out
of the twenty-four.
 The National League for Woman’s Service, by which one of these
corps was recruited and directed, was organized for patriotic
purposes two months before America entered the war and upon that
event was ready to begin active work in the coördinating of women’s
organizations and the enlisting and directing of all manner of
women’s resources and abilities that would aid the nation in the
prosecution of the war. Its organization spread into almost every
state of the Union and numbered 300,000 members. Its Motor Corps
Service, which was recognized by the Surgeon General of the Army,
had throughout the country seventy-eight chapters with a
membership of about five hundred women car owners. Its social and
welfare division established many soldiers’ and sailors’ club rooms
and club houses, with reading and lounging rooms, billiard and pool
tables, dances and entertainments, and classes in French and
English. It also conducted classes for the instruction of women in
occupational therapy and handicraft who worked in hospitals and
camps, recruited and trained women to serve as nurses’ aids, and
coöperated with the War Camp Community Service in many ways.
Its members worked in canteens and clubs, gave their services in
workrooms where clothing and supplies were made for hospitals and
for soldiers and sailors, distributed the thousands upon thousands of
flower donations made to hospitals by florists, worked with the Food
Administration by distributing food pledges, establishing emergency
and community kitchens and providing experts in home economics
who gave instruction in food conservation. The League collected
books, magazines, games and tobacco for the fighting forces,
recruited a Woman’s Reserve Camouflage Corps which gave some
important services, enlisted the aid of authors and artists for the
publicity needs of one or another department of the Government,
and served, in general, as a means of mustering and directing the
resources and abilities of women for war work.
Women’s clubs of every sort all over the country had their war
service committees, or mobilized all their members for that purpose,
and these were closely linked together through their federations so
that their work, which included assistance for every war making and
war assisting agency of Government or people, could be done
without overlapping or waste. Women’s colleges and women
students in co-educational institutions also took up war work, as
described in “Feeding the Nations.” As the men students of the
colleges mobilized for training for the war in the Students’ Army
Training Corps, the women students mobilized for work to uphold the
war. The Association of Collegiate Alumnæ, with membership spread
all through the Union, organized itself for war effort with especial
reference to the task of bringing home to people everywhere the
fundamental issues involved in the war, the necessity of fighting it
through to a completely victorious conclusion and the dangers that
would lurk in a premature peace. The Association coöperated with
the Committee on Public Information, held college women’s rallies,
formed local speakers’ bureaus, helped to procure trained workers
for various forms of national service, set on foot a movement to
provide in colleges preparatory nursing courses for women, and
worked with and for all of the war sustaining agencies of the
Government.
Coöperating with all these and with the many other women’s
organizations for war effort and comprehending in its nation-wide
scope all the women of the country was the Woman’s Committee of
the Council of National Defense, which interlocked in effective team-
work all organizations of women and, reaching out to almost every
community in the land, inspired those outside such organizations to
definite, regular, organized effort for war service especially fitted for
women’s hands. It served solely among women, just as the Council
of National Defense, of which it was a part, joined in team-work all
war sustaining and war producing agencies and organized the
communities, as told in “Organizing the Nation.”
The Woman’s Committee was created in April, 1917, and very
soon had its divisions organized in each of the forty-eight states and
also in Alaska, Hawaii, Porto Rico and the District of Columbia. Upon
each State Committee were represented both the state-wide
women’s organizations and the women not connected with any
organization, and these committees organized the states into small
units. Over 15,000 of these subordinate units had been formed and
were at work by mid-summer of 1918, including 2,500 counties and
8,500 cities, towns and townships and, in addition, many thousand
smaller units, such as school districts, wards, precincts, city blocks.
These small units brought the organization into direct touch with
women everywhere and enlisted them as individuals and as groups
in the great army of patriotic women who were giving everything in
their power for the prosecution of the war.
In half or more of the states women registered for war work,
stating the amount of time they could give, the special service for
which they were fitted and the kinds of work they could do. When the
request came for volunteers for any particular service, or when it
became known that there was some new need for woman’s
assistance, the leader of each unit knew just where to look for the
necessary help. The Woman’s Committee, from its central offices in
Washington to the members of local units in city block or country
district, worked with the Food Administration for the increased
production and the conservation of food and, similarly, gave their
help to the conservation program of the Fuel Administration. So also,
they coöperated with the War Camp Community Service and the
Training Camp Commissions, with the Liberty Loan and War Savings
Stamps campaigns, aided in the campaign to recruit nurses and in
that to secure workers for the ship yards, and helped to find trained
women workers who were needed at once by the rapidly expanding
departments and the new boards and commissions at Washington.
The Woman’s Committee endeavored always, while aiding in the
work of the war agencies, to preserve and improve the peace time
standards and values of life. And therefore not a little of its work was
along the lines of maintaining the health and protecting the welfare of
women and children. It had a department of Child Welfare and
carried on a vigorous campaign to further these aims while it
endeavored to promote public sentiment in favor of proper living and
working conditions for women in industry.
The Woman’s Committee of the Council of National Defense, in
short, mobilized in one great, enthusiastic, democratic army the
women of all the land, rich and poor, ignorant and cultured, of many
races, of foreign birth and of American ancestry, and by organization
enabled them to use their time, ability and effort in the way and at
the time when they would be of best service.
 CHAPTER XXXIV
FIGHTING THE UNDERGROUND ENEMY

For years before it plunged the world into war the German
Government, as in every country in which it could obtain the
necessary foothold, had been applying in the United States its policy
of “peaceful penetration.” Toward that end it had endeavored by
many apparently innocent means to hold the loyalty of American
citizens or residents of German birth or extraction, to create a
dominant body of sentiment in favor of anything and everything
German, and to secure the open or concealed control of vast
quantities of business through which it could operate for the
furtherance of German interests, political, industrial, financial or
cultural. German methods and ideals accepted in schools and
colleges; German departments in universities that were centers of
influence for the spreading of admiration of everything German; in
some regions Germanized public schools; a country-wide net-work
of German societies and associations through which love and loyalty
for the “fatherland” were kept alive; millions of dollars of German
money invested in American business, frequently under disguised
ownership; German and Austro-Hungarian diplomats using their
offices and privileges for the promotion of all manner of intrigue
against the interests of the country; plots for the control of industry,
the destruction of property, the inciting of sedition, the hatching of
conspiracies, the rousing of enmity against us in friendly nations—
these were some of the things the American people found had been
going on under their very noses, many of which they had
thoughtlessly aided, when the shock of war opened their eyes to the
character and the methods of the enemy who, for the sake of
civilization, had to be rendered innocuous. It was an enemy who had
not only to be fought on the open battlefield but foiled in all the
underground tricks and activities in which he was exceptionally
expert and incessantly busy.
Before our entrance into the war Germany had used her own and
the Austro-Hungarian embassies and her well organized spy system
to carry on operations against England and France, her diplomatic
representatives and her agents secretly concocting and directing
activities that would interfere with the efficiency of the Entente Allies
and might also be depended upon to create friction and possibly
even war between them and the United States. After the two
ambassadors and their staffs had been sent home because of these
machinations and the United States had declared war, there still
remained the spy system, which had been greatly increased and
strengthened during the first years of the war. Huge sums of money
financed it and it was directed and carried on by some of the most
experienced agents of the German Foreign Office. To aid them
Germany had sent to this country many professional men, scientists
and others with instructions to advance German interests and to
assist in the carrying on of her underground activities in every
possible way. The Intelligence Division of the United States War
Department estimated that Germany maintained in this country,
before and after our entrance into the war, an immense, secretly
operating force of between 200,000 and 300,000 paid and volunteer
workers. There was also the wide-spreading net-work of business
firms, apparently innocent, but really a cover and medium for enemy
machinations.
Emissaries to blow up bridges and railroads and do other damage
were sent into Canada. Malcontents from Ireland and India were
sought out and financed and aided in the laying of plots to create
dissatisfaction, riots and, if possible, revolution in their home
countries. A French traitor was brought to the United States and
furnished with money for setting on foot a traitorous scheme in
France. Much ingenuity was expanded in the endeavor to create
friction between this country and Japan. In Mexico Germany
diligently spread propaganda to influence the people and
government of that country against the United States and aided and
financed terroristic movements and activities whose purpose was to
embroil the two nations in war.
Germany’s underground activities in the United States, some of
them dating before our entrance into the war, some of them carried
into the period of our war participation, and others not begun until
after we became a belligerent, included many and varied schemes to
prevent this country from exercising its rights under international law,
to interfere with its effective prosecution of the war and to undermine
its political and trade relations with other countries. An effort was
made to gain control of airplane building. There was an attempt to
secure a similar hold upon the munitions industry, by maneuvering it
into the hands of German capital so camouflaged that its character
would not be recognized. A particularly well organized and cunningly
concealed scheme, directed and financed in the United States, was
set on foot to buy up and hoard wool and woolen and other textiles,
in both North and South America, needed for the clothing of our own
and our associates’ armies.
Plots were laid and feverishly pushed forward for blowing up ships
bearing troops or war cargoes across the Atlantic and for wrecking
munition plants and other war industries. German agents sent
throughout the Southern states did their best to incite race riots
among the negroes and to instigate a race war, working among them
in their homes and churches and following them into cotton fields
and mills and even into the army camps. Much effort and ingenuity
were expended in the attempt to cause dissatisfaction and strikes
among the workers in war industries and strife among those of
different nationalities.
Propaganda, both open and concealed, was carried on by
innumerable methods in the hope of influencing sentiment against
the war, in favor of Germany, or against our war associates. For this
purpose there were used moving pictures, the pastors of German
churches, the German language press, the newspapers of other
languages, writers in German pay who contributed articles and
correspondence to American newspapers and magazines, German
owned or controlled periodicals whose directing influence was well
concealed, and a great number of societies having for their
ostensible purpose the aiding of the aims of labor, or of pacifist
sentiment, or of socialism.
The United States Department of Justice discovered, in the course
of its investigations, that the German Government had placed in this
country for the use of these various underground activities over
$27,000,000, of which $7,500,000 had been spent in propaganda.
For measuring forces with an enemy of this sort the United States,
when it entered the war, was inadequately equipped with laws. A
friendly people, believing in square and open dealing between
nations as between individuals and trusting those to whom it had
given citizenship and business and professional hospitality, as it
would expect to be trusted in another land, had never thought it
necessary to enact such laws as this emergency demanded. The
only weapon of consequence which the Government had ready for
conflict with the underground enemy was a statute which had been
in force more than a hundred years permitting the arrest and
internment by executive order of an alien enemy believed to be a
menace to the public safety. Advantage was taken of this at once
and some of the most dangerous agents of Germany were soon
under guard and innocuous for the duration of the war. This
internment statute was a powerful weapon in putting down enemy
activities, while the severity with which it was enforced from the very
beginning was effective in discouraging the continued hatching of
plots.
An espionage law enacted two months after our declaration of war
and strengthened later on and a sabotage bill dealing with injury to
property gave the needed means for dealing with a difficulty the
nation had never before encountered. The espionage act was
effective against organized or deliberate enemy or disloyal
propaganda, but it was not intended to curtail the rights of free
speech or of a free press and in its enforcement the courts made
every effort to protect these rights as the basis of our political
institutions. In the emotional tensity of the time it was inevitable that
there should be bitter charges of excessive leniency on one side
and, on the other, of unnecessary severity from those who feared the
undermining of our principles of freedom. But in the end there were
few who did not recognize that substantial justice had been meted
out in most of the many cases. German alien enemies were required
to register and 480,000 men and women were thus listed. A system
of permits governed their movements and debarred them, without
special permission, from the District of Columbia and from specified
zones surrounding fortifications, docks, piers, wharves, warehouses,
and other places important for war purposes. They were forbidden
also to enter or leave the United States.
Much more lenient treatment was given to the subjects of Austro-
Hungary, upon whom the only restriction was that of not leaving the
country, although they were also subject to arrest and internment if
guilty of dangerous activities. They proved to be worthy of the trust
placed in them, for, although there were seven or eight times as
many of these enemy aliens as of those of German citizenship, they
gave little trouble of any sort, their labor helped importantly in much
of our war production and throughout the war they were quiet,
industrious and law-abiding.
Germany’s spies and agents were of several nationalities and in
order to keep an effective watch upon their movements a stringent
passport system was instituted which made impossible the departure
from this country of any one whose purpose was not clear and
proper. Private persons were forbidden to carry mail out of or into the
country, as a means of preventing enemy agents from sending
reports by others. Officers and crews of neutral ships were not
allowed to land at United States ports without permits from the
Department of State. A large force of picked and trained men,
numbering several hundreds, scrutinized every ship coming into or
going out of the important ports, her cargo and her passengers, to
make sure that no enemy agent was among them or material of any
sort intended for the enemy secreted in hold, or quarters, or cabins.
Supplementing the six secret service agencies of the Government,
all of which were immediately and very greatly increased to enforce
these provisions and deal with enemy activities, there sprang into
life, almost over-night, the American Protective League. An
organization of citizen volunteers, it was a unique development of
the situation and in spirit and methods thoroughly characteristic of
the American people. The League was born out of a realization of
the danger the country faced, overrun as it was with enemy agents
directed by some of the most skillful intriguers and spy captains that
a nation specializing in spying and intrigue had been able to train,
and out of the loyal wish to serve.
The American Protective League, which had its beginning almost
simultaneously with our declaration of war, was a volunteer auxiliary
of the Department of Justice. Its organizer, a private citizen who saw
the necessity of such service and the possibility of securing the
effective coöperation of selected persons everywhere, had it in
operation within a few weeks, with several thousand members. It
grew rapidly and within a year had 250,000 members working for it in
their own communities. The organization was established in every
state in the Union, the country being cut up into divisions, each
under a chief, and each division into districts with a captain in
command of each one, while each captain recruited his own working
squads and put them under lieutenants. This organization by territory
was reënforced by another whose divisions were along the lines of
important industries, trades and professions, the two bureaus
working constantly in coöperation. In the membership of the League
was represented every section and phase of American life—college
professors and day laborers, bank presidents and mechanics,
journalists, lawyers, janitors, ministers, carpenters, judges. The very
great value of its service was due to this variety and to the
intelligence and character of its membership, for it was able to
penetrate into all circles, to be on the watch everywhere in city, town
and country and to follow a suspect through the most devious of
wanderings. It investigated pro-German propaganda of every sort,
sabotage cases, suspected spies and their activities, seditious
speeches and printed matter, efforts to evade the selective service
act, lying reports circulated by the “whispering propaganda” method
about American organizations or individuals, and suspected
treasonable conspiracies.
The members of the League, undertaking its work in addition to
the duties of their regular occupations, served without pay and
without rendering expense accounts. It carried on 3,000,000
investigations upon which it made reports, a great many resulting in
the uncovering of serious disloyalties or enemy activities. So efficient
was the organization that it won the warmest praise from the
Attorney General of the United States, who declared that not only
were its active services of very great value but that its passive effect
was of equal importance, because the knowledge that its eyes and
ears were everywhere had a most discouraging influence upon
enemy and disloyal intentions.
Under the Trading with the Enemy Act, passed early in our war
progress, an Alien Property Custodian took charge of properties and
businesses belonging to enemy aliens in this country or operated for
the benefit of enemy subjects elsewhere. The investigations which
uncovered these business operations, many of them deeply and
cleverly concealed, revealed startling facts as to the extent to which
German subjects had gained commercial and industrial footholds in
the United States, the methods which they had used and the
purposes to which they had applied their resources and their
knowledge of the nation’s business and industrial life. More than
thirty thousand cases of enemy owned business were handled by
the Custodian, while enemy owned stock, ranging in the several
cases from fifteen to one hundred per cent of the total, was found in
nearly three hundred corporations. He seized enemy owned property
in the first year of his work to the value of more than $700,000,000,
the businesses in which it was engaged running the whole gamut of
American industry in mining, manufacturing, buying and selling.
Frequently the enemy ownership was so cleverly and persistently
concealed that months of investigation were necessary to uncover
the truth. A great many of these German owned industrial
establishments were used as spy centers and were filled with the
agents of Germany plotting for political and industrial domination. In
order to protect the country in the future and prevent a repetition of
this attempt to conceal a knife meant for her heart, the Alien Property
Custodian was authorized by Congress to sell to American citizens
all enemy owned businesses, the proceeds to be deposited in the
United States Treasury to await decision concerning it by the Peace
Congress which should settle the problems growing out of the war.
Not only did the volunteer organization of the American Protective
League undertake to uncover and stop enemy and disloyal activities,
but a large percentage of the American people individually
endeavored to aid the authorities in the same way. So intense was
the general indignation against Germany and the Germans because
of their insidious methods and the extent to which they had abused
the friendly attitude of America and so high was the spirit of loyalty
that young and old, rich and poor, were everywhere on the watch for
signs of disloyal sentiment. Sometimes this eagerness overstepped
common sense and degenerated into unthinking persecution of
people of German birth or extraction who were good and loyal
citizens. It resulted also in the circulation of many wild rumors of spy
activity without basis of truth. But it also had undoubted good result
in the discouraging of the underground activities of the enemy.
Germany expected confidently that her well organized and richly
provided spy service, her extensive propaganda and her hold upon
business would enable her to undermine and palsy America’s war
effort. But all her careful preparations and the huge sums of money
she expended profited her scarcely at all. The great majority of
American citizens of German blood or birth proved to be loyal to the
United States. The swift hand of justice at once grasped and put
under guard so many of Germany’s agents that the rest were
unwilling to run the risk of continued activity. Over 6,000 enemy
aliens were arrested under warrants and 4,000 were interned in
army detention camps for the period of the war. Systematic disloyal
propaganda failed so completely to produce its desired results, was
everywhere so frowned upon and was so likely to be fraught with
danger for those behind it that it dwindled rapidly. By the end of our
first year of war pro-German and anti-American propagandists had
realized the futility of their attempts.
Notwithstanding all the preparations and efforts of the enemy to
breed disloyalty and create disorder and lawlessness and our own
lack at first of legal machinery with which to meet the situation,
Germany’s underground operations were squarely met and wholly
defeated and the country was never more quiet and law-abiding than
it was during all the period of the war.
 CHAPTER XXXV
AT THE HEART OF THE NATION

In the memory of those who knew it during the war Washington will
ever stand out as an epitome of the titanic achievements of the
country. There beat the heart of the nation and there could be felt, as
nowhere else, its mighty and determined pulses. There was the
source of every great activity and there, with the burning intensity of
sunbeams focused through a lens, the spirit of the people was
making itself manifest.
The war found the capital of the United States, just as it had been
for many years, quiet and leisurely, aloof from business and industry,
spacious and restful and lovely. And the war transformed it with
lightning speed into a busy hive of war making industry, crammed
with people, humming with prodigious labors, striving mightily to
achieve what seemed the impossible in a hundred different ways at
the same time.
The vast expansion in every war making or war administration
agency of the Government and the creation of new agencies that
began at once had, of course, their source and direction in
Washington and there their machinery had to be housed and
operated. First to outgrow its former allocation of space in the huge
State, War and Navy Building, ample for the peace time needs of all
three Departments, was the War Department. As the expansion in
each of its divisions increased from day to day, it overflowed into
other buildings, and one immense structure after another, nearly a
dozen in all, was rushed to completion to house its activities. The
Navy Department and the Treasury Department each had its own
difficulties, although in neither was the expansion so great as in that
of War. In the great Treasury building entrances were closed and
corridors screened to make more desk room and buildings and office
space were leased elsewhere to accommodate the many thousands
of new employees who were needed for the vast amount of expert
and clerical work suddenly made necessary in connection with the
income tax, the War Risk Insurance, the Liberty Loan bonds, the War
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com