Real-World Image Datasets for Federated Learning

Jiahuan Luo1* , Xueyang Wu2* , Yun Luo2,3 , Yunfeng Huang4* , Yang Liu5† ,
Aubu Huang5 , Qiang Yang2,5
1
South China University of Technology, China
2
Hong Kong University of Science and Technology, Hong Kong SAR, China
3
Extreme Vision Co. Ltd., Shenzhen, China
4
Shenzhen University, China
5
WeBank Co. Ltd., Shenzhen, China
arXiv:1910.11089v2 [cs.CV] 12 Dec 2019

[email protected], {xwuba, yluoav}@cse.ust.hk, [email protected]

{yangliu, stevenhuang}@webank.com, [email protected]

Abstract
Federated learning is a new machine learning
paradigm which allows data parties to build ma-
chine learning models collaboratively while keep-
ing their data secure and private. While research
efforts on federated learning have been growing
tremendously in the past two years, most existing
works still depend on pre-existing public datasets
and artificial partitions to simulate data federations
due to the lack of high-quality labeled data gen-
erated from real-world edge applications. Conse-
quently, advances on benchmark and model evalu-
ations for federated learning have been lagging be-
hind. In this paper, we introduce a real-world im-
age dataset. The dataset contains more than 900
images generated from 26 street cameras and 7 ob-
ject categories annotated with detailed bounding
box. The data distribution is non-IID and unbal-
Figure 1: Examples taken from Street Dataset. The green
anced, reflecting the characteristic real-world fed-
bounding boxes represent the target objects.
erated learning scenarios. Based on this dataset, we
implemented two mainstream object detection al-
gorithms (YOLO and Faster R-CNN) and provided
an extensive benchmark on model performance, ef- greatly improved in the past few decades [Ren et al., 2017;
ficiency, and communication in a federated learning Redmon and Farhadi, 2018a; Zhao et al., 2018]. A traditional
setting. Both the dataset and algorithms are made object detection approach requires collecting and centralizing
publicly available1 . a large amount of annotated image data. Image annotation
is very expensive despite crowd-sourcing innovations, espe-
cially in areas where professional expertise is required, such
1 Introduction as disease diagnose. In addition, centralizing these data re-
Object detection is at the core of many real-world artificial quires uploading bulk data to database which incurs tremen-
intelligence (AI) applications, such as face detection [Sung dous communication overhead. For autonomous cars, for ex-
and Poggio, 1998], pedestrian detection [Dollar et al., 2012], ample, it is estimated that the total data generated from sen-
safety controls, and video analysis. With the rapid develop- sors reach more than 40GB/s. Finally, centralizing data may
ment of deep learning, object detection algorithms have been violate user privacy and data confidentiality and each data
∗
party has no control over how the data would be used af-
Authors performed the work while they were research interns at ter centralization. In the past few years, there has been a
WeBank Co. Ltd., Shenzhen, China
† strengthening private data protection globally, with law en-
Corresponding author
1 forcement including the General Data Protection Regulation
Dataset and code are available at https://dataset.
(GDPR) [Voigt and Bussche, 2017], implemented by the Eu-
fedai.org and https://github.com/FederatedAI/
FATE/tree/master/research/federated_object_ ropean Union on May 25, 2018.
detection_benchmark To overcome the challenge of data privacy and security in
Table 1: The object category distribution over the Street
Dataset
Object Category Sample Frequency 3
15
Basket 162 211
Carton 164 275
Chair 457 619 14
Electromobile 324 662 2
6
Gastank 91 137 18 20
Sunshade 314 513 5
Table 88 127 13
1 10
17
19 12 9
4 8

Figure 3: The schematic view of cameras’ geometric infor-

mation.

learning. There is a lack of real-world edge datasets repre-

senting the truly non-IID and unbalanced data distributions in
federated learning. Consequentially, there are still significant
lags in model evaluations and benchmark for federated learn-
ing. In this paper, we present a real-world image dataset gen-
erated from street cameras and then manually annotated and
selected. The dataset is a realistic representation of real-world
image data distributions and therefore are non-IID and unbal-
anced. We carefully evaluate these images and analyze the
detailed statistics of the object distributions. In addition, we
Figure 2: The distribution of labels per image. implement two state-of-the-art object detection algorithms,
YOLOv3 and Faster R-CNN, and integrate them with feder-
ated learning technique. We evaluate the model performance
the machine learning process, many privacy-preserving ma- comprehensively and provide baselines and comparisons us-
chine learning methods have been proposed in the past, such ing this federated dataset. We make this dataset and evalua-
as secure multi-party computing (MPC) in the 1980s [Yao, tion source code public that will be available.
1982; Yao, 1986]. MPC allows multiple parties to collabora-
tively calculate a convention function safely without reveal-
ing their data to each other or a trusted third party. How- 2 Related Work
ever, traditional MPC protocols require high communication As an emerging technology, federated learning introduces
overhead between parties, making it difficult to be applied to new challenges in system design, communication efficiency
industry. Differential Privacy, proposed by Dwork in 2006 and model evaluation. Google presented an overview of fed-
[Dwork, 2008], protects user data by adding noise, but incurs erated learning system design in 2019 [Bonawitz et al., 2019],
a trade-off between model accuracy and the risk of data leak- highlighting some of the design challenges for large-scale
age. problems. The complexity of federated learning systems
Federated learning [McMahan et al., 2016], an emerging poses challenges for both implementation and experimental
machine learning paradigm, allows users to collaboratively evaluation. As a result, many open-source projects were pro-
train a machine learning model while protecting user data posed to lower this barrier to entry. TensorFlow Federated
privacy. In essence, federated learning is a collaborative com- (TFF), led by Google is an open-source framework on top of
puting framework. The idea is to train through model aggre- TensorFlow for flexibly expressing arbitrary computation on
gations rather than data aggregation and keep local data stay decentralized data. Federated AI Technology Enabler (FATE)
at the local device. Since most of data (images, videos, text...) led by WeBank is an open-source technical framework that
are generated from edge devices, federated learning is an at- enables distributed and scalable secure computation protocols
tractive approach to enable end-to-end computer vision tasks based on homomorphic encryption and multi-party computa-
with image annotation and training tasks moved on the edge tion. Moreover, OpenMined proposes a python Library for
whereas only model parameters are sent to the central cloud secure, private deep learning [Ryffel et al., 2018]. It provides
for aggregation. a general frameowrk to incoorporate federated learning with
Despite the rapid growth of research interest on federated deep learning models implemented with PyTorch [Paszke et
learning, current research work still rely on the pre-existing al., 2017]. These federated learning framework allows re-
public datasets which are designed for centralized machine searchers to explore federated learning algorithms and appli-
cations in a more convenient and efficient way. LEAF [Caldas
et al., 2018] is a benchmark framework that contains prepro- Table 2: Detailed distribution of the object labels in the train-
cessed datasets, each with a “natural” partitioning that aims ing set of Street-5 Dataset and Street-20 Dataset
to reflect the type of non-identically distributed data partitions Street-5 Images/Client Frequency/Client
encountered in practical federated environments. Dataset total mean stdev total mean stdev
Basket 127 25.40 15.08 165 33.00 22.20
So far, federated learning has been implemented with many Carton 133 26.60 27.94 215 43.00 55.13
state-of-the-art algorithms such as long short-term memory Chair 369 73.80 59.36 494 98.80 87.60
(LSTM) networks [McMahan et al., 2017], support vector Electro
machines (SVMs) [Rubinstein et al., 2009], gradient boost- 257 51.40 45.23 510 102.00 105.27
-mobile
ing trees [Cheng et al., 2019], etc.. Federated learning algo- Gastank 71 14.20 26.42 106 21.20 36.19
rithms are also applied to industrial scenarios including next- Sunshade 255 51.00 31.89 413 82.60 56.02
word-prediction [Hard et al., 2018], speech keyword spot- Table 73 14.60 29.20 102 20.40 40.80
ting [Leroy et al., 2018], and images classification [Liu et
al., 2018]. [Chen et al., 2019] further improves next-word- Street-20 Images/Client Frequency/Client
prediction by addressing the out-of-vocabulary problem in Dataset total mean stdev total mean stdev
language model personalization. To the best of our knowl- Basket 127 6.35 11.06 165 8.25 14.93
edge, this is the first time that object detection algorithms Carton 133 6.65 12.03 215 10.75 26.23
are implemented with federated learning providing a reli- Chair 369 18.45 17.70 494 24.70 28.54
able benchmark framework and end-to-end solutions for real- Electro
257 12.85 14.81 510 25.50 37.55
world vision tasks. -mobile
Gastank 71 3.55 11.14 106 5.30 16.63
As one of the fundamental problems of computer vision, Sunshade 255 12.75 19.14 413 20.65 37.87
object detection forms the basics of many other computer Table 73 3.65 13.06 102 5.10 19.26
vision tasks, such as instance segmentation [Hariharan et
al., 2014], image caption [Karpathy and Fei-Fei, 2015], ob-
ject tracking [Kang et al., 2017], etc. Object detection is 3.2 Data Division
able to provide valuable information for semantic understand- We split the Street Dataset according to the geographic infor-
ing of images and videos, and is related to many applica- mation2 of the camera. We naturally have 26 street monitor-
tions, including face recognition, pedestrian detection and au- ing videos, but some of the cameras have very few images.
tonomous driving [Chen et al., 2015]. In recent years, the We split the whole dataset with 956 images into around 80%
rapid development of deep learning techniques has brought (765 images) for training and 20% (191 images) for testing.
new blood into object detection, leading to remarkable break- The testing set consists of images from 6 cameras with very
throughs and making it widely used in many real-world ap- few images, and random sample from other 20 cameras. By
plications. splitting the dataset this way, our testing set is able to jointly
evaluate the predictability (with images from existed cam-
eras) and generalization capability (with images from unseen
3 Street Dataset Description cameras).
We further divide the remaining training set into several
clients according to its real world attribute, i.e., geographic
3.1 Source Description information. Figure 3 reflects the schematic location infor-
mation of each device, denoted as device ID. We consider
two scenarios: 1) Each camera device is a client, named as
We randomly capture these images of different scenes at dif- Street-20; 2) Nearby cameras are grouped into a single client
ferent time from 26 street monitoring videos with 704 × 576 to form in total 5 clients, named as Street-5. And we pub-
pixels. Similar and night scene images are removed from lish two datasets based on this method of division. Street-5
them. The remaining 956 images are legible and have obvious mimics the scenario where several edge devices are commu-
distinction in content. Eventually, we select a total of 2,544 nicated and controled by a central node. Both datasets share
items from these images with 7 object categories. Each image the same testing set for evaluation.
has at least one labeled object, and may have multiple labels
of this same category in one image. The object labels are bas- Street-5
ket, carton, chair, electromobile, gastank, sunshade and table. We cluster the remaining cameras into 5 groups, each of
The distribution of these object labels is shown in Table 1, which represents as a client in the federated learning, accord-
which demonstrates that the class distribution is unbalanced. ing to the following considerations:
In addition, we also calculate the object frequency per image,
shown in Figure 2. 1. The cameras should be clustered according to their ge-
ographic information, i.e., a client possesses the images
Visual objects are annotated with bounding boxes. We from nearby cameras or cameras from the same region.
use bounding box of description (xmin , ymin , xmax , ymax ),
where (xmin , ymin ) is the top-left coordinates of the bound- 2
Notably, the geographic information is hashed when the dataset
ing box and (xmax , ymax ) the bottom-right. is published.
Table 3: Detailed distribution of the object frequency on each client in the training set of Street-5 Dataset and Street-20 Dataset
Client ID Basket Carton Chair Electromobile Gastank Sunshade Table
1 23 145 148 209 12 35 0
2 64 57 249 249 93 102 102
3 0 0 38 12 0 1 0
4 50 13 36 40 1 151 0
5 28 0 23 0 0 124 0
1 33 35 124 44 75 76 88
2 50 13 21 40 0 119 0
3 28 0 23 0 0 124 0
4 0 0 50 0 18 0 0
5 22 2 51 144 0 0 0
6 1 0 2 14 0 1 0
7 0 116 21 0 0 18 0
8 0 0 0 11 0 0 0
9 0 0 0 40 0 2 0
10 0 0 0 92 0 0 0
11 0 0 0 77 0 0 0
12 31 0 0 11 0 0 0
13 0 27 44 0 0 0 0
14 0 0 13 0 1 0 0
15 0 0 38 12 0 1 0
16 0 22 22 18 0 11 0
17 0 0 38 1 0 0 14
18 0 0 2 0 0 32 0
19 0 0 15 6 0 15 0
20 0 0 30 0 12 14 0

2. The number of samples in each client should have large CNN3 [Ren et al., 2015] and YOLOv34 [Redmon and
divergence. Farhadi, 2018b] for their excellent performance. Note that,
the backbone networks of Faster R-CNN is VGG16 [Si-
This split aims to simulate the federated learning running
monyan and Zisserman, 2014] and Darknet-53 for YOLOv3.
on different monitoring camera owners, such as different se-
curity companies. In this case, each company usually has 4.1 Baseline Implementation
more than one nearby cameras, and they can contribute to the
The code of our benchmark will be released, where two object
federated learning with more images as a whole.
detection models are implemented using PyTorch [Paszke et
Street-20 al., 2017], on a GPU server with CPU of Intel Xeon Gold
This dataset division is based on the minimal unit of our 61xx and 8 GPUs of Tesla V100.
raw dataset, which aims to simulate the case where federated Training Faster R-CNN was via SGD with a fixed learning
learning algorithms run on each device. In this case, the data rate of 1e-4 and a momentum of 0.9, while training YOLOv3
are kept and processed within each client with the minimal was via Adam with an initialization learning rate of 1e-3. No-
risk to reveal the raw data. tably, we use pretrained VGG16 model for Faster R-CNN for
faster convergence. In terms of model size, the YOLOv3
Since our data division is based on real-world distribution
model has 61,556,044 parameters, and the Faster R-CNN
of cameras, our datasets suffer from non-IID data distribu-
model has 137,078,239 parameters with backbone network
tion problem. Table 2 shows the detailed distributions of the
of VGG16.
distribution of annotated boxes among different clients, from
We adapt the original FederatedAveraging
which we can derive the unbalanced distribution of boxes,
(FedAvg) algorithm [McMahan et al., 2016] to frame-
which may lead to learning bias of each client. From Ta-
work, shown in Algorithm 1. As our purpose is to examine
ble 3 we can learn the number of classed in each client more
the effect of different data division and federated learning set-
intuitively. Therefore, our published datasets can serve as
tings, we modified FedAvg algorithm to a pseudo FedAvg
good benchmarks for researchers to examine their federated
algorithm, by replacing the server-client communication
learning algorithm’s ability to address the non-IID distribu-
framework such as SocketIO with saving and restoring
tion problem in real-world applications.
checkpoints on hard-devices, which simplifies the processing
3
4 Experiments based on the implementation https://github.com/
chenyuntc/simple-faster-rcnn-pytorch
We evaluate two object detection methods on the proposed 4
based on the implementation https://github.com/
datasets as our benchmark algorithms, including Faster R- eriklindernoren/PyTorch-YOLOv3
 Algorithm 1 Pseudo FedAvg
Input: N client parties {ck }k=1..N , total rounds T , and Server
side S;
Output: Aggregated Model w
S initializes federated model parameters, and saves as check-
point. Client parties {ck }k=1..N load the checkpoints.
for t = 1, ..., T do
for k = 1, ..., N do
wk = w(t)
each client {ck } do local training:
for i = 0, 1, ..., Mk do
(Mk is the number of data batches b in the client ck )
client {ck } computes gradients ∇`(wk , bi )
update with wk = wk − η∇`(wk , bi )
end for
save wk results to checkpoints.
end for
S loads checkpoints and get averaged model with w(t) =
1
PN
N k=1 wk
end for
return w(T )

Table 4: Number of communication rounds to reach a target

Figure 4: Test set mAP vs number of megabytes uploaded for mAP (75%)
different models. Model C E Rounds Amount (MB)
5 1 158 186,440
of model aggregation. However, our implementation can also 5 5 48 56,640
be easily migrated to FedAvg. YOLOv3 5 10 90 106,200
There are three key parameters of the FedAvg algorithm: (w/o pretrained) 20 1 83 391,760
C, the number of clients that participate in training on each 20 5 448 2,114,560
round; E , the number of times that each client performs over 20 10 346 1,633,120
its local dataset on each round; and B, the minibatch size 5 1 45 117,675
used for client updates. All these three parameters control the 5 5 30 78,450
computation. We mainly set B = 1 for the experiments when 5 10 189 494,235
Faster R-CNN
running Faster R-CNN. 20 1 161 1,684,060
As for FedAvg algorithm, it can select a R-fraction of 20 5 119 1,244,740
clients on each round, and compute the gradient of the loss 20 10 90 941,400
over all the data held by these clients. Thus, in this algorithm
R controls the global batch size, with R = 1 corresponding
to full-batch gradient descent. In order to produce a better tion, in which mean is taken over per-class APs:
result, we fix R = 1, which means the server waits for the Pk
updates of all the clients participating in training. APi
mAP = i=1
k
4.2 Evaluation Metrics where Average Precision (AP) is calculated for each class re-
In this section, we summarize some common metrics for ob- spectively, k is number of classes.
ject detection in our federated learning framework.
Intersection Over Union Intersection Over Union (IOU) 4.3 Results
is used to evaluate the overlap between two bounding boxes. In this section, we report the baseline results. In order to
It requires a ground truth bounding box Bgt and a predicted thoroughly investigate the hyper-parameters of the FedAvg
bounding box Bp . By applying the IOU we can tell if a de- algorithm and evaluate the effect of different parameters, we
tection is valid (True Positive) or not (False Positive). IOU is conducted experiments in different parameter configurations.
given by the overlapping area between the predicted bound- Note that, we use a threshold of 0.5 for IOU in our experi-
ing box and the ground truth bounding box divided by the ments when computing mAP.
area of union between them: Figure 5(a) shows the test-set mAP for YOLOv3 without
area(Bgt ∩ Bp ) pretrained model. We fix C = 5, which means the data is
IOU = split into five parts according to the geographic location, add
area(Bgt ∪ Bp )
more computation per client by increasing E, which controls
mean Average Precision We choose the standard PAS- the number of training passes. We see that increasing local
CAL VOC 2010 mean Average Precision (mAP) for evalua- epoch E is effective. Figure 5(a) demonstrates that adding
 (a) YOLOv3 (b) Pretrained YOLOv3 (c) Pretrained Faster R-CNN

Figure 5: Test set mAP vs. number of communication rounds using different models

(a) YOLOv3 (b) Pretrained YOLOv3 (c) Pretrained Faster R-CNN

Figure 6: Training loss vs. number of communication rounds using different models

more local SGD updates per round and then averaging the re- E gets the best result. This result suggests that for non-IID
sulting models can reach a higher mAP at the beginning of datasets, especially in the later stages of convergence, it may
the training procedure. We report the number of communi- be useful to decay the amount of local computation per round
cation rounds necessary to achieve a target test-set mAP. To if we start at a large E to lower communication costs.
achieve this, we evaluate the averaged model on each round The initial success of deep learning in computer vision can
to monitor the performance. Table 4 quantifies this speedups. be largely attributed to transfer learning. ImageNet pretrain-
Expectantly, the FedAvg algorithm should have the same ing was crucial to obtain improvements over state-of-the-art
performance as centralized training. When it comes to non- results. Due to the importance of pretraining, we conduct
IID datasets, it is difficult for FedAvg to reach the same score additional experiments with pretrained models. Figure 5(b)
as that of centralized. Using the more non-IID dataset, setting demonstrates that initialing with pretrained model weights
C = 20, shows a lower performance compared to the C = 5 produces a significant and stable improvement, especially for
one. When we fix C = 5, we get a comparable performance the Street-20 dataset, which has a small amount of pictures
compared to centralized training. Though we stopped train- on each client. This shows that pretraining on large datasets
ing at a number of communication rounds of 1000, it seems is crucial for fine-tuning on small detection datasets. Fur-
that the algorithm has not converged and they can get higher thermore, Figure 5(b) shows the impact of batch size of each
mAP if the training procedure continues. For both C = 5 and client. It is not very effective when we increase the batch
C = 20, larger E usually converges faster. But as the train- size for each client. We conjecture this is largely due to the
ing procedure goes on, when it comes to more non-IID cases, amount of pictures on each client is small. Especially on the
different E leads to different performance and not the largest Street-20 dataset, larger batch size even leads to lower perfor-
mance, since each client contains only dozens of pictures. References
In addition to one-stage approach towards object detec- [Bonawitz et al., 2019] Keith Bonawitz, Hubert Eichner, Wolfgang
tion, we contain Faster R-CNN as our benchmark, which is Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov,
a popular two-stage approach. Figure 5(c) reports the per- Chlo M Kiddon, Jakub Konen, Stefano Mazzocchi, Brendan
formance for Faster R-CNN with backbone network of pre- McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage,
trained VGG-16. For Faster R-CNN the C = 5, E = 1, and Jason Roselander. Towards federated learning at scale: Sys-
FedAvg model eventually reaches almost the same perfor- tem design. In SysML 2019, 2019. To appear.
mance as the centralized training, while the C = 5, E = [Caldas et al., 2018] Sebastian Caldas, Peter Wu, Tian Li, Jakub
5, FedAvg model reaches a considerable mAP after 400 Konecný, H. Brendan McMahan, Virginia Smith, and Ameet
rounds. Training with pretrained model shows faster conver- Talwalkar. LEAF: A benchmark for federated settings. CoRR,
gence. With C = 5, small local epoch got better performance. abs/1812.01097, 2018.
We also compare the training loss of different models. As [Chen et al., 2015] Chenyi Chen, Ari Seff, Alain Kornhauser, and
shown in Figure 6, FedAvg is effective at optimizing the Jianxiong Xiao. Deepdriving: Learning affordance for direct per-
training loss as well as the generalization performance. Note ception in autonomous driving. In Proceedings of the IEEE In-
ternational Conference on Computer Vision, pages 2722–2730,
the y-axes of different models are on different scales and loss 2015.
is the average of all the clients. From Figure 6(a), we can see
that in training, large local epoch E always produces small [Chen et al., 2019] Mingqing Chen, Rajiv Mathews, Tom Ouyang,
loss and smooth training loss curve. We observed similar be- and Françoise Beaufays. Federated learning of out-of-vocabulary
words. arXiv preprint arXiv:1903.10635, 2019.
havior for all three models. This is reasonable, because for
large numbers of local epochs client would over-optimize on [Cheng et al., 2019] Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu,
local dataset. One might conjecture large numbers of local Tianjian Chen, and Qiang Yang. Secureboost: A lossless feder-
epochs would bring about over-fitting. But they eventually ated learning framework. CoRR, abs/1901.08755, 2019.
reach a fairly similar mAP. Interestingly, for all three mod- [Dollar et al., 2012] Piotr Dollar, Christian Wojek, Bernt Schiele,
els, training with FedAvg converges to a high level of mAP. and Pietro Perona. Pedestrian detection: An evaluation of
This trend continues even if the lines are extended beyond the the state of the art. IEEE Trans. Pattern Anal. Mach. Intell.,
plotted ranges. For example, for the YOLOv3 the C = 5, E 34(4):743–761, April 2012.
= 1, FedAvg model reaches 88.86% mAP after 1400 rounds, [Dwork, 2008] Cynthia Dwork. Differential privacy: A survey of
which is the best performance of centralized training. results. In TAMC, pages 1–19, 2008.
We are also concerned with the communication costs when [Hard et al., 2018] Andrew Hard, Kanishka Rao, Rajiv Mathews,
using different models. We choose the Faster R-CNN as our Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé
cumbersome model and YOLOv3 as our lightweight model. Kiddon, and Daniel Ramage. Federated learning for mobile key-
The size of the parameters of Faster R-CNN is more than board prediction. arXiv preprint arXiv:1811.03604, 2018.
twice that of YOLOv3. Note that the backbone network of [Hariharan et al., 2014] Bharath Hariharan, Pablo Arbeláez, Ross
Faster R-CNN is VGG16. Figure 4 and Table 4 demonstrate Girshick, and Jitendra Malik. Simultaneous detection and seg-
the communication rounds and costs to reach a target mAP of mentation. In European Conference on Computer Vision, pages
different models. 297–312. Springer, 2014.
The unbalanced and non-IID distribution of the datasets are [Kang et al., 2017] Kai Kang, Hongsheng Li, Junjie Yan, Xingyu
representative of the kind of data distribution for real-world Zeng, Bin Yang, Tong Xiao, Cong Zhang, Zhe Wang, Ruohui
applications. Encouragingly, it is impressive that naively av- Wang, Xiaogang Wang, et al. T-cnn: Tubelets with convolu-
tional neural networks for object detection from videos. IEEE
erage the parameters of models trained on clients respectively Transactions on Circuits and Systems for Video Technology,
provides considerable performance. We conjecture that tasks 28(10):2896–2907, 2017.
like object detection and speech recognition, which usually
[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei-Fei.
require cumbersome model, are suitable and show significant
Deep visual-semantic alignments for generating image descrip-
result on Federated Learning. tions. In Proceedings of the IEEE conference on computer vision
and pattern recognition, pages 3128–3137, 2015.
5 Conclusions and Future Work [Leroy et al., 2018] David Leroy, Alice Coucke, Thibaut Lavril,
Thibault Gisselbrecht, and Joseph Dureau. Federated learning
In this paper we release a real-world image dataset to evalu- for keyword spotting. arXiv preprint arXiv:1810.05512, 2018.
ate federated object detection algorithms, with reproducible [Liu et al., 2018] Yang Liu, Tianjian Chen, and Qiang Yang. Secure
benchmark on Faster R-CNN and YOLOv3. Our released federated transfer learning. CoRR, abs/1812.03337, 2018.
dataset contains common object categories on the street, [McMahan et al., 2016] H. Brendan McMahan, Eider Moore,
which are naturally collected and divided according to the ge- Daniel Ramage, and Blaise Agüera y Arcas. Federated learning
ographical information of the cameras. The dataset also cap- of deep networks using model averaging. CoRR, abs/1602.05629,
tures the realistic non-IID distribution problem in federated 2016.
learning, so it can serve as a reliable benchmark for further [McMahan et al., 2017] H Brendan McMahan, Daniel Ramage,
federated learning research on how to alleviate the non-IID Kunal Talwar, and Li Zhang. Learning differentially private
problem. In the future, we will keep augmenting the dataset recurrent language models. arXiv preprint arXiv:1710.06963,
as well as presenting more benchmarks on these datasets. 2017.
[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala,
and Gregory Chanan. Pytorch: Tensors and dynamic neural net-
works in python with strong gpu acceleration. PyTorch: Tensors
and dynamic neural networks in Python with strong GPU accel-
eration, 6, 2017.
[Redmon and Farhadi, 2018a] Joseph Redmon and Ali Farhadi.
Yolov3: An incremental improvement. CoRR, abs/1804.02767,
2018.
[Redmon and Farhadi, 2018b] Joseph Redmon and Ali Farhadi.
Yolov3: An incremental improvement. arXiv preprint
arXiv:1804.02767, 2018.
[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and
Jian Sun. Faster r-cnn: Towards real-time object detection with
region proposal networks. In Advances in neural information
processing systems, pages 91–99, 2015.
[Ren et al., 2017] Shaoqing Ren, Kaiming He, Ross Girshick, and
Jian Sun. Faster r-cnn: Towards real-time object detection with
region proposal networks. IEEE Trans. Pattern Anal. Mach. In-
tell., 39(6):1137–1149, June 2017.
[Rubinstein et al., 2009] Benjamin IP Rubinstein, Peter L Bartlett,
Ling Huang, and Nina Taft. Learning in a large function space:
Privacy-preserving mechanisms for svm learning. arXiv preprint
arXiv:0911.5708, 2009.
[Ryffel et al., 2018] Theo Ryffel, Andrew Trask, Morten Dahl,
Bobby Wagner, Jason Mancuso, Daniel Rueckert, and Jonathan
Passerat-Palmbach. A generic framework for privacy preserving
deep learning. arXiv preprint arXiv:1811.04017, 2018.
[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew
Zisserman. Very deep convolutional networks for large-scale im-
age recognition. arXiv preprint arXiv:1409.1556, 2014.
[Sung and Poggio, 1998] Kah-Kay Sung and Tomaso Poggio.
Example-based learning for view-based human face detection.
IEEE Trans. Pattern Anal. Mach. Intell., 20(1):39–51, January
1998.
[Voigt and Bussche, 2017] Paul Voigt and Axel von dem Bussche.
The EU General Data Protection Regulation (GDPR): A Practi-
cal Guide. Springer Publishing Company, Incorporated, 1st edi-
tion, 2017.
[Yao, 1982] Andrew C. Yao. Protocols for secure computations. In
Proceedings of the 23rd Annual Symposium on Foundations of
Computer Science, SFCS ’82, pages 160–164, Washington, DC,
USA, 1982. IEEE Computer Society.
[Yao, 1986] Andrew Chi-Chih Yao. How to generate and exchange
secrets. In Proceedings of the 27th Annual Symposium on Foun-
dations of Computer Science, SFCS ’86, pages 162–167, Wash-
ington, DC, USA, 1986. IEEE Computer Society.
[Zhao et al., 2018] Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu,
and Xindong Wu. Object detection with deep learning: A review.
CoRR, abs/1807.05511, 2018.