Journal of Physics: Conference Series

PAPER • OPEN ACCESS You may also like

- Research on Anomaly Detection Method
Telemetry Data Processing and Analysis Platform for Satellite Power Supply Based on
Bayesian Model
for Flight Test Based on Flink Hui Li, Jing He and Fuqiang Cheng

- Research on telemetry data compression

technology based on inter frame
To cite this article: Mingzhe Hao and Xi Wang 2023 J. Phys.: Conf. Ser. 2480 012020 differential adaptive run length encoding
Shi Fenglei

- Improved Image Quality over 10 Fields

with the Imaka Ground-layer Adaptive
Optics Experiment
View the article online for updates and enhancements. F. N. Abdurrahman, J. R. Lu, M. Chun et
al.

This content was downloaded from IP address 213.188.69.241 on 19/04/2023 at 16:48

ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

Telemetry Data Processing and Analysis Platform for Flight

Test Based on Flink

Mingzhe Hao, Xi Wang

Chinese Flight Test Establishment, Xi’an, China
[email protected]

Abstract: Telemetry data processing has extremely high real-time requirements in flight tests.
In response to the problems faced by the traditional telemetry data processing system, such as
insufficient computing capacity and low scalability, the Flink-based telemetry data processing
and analysis technology is proposed to give full play to the cluster computing capacity of Flink,
improve the throughput of real-time processing and reduce the processing delay. After flight
test verification, the platform meets the real-time processing of multiple telemetry data and has
significantly improved the stability, processing efficiency, and scalability compared with the
traditional system.

1. Introduction
Flight test telemetry is a method of tracking measurement and communication for aircraft, and the
collected measurement data is transmitted to the flight test command and control center. Then the
telemetry data is received, demodulated, and calculated by the ground real-time monitoring system,
and converted into the significance of engineering data. Finally, it is sent to the monitoring terminal
for the flight commander and flight test engineer to grasp the real-time state of the aircraft [1].
Telemetry real-time monitoring is the core means of providing flight test risk prediction and decision
support assurance, which is of great practical significance for flight test safety and improving
efficiency [2].
At present, the traditional telemetry data processing system is mainly designed based on C/S
architecture. Reference [3] proposed the technical implementation idea and scheme of multi-objective
telemetry monitoring, which adopts virtual LAN and multicast communication technology to realize
multi-objective integrated telemetry monitoring. Reference[4] designed a new multi-redundant
telemetry data processing system to increase the status information interaction function between server
and client software and to improve the reliability of system operation and the accuracy of telemetry
data. Reference[5] uses container technology to distribute and execute flight test data, which solves the
problems of distributed real-time receiving and analyzing telemetry data and centralized control of
flight tasks. Reference[6] adopts the design ideas of modularity, multithreading concurrency, and
multi-buffer to ensure the real-time and functional scalability of telemetry video data. To improve the
efficiency of real-time safety monitoring and management of flight tests, Reference 7 designs and
implements a real-time test flight monitoring and management system based on B/S architecture,
further simplifies the steps of system deployment and maintenance and improves the cross-platform
operation capability of the system [7].
In summary, with the development of various new telemetry systems and technologies, the
telemetry data processing system has developed from the previous stand-alone and specialized design
to the direction of clustering and modularization. This has put forward higher requirements on the

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

processing capability, versatility, reconfigurability, and scalability of the telemetry data processing and
analysis platform. Meanwhile, with the complexity of future flight test missions and the diversification
of telemetry transmission systems, telemetry data processing and analysis face unprecedented data
volume, multiple protocol types, and complex application scenarios. Therefore, building a real-time,
general-purpose, and reconfigurable deep flight test data mining and application platform is the focus
of this paper’s research.
With the rapid development of cloud computing and big data technology, the new generation of big
data computing engine and framework, with advantages such as low latency and high throughput, is
triggering technological changes in various industries, including aerospace, electric power, finance,
and other fields [8], for example, the space TT&C signal processing platform in the cloud environment
[9], the online early warning system based on distributed real-time computing framework for abnormal
feature extraction of electric energy data [10], and the real-time recommendation business of
commodities in the e-commerce industry [11]. For different application scenarios, there is no perfect
method of technology selection and design. Therefore, using cloud computing and big data processing
technologies, this paper proposes a flink-based flight test telemetry data processing and analysis
platform and uses the streaming real-time computing framework to realize the concurrent real-time
processing capability of multiple flight data streams. Using component-based data analysis flow
choreography, the dynamic assembly of telemetry protocol parsing, fault alarm, and other data
processing logic is realized. With k8s+docker, the cloud-based software layer enables horizontal and
elastic platform scaling.

2. Key Technology Selection

At present, in the aspect of real-time data processing, the commonly used processing technologies
include Storm, Spark Streaming, and Flink [12]. Each technology has different features and
application scenarios. It is mainly applicable to real-time data collection, analysis, and monitoring of
devices, and real-time hotspot data push. Scenarios with high-time requirements, such as real-time
alarms of abnormal data collection failures, are described in detail below.
Apache Storm[13,14] is a free open-source distributed real-time computing system, which can
easily and reliably process unlimited data streams and support real-time processing at the second or
even millisecond level. It is often used in the fields of real-time analysis and online machine learning,
with built-in integration of common message queues and databases. In a small Storm cluster, each
node can process messages at a rate of millions per second, and the parallelism of the real-time
computing program can be dynamically adjusted to maximize cluster resources.
Apache Flink[15] is a distributed processing engine and framework for the stateful computation of
unbounded and bounded data streams.
Through the above analysis, to meet the requirements of latency considering multiple flight
monitoring scenes at the same time, considering the characteristics of the flight control, based on the
Flink framework design and building the platform, to quickly adapt multiple types of data access in
the complex and variable safety monitoring scenarios of telemetry data types, modular components for
parsing data, and efficiently and stably outputting the results needed for test flight tasks.
After a comprehensive comparison and analysis in terms of latency, throughput, and modularity,
this paper implements the telemetry data processing and analysis based on the Flink framework to
achieve low latency and high throughput telemetry data processing, and the flight test data results are
correct and reliable.

3. Platform design and implementation

Flink-based flight test telemetry data processing and analysis platform are based on Hadoop big data
platform. WithFlink computing framework as the core, it uses Kafka as distributed message
Publish/Subscribe channel for decoupling, HDFS distributed file system to store the raw telemetry
data, and HBase to store the calculated data. In addition, the information related to the flight subject

2
ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

analysis algorithm is written into Redis to realize the functions of telemetry real-time data collection,
calculation, and analysis.

3.1. Overall platform architecture design

The overall architecture of the Flink-based flight test telemetry data processing and analysis platform
is divided into five layers, including the physical device layer, data source access layer, data storage
layer, data processing layer, and data transfer layer. The overall architecture is shown in Figure 1.

Figure 1. Overall platform architecture diagram

(1) Physical Device layer
The physical device layer is mainly composed of high-performance servers, storage devices, and
communication devices, which form the computing basis of the Flink platform.
(2) Data source access layer
The data source access layer mainly completes the telemetry data receiving and provides different
types of data sources for the upper layer. At the same time, a new telemetry data protocol format is
considered to support dynamic expansion.
(3) Data storage layer
The data storage layer uses HBase to store the data decoded by telemetry and the data derived from
further calculations, meeting the requirements for rapid post-operation data analysis and full-course
data playback.
(4) Data processing layer
The data processing layer is the core of the whole platform, which mainly completes the processing
of telemetry data, including the decoding, verification, calculation of telemetry parameters, and
integration of fault prediction algorithms.
(5) Data transfer layer
The data transfer layer is mainly responsible for pushing the computed telemetry data and
predictive analysis data to other applications for driving visualization tools such as curves, meters, and
3D.

3.2. Kafka-based multi-protocol data access

Most avionics bus data are transmitted by telemetry devices on the aircraft after some important
parameters are coded and modulated by the collector. The data gateway is responsible for receiving
different telemetry data sources. The Kafka middleware receives the data sent from the data gateway
through the listening port and caches it to the message Queue. Thus a one-to-one mapping relationship
between telemetry data and the message queue is established. The data processing module can then
fetch messages from the corresponding message queue in Kafka for subsequent processing. The entire
access deployment architecture is shown in Figure 2. For streaming data, Kafka message queues use a
buffer layer to facilitate the most efficient execution of data, helping to control and optimize the speed
at which data flows through the system.

3
ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

Figure 2. Telemetry data access architecture

3.3. Real-time data processing flow

Real-time data processing is mainly responsible for telemetry data analysis and calculation. At present,
the telemetry data encoding in the field of flight test mainly follows the PCM mode defined in the
IRIG-106 telemetry standard, so Flink distributed computing is adopted to realize the PCM decoding
operation.
Firstly, the Flink consumption Kafka processing is implemented through the Kafka Flink
Connector API.
Secondly, when analyzing PCM data, it is necessary to divide the PCM data into several short
frames and send them to several Slot computing units for calculation. After the calculation is
completed, multiple calculation results are synthesized into a completed PCM long frame analysis
result according to the time stamp, and the system computation delay is reduced through this parallel
calculation method of short frames. The flight fault alarm algorithm is managed by Redis cache, and
the compiled algorithm library is called by the Flink computing framework when running for in-depth
data analysis.
Finally, the calculated data is stored in the HBase database using the push module and sent to other
applications in real time in publish/subscribe mode for a visual display of telemetry data. The
processing flow is shown in Figure 3.

Figure 3. Real-time data processing flow

3.4. Customized process choreography

The telemetry data processing and analysis platform include modules of data access, data analysis and
data delivery, which realizes the business of decoding telemetry data, fault alarm and data processing,
etc. The platform divides these modules into multiple dynamic components and provides standard
interfaces for each component, which can realize the interface methods according to the requirements
in the specific development process, improving the secondary development efficiency and scalability
of the platform, as shown in Figure 4.

4
ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

Figure 4. Customized process choreography

(1) Data access unit. It is used to receive external data and is the starting point of the entire data
processing flow. Each processing flow must contain only one access unit. When the access unit no
longer receives data, it can send a command through the client to determine whether the process is
complete.
(2) Data analysis unit. The conversion process can be basic displacement or format extraction, or
complex logical operations, depending on the requirements to determine the process. The analysis unit
can contain more than one in the process, and each analysis unit can receive data from the access unit
or other analysis units, either in parallel or serially with each other.
(3) Data delivery unit. It is used to send the processed data to the data monitoring terminal. The
unit can receive the data of the access unit or the analysis unit, and send the data to the UDP multicast,
message queue, and other terminals.

4. Experimental results

4.1. Experimental environment

To further verify the effectiveness of the Flink platform and consider the scalability and load
balancing of the platform, a distributed cluster environment is built, including the deployment,
monitoring, and management of HDFS, Yarn, Zookeeper, Kafka, and HBase components. One
primary node and three compute nodes were selected to test the telemetry data processing capability of
multiple aircraft sorties in the cluster application environment. The specific experimental environment
is shown in Table 1.
Table 1. Experimental environment list
Configuration Name Configuration Info
CPU Intel Xeon E5-2650
Memory 64GB
Disk 4TB
Operating System CentOS 6.5 64Bit
Flink 1.8.2
HDFS 2.7.3
Kafka 0.10.1
JDK 1.8

4.2. Results analysis

In a small cluster environment, the latency from receiving to delivering multi-channel telemetry data
(1/5/10/15) is verified respectively. The experimental results show that the average latency is 23ms,

5
ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

40ms, 54ms, and 64ms, as shown in Figure 5. According to the experimental results, the Flink-based
flight test telemetry data processing and analysis platform can realize the decoding and analysis of
multi-channel data through the distributed computing framework, and meet the real-time requirements
of real flight tests.

(a) (b)

(c) (d)
Figure 5. Latency test

5. Conclusion
By analyzing the advantages and disadvantages of traditional flight test telemetry data processing
systems, this paper designs and implements a telemetry data processing and analysis platform based
on Flink distributed real-time computing architecture. Firstly, the distributed processing platform is
built based on Hadoop technology. Secondly, based on Flink and Kafka, high throughput and low
latency telemetry data processing requirements are realized. Finally, the Flink framework is used to
design the independent choreography of the process and realize the flexible configuration of data
source access, processing, and parameter alarm components. Through the real flight test, the results
show that the platform can effectively process and analyze the multi-channel remote measurement
data, and meet the requirements of real-time and high-concurrency test flight scenarios.

References
[1] In Jong Kim and Sungpil Lee, “Development of telemetry data processing program,” 2013
International Conference on ICT Convergence (ICTC), 2013, pp. 468-473.
[2] Ke, S., and W. Qian.Design and application of automatic tracking antenna for the flight-test[J].
Foreign Electronic Measurement Technology (2018).
[3] Xiao-Ya LI. Design of multi-target comprehensive telemetry monitoring system architecture[J].
Modern Electronics Technique, 35.20(2012):124-127.
[4] Liu S, Jia Y, Yang Z. Design of a Multi-redundant Telemetry Data Processing System[J].
Computer Measurement & Control, 25.11(2017):190-192.
[5] Shan S,Peng W and Nie Y J, Research on Multi-source Data Real-time Monitoring System Based
on TaskContainer[J]. Computer Measurement & Control, 28.03(2020):98-102.
[6] Hao Z and Wang H. Design of Real-time Monitoring System of Airborne Net Videos Based on
RTP[J]. Computer Technology and Development, 29.05(2019):131-134.
[7] Nie R, Huang P, Liu Y Q. Research on the real-time monitoring and management in flight test
based on WEB platform[J]. Electronic Design Engineering, 26.08(2018):88-92.
[8] A. Verma, A. H. Mansuri and N. Jain. Big data management processing with Hadoop MapReduce
and spark technology: A comparison, 2016 Symposium on Colossal Data Analysis and

6
ECCST-2022 IOP Publishing
Journal of Physics: Conference Series 2480 (2023) 012020 doi:10.1088/1742-6596/2480/1/012020

Networking (CDAN), 2016, pp. 1-4.

[9] Yan D, Wang Y, Hong M A, et al. Aerospace TT&C Signal Processing Platform on Cloud
Computing[J]. Telecommunication Engineering, 58.05(2018):493-499.
[10] Chong W, Xiao Z. A data cleaning model for electric power big data based on Spark
framework[J]. Electrical Measurement & Instrumentation, 2017.
[11] Cen K L, Hong-Yan Y U, Yang T X. Design and Implement of E-Commerce Real-Time
Recommender System with Spark Based on Big Data[J]. Modern Computer, 2016.
[12] S. Chintapalli et al., “Benchmarking Streaming Computation Engines: Storm, Flink and Spark
Streaming,” 2016 IEEE International Parallel and Distributed Processing Symposium
Workshops (IPDPSW), 2016, pp. 1789-1792.
[13] Apache, 2022. Apache Storm.https://storm.apache.org/.
[14] J. Bang and M. -J. Choi, “Docker environment based Apache Storm and Spark Benchmark Test,”
2020 21st Asia-Pacific Network Operations and Management Symposium (APNOMS), 2020,
pp. 322-325.
[15] Apache, 2022. What is Apache Flink? — Architecture. https://flink.apache.org/flink-
architecture.html.