Calhoun: The NPS Institutional Archive

DSpace Repository

NPS Scholarship Theses

2024-06

AN ACOUSTIC APPROACH TO DRONE

IDENTIFICATION USING MACHINE LEARNING

Kenagy, Rachel M.
Monterey, CA; Naval Postgraduate School

https://hdl.handle.net/10945/73158

This publication is a work of the U.S. Government as defined in Title 17, United
States Code, Section 101. Copyright protection is not available for this work in the
United States.

Downloaded from NPS Archive: Calhoun

 NAVAL
POSTGRADUATE
SCHOOL
MONTEREY, CALIFORNIA

THESIS

AN ACOUSTIC APPROACH TO DRONE

IDENTIFICATION USING MACHINE LEARNING

by

Rachel M. Kenagy

June 2024

Co-Advisors: Fabio Durante Pereira Alves

Thor Martinsen
Distribution Statement A. Approved for public release: Distribution is unlimited.
 THIS PAGE INTENTIONALLY LEFT BLANK

_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Form Approved OMB
REPORT DOCUMENTATION PAGE No. 0704-0188
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing
instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of
information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions
for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson
Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project
(0704-0188) Washington, DC, 20503.
1. AGENCY USE ONLY 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED
(Leave blank) June 2024 Master’s thesis
4. TITLE AND SUBTITLE 5. FUNDING NUMBERS
AN ACOUSTIC APPROACH TO DRONE IDENTIFICATION USING
MACHINE LEARNING
6. AUTHOR(S) Rachel M. Kenagy

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING

Naval Postgraduate School ORGANIZATION REPORT
Monterey, CA 93943-5000 NUMBER
9. SPONSORING / MONITORING AGENCY NAME(S) AND 10. SPONSORING /
ADDRESS(ES) MONITORING AGENCY
N/A REPORT NUMBER
11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the
official policy or position of the Department of Defense or the U.S. Government.
12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE
Distribution Statement A. Approved for public A
release: Distribution is unlimited.
13. ABSTRACT (maximum 200 words)
Drone use in contested spaces worldwide demonstrates an urgent need for fast and accurate drone
detection, localization, and identification. The miniature near-resonant MEMS acoustic vector sensor
developed at NPS provides drone detection and direction of arrival but cannot identify the source of a sound.
The objective of this research is to augment the sensor with a real-time identification component using
machine learning (ML). Data used in this work includes 350 drone and background audio
files, pre-processed into one-second overlapping windows, resulting in 22,488 samples. Twenty-
seven (27) features were extracted from each sample including signal and frequency statistics and
mel frequency cepstral coefficients. Correlation analysis and neighborhood component analysis were
used to select useful features for ML, decreasing feature dimensionality from 27 to 13. Thirty-two (32)
ML algorithms were trained and tested using standard MATLAB parameters to establish a baseline
performance for a variety of algorithms. Six algorithms resulted in an accuracy above 90% with
acceptable prediction speed. Optimization did not result in higher accuracy or faster prediction speed. A
simple neural network using the rectified linear unit (ReLu) activation function and the ensemble bagged
trees algorithm are recommended for integration with the sensor based on their accuracy, speed, and
potential for extension to multi-class problems.

14. SUBJECT TERMS 15. NUMBER OF

MEMS sensors, acoustic vector sensors, machine learning, mel frequency cepstral PAGES
coefficients 127
16. PRICE CODE

17. SECURITY 18. SECURITY 19. SECURITY 20. LIMITATION OF

CLASSIFICATION OF CLASSIFICATION OF THIS CLASSIFICATION OF ABSTRACT
REPORT PAGE ABSTRACT
Unclassified Unclassified Unclassified UU
NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89)
Prescribed by ANSI Std. 239-18

i
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

ii
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Distribution Statement A. Approved for public release: Distribution is unlimited.

AN ACOUSTIC APPROACH TO DRONE IDENTIFICATION USING MACHINE

LEARNING

Rachel M. Kenagy
Captain, United States Army
BS, United States Military Academy, 2016

Submitted in partial fulfillment of the

requirements for the degrees of

MASTER OF SCIENCE IN APPLIED MATHEMATICS

and

MASTER OF SCIENCE IN APPLIED PHYSICS

from the

NAVAL POSTGRADUATE SCHOOL

June 2024

Approved by: Fabio Durante Pereira Alves

Co-Advisor
Thor Martinsen
Co-Advisor
Francis X. Giraldo
Chair, Department of Applied Mathematics
Frank A. Narducci
Chair, Department of Physics

iii
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

iv
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 ABSTRACT

Drone use in contested spaces worldwide demonstrates an urgent need for fast and
accurate drone detection, localization, and identification. The miniature near-resonant
MEMS acoustic vector sensor developed at NPS provides drone detection and direction
of arrival but cannot identify the source of a sound. The objective of this research is to
augment the sensor with a real-time identification component using machine learning
(ML). Data used in this work includes 350 drone and background audio files,
pre-processed into one- second overlapping windows, resulting in 22,488
samples. Twenty-seven (27) features were extracted from each sample including
signal and frequency statistics and mel frequency cepstral coefficients. Correlation
analysis and neighborhood component analysis were used to select useful features for
ML, decreasing feature dimensionality from 27 to 13. Thirty-two (32) ML algorithms
were trained and tested using standard MATLAB parameters to establish a baseline
performance for a variety of algorithms. Six algorithms resulted in an accuracy above
90% with acceptable prediction speed. Optimization did not result in higher
accuracy or faster prediction speed. A simple neural network using the rectified linear
unit (ReLu) activation function and the ensemble bagged trees algorithm are
recommended for integration with the sensor based on their accuracy, speed, and
potential for extension to multi-class problems.

v
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

vi
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Table of Contents

1 Introduction 1

2 Background 9
2.1 Machine Learning in Drone Detection . . . . . . . . . . . . . . . . 9
2.2 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Classification Work for the NPS MEMS Sensor . . . . . . . . . . . . 19
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3 Methods 21
3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . 24
3.5 Algorithm Training and Analysis . . . . . . . . . . . . . . . . . . 26

4 Results 29
4.1 Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5 Conclusion 41
5.1 Conclusions and Limitations . . . . . . . . . . . . . . . . . . . . 41
5.2 Recommendations for Future Work . . . . . . . . . . . . . . . . . 42

Appendix A Data Files 45

Appendix B MATLAB Code 57

vii
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Appendix C Feature Tables 99

List of References 101

Initial Distribution List 109

viii
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 List of Figures

Figure 1.1 Image of MEMS sensor . . . . . . . . . . . . . . . . . . . . . . 3

Figure 1.2 Image of acoustic vector sensor . . . . . . . . . . . . . . . . . . 4

Figure 1.3 Image of most current sensor design . . . . . . . . . . . . . . . . 5

Figure 1.4 Image of most current acoustic vector sensor design . . . . . . . 6

Figure 1.5 Measured sensor response . . . . . . . . . . . . . . . . . . . . . 6

Figure 2.1 Sample neural network . . . . . . . . . . . . . . . . . . . . . . . 12

Figure 3.1 Frequency response plots of background and drone samples created
for a basic exploration of the data prior to processing. . . . . . . 22

Figure 3.2 Features with high correlation to the target variable and low correla-
tion to each other are considered good for machine learning. Features
here generally show low correlation to the target variable ‘Class’ and
low correlation other features. This indicates that a large amount of
data will be required to train a machine learning algorithm. . . . 25

Figure 3.3 Neighborhood component analysis for each extracted feature shows
responses above 0.1 only for MFCCs. Features with a weighted re-
sponse below this threshold are unlikely to contribute to classifica-
tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Figure 4.1 Test confusion matrix for the SVM algorithm with a quadratic kernel
function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Figure 4.2 Test confusion matrix for the SVM algorithm with a Gaussian kernel
function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Figure 4.3 Test confusion matrix for the weighted k-Nearest Neighbors algo-
rithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Figure 4.4 Test confusion matrix for the ensemble algorithm with a bag method
and decision tree learner type. . . . . . . . . . . . . . . . . . . . 34

ix
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 4.5 Test confusion matrix for the ensemble algorithm with a subspace
method and nearest neighbors learner type. . . . . . . . . . . . . 35

Figure 4.6 Test confusion matrix for a neural network with one fully connected
layer consisting of 100 neurons using the ReLU activation function. 36

x
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 List of Tables

Table 3.1 Data set sample sizes . . . . . . . . . . . . . . . . . . . . . . . . 22

Table 3.2 Feature Extraction Time . . . . . . . . . . . . . . . . . . . . . . 23

Table 4.1 Best performing classifiers . . . . . . . . . . . . . . . . . . . . . 29

Table 4.2 Optimization using cost matrix manipulation . . . . . . . . . . . 37

Table 4.3 Neural Networks using ReLu Activation . . . . . . . . . . . . . . 38

Table 4.4 Neural Networks using Hyperbolic Tangent Activation . . . . . . 38

Table 4.5 Neural Networks using Sigmoid Activation . . . . . . . . . . . . 39

Table 4.6 Bagged Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Table 4.7 Boosted Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Table C.1 Training Set Feature Table . . . . . . . . . . . . . . . . . . . . . 99

Table C.2 Test Set Feature Table . . . . . . . . . . . . . . . . . . . . . . . . 99

xi
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

xii
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 List of Acronyms and Abbreviations

AVS acoustic vector sensor

CNN convolution neural network

CW continuous wave

CWT continuous wavelet transformation

DOA direction of arrival

DOD Department of Defense

FNR false negative rate

FPR false positive rate

GCC gammatone cepstral coefficients

MEMS micro-electro mechanical systems

MFCC mel frequency cepstral coefficients

NCA neighborhood component analysis

NPS Naval Postgraduate School

ReLU rectified linear unit

RF radio frequence

SCF spectral correlation function

SVM support vector machine

TPR true positive rate

USA U.S. Army

USN U.S. Navy

xiii
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

xiv
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Acknowledgments

Thank you to my advisors, CAPT Thor Martinsen and Dr. Fabio Alves, and to Dr. Wei Kang
for your insight, guidance, and patience.

xv
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

xvi
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 CHAPTER 1:
Introduction

Wide availability, low cost, and innovative use have led to drone proliferation across many
contested spaces. In Ukraine, drones provide a cheap capability that can provide invaluable
intelligence, surveillance, and reconnaissance, and destroy much more expensive equip-
ment, industry, and intelligence assets without the traditional risk to soldiers or material
[1]. Sensational drone footage from the early days of the war is rife with videos of drones
destroying tanks, trucks, and more recently, attacking oil refineries deep inside Russian bor-
ders [2]. Small commercial drones retrofitted for military operations or nefarious purposes
present a significant challenge to traditional security protocols [3]. On January 28, 2024 an
incoming drone at a U.S. base in Jordan was misidentified as friendly, killing three U.S. ser-
vice members and injuring 40 more [4]. And in March of 2024, a Chinese man posted videos
to social media from a small drone flying over the JS Izumo, a Japanese aircraft carrier
[3]. Japan’s defense ministry confirmed that the videos of the JS Izumo are likely authentic
and represent a significant breach in security [5]. The ministry indicated that future base
security depends on the introduction of "more capable drone countermeasure equipment"
[5]. Drone incidents are not limited to military interactions. On the U.S.-Mexico border,
human traffickers and drug smugglers regularly use drones to surveil U.S. border patrol and
transport illegal drugs [6], [7]. Outdoor sporting events now require drone specific security
measures to prevent unauthorized access and ensure the safety and privacy of attendees [8].
Individuals seeking to cause chaos or access secure events routinely fly drones into private
spaces and no-fly areas causing travel delays at major airports around the globe [7]. These
incidents point to an urgent need to quickly detect, identify, and locate drones in contested
spaces.

There are five common types of drone detection systems: radio frequency, optical, thermal,
radar, and acoustic sensors [9]. Radio frequency (RF) systems detect, and in some cases
jam, radio communication between a drone and its pilot [10]. Radio frequency sensors
can provide a passive option for detecting piloted drones and even triangulating the pilot’s
location, but cannot detect autonomous drones and suffer from signal attenuation in crowded
areas [9]. Optical sensors provide visual confirmation of the drone and its cargo and can

1
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 record evidence, but require significant data processing and rely on line of sight and clear
weather conditions [9], [10]. Thermal imagery is a type of optical detection which can
operate at night, overcoming one of the main drawbacks of optical sensors [9]. However,
the thermal imagery depends on a drone’s heat signature and the thermal characteristics of
the background environment, which are highly variable. Many drones are made of plastic
with small motors and low resolution thermal cameras cannot provide any actionable
identification from the heat production [9]. Radar provides long range tracking and accurate
localization under most weather conditions and can easily detect many drones at once,
but detection relies on the radar cross section of the target [10]. Radar is not suitable for
detecting small drones hovering or flying at low altitudes [9]. Acoustic sensors provide a
passive capability that can augment each of the other methods. They can operate in poor
weather conditions, detect autonomous drones with no RF emissions, and do not require
line of sight [10]. Typically, acoustic sensors require microphone arrays to provide direction
of arrival (DOA) information. Such arrays can be large and cumbersome, and often have
short range and struggle in acoustically noisy environments [10]. Small microphone vector
sensors can be used for drone localization, but with limited detection range.

In order to implement a miniature, accurate, and low power consumption solution to acousti-
cally detect drones, NPS researchers from the sensor research lab are developing an acoustic
vector sensor (AVS) based on a MEMS sensor. The MEMS sensor, shown in Figure 1.1,
uses a pair of wings that vibrate when the sensor is exposed to acoustic waves [11]. The
wings are connected by a bridge in the center of the sensor [11]. At the edge of each wing,
interdigitated comb finger capacitors allow for transducing vibration to capacitance that can
be read by an electronic circuit.

2
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 1.1. Source: [11] "Image of MEMS sensor: (a) microscope image of
MEMS sensor: (1) wing, (2) bridge, (3) torsional legs, (4) substrate, (5) SEM
image of comb fingers (for capacitive sensing), (6) diagram showing the 5 µm
gap between comb fingers; and (b) laser vibrometry image of MEMS sensor
vibrating in bending mode. Inset: normalized sensitivity response of sensor
showing a resonance at about 680 Hz."

The wings exhibit two predominant vibrational modes: rocking and bending. Various con-
figurations have been tested to optimize the sensor’s performance in air and underwater [11],
[12], [13], [14], [15]. Direction of arrival estimation can be accomplished by combining two
of the MEMS sensors in a perpendicular configuration and an omnidirectional microphone
into an acoustic vector sensor (AVS) [11]. Figure 1.2 shows this configuration.

3
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 1.2. Acoustic vector sensor. Source: [11] "(a) top-down view. The
AVS can be used to determine the DOA of an incoming sound and consists
of a cosine sensor (blue), orthogonal sine sensor (red), and omnidirectional
sensor (green); and (b) 3-D view: (1) cosine sensor, (2) sine sensor, and (3)
omnidirectional sensor."

The sensor operates near resonance, which acts as a filter for unwanted frequencies and helps
achieve detection of quiet or distant sources [14], [16]. Desired frequencies can be targeted
through alterations to the dimensions of the wings and bridge [14]. The most current design
of the sensor from Ivancic et al. [16] is based on independent multi-resonant wings to obtain
a high signal-to-noise ratio and broaden the sensor’s bandwidth. Figure 1.3 shows the latest
design of an individual sensor.

4
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 1.3. Source: [16] "Layout of single wing of the sensor design. (A)
Layout of entire wing: (1) wing, (2) bridge, (3) torsional leg, (4) comb fingers,
(5) gold wire bonding pad, and (6) a groove in the device layer electrically
separating the wing from the substrate. (B) Zoomed-in view of fishbone shape
of comb fingers. The dark grey areas under the wing and surrounding the
bridge and torsional legs represent a trench that passes through the base layer
of the sensor."

This design is similar in function to the MEMS design in Figure 1.1; when the wing vibrates
in response to acoustic waves the capacitance between the wing and substrate comb fingers
changes with the deflection [16]. However, in a single wing configuration the bending mode
is dominant and the fishbone shaped fingers create more surface area to increase capacitance
[16]. Single wings with different resonant frequencies are combined as in Figure 1.4 to form
a multi-resonant sensor with a broader bandwidth than a single wing [16].

5
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 1.4. Source: [16] "Multi-resonant acoustic sensor. (A) Microscope
image of V12 sensor; (B) image of MEMS sensor mounted in PCB; (1)
multi-resonant MEMS sensor, (2) capacitive readout circuitry, and (3) wire
connections for power and readout; (C) SEM image of fishbone-patterned
comb fingers. Note that residual stress on wings from fabrication causes
wings to bend slightly, lifting wing comb fingers while at rest, approximately
halfway up from substrate comb fingers."

In this configuration each individual wing acts as its own harmonic oscillator, resulting in
the mechanical sensitivity, acoustic sensitivity, and phase response depicted in Figure 1.5
[16].

Figure 1.5. Source: [16] "Measured sensor response: (A) mechanical sensitiv-
ity measured via laser vibrometry. Overall sensor response is calculated. (B)
Acoustic sensitivity measured from output of capacitive sensing circuit. (C)
Phase associated with acoustic sensitivity. Overall sensor response is mea-
sured for (B,C)."
6
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Research on the MEMS sensor project has primarily been focused on improving the sensor’s
sensitivity and frequency response to accomplish drone detection and DOA. Limited work
has been done to identify the source of incoming sounds. While the sensor provides critical
detection and DOA information, it cannot currently distinguish between a signal of interest
and background noise. This project aims to augment the MEMS sensor capability and
provide this essential identification component. The objective is to distinguish between
drone and background noise using a machine learning classifier and data already collected
on the MEMS project. Integration of an intelligent identification algorithm into the sensor
could provide a lightweight system that significantly increases battlefield awareness [11].
This work will focus on data processing methods and algorithms that do not require large
processing capabilities so that the final results are operationally useful. The results serve
as a proof of concept and starting point to run a real time classification algorithm on the
sensor’s microcontroller and eventually distinguish between types of drones.

This work is organized into five parts: introduction, background, methods, results, and
conclusion. The background chapter provides information on machine learning and signal
processing techniques relevant to this project, as well as a summary of previous classification
work done for the MEMS sensor project. The methods section details data collection,
feature selection, and algorithm analysis. The results section presents analysis of the six
best performing classification algorithms and options for optimization with comparison to
standard parameter algorithm performance. The conclusion provides a recommendation
for a classifier to be used with the NPS MEMS sensor, the limitations of this work, and
suggestions for future research.

7
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

8
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 CHAPTER 2:
Background

2.1 Machine Learning in Drone Detection

Machine learning algorithms aim to learn a function which describes a data set in order
to make predictions about new independent data. The success of learning algorithms is
largely dependent on quality data sets: extreme values, biased, or misrepresentative data
can all lead to poor performance [17]. Data with extreme values is often skewed by outliers
which cause learning algorithms to predict incorrectly. Inconsistent data collection or data
that does not represent the target of a learning algorithm could also lead to bias and false
performance results. Learning algorithms can be divided into two categories: supervised
and unsupervised learning [17], [18], [19]. Supervised learning maps an input to an output
based on known examples in a labeled data set [18]. Large data sets containing descriptive
features and a target variable are typically divided into training and test sets. An algorithm
then learns from features in the training set to form a predictive model and can be tested for
accuracy using the test set. In supervised learning, the target variable has a known value.
This often consists of a binary label: ‘0/1,’ ‘-1/1,’ or ‘Drone/No Drone.’ In multi-class
classification, target labels may be a list of the relevant classes: ‘Drone Type 1,’ ‘Drone
Type 2,’ ‘Drone Type 3,’ etc. Unsupervised learning algorithms take unlabeled data as
input and learn patterns without human instruction [18], [20], [21]. Both types of machine
learning result in a model that can be evaluated based on how well it predicts the outputs in
the training and test data. Machine learning models often suffer from two errors: underfitting
or overfitting [17]. Underfitting can occur when the model is too simple or data points do
not provide enough information, and results in predicted values that are far from the known
values of the training set. If a model is too complex and overfits the data, then it performs
well on the training set but predicts poorly on the test set because it is trained too specifically
to the known data [17].

As drones proliferate, more data becomes available to train machine learning algorithms
to predict the presence of a drone or distinguish between different types of drones. Recent
studies demonstrate the viability of drone detection using machine learning with a variety

9
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 of data sources including image, radar, and acoustic data [7]. Image based data includes
standard video and thermal imagery and requires cameras with a line of sight to the drone
[7]. In their 2017 work, Rozantsev et al. [22] proposed a convolution neural network (CNN)
model to detect flying objects using image based data. Samples from two video datasets of
drones and aircraft were divided into overlapping frames then processed with a muti-scale
sliding window. The authors trained boosted tree regression algorithms to compensate for
motion in the data and two convolution neural networks for classification. This method
achieved an average result of 84.9% accuracy for the drone dataset and 86.4% accuracy for
the aircraft dataset. Lee et al. [23] also proposed a CNN model to identify drones from a
camera mounted on another drone. A Haar feature cascade classifier detects drones in images
received from the camera and a CNN identifies the drone model. Detection accuracy and
identification accuracy for this system were 89% and 91.6% respectively. In 2018 Yoshihashi
et al. [24] proposed a recurrent correlational network for drone identification. The system
was comprised of four layers: a convolution layer, a ConvLSTM, a cross-correlation layer,
and fully connected layers which result in final confidence scores. Each of these methods
used learned features and deep learning models, which require large labeled datasets to train
accurate models [25]. There are few publicly available image based datasets large enough
to build robust predictive models. The datasets that are large enough to train deep learning
models often suffer from inconsistent acquisition. A single dataset could be collected from
many different sources, making results difficult to reproduce [25]. Additionally, because
image based data requires a line of sight from the collection point, it is often operationally
restricted by weather and terrain.

Radar methods do not require line of sight and can provide the added benefit of localization
[7]. In their 2020 work, Zhang and Li [26] achieved drone identification accuracy of 93%
using reflected radar signatures from multiple micro-drones using Cadence Frequency
Spectrum features and a k-means classifier. Molchanov et al. [27] used data collected from
a CW radar after flying different types of drone for 30 seconds each. The authors used
eigenvector and eigenvalue based features to train a Naive Bayes classifier and both linear
and non-linear support vector machines (SVMs). Their method resulted in an accuracy
rate of 95%. Mohajerin et al. [28] used data collected from an air traffic control radar to
classify drones, birds, and manned aircraft. The research used real world data for birds and
manned aircraft, but drone data was artificially simulated. Mohajerin used statistical features

10
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 target radar cross section, mean and variance of speed to train a neural network resulting
in 100% accurate drone identification. The results from this work rely on simulated data
with assumptions regarding range and radar cross section which likely inflate the reported
accuracy [25]. Mendis et al. [29] extracted the spectral correlation function (SCF) from the
micro-Doppler signature to train a deep belief network. In addition to a reference class with
no drone, the authors flew an artificial bird, a helicopter, and a quad-copter micro-drone at a
fixed point three meters from a continuous wave (CW) radar in a lab environment to create
the SCF images. Gaussian noise was added to a number of the images to imitate real-world
conditions. This method resulted in classification accuracy above 90%. While radar based
data achieves a high classification rate, radar detection is limited when identifying drones
with small radar cross sections or flying at low altitudes [7].

Machine learning using acoustic data is of particular interest for this work because the
results will be integrated with an acoustic sensor which may provide an alternative to
image or radar based detection methods. Acoustic data provides a passive capability that is
advantageous to drone detection under low visibility and low altitude conditions common
on the battlefield [7], [11], [30]. In their 2021 paper, Kolumunna et al. [7] created a
classification system called ‘DronePrint’ which uses acoustic data and a neural network to
identify the make and model of a drone with an accuracy rate of 90%. The authors used
a mix of experimentally collected and online scraped data to build a large dataset. After
exploring the power spectrum of the drones and background noise, they concluded that using
power spectrum characteristics for machine learning would be ambiguous and lead to poor
performance. Instead, mel frequency ceptral coefficients (MFCCs) were used to process the
data and extract features for classification. The DronePrint system uses a series of three long
short-term memory, a type of recurrent neural network, classifiers to first detect a drone and
then identify the model. Bernardini et al. [31] processed sound data broken into 5 second
frames and computed the short-time energy, zero crossing rate, temporal centroid, spectral
centroid, spectral roll-off, and MFCCs for each frame. The resultant data was used to train
support vector machine classifiers to distinguish between drone noise and different types
of background noise. The SVM classifiers successfully identified drones with accuracy
rates between 96.4% and 99.2%. Al-Emadi et al. [32] recorded audio of drones flying and
hovering in a quiet indoor environment using a smartphone microphone. The authors pre-
processed the data by splitting each sample into one, two, and five second frames, observing

11
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 that the one second frames outperformed the other time windows. After splitting the data the
authors converted it into spectrograms to train three different neural networks for detection
of a drone and identification of the type of drone. A convolution neural network resulted in
the highest detection accuracy rate of 96.38% and the highest identification rate of 92.94%.
Additional information on neural networks and SVMs is provided in sections 2.1.1 and
2.1.2 because both algorithms appear prominently in studies using acoustic data in machine
learning.

2.1.1 Neural Networks

Neural networks are machine learning algorithms that mimic neural structures [19]. Each
node in a neural network is a computational unit that takes input from previous nodes,
computes an activation function, and passes the output to the next node [19]. Layers of
these units are linked together to create a network as shown in Figure 2.1.

Figure 2.1. A sample neural network from the Stanford Deep Learning Tutorial
[19] depicting a network with input layer 𝐿 1 , two fully connected, often called
hidden, layers, 𝐿 2 and 𝐿 3 , and two outputs in layer 𝐿 4 .

The first and last layers of a neural network are its input and output layers, respectively, and
any intermediate layers are called hidden layers [19]. A network may have any number of
hidden layers and computational units. Computation of the activation function and passing
this output forward is called forward propagation [19]. Common activation functions include
sigmoid
1
𝑓 (𝑧) = , (2.1)
1 + 𝑒 (−𝑧)

12
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 hyperbolic tangent
𝑒 (𝑧) − 𝑒 (−𝑧)
𝑓 (𝑧) = 𝑡𝑎𝑛ℎ(𝑧) = (𝑧) , (2.2)
𝑒 + 𝑒 (−𝑧)
and the rectified linear unit (ReLU) function

𝑓 (𝑧) = 𝑚𝑎𝑥(0, 𝑧) (2.3)

which is zero for negative input [19]. While sigmoid has been used for many years, empirical
observation shows that the ReLU function is much faster than other activation functions,
and is now widely used in deep learning applications [33]. Neural networks are often
trained using a cost function and gradient descent. A common cost function for 𝑚 samples
{(𝑥 (1) , 𝑦 (1)) , ..., (𝑥 (𝑚) , 𝑦 (𝑚) )} is the one-half squared error

𝑚 𝑛𝑙 −1 ∑︁
𝑠𝑙 ∑︁
𝑠𝑙+1
1 ∑︁ 1 𝜆 ∑︁
𝐽 (𝑊, 𝑏) = [ ( (ℎ𝑊,𝑏 (𝑥 (𝑖) ) − 𝑦 (𝑖) ) 2 ] + (𝑊 𝑗𝑖(𝑙) ) 2 (2.4)
𝑚 𝑖=1 2 2 𝑙=1 𝑖=1 𝑗=1

which computes the average sum of squares error in the first term based on the difference
between the network hypothesis ℎ𝑊,𝑏 (𝑥 (𝑖) ) and the true value of a sample 𝑦 (𝑖) , and a weight
decay term which helps prevent over-fitting [19]. The weight decay term depends on the
parameter 𝜆 which determines the relative importance of the two terms and 𝑊, the output
of the activation function in each layer 𝑙 [19]. Some neural networks include the bias term
𝑏 in each layer which can be used to shift the activation function [19]. The goal of training
is to minimize the error term, 𝐽 (𝑊, 𝑏), using an optimization algorithm. Gradient descent
optimization using back-propagation works well for many classification problems. The first
round of gradient descent amends 𝑊 by computing the activation function and 𝑏 with a
learning rate of 𝛼 to

𝜕
𝑊𝑖(𝑙) (𝑙)
𝑗 = 𝑊𝑖 𝑗 − 𝛼 𝐽 (𝑊, 𝑏)
𝜕𝑊𝑖(𝑙)
𝑗
(2.5)
𝜕
𝑏𝑖(𝑙) = 𝑏𝑖(𝑙) −𝛼 𝐽 (𝑊, 𝑏),
𝜕𝑏𝑖(𝑙)

13
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 𝜕 𝜕
which requires the computation of partial derivatives (𝑙) and (𝑙) [19]. The learning rate
𝜕𝑊𝑖 𝑗 𝜕𝑏 𝑖
parameter, 𝛼, controls how much each weight is updated during training [19]. These partial
derivatives can be computed for the overall cost function using

𝑚
𝜕 1 ∑︁ 𝜕
(𝑙)
𝐽 (𝑊, 𝑏) = [ (𝑙)
𝐽 (𝑊, 𝑏; 𝑥 (𝑖) , 𝑦 (𝑖) )] + 𝜆𝑊𝑖(𝑙)
𝑗
𝜕𝑊𝑖 𝑗 𝑚 𝑖=1 𝜕𝑊
𝑖𝑗
𝑚
(2.6)
𝜕 1 ∑︁ 𝜕
(𝑙)
𝐽 (𝑊, 𝑏) = (𝑙)
𝐽 (𝑊, 𝑏; 𝑥 (𝑖) , 𝑦 (𝑖) )
𝜕𝑏𝑖 𝑚 𝑖=1 𝜕𝑏 𝑖

[19]. Back-propagation allows for the computation of an error term that determines each
node’s responsibility for the error of the output. Optimization tunes the weight of each
node to minimize this error. The Stanford Deep Learning tutorial created by Ng et al. [19],
from which these equations originate, provides an excellent condensed version of both the
feed-forward and back-propagation steps in matrix notation.

2.1.2 Support Vector Machines

A support vector machine is a machine learning algorithm, originally proposed by Vapnik
in the 1990s, which is still frequently used for classification of large binary data sets due to
its learning speed and performance in pattern recognition [21], [34], [35]. Support vector
machines use a kernel function to map training data into a feature space to determine
the optimal hyperplane that can divide the binary data into separate classes [34]. Vapnik
[35] described this method of machine learning as the solution to the following primal
optimization problem using a given set of labeled data (𝑥𝑖 , 𝑦𝑖 ), 𝑖 = 1, ..., 𝑙 where 𝑥𝑖 ∈ 𝑅 𝑛
and 𝑦𝑖 ∈ {1, −1}𝑙
𝑁
1 𝑇 ∑︁
min 𝑤 𝑤+𝐶 𝜉𝑖
𝑤,𝑏,𝜉 2 𝑖=1
(2.7)
s.t. 𝑦𝑖 (𝑤𝑇 𝜙(𝑥𝑖 + 𝑏)) + 𝜉𝑖 ≥ 1
𝜉𝑖 ≥ 0, 𝑖 = 1, ..., 𝑙.
The objective function contains weights 𝑤 and a penalty term, 𝐶 > 0, for the error, 𝜉 [35],
[36]. The objective is constrained by 𝜙, a function that maps training data and a bias unit
into a higher dimensional space [36].

14
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 This is often solved with the Lagrangian dual of 2.7, using dual variable 𝛼. The dual problem

1 𝑇
max 𝛼 𝑄𝛼 − 𝑒𝑇 𝛼
𝛼 2
s.t. 0 ≤ 𝛼𝑖 , 𝑖 = 1, ..., 𝑙 (2.8)

𝑦𝑇 𝛼 = 0

where 𝑒 is a vector of ones and 𝑄 is an 𝑙 by 𝑙 matrix given by 𝑄 𝑖 𝑗 = 𝑦𝑖 𝑦 𝑗 𝐾 (𝑥𝑖 , 𝑥 𝑗 ) is less

computationally expensive and, if feasible, results in an equal optimal value as the primal
[36]. 𝐾 is called the kernel function and is given by

𝐾 (𝑥𝑖 , 𝑥 𝑗 ) = 𝜙(𝑥𝑖 )𝑇 𝜙(𝑥 𝑗 ) (2.9)

[35], [36]. While kernel function selection is limited only by the satisfaction of the Mer-
cer condition, different functions result in disparate outcomes [34], [37]. Support vector
machines with a Gaussian kernel function
− ∥ 𝑥𝑖 − 𝑥 𝑗 ∥ 2
𝐾 (𝑥𝑖 , 𝑥 𝑗 ) = 𝑒 2𝜎 2 (2.10)

or cubic kernel function

𝐾 (𝑥𝑖 , 𝑥 𝑗 ) = (𝑥𝑖𝑇 𝑥 𝑗 + 1) 3 (2.11)

are widely used in machine learning problems dealing with image and acoustic data [7],
[21], [31], [34], [36], [38].

2.2 Data Processing

Data processing is the first step in machine learning. Unprocessed data often requires large
storage capacity and results in high dimensionality when used in machine learning. Audio
data is typically processed by extracting signal statistics and frequency characteristics, in-
cluding MFCCs [7], [31], [39]. Kolamunna et al. [7] created their ‘DronePrint’ system to
distinguish between types of drone using MFCCs to train neural networks with a detection
accuracy up to 96%. Similarly, Djeddou et al. [40] used MFCCs to train a Gaussian mixture
model and the bagged decision trees algorithm in their gunshot classifier with an overall
accuracy rate of 96.29%. In 2021, Pawar et al. [41] calculated pitch, energy, MFCCs, and

15
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 mel energy spectrum dynamic coefficients to train a CNN for emotion recognition in speech.
Salman et al. [42] analyzed five data processing methods: mel frequency cepstral coeffi-
cients, Gammatone cepstral coefficients, linear prediction coefficients, spectral roll-off, and
zero-crossing rate for their efficiency in drone classification using a support vector machine
classifier. The authors found that when combined with a quadratic polynomial SVM kernel,
Gammatone cepstral coefficients and mel frequency coefficients are equally efficient fea-
tures with a 99.8% accuracy rate and comparable speeds. When using a fine Gaussian SVM
kernel Gammatones were the most efficient of all five features. These processing methods
effectively extract meaningful features from unprocessed data, helping to distinguish be-
tween classes and decrease dimensionality in machine learning [43]. Because it is a widely
used technique, more detail on MFCC calculation is provided in section 2.2.1.

2.2.1 Mel Frequency Cepstral Coefficients

Mel frequency cepstral coefficients are sets of mathematically extracted characteristics
meant to mimic human auditory perception [7]. They are widely used in speech, speaker,
and emotion recognition, drone and gunshot identification, and medical applications [7],
[39], [40], [43], [44]. Mathematically, the coefficients are the discrete cosine transform of
the mel scaled log-power spectrum [31]. MFCCs are most commonly calculated via a five
step process: pre-emphasis, signal framing and windowing, power spectrum, mel filter bank,
and discrete cosine transform. Pre-emphasis is a pre-processing step which is accomplished
by applying a high-pass filter to signal data [45]. This can compensate for suppression of
the high frequency of a signal during production [43]. While high-pass filters are common,
novel work in MFCC calculation proposes the use of both low-pass and band-pass filters
depending on the goal of signal processing [46]. Following pre-processing, the signal is split
into discrete frames and a windowing function is applied, typically the Hamming window
function for MFCC calculation [44]. The discrete Fourier transform for signal length 𝑁
where 𝑥(𝑛) is the discrete signal

𝑁−1
−𝑖2 𝜋 𝑘𝑛
∑︁
𝑋𝑘 = 𝑥𝑛 𝑒 𝑁 𝑘 = 0, ...𝑁 − 1 (2.12)
𝑛=0

is then applied to find the power spectrum [43]. This is commonly done using a fast Fourier
transform algorithm to increase computational speed. Next the mel filter, a bank of triangular

16
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 filters, is applied to extract a non-linear representation of the signal. The mel scale response
of frequency 𝑓 is then calculated by

𝑓
𝑚 = 1127 ∗ log(1 + ). (2.13)
700

Finally, the discrete cosine transform is applied for signal length 𝑀 and discrete signal 𝑥 𝑚

𝑀−1
∑︁ 𝑖2 𝜋 𝑘𝑚
𝑋𝑘 = 𝑥 𝑚 𝑐𝑜𝑠 𝑀 𝑘 = 0, ...𝑀 − 1 (2.14)
𝑚=0

to find the mel coefficients [43]. This process synthesizes large datasets into a manageable
set of features that have proven useful across many machine learning applications.

2.2.2 Statistical and Frequency Features

In addition to MFCCs, statistical and frequency characteristics have been used with varying
success to process audio data for use in machine learning algorithms [47], [48]. Common
statistics and frequency features include skewness, kurtosis, dominant frequency value
and magnitude, signal mean value, and signal and spectral entropy [39]. Each processing
method, or the combination of methods, results in a set of features that can be used to
differentiate between samples and train a machine learning classifier.

2.3 Feature Selection

Signal processing techniques provide many different features that can be used to distinguish
between data, but some are more useful than others for classification. Redundant features,
or features that provide no discriminating information, require unnecessary processing
power and should not be computed. Feature selection can help eliminate irrelevant features
and reduce dimensionality to improve computational efficiency. However, it is difficult to
tell which features contribute to classification and which are superfluous. Neighborhood
component analysis (NCA) and correlation analysis are common methods used to determine
effective predictor variables in machine learning algorithms [49], [50].

17
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 2.3.1 Neighborhood Component Analysis
In their 2012 article, Yang et al. [49] proposed using neighborhood component analysis
(NCA) to weight relevant machine learning features. Neighborhood component analysis is a
learning algorithm that aims to maximize the distance calculation from the k-nearest neigh-
bors classification algorithm [48]. Given a set of data points 𝑇 = {(𝑥 1 , 𝑦1 ), ..., (𝑥 𝑁 , 𝑦𝑁 )}
where 𝑥𝑖 is a 𝑑-dimensional feature vector and 𝑦𝑖 ∈ {1, ..., 𝐶} is the corresponding class
label and 𝑁 is the number of samples, the objective of NCA is to find a weighting vector 𝑤
which maximizes the following distance

𝑑
∑︁
𝐷 𝑤 (𝑥𝑖 , 𝑥 𝑗 ) = 𝑤𝑙2 |𝑥𝑖𝑙 − 𝑥 𝑗𝑙 | (2.15)
𝑙=1

where 𝑤𝑙 is the weight of the 𝑙 𝑡ℎ feature [49]. The algorithm requires two parameters: 𝜎, an
input parameter to the kernel function, and 𝜆 > 0, a regularization function. Neighborhood
component analysis performs well but is generally a computationally expensive feature
selection algorithm [48]. Suhm’s [39] heart sound classifier, which identifies abnormal
heart function using acoustic data, initially calculates statistical and frequency features, as
well as MFCCs in an attempt to find useful machine learning features. Suhm uses NCA to
eliminate all statistical and frequency features and reduce feature extraction to MFCCs only.
Malan et al. [48] also successfully executed NCA to reduce the feature dimension from 42
to 6 in their study on classification performance of motor imagery signals. Similar methods
have been used in studies to detect abnormal crowd behavior, and diagnose breast cancer,
Alzheimer’s disease, and COVID-19 [51], [52], [53], [54].

2.3.2 Correlation Analysis

Correlation analysis between predictive features and the target variable is a simple method
to select features that best characterize the target and reduce dimensionality [50], [55], [56].
In their study on predictor correlation and implications for fenomic studies, Nicodemus et al.
[55], showed that predictive features that were highly correlated with each other negatively
impacted the performance of three learning algorithms. The results of this study indicate
that feature selection should eliminate within-predictor correlation. Assegie et al. [56] used
correlation analysis to identify relevant features in a heart failure dataset used to train a k-
nearest neighbor algorithm to identify heart disease. Features that were strongly correlated

18
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 to the target variable significantly improved the classifier’s performance, indicating that
feature selection should attempt to retain features that are strongly correlated to the target
variable. Djeddou et al. [40] used correlation analysis between predictive features to reduce
the number of calculated cepstral coefficients in their gunshot classifier from 16 to 14.
Kumar et al. [50] also performed correlation analysis in their study of machine learning to
predict depressive disorder. The authors executed a ranking technique to select and weight
features based on their within-predictor correlation and correlation to the target variable.
Their method resulted in a reduction in features from 24 to 9 and an increase in accuracy
from 85.7% to 91.1%. Correlation analysis requires calculation of the correlation coefficient
between each two variables in the data set

𝑐𝑜𝑣( 𝐴, 𝐵)
𝐶 𝐴,𝐵 = (2.16)
𝜎𝐴 𝜎𝐵

and is a function of the covariance, 𝑐𝑜𝑣( 𝐴, 𝐵), and standard deviation, 𝜎 [50]. A relevant
feature should not be correlated with other features but should be highly correlated to the
target variable [50], [56].

2.4 Classification Work for the NPS MEMS Sensor

Previous work on the sensor project at NPS used continuous wavelet transformation (CWT)
and AlexNet to distinguish between drone and background noise [30]. AlexNet is a convo-
lution neural network (CNN) used for image classification which requires images of specific
dimensions as input [30]. Fleming used CWT to transform audio data into scaleogram im-
ages, which were then used to train the AlexNet CNN interface in MATLAB. This approach
resulted in a 94.2% accuracy rate when distinguishing between drone and background
noise, but is computationally expensive [30]. The CWT signal processing method required
60 hours to generate and store 3,000 images and 1 hour to train the CNN using a common
desktop computer [30]. This processing time is impractical for operational use. Any data
processing scheme deployed with a machine learning algorithm on the MEMS sensor must
be able to quickly extract relevant features to be of any use on the battlefield.

19
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 2.5 Summary
There is a robust body of work with successful results in drone classification using a variety
of data sources and processing methods. Prior research done to classify drone noise for
the NPS MEMS sensor project focused on a data processing method that is unsuitable for
operational use and was limited to one classification algorithm. Success in applications
with similar data types indicates that MFCCs may be useful in drone classification for
the MEMS sensor. The success of neural networks and SVMs in similar classification
applications suggests that neural networks and SVMs may also work for classification on
this project. The supervised machine learning techniques and data processing methods
explored here may provide an alternative to Fleming’s approach.

20
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 CHAPTER 3:
Methods

3.1 Data Collection

Data for this work was collected during field and anechoic chamber testing for the MEMS
sensor. The same data was used in Fleming’s initial work on drone classification for the
MEMS sensor project [30]. Sound files were recorded using broadband microphones in
different positions with respect to the recorded drones and background noise. Background
noise collected at a military exercise at Camp Pendleton includes passing helicopters,
assault ships, generators, and ocean noise. Drones recorded at the same exercise in various
flight patterns include the navigator, parrot, sky raider, and Lawrence Livermore National
Laboratory drones. Drones recorded in the anechoic chamber at NPS include the DJI Mavic,
Phantom 3 and 4 PRO, Typhoon, Inspire, Matrice 1000, Shield AI, and Matrice drones in
various loading configurations. A comprehensive review of the drones recorded for in data
can be found in Fleming’s work [30]. Refer to Appendix A for a complete list of the 350
total data files.

3.2 Data Processing

Data processing for this work began with exploration of the raw sound files. Figure 3.1 shows
frequency response plots of generator background noise and a hovering parrot drone created
for an initial investigation of the data. From a visual inspection, the frequency responses
show clear differences in drone and background noise but do not provide practical features
to use in machine learning.

21
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 (a) Background sample (b) Drone sample
Figure 3.1. Frequency response plots of background and drone samples created
for a basic exploration of the data prior to processing.

Prior to pre-processing, sample data was divided into training and test sets using 79% for
training and 21% for testing. This division follows the common rule of thumb that training
and test data should be between an 80/20 and 70/30 split. Test and training sets were separated
prior to pre-processing to ensure that windows from the same sample were not included
in both the training and test data. This prevented bias and false accuracy in the resultant
algorithms. During pre-processing each sound file was split into one second windows with
a 20% overlap to avoid edge effects, resulting in 22,488 total samples. Partitioning the raw
files prior to processing increases the number of samples for use in classification. Table 3.1
shows the total number of samples in each set.

Table 3.1. Data set sample sizes

Data Set Background Drone Total
Training 3216 14542 17758
Test 844 3886 4730

22
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 3.3 Feature Extraction
Feature extraction and algorithm development were executed in MATLAB because the
existing work for detection and DOA on the MEMS sensor already use MATLAB. A variety
of 27 easily computed signal and frequency features known to be useful for classification
of audio data were extracted to provide a large number of distinguishing characteristics per
sample [7], [39], [40], [41], [43], [44], [46]. Many of these features were found to provide
very little value to classification and were discarded after feature selection. Sample tables
containing examples of the final 13 features for the training and test data can be found in
Appendix C.

3.3.1 Signal Statistics

Extracted signal statistics include: mean and median value, standard deviation, mean ab-
solute deviation, 25th, 75th, and interquartile range, sample skewness and kurtosis, signal
entropy, and spectral entropy. These features were extracted in the hopes that they would
provide distinguishing characteristics because they are extremely fast to compute. How-
ever, as detailed in feature selection, all signal statistics were determined to be irrelevant to
classification and discarded.

3.3.2 Frequency Characteristics

Frequency characteristics included dominant frequency value, dominant frequency magni-
tude, and dominant frequency ratio which gives the ratio of the energy of the maximum to the
total energy [39]. Additionally, MFCCs were calculated for each sample. These frequency
characteristics were chosen because they are known to be useful for audio data classification
and can be computed quickly using computations that the MEMS sensor already executes
to calculate DOA. Table 3.2 shows the processing time required to extract features from the
training and test sets, as well as a single, 42 second, drone audio file resulting in 52 samples.

Table 3.2. Feature Extraction Time

Data Set Sample Size Processing Time (s)
Training 17758 375.59
Test 4730 69.68
One Drone File 52 2.65

23
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 These processing times represent a 99% decrease in the processing time required from
the 60 hours required for Fleming’s scaleogram approach [30]. Mel frequency cepstral
coefficients and frequency characteristics were shown to provide value to classification and
were carried forward after feature selection.

3.4 Feature Selection

Feature selection included both correlation analysis and NCA. Both feature selection meth-
ods were executed with the goal of maximizing prediction accuracy of the final classifiers
and minimizing computational expense of data processing. Computational expense was
of particular interest because the resulting features will eventually be computed using the
limited processing power of sensor’s microcontroller.

3.4.1 Correlation Analysis

Correlation analysis was executed in MATLAB and used to assess the correlation of each
feature to the target variable ‘Class,’ which indicates whether a sample is background or
drone noise. This analysis also assessed correlation between features. Features with high
correlation to the target variable and low correlation to each other are considered good
for machine learning. Figure 3.2 shows the results of the correlation matrix calculation.
Statistical features show high correlation to each other and low correlation to the target
variable. Similarly, most frequency related features show high correlation to each other and
low correlation to the target variable.

24
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 3.2. Features with high correlation to the target variable and low corre-
lation to each other are considered good for machine learning. Features here
generally show low correlation to the target variable ‘Class’ and low correla-
tion other features. This indicates that a large amount of data will be required
to train a machine learning algorithm.

Mel frequency cepstral coefficients and dominant frequency value showed the highest
correlation to the target variable and were carried forward into neighborhood component
analysis.

3.4.2 Neighborhood Component Analysis

Neighborhood component analysis was executed in MATLAB using a regularization pa-
rameter of 𝜆 = 0.005. Figure 3.3 shows the weighted response of each extracted feature.

25
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 3.3. Neighborhood component analysis for each extracted feature shows
responses above 0.1 only for MFCCs. Features with a weighted response below
this threshold are unlikely to contribute to classification.

Features with a weighted response 𝑤 > 0.1 were carried forward into classification. Based on
a combination of correlation analysis and NCA, dominant frequency value was also carried
forward. Other features with a weighted response 𝑤 ≤ 0.1 were considered irrelevant to
classification.

3.5 Algorithm Training and Analysis

Algorithm training and analysis was performed using the MATLAB Classification Learner
App. This enables the user to simultaneously train many different algorithms using the
same training dataset to compare performance. Thirty two (32) algorithms were trained
using standard MATLAB parameters and 5-fold cross-validation. This validation scheme
shuffles the training dataset and then randomly partitions the data into five roughly equal
folds. Each fold is divided into five parts, four of which are used to train, and one to validate.

26
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Each data point is used once and each training cycle yields an evaluation score. The average
evaluation score is reported as average error [57]. This "shotgun" approach to algorithm
training was an experimental decision made to avoid a biased choice of algorithm. After
training, each algorithm was tested using the 21% of data separated before pre-processing.
Testing using new, unseen data provided an assessment of true prediction performance.
Algorithm performance was evaluated based on test accuracy, training time, and prediction
speed. Validation accuracy was compared to test accuracy to identify cases of over or
under-fitting. Algorithms with a test accuracy above 90% are reported in the results section.
Analysis of the best-performing algorithms included an assessment of the true positive
rate (TPR) and false negative classification rate (FNR), and an exploration of parameter
optimization to increase accuracy and decrease FNR.

27
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

28
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 CHAPTER 4:
Results

4.1 Classifiers
Six classifiers resulted in an overall test accuracy above 90%. All six classifiers resulted in
accuracy rates similar to Fleming’s result of 94.2% using the AlexNet CNN [30]. However,
training time was greatly improved over the hour required to train CNN. The maximum
training time required was for the neural network, which required just under four minutes
to train, a 93% decrease in training time. Table 4.1 displays the performance measures for
each of the six best performing classifiers.

Table 4.1. Best performing classifiers

Classifier Training Validation Accuracy (%) Test Accuracy (%) Training Time (s) Prediction Speed (observations/s)
Quadratic SVM 95.2 93.8 33.37 46,000
Medium Gaussian SVM 95.3 94.5 23.34 25,000
Weighted KNN 96.6 95.4 26.92 4,000
Ensemble Bagged Trees 96.4 94.7 53.82 36,000
Ensemble Subspace KNN 96.7 95.4 59.81 2,000
Wide Neural Network 96.2 94.5 227.84 340,000

These results show minimal difference between validation and test accuracy, indicating
that none of the six selected algorithms are severely over or under-fitted. The following
sections detail the testing results of each algorithm. Each classifier is presented with a
confusion matrix that shows the true positive rate (TPR) and the false negative rate (FNR)
of predictions using the test dataset. The total TPR is the rate of correct classification;
the FNR is the percentage of background noise identified as drone noise, combined with
the percentage of drone noise identified as background noise. The FNR for drone noise
identified as background noise is of particular operational concern because it represents the
most dangerous misclassification outcome on the battlefield.

29
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 4.1.1 Quadratic Support Vector Machine
The quadratic SVM in MATLAB Classification Learn App is an SVM that uses a quadratic
polynomial as a kernel function to reshape the data and find an optimal hyperplane sep-
arating two binary classes [58]. No adjustments were made to the standard MATLAB
parameters. The quadratic SVM resulted in an overall test accuracy of 93.8%, a training
time of 33.37 seconds, and a prediction speed of 46,000 observations/second. This is the
lowest overall accuracy of all six algorithms but with the second fastest prediction speed
and moderate training time. Figure 4.1 gives a more detailed breakdown of this algorithm’s
overall accuracy.

Figure 4.1. Test confusion matrix for the SVM algorithm with a quadratic
kernel function.

The quadratic SVM correctly identified 97.6% of drone sound as drones, and 76.1% of
background noise as background. The disparity between TPR of drones and background is
common to all six algorithms and is likely due to the imbalance of drone and background
samples in the available data set. The data imbalance is a limitation of this research and can
be remedied by additional data collection. This algorithm resulted in a drone FNR of 2.4%

30
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 (drones classified as background) and a background FNR of 23.9% (background classified
as drone), the second highest overall FNR for all six algorithms.

SVMs are excellent for labeled, binary data sets, but cannot be used for multiclass clas-
sification. So while the quadratic SVM resulted in excellent overall performance for the
problem of distinguishing between drones and background noise, it should not necessarily
be employed on the MEMS sensor when the intent is to eventually distinguish between
different types of drones. It could be employed as a proof of concept but provides limited
utility for future work.

4.1.2 Medium Gaussian Support Vector Machine

The medium Gaussian SVM in MATLAB Classification Learn App is an SVM that uses a
Gaussian function as a kernel function, with a width parameter 𝜎 = 3.6, to reshape the data
and find an optimal hyperplane separating two binary classes [58]. No adjustments were
made to the standard MATLAB parameters. The medium Gaussian SVM resulted in an
overall test accuracy of 94.5%, a training time of 23.34 seconds, and a prediction speed of
25,000 observations/second. Figure 4.2 gives a more detailed breakdown of this algorithm’s
overall accuracy.

Figure 4.2. Test confusion matrix for the SVM algorithm with a Gaussian
kernel function.
31
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 The Gaussian SVM correctly identified 99.0% of drones and 73.8% of background noise,
again showing the disparity between TPR in drones and background noise as a result of
under-representation of background noise in the data set. The Gaussian SVM resulted in
the lowest rate of drones classified as background, but the highest rate of background
noise classified as drones. This demonstrates a common trade off between false negative
rates: when one decreases the other increases. This relationship is explored explicitly in the
optimization section.

Similarly to the quadratic SVM, the Gaussian SVM shows excellent performance on the
binary classification problem of distinguishing between drone and background noise. How-
ever, the Gaussian SVM suffers from the same limitation in that it cannot be extended to a
multi-class classification task [59]. As such, it should not be employed on the MEMS sensor
when the intent is to eventually distinguish between different types of drones.

4.1.3 Weighted k-Nearest Neighbors

The weighted k-nearest neighbors algorithm in the MATLAB Classification Learner App is
a k-nearest neighbors classification algorithm that uses 10 nearest neighbors, the Euclidean
distance metric, and the squared inverse distance weight [60]. No adjustments were made to
the standard MATLAB parameters. The weighted k-nearest neighbors resulted in an overall
test accuracy of 95.4%, a training time of 26.92 seconds, and a prediction speed of 4,000
observations/second. Both k-nearest neighbors resulted in the highest overall accuracy. The
weighted k-nearest neighbor algorithm has the fastest training time, but the lowest prediction
speed. Figure 4.3 gives a more detailed breakdown of this algorithm’s overall accuracy.

32
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Figure 4.3. Test confusion matrix for the weighted k-Nearest Neighbors algo-
rithm.

The weighted k-nearest neighbors algorithm correctly identified 98.0% of drones and 83.5%
of background noise, with a relatively low total FNR in both categories. While this algorithm
is one of the two most accurate algorithms, its prediction speed is slow because of the k-
nearest neighbors lazy learning method. This method stores training data within the model
to access during classification, resulting in short training time and long prediction time [59].
This method makes the k-nearest neighbors algorithm unsuitable for further exploration on
the MEMS sensor project. The MEMS sensor system does not have the capacity to store
training data for retrieval during classification and the operational intent of the project
requires speedy prediction.

4.1.4 Bagged Tree Ensemble

An ensemble algorithm trains many individual learner algorithms and then combines them
into an overall predictive model [61]. The bagged tree ensemble trains individual decision
trees and then combines them using the bootstrap aggregation technique to prevent over-
fitting [62]. A decision tree is a simple algorithm that follows a series of logical arguments
with binary outputs to arrive at a classification prediction, much like a common decision
flow chart [63]. The standard bagged tree ensemble classifier in the MATLAB Classification

33
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Learner App combines 30 decision tree learners. The bagged tree ensemble resulted in an
overall test accuracy of 94.7%, a training time of 53.82 seconds, and a prediction speed of
36,000 observations/second. Figure 4.4 gives a more detailed breakdown of this algorithm’s
overall accuracy.

Figure 4.4. Test confusion matrix for the ensemble algorithm with a bag
method and decision tree learner type.

The bagged tree ensemble algorithm correctly identified 97.6% of drones and 81.3% of
background noise. The disparity between true positive rates of drones and background is
again, likely caused by the lack of background samples in the data. The bagged tree ensemble
is relatively quick, requires little training time, and can be easily interpreted to an interested
party with little experience in machine learning.

4.1.5 Subspace k-Nearest Neighbors Ensemble

The subspace KNN ensemble is similar to the bagged tree ensemble but uses the nearest
neighbors algorithm as individual learners and a subspace ensemble method [61]. The
Learner App uses 30 KNNs and a subspace of dimension 7; no adjustments were made to
the standard MATLAB parameters. The subspace k-nearest neighbors ensemble resulted in

34
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 an overall test accuracy of 95.4%, a training time of 59.81 seconds, and a prediction speed of
2,000 observations/second. Figure 4.5 gives a more detailed breakdown of this algorithm’s
overall accuracy.

Figure 4.5. Test confusion matrix for the ensemble algorithm with a subspace
method and nearest neighbors learner type.

The ensemble subspace k-nearest neighbors correctly identified 96.3% of drones and 82.1%
of background noise with the second lowest false negative rate. Similarly to the weighted k-
nearest neighbors algorithm, the ensemble subspace k-nearest neighbors results in high test
accuracy and low training time, but slow prediction speed. This is again a result of the lazy
learning method. The issue is compounded because the ensemble method executes multiple
k-nearest neighbor algorithms and averages the results. The slow prediction time makes
the ensemble subspace k-nearest neighbors algorithm inappropriate for further exploration
on the MEMS sensor project for the same reasons the weighted k-nearest neighbors was
unsuitable: the MEMS sensor system does not have the capacity to store training data
for retrieval during classification and the operational intent of the project requires speedy
prediction.

35
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 4.1.6 Wide Neural Network
The wide neural network classification algorithm in MATLAB is a neural network with
one fully connected layer consisting of 100 neurons which compute the ReLU activation
function. The softmax function computes the final output [64]. The neural network algorithm
resulted in an overall test accuracy of 94.5%, a training time of 227.84 seconds, and a
prediction speed of 340,000 observations/second. The overall accuracy is comparable to the
other five algorithms, but the wide neural network required the longest training time and
resulted in the fastest overall prediction speed. Figure 4.6 gives a more detailed breakdown
of this algorithm’s overall accuracy.

Figure 4.6. Test confusion matrix for a neural network with one fully connected
layer consisting of 100 neurons using the ReLU activation function.

The wide neural network correctly identified 96.8% of drones in the test data and 83.9%
of background noise. This algorithm resulted in the highest rate of drones classified as
background, the most dangerous battlefield result. Neural networks provide a wide variety
of optimizable parameters. The neural network tested here has only one hidden layer with
100 nodes. The number of nodes, hidden layers, the activation function(s), addition of a bias
unit, and the final output function could all be changed in order to optimize this network.

36
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 These options highlight a key drawback of neural networks: uncertainty of interpretation.
Where the decision tree algorithm is simple to interpret, neural networks are ambiguous.
It can be difficult to determine exactly what parameter, or even what data features, are
contributing to classification.

4.2 Optimization
As briefly mentioned in Section 4.1, FNR for drone noise identified as background noise
is of particular operational concern because it represents the most dangerous misclassifi-
cation outcome. A misidentified drone could lead to infiltration and real-world casualties
as demonstrated in Jordan in early 2024 when an enemy drone was mistaken for a friendly
U.S. drone, killing three U.S. service members and injuring many more [4]. Five of the
six algorithms reported here can be optimized to reduce the FNR of drones misclassified
as background via manipulation of the standard MATLAB cost matrix used in training.
However, a decrease in FNR of drones as background is always accompanied by an increase
in false positive rates (FPR) of background misclassified as drones. While a high FPR is
not as dangerous as a high FNR, background noise identified as drone noise could result in
many wasted man-hours as troops respond to false alarms. Ultimately, false negatives and
false positives must be balanced according to operational requirements.
" #
0 1
Table 4.2 shows the results of optimization using a cost matrix of rather than the
5 0
" #
0 1
MATLAB default . The change in the cost matrix applies a larger penalty, chosen
1 0
arbitrarily for experimentation, to incorrect identification of drones as background, keeping
all other costs the same as the original training method.

Table 4.2. Optimization using cost matrix manipulation

Classifier Initial FNR (%) Optimized FNR (%) Initial FPR (%) Optimized FPR (%)
Quadratic SVM 2.4 0.2 23.9 39.0
Medium Gaussian SVM 1.0 0.1 26.2 42.8
Weighted KNN 2.0 0.1 16.5 29.4
Ensemble Bagged Trees 2.4 0.4 18.7 30.3
Ensemble Subspace KNN 1.7 1.6 17.9 17.1

37
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 As expected, increasing the penalty for drones identified as background noise decreased the
FNR for each algorithm, while simultaneously increasing the FPR. This demonstrates that
manipulation of the cost matrix is an effective optimization method, but requires operational
input to balance the danger of false negatives and time cost of false positives.

In addition to manipulating the cost matrix used in training, each of the six algorithms has
parameters that can be investigated for better accuracy and speed. Given their suitability for
extension to multi-class classification and overall performance, exploration into parameter
optimization was focused on neural networks and ensemble tree classifiers. Tables 4.3, 4.4,
and 4.5 show the results of varying neural network architecture and activation function. Table
4.3 uses the ReLU function, the standard MATLAB neural network activation function, with
networks comprised of 1,2, or 3 hidden layers and 1,25, or 100 nodes per layer.

Table 4.3. Neural Networks using ReLu Activation

Number of Connected Layers Nodes/Layer Training Validation Accuracy (%) Test Accuracy (%) Training Time (s) Prediction Speed (observations/s)
1 10 95.7 93.5 101.77 210,000
1 25 96.4 93.9 117.24 340,000
1 100 96.2 94.5 227.84 340,000
2 10 95.9 93.9 100.19 380,000
2 100 96.4 94.5 338.9 150,000
3 10 95.8 93.1 162.5 470,000
3 100 96.3 94.4 416.71 210,000

Additional hidden layers and nodes do not show improvement in accuracy rate over the
standard MATLAB parameters. Table 4.4 uses the hyperbolic tangent activation function,
with networks comprised of 1,2, or 3 hidden layers and 1,25, or 100 nodes per layer.

Table 4.4. Neural Networks using Hyperbolic Tangent Activation

Number of Connected Layers Nodes/Layer Training Validation Accuracy (%) Test Accuracy (%) Training Time (s) Prediction Speed (observations/s)
1 10 96.2 93.8 104.09 190,000
1 25 96.1 93.4 185.11 320,000
1 100 95.8 94.1 263.79 220,000
2 10 96.1 93.5 155.6 250,000
2 100 96.4 94.1 228.86 150,000
3 10 96.1 93.6 249.69 330,000
3 100 96.5 94.2 387.92 150,000

38
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 The hyperbolic tangent activation function yields similar results to the ReLu activation
function, showing no improvement in accuracy and a decrease in prediction speed. Table
4.5 uses the sigmoid activation function with networks comprised of 1,2, or 3 hidden layers
and 1,25, or 100 nodes per layer.

Table 4.5. Neural Networks using Sigmoid Activation

Number of Connected Layers Nodes/Layer Training Validation Accuracy (%) Test Accuracy (%) Training Time (s) Prediction Speed (observations/s)
1 10 96.2 93.7 94.409 260,000
1 25 96.2 94.3 163.14 300,000
1 100 95.8 93.7 256.8 300,000
2 10 96.3 94.0 136.52 380,000
2 100 96.4 94.7 470.66 180,000
3 10 96.3 93.8 219.04 390,000
3 100 96.3 94.8 924.01 170,000

None of the tested network architectures or activation functions improved upon the accuracy
of the standard MATLAB parameters for a wide neural network. This may demonstrate that
the current accuracy rate of 94.7% for the wide neural network algorithm is the best possible
rate for the given data set using a neural network, but given the near endless possibilities
for network architecture, it may also mean that the optimal design was not tested.

The ensemble bagged tree classifier can be optimized by varying the number of decision tree
learners aggregated in each ensemble. Table 4.6 shows the results of varying this parameter
from 10 to 50 trees.

Table 4.6. Bagged Trees

Number of Trees Training Validation Accuracy (%) Test Accuracy (%) Training Time (s) Prediction Speed (observations/s)
10 96.1 93.8 18.299 100,000
20 96.5 94.8 35.976 41,000
30 96.2 94.7 48.267 33,000
40 96.4 94.8 56.363 29,000
50 96.4 94.7 68.219 28,000

The number of decision tree learners does not improve test accuracy but significantly
decreases prediction speed. In addition to the number of learners, the aggregation, or
boosting, method for an ensemble classifier can also be optimized. Table 4.7 shows the
results of several common boosting algorithms.

39
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Table 4.7. Boosted Trees
Number of Trees Boost Algorithm Training Validation Accuracy (%) Test Accuracy (%) Training Time (s) Prediction Speed (observations/s)
30 AdaBoost 93.7 91.6 41.385 27,000
30 RUSBoost 92.4 88.4 33.2 29,000
30 Subspace 82.9 83.2 34.898 2,500
30 LogitBoost 94.4 92.4 36.988 31,000
30 GentleBoost 96.4 95 44.475 49,000

The GentleBoost algorithm shows improvement in both test accuracy and prediction speed
from the standard bootstrap aggregation method. GentleBoost is an adaptive algorithm that
combines elements of AdaBoostM1 and LogitBoost [65]. It minimizes exponential loss and
uses a regression model to combine learners [65]. The results for the ensemble GentleBoost
trees classifier indicate that it is better for the binary problem of identifying "drone" or "not
drone," however it cannot be extended to multi-class problems [65].

4.3 Summary
Of the 32 algorithms tested, 6 resulted in test accuracy above 90%. Of those six, two were
variations of the support vector machine, which cannot be extended to multi-class problems,
and two were k-nearest neighbors algorithms with poor prediction speed. The wide neural
network and ensemble bagged trees algorithms were considered for optimization, done by
experimenting with each algorithm’s parameters, which did not result in improved perfor-
mance. Cost matrix manipulation was also considered as an optimization tool, assigning
a higher penalty to false negative identification, but requires operational input to properly
weigh the dangers of false negatives against the cost of false positives.

40
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 CHAPTER 5:
Conclusion

5.1 Conclusions and Limitations

The objective of this work was to provide a proof of concept that a machine learning
algorithm could distinguish between drone sound and background noise using data collected
on the MEMS sensor project. Integration of an intelligent identification algorithm into the
sensor could provide a lightweight system to detect, locate, and identify drones which
significantly increases battlefield awareness [11]. This work focused first on implementing
a data processing method that efficiently extracts features for machine learning. A wide
variety of signal and frequency features were extracted from the data based on their success
in the robust body of work in drone classification. Mel frequency cepstral coefficients and
dominant frequency value were most strongly correlated to the target variable and highly
weighted in NCA. Computation of these features is efficient and feasible to execute on the
sensor microcontroller. Thirty two machine learning algorithms were trained using these
features, resulting in six classification algorithms with accuracy rates above 90%.

The performance of the top six algorithms is comparable to that of similar work classifying
drones using audio data. Kolumunna et al.’s [7] ‘DronePrint’ achieved an accuracy rate of
90% using MFCCs to identify the make and model of a drone with a series of neural net-
works. Bernardini et al.’s [31] SVM classifiers successfully identified drones with accuracy
rates between 96.4% and 99.2% using a combination of MFCCs, signal, and frequency
characteristics. Al-Emadi et al. [32] achieved accuracy and identification rates of 96.38%
and 92.94% respectively using spectrograms to train their convolution neural network. All
of the six best performing algorithms are able to distinguish between drones and back-
ground noise with acceptable accuracy. However operational requirements and processing
capability restrict the algorithms that should be carried forward for testing with the MEMS
sensor to the bagged tree ensemble and neural network algorithms. The two most accurate
algorithms, the weighted k-nearest neighbors and ensemble subspace k-nearest neighbors,
are also the slowest and require too much storage to operate on the MEMS sensor. Both
SVMs achieve accuracy above 93% and adequate prediction speed, but can only be used for

41
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 binary classification, so they cannot be extended to classification between different types
of drones in future work. The ensemble bagged trees classifier achieved similar accuracy
to the neural network, but the neural network was seven times faster than the next highest
prediction speed. Conversely, the neural network had the slowest overall training time by
a wide margin. Based on overall performance and potential for extension to more classes
with higher dimension data, both neural networks and the ensemble bagged trees algorithm
should be investigated further for employment on the MEMS sensor. Both algorithms were
considered for optimization by experimenting with each algorithm’s parameters, which did
not result in improved performance. Cost matrix manipulation is a viable optimization tool
to reduce FPR, the most dangerous battlefield outcome, but requires operational input to
properly weigh the dangers of false negatives against the cost of false positives.

The limitations of this work are a result of inadequate data. Machine learning requires large
data sets to develop accurate predictive algorithms. This project relied on data from 350
recordings, with limited variety and an imbalance in drone and background samples, and
so is restricted in its applicability until the recommended classifiers can be trained with
more data. The conclusions here are limited to environments similar to those represented
in the recorded samples; classifier performance will be severely degraded outside of these
conditions.

Global drone proliferation and innovative use, both military and civilian, indicate an acute
need to quickly and accurately detect, identify, and locate drones. Acoustic sensors provide
a passive capability that can augment RF, optical, thermal, and radar drone sensors, but
typically require large microphone arrays or have limited detection range. The NPS MEMS
sensor presents a miniature, accurate, and low power consumption solution to acoustic
detection, but cannot identify the source of a sound. The data processing protocols and
machine learning algorithms recommended here could augment the MEMS sensor with a
real-time drone identification capability. This fully realized capability may well play a vital
role in battlefield awareness and force protection for the Department of Defense.

5.2 Recommendations for Future Work

This work serves as a proof of concept, showing that classification between drone and
background noise is possible using the data collected on the MEMS sensor project. Future

42
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 research should focus on integration with the MEMS sensor, additional data collection, new
feature extraction methods, and extending the project from distinguishing between drone
and background noise, to distinguishing between different types of drones.

5.2.1 Integrate with the MEMS Sensor

This project stopped short of running a classification algorithm on the MEMS sensor. Future
work should adapt the data processing procedures and classification algorithm for use on
the TEENSY microcontroller. Data processing for the sensor requires similar computations
to those required to extract MFCCs, so integration with the sensor should only be limited
by the processing required by the selected algorithm.

5.2.2 Additional Data Collection

This research used a total of 350 samples of drone and background noise. Dividing each
sample into one second windows created 22,488 samples, which is enough data to train
without under-fitting, but is an artificial representation of the sample size. More data from
different drone and background sources in different environments will help improve algo-
rithm accuracy across a variety of conditions. Additionally, background noise comprised
only 18% of the total data used for this project. This difference between the amount of
background and drone noise exposed the algorithms to significantly less background than
drone samples during training. Future data collection should attempt to balance drone and
background sample collection.

5.2.3 Feature Extraction

This work focused mainly on signal and frequency statistics and mel-frequency cepstral
coefficients during feature extraction. Future work should explore other signal characteristics
that show potential for drone classification. Gammatone frequency cepstral coefficients
(GFCCs) have similar potential to MFCCs in audio classification, especially in underwater
applications, and Salman et al. [42] showed them to be slightly more efficient predictive
features than MFCCs [66], [67]. These characteristics could be compared to MFCCs in
future work as the MEMS sensor underwater capabilities expand. Gammatone frequency
cepstral coefficients may prove to be more efficient to calculate and correlate more strongly
to the target variable.

43
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 5.2.4 Distinguish Between Drones
Future work should expand this project from classification of drone and background noise,
to classification of different types of drones. Adding intelligent identification of a type of
drone would significantly expand the capability and battlefield usefulness of the MEMS
sensor. Loss of life, such as what happened in Jordan in January 2024, which resulted from
the misidentification of an enemy drone, could be prevented by this fully realized capability
[4].

44
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 APPENDIX A:
Data Files

Drone Files:
GeorgiaTech Drone_tgt of opportunity_distant and quiet_Ch_0.wav
GeorgiaTech Drone_tgt of opportunity_distant and quiet_Ch_1.wav
GeorgiaTech_inbound and outbound high_Ch_0.wav
GeorgiaTech_inbound and outbound high_Ch_1.wav
GeorgiaTech_outbound high_Ch_0.wav
GeorgiaTech_outbound high_Ch_1.wav
LLNL Drone_dynamic_movements_and_flybys_20ft_altitude_Ch_0.wav
LLNL Drone_dynamic_movements_and_flybys_20ft_altitude_Ch_1.wav
LLNL Drone_fast_movements_and_flybys_20ft_altitude_Ch_0.wav
LLNL Drone_fast_movements_and_flybys_20ft_altitude_Ch_1.wav
Navigator Drone_hovering 6m away_15ft_high_Ch_0.wav
Navigator Drone_hovering 6m away_15ft_high_Ch_1.wav
Navigator Drone_hovering 6m away_15ft_high-2_Ch_0.wav
Navigator Drone_hovering 6m away_15ft_high-2_Ch_1.wav
Parrot_hovering near_Ch_0.wav
Parrot_hovering near_Ch_1.wav
Parrot_passing by back and forth_Ch_0.wav
Parrot_passing by back and forth_Ch_1.wav
SkyRaider-2-4m away_Ch_0.wav
SkyRaider-2-4m away_Ch_1.wav
SkyRaider-4m away_Ch_0.wav
SkyRaider-4m away_Ch_1.wav
SkyRaider-7m away_Ch_0.wav
SkyRaider-7m away_Ch_1.wav
SkyRaider and helicopter flying by_Ch_0.wav
SkyRaider and helicopter flying by_Ch_1.wav
SkyRaider-flaring around the mics_Ch_0.wav
SkyRaider-flaring around the mics_Ch_1.wav

45
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 SkyRaider-free flight around_Ch_0.wav
SkyRaider-free flight around_Ch_1.wav
SkyRaider-hovering 15m away and 20ft high_Ch_0.wav
SkyRaider-hovering 15m away and 20ft high_Ch_1.wav
SkyRaider-hovering 35m away and 20ft high_Ch_0.wav
SkyRaider-hovering 35m away and 20ft high_Ch_1.wav
SkyRaider-hovering on top 20ft high_Ch_0.wav
SkyRaider-hovering on top 20ft high_Ch_1.wav
SkyRaider-hovering on top_Ch_0.wav
SkyRaider-hovering on top_Ch_1.wav
SkyRaider-passing by 38ft with loud ocean background_Ch_0.wav
SkyRaider-passing by 38ft with loud ocean background_Ch_1.wav
SkyRaider-passing by 38ft-and helocopter passing by 300ft_Ch_0.wav
SkyRaider-passing by 38ft-and helocopter passing by 300ft_Ch_1.wav
Mavic Pro no bip_Ch_0.wav
Mavic Pro no bip_Ch_1.wav
Mavic Pro no bip_Ch_2.wav
Phantom 3 Pro_Ch_0.wav
Phantom 3 Pro_Ch_1.wav
Phantom 3 Pro_Ch_2.wav
Phantom 3 Pro 1kg_Ch_0.wav
Phantom 3 Pro 1kg_Ch_1.wav
Phantom 3 Pro 1kg_Ch_2.wav
Phantom 3 Pro 1kg_No guards_Ch_0.wav
Phantom 3 Pro 1kg_No guards_Ch_1.wav
Phantom 3 Pro 1kg_No guards_Ch_2.wav
Phantom 4 Pro plus_Ch_0.wav
Phantom 4 Pro plus_Ch_1.wav
Phantom 4 Pro plus_Ch_2.wav
Phantom 4 Pro plus 940g_Ch_0.wav
Phantom 4 Pro plus 940g_Ch_1.wav
Phantom 4 Pro plus 940g_Ch_2.wav
Typhoon_Ch_0.wav

46
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Typhoon_Ch_1.wav
Typhoon_Ch_2.wav
Typhoon 940g_Ch_0.wav
Typhoon 940g_Ch_1.wav
Typhoon 940g_Ch_2.wav
DJI_Inspire_Ch_0.wav
DJI_Inspire_Ch_1.wav
DJI_Inspire_Ch_2.wav
DJI_Inspire_pole2_Ch_0.wav
DJI_Inspire_pole2_Ch_1.wav
DJI_Inspire_pole2_Ch_2.wav
DJI_Inspire_pole3_Ch_0.wav
DJI_Inspire_pole3_Ch_1.wav
DJI_Inspire_pole3_Ch_2.wav
Matrice 600_Ch_0.wav
Matrice 600_Ch_1.wav
Matrice 600_Ch_2.wav
Matrice 600_pole2_Ch_0.wav
Matrice 600_pole2_Ch_1.wav
Matrice 600_pole2_Ch_2.wav
Matrice 600_pole3_Ch_0.wav
Matrice 600_pole3_Ch_1.wav
Matrice 600_pole3_Ch_2.wav
Roberts 02_26_2pole_Ch_0.wav
Roberts 02_26_2pole_Ch_1.wav
Roberts 02_26_2pole_Ch_2.wav
Roberts 02_26_3pole_Ch_0.wav
Roberts 02_26_3pole_Ch_1.wav
Roberts 02_26_3pole_Ch_2.wav
Roberts 02_26_Ch_0.wav
Roberts 02_26_Ch_1.wav
Roberts 02_26_Ch_2.wav
DJI_Shield_AI_NOA_Ch_0.wav

47
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 DJI_Shield_AI_NOA_Ch_1.wav
DJI_Shield_AI_NOA_Ch_2.wav
DJI_Shield_AI_NOA_pole2_Ch_0.wav
DJI_Shield_AI_NOA_pole2_Ch_1.wav
DJI_Shield_AI_NOA_pole2_Ch_2.wav
DJI_Shield_AI_NOA_pole3_Ch_0.wav
DJI_Shield_AI_NOA_pole3_Ch_1.wav
DJI_Shield_AI_NOA_pole3_Ch_2.wav

DJI Inspiere PRO:

Ascention 1_Ch_0.wav
Ascention 1_Ch_1.wav
Ascention 1_Ch_2.wav
Ascention 2_Ch_0.wav
Ascention 2_Ch_1.wav
Ascention 2_Ch_2.wav
Centerl pole_Ch_0.wav
Centerl pole_Ch_1.wav
CeCone 1_Ch_0nterl pole_Ch_2.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

AC-DJI Mavic:
Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav

48
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone 1_Ch_0.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by 1_Ch_0.wav
Fly by 1_Ch_1.wav
Fly by 1_Ch_2.wav
Fly by 2_Ch_0.wav
Fly by 2_Ch_1.wav
Fly by 2_Ch_2.wav

IC-DJI Phanton 3 PRO:

Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav
Background_Ch_0.wav
Background_Ch_1.wav
Background_Ch_2.wav
Central pole_Ch_0.wav
Central pole_Ch_1.wav
Central pole_Ch_2.wav
Cone 1_Ch_0.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav

49
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

IC-DJI Phanton 4 PRO:

Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav
Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone 1_Ch_0.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

IC-Yoneek Typhoon:
Ascention 1_Ch_0.wav
Ascention 1_Ch_1.wav
Ascention 1_Ch_2.wav
Ascention 2_Ch_0.wav
Ascention 2_Ch_1.wav
Ascention 2_Ch_2.wav
Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone 1_Ch_0.wav

50
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav
flight no load 4_2_2-1_Ch_1.wav
flight no load 4_2_22 lower_Ch_1.wav
flight no load 4_2_2-3_Ch_1.wav
flight no load 4_2_2-4_Ch_1.wav
flight w load 4_2_2_5_Ch_1.wav
flight w load 4_2_2_6_Ch_1.wav
flight w load 4_2_2_7_Ch_1.wav
flight w load 4_2_2_8_Ch_1.wav
flight w load 4_2_2_9_Ch_1.wav
flight w load 4_2_2_10_Ch_1.wav
flight w load 4_2_2_11_Ch_1.wav
flight w load 4_2_2_12_Ch_1.wav
flight w load 4_3_1_1_Ch_1.wav
flight w load 4_3_1_2_Ch_1.wav
flight w load 4_3_1_3_Ch_1.wav
flight w load 4_3_1_4_Ch_1.wav
flight w load 4_3_1_5_Ch_1.wav
flight w load 4_3_1_6_Ch_1.wav
flight w load 4_3_1_7_Ch_1.wav
flight w load 4_3_1_8_Ch_1.wav
flight w load 4_3_1_9_Ch_1.wav
flight w load 4_3_1_10_Ch_1.wav
flight w load 4_3_1_11_Ch_1.wav
flight w load 4_3_1_12_Ch_1.wav
flight w load 4_3_1_13_Ch_1.wav

51
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 flight w load 4_3_1_14_Ch_1.wav
flight w load 4_3_1_15_Ch_1.wav
flight w load 4_3_1_16_Ch_1.wav
flight w load 4_3_1_17_Ch_1.wav
flight w load 4_3_1_18_Ch_1.wav
flight w load 4_3_1_19_Ch_1.wav
flight w load 4_3_1_20_Ch_1.wav
flight w load 4_3_1_21_Ch_1.wav
flight w load 4_3_1_22_Ch_1.wav
flight w load 4_3_1_23_Ch_1.wav
flight w load 4_3_1_24_Ch_1.wav
flight w load 4_3_1_25_Ch_1.wav
flight w load 4_3_1_26_Ch_1.wav
flight w load 4_3_1_27_Ch_1.wav

LF-DJI Mavic:
Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav
Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone1_Ch_0.wav
Cone1_Ch_1.wav
Cone1_Ch_2.wav
Cone2_Ch_0.wav
Cone2_Ch_1.wav
Cone2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

52
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 LF-DJI Phantom 3:
Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav
Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone 1_Ch_0.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

LF-DJI Phantom 3 with 1 kg load:

Brass load_Ch_0.wav
Brass load_Ch_1.wav
Brass load_Ch_2.wav

LF-DJI Phantom 4 Pro:

Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav
Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone 1_Ch_0.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav

53
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

LF- DJI Phantom 4 Pro with 1 kg load:

1kg_Ch_0.wav
1kg_Ch_1.wav
1kg_Ch_2.wav

LF - Yoneek Typhoon:
Ascention_Ch_0.wav
Ascention_Ch_1.wav
Ascention_Ch_2.wav
Center Pole_Ch_0.wav
Center Pole_Ch_1.wav
Center Pole_Ch_2.wav
Cone 1_Ch_0.wav
Cone 1_Ch_1.wav
Cone 1_Ch_2.wav
Cone 2_Ch_0.wav
Cone 2_Ch_1.wav
Cone 2_Ch_2.wav
Fly by_Ch_0.wav
Fly by_Ch_1.wav
Fly by_Ch_2.wav

LF - Yoneek Typhoon with 1 kg load:

1kg_Ch_0.wav

54
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 1kg_Ch_1.wav
1kg_Ch_2.wav

NG - Matrice:
matrice_20221012T133218_chan01.wav
matrice_20221012T133218_chan02.wav
matrice_20221012T133447_chan01.wav
matrice_20221012T133447_chan02.wav
matrice_20221012T133619_chan01.wav
matrice_20221012T133619_chan02.wav
matrice_20221012T133644_chan01.wav
matrice_20221012T133644_chan02.wav

NG - Phantom 4:
phantom_20221012T132547_chan01.wav
phantom_20221012T132547_chan02.wav
phantom_20221012T132713_chan01.wav
phantom_20221012T132713_chan02.wav
phantom_20221012T132856_chan01.wav
phantom_20221012T132856_chan02.wav

Background Files:
3 helos passing by_Ch_0.wav
3 helos passing by_Ch_1.wav
Assault_ship 100m away_Ch_0.wav
Assault_ship 100m away_Ch_1.wav
Assault_ship 100m away-2_Ch_0.wav
Assault_ship 100m away-2_Ch_1.wav
Bell_214_helo_passing by_Ch_0.wav
Bell_214_helo_passing by_Ch_1.wav
Cobra_helo_passing by_Ch_0.wav

55
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Cobra_helo_passing by_Ch_1.wav
Generator_Ch_0.wav
Generator_Ch_1.wav
Ocean background - no human noise_Ch_0.wav
Ocean background - no human noise_Ch_1.wav
Ocean background - no human noise-2_Ch_0.wav
Ocean background - no human noise-2_Ch_1.wav
Ocean background - loud human noise_Ch_0.wav
Ocean background - loud human noise_Ch_1.wav
Backg long_Ch_0.wav
Backg long_Ch_1.wav
Backg long_Ch_2.wav
Background 2_Ch_0.wav
Background 2_Ch_1.wav
Background 2_Ch_2.wav
Background A2_rede_Ch_0.wav
Background A2_rede_Ch_1.wav
Background A2_rede_Ch_2.wav
Background_Ch_0.wav
Background_Ch_1.wav
Background_Ch_2.wav
Background Roberts AI0_02_26.wav
Background Roberts AI1_02_26.wav
Background Roberts AI2_02_26.wav
Background_Ch_0.wav
Background_Ch_1.wav
Background_Ch_2.wav
Bckg disturbed_Ch_0.wav
Bckg disturbed_Ch_1.wav
Bckg disturbed_Ch_2.wav

56
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 APPENDIX B:
MATLAB Code

Frequency Response
clc;
clear all;
%close all;
[y,fs] = audioread ('b001.wav ');
ydft = fft(y);
ydft = ydft (1: length (y) /2+1) ;
freq = 0:fs/ length (y):fs /2;
% plot magnitude
figure (4)
plot(freq ,abs(ydft)/500) ;
axis ([10 1000 0 1]);
xlabel ('Hz ');
% csvwrite ('File Name ', [freq ' abs(ydft)])
set(gca , 'Fontsize ', 14,'Box ', 'on ', 'LineWidth ', 1.5); grid
on;
set(gca , 'GridAlpha ', 0.2, 'GridColor ', 'black ')
xlabel ('Frequency (Hz)'); % x,y labels
ylabel (['Normalized SPL ']);

57
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Training and Test Data Import and Feature Extraction [39]
mkdir('')
addpath ( genpath (''))
training_fds = fileDatastore ( fullfile (pwd , 'droneData -blind '
, 'training '),'ReadFcn ', @ importAudioFile , '
FileExtensions ', '.wav ','IncludeSubfolders ',true);

% create a table with field names and labels

data_dir = fullfile (pwd , 'droneData -blind ', 'training ');
folder_list = dir ([ data_dir filesep 'training *']);
reference_table = table ();
for ifolder = 1: length ( folder_list )
disp ([ 'Processing files from folder : ' folder_list (
ifolder ).name ])
current_folder = [ data_dir filesep folder_list ( ifolder ).
name ];
% Import labels 1 = Drone , -1 = Background
reference_table = [ reference_table ; importReferencefile
([ current_folder filesep 'REFERENCE .csv '])];
end
%% Extract Features from training data
%this calls the MathWorks extractFeatures helper function
which extracts signal and frequency statistics and MFCCs

runExtraction = true; % control whether to run feature

extraction
if runExtraction | ~ exist('FeatureTable .mat ')
% Window length for feature extraction in seconds
win_len = 1;
% Specify the overlap between adjacent windows for
feature extraction in percentage
win_overlap = 20;
% Initialize feature table to accumulate observations

58
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 feature_table = table ();
% Use Parallel Computing Toobox
n_parts = numpartitions ( training_fds , gcp);
parfor ipart = 1: n_parts
subds = partition ( training_fds , n_parts , ipart);
[ feature_win , sampleN ] = extractFeatures (subds ,
win_len , win_overlap , reference_table );
feature_table = [ feature_table ; feature_win ];
disp ([ 'Part ' num2str (ipart) ' done.'])
end
save('FeatureTable ', 'feature_table ');

else %load the precomputed features

load('FeatureTable .mat ');
end

% create an excel file with features and display class totals

writetable ( feature_table ,'FeatureTable .xlsx ');
grpstats_all = grpstats ( feature_table , 'class ', 'mean ');
disp( grpstats_all (:,'GroupCount '))

%% Create data structure and import test data

test_fds = fileDatastore ( fullfile (pwd , 'droneData - blind ', '
test '),'ReadFcn ', @ importAudioFile , 'FileExtensions ', '.
wav ','IncludeSubfolders ',true);
test_data_dir = fullfile (pwd , 'droneData -blind ', 'test ');
test_folder_list = dir ([ test_data_dir filesep 'test*']);

test_reference_table = table ();

for ifolder = 1: length ( test_folder_list )

disp ([ 'Processing files from folder : ' test_folder_list (
ifolder ).name ])

59
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 current_folder = [ test_data_dir filesep test_folder_list
( ifolder ).name ];

% Import labels 1 = Drone , -1= Background

test_reference_table = [ test_reference_table ;
importReferencefile ([ current_folder filesep '
REFERENCE .csv ']) ];
end

%% Extract features from test data

runExtraction = true;

if runExtraction | ~exist('TestFeatureTable .mat ')%#ok

win_len = 1;
win_overlap = 20;
test_feature_table = table ();
n_parts = numpartitions (test_fds , gcp);
parfor ipart = 1: n_parts
subds = partition (test_fds , n_parts , ipart);
[ feature_win , sampleN ] = extractFeatures (subds ,
win_len , win_overlap , test_reference_table );
test_feature_table = [ test_feature_table ;
feature_win ];
disp ([ 'Part ' num2str (ipart) ' done.'])
end
save('TestFeatureTable ', 'test_feature_table ');

else %load the precomputed features

load('TestFeatureTable .mat ');
end

writetable ( test_feature_table ,'TestFeatureTable .xlsx ');

60
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 test_grpstats_all = grpstats ( test_feature_table , 'class ', '
mean ');
disp( test_grpstats_all (:,'GroupCount '))

61
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Data import helper function: importfile.m
function data = importfile ( fileToRead1 )
data = importdata ( fileToRead1 );
vars = fieldnames (data);
for i = 1: length (vars)
assignin ('base ', vars{i}, data .( vars{i}));
end

62
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Data import helper function: importAudioFile.m
function newData1 = importAudioFile ( fileToRead1 )
newData1 = importdata ( fileToRead1 );
[~,fname ,~] = fileparts ( fileToRead1 );
newData1 . filename = fname;

63
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Data import helper function: importReferenceFile.m [39]
function REFERENCE = importReferencefile (filename , startRow ,
endRow )
delimiter = ',';
if nargin <=2
startRow = 1;
endRow = inf;
end
formatSpec = '%s%f%[^\n\r]';
fileID = fopen (filename ,'r');
dataArray = textscan (fileID , formatSpec , endRow (1) -startRow
(1) +1, 'Delimiter ', delimiter , 'HeaderLines ', startRow (1)
-1, 'ReturnOnError ', false);
for block =2: length ( startRow )
frewind ( fileID );
dataArrayBlock = textscan (fileID , formatSpec , endRow (
block)-startRow ( block)+1, 'Delimiter ', delimiter , '
HeaderLines ', startRow (block) -1, 'ReturnOnError ',
false);
for col =1: length ( dataArray )
dataArray {col} = [ dataArray {col }; dataArrayBlock {col
}];
end
end
fclose ( fileID );
REFERENCE = table ( dataArray {1: end -1}, 'VariableNames ', {'
record_name ','record_label '});

64
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Feature extraction helper function: extractFeatures.m [39]
function [ feature_table_all ,N] = extractFeatures (fds ,
window_length , window_overlap , reference_table )
warning off
overlap_length = window_length * window_overlap / 100;
step_length = window_length - overlap_length ;
feature_table_all = table ();
labelMap = containers .Map('KeyType ','int32 ','ValueType ','any
');
keySet = {-1, 1};
valueSet = {'Background ','Drone '};
labelMap = containers .Map(keySet , valueSet );
while hasdata (fds)
PCG = read(fds);

signal = PCG.data;
fs = PCG.fs;
current_class = reference_table ( strcmp ( reference_table .
record_name , PCG. filename ), :). record_label ;

N = length ( signal ); % to keep track of # samples

in files we process
number_of_windows = floor( (N - overlap_length *fs) / (fs
* step_length ));

feature_table = table ();

for iwin = 1: number_of_windows
current_start_sample = (iwin - 1) * fs * step_length
+ 1;
current_end_sample = current_start_sample +
window_length * fs - 1;
current_signal = signal ( current_start_sample :
current_end_sample );

65
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % Calculate mean value feature
feature_table . meanValue (iwin , 1) = mean(
current_signal );

% Calculate median value feature

feature_table . medianValue (iwin , 1) = median (
current_signal );

% Calculate standard deviation feature

feature_table . standardDeviation (iwin , 1) = std(
current_signal );

% Calculate mean absolute deviation feature

feature_table . meanAbsoluteDeviation (iwin , 1) = mad(
current_signal );

% Calculate signal 25th percentile feature

feature_table . quantile25 (iwin , 1) = quantile (
current_signal , 0.25);

% Calculate signal 75th percentile feature

feature_table . quantile75 (iwin , 1) = quantile (
current_signal , 0.75);

% Calculate signal inter quartile range feature

feature_table . signalIQR (iwin , 1) = iqr(
current_signal );

% Calculate skewness of the signal values

feature_table . sampleSkewness (iwin , 1) = skewness (
current_signal );

66
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % Calculate kurtosis of the signal values
feature_table . sampleKurtosis (iwin , 1) = kurtosis (
current_signal );

% Calculate Shannon 's entropy value of the signal

feature_table . signalEntropy (iwin , 1) =
signal_entropy ( current_signal ');

% Calculate spectral entropy of the signal

feature_table . spectralEntropy (iwin , 1) =
spectral_entropy ( current_signal , fs , 256);

% Extract features from the power spectrum

[maxfreq , maxval , maxratio ] =
dominant_frequency_features ( current_signal , fs ,
256 , 0);
feature_table . dominantFrequencyValue (iwin , 1) =
maxfreq ;
feature_table . dominantFrequencyMagnitude (iwin , 1) =
maxval ;
feature_table . dominantFrequencyRatio (iwin , 1) =
maxratio ;

% Extract Mel - frequency cepstral coefficients

Tw = window_length *1000; % analysis frame duration (
ms)
Ts = 10; % analysis frame shift (ms)
alpha = 0.97; % preemphasis coefficient
M = 20; % number of filterbank
channels
C = 12; % number of cepstral
coefficients

67
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 L = 22; % cepstral sine lifter
parameter
LF = 5; % lower frequency limit (Hz)
HF = 500; % upper frequency limit (Hz)

[MFCCs , ~, ~] = mfcc( current_signal , fs , Tw , Ts ,

alpha , @hamming , [LF HF], M, C+1, L);
feature_table . MFCC1(iwin , 1) = MFCCs (1);
feature_table . MFCC2(iwin , 1) = MFCCs (2);
feature_table . MFCC3(iwin , 1) = MFCCs (3);
feature_table . MFCC4(iwin , 1) = MFCCs (4);
feature_table . MFCC5(iwin , 1) = MFCCs (5);
feature_table . MFCC6(iwin , 1) = MFCCs (6);
feature_table . MFCC7(iwin , 1) = MFCCs (7);
feature_table . MFCC8(iwin , 1) = MFCCs (8);
feature_table . MFCC9(iwin , 1) = MFCCs (9);
feature_table . MFCC10 (iwin , 1) = MFCCs (10);
feature_table . MFCC11 (iwin , 1) = MFCCs (11);
feature_table . MFCC12 (iwin , 1) = MFCCs (12);
feature_table . MFCC13 (iwin , 1) = MFCCs (13);

% Assign class label to the observation

if iwin == 1
feature_table .class = { labelMap ( current_class )};
else
feature_table .class{iwin , :} = labelMap (
current_class );
end

end

feature_table_all = [ feature_table_all ; feature_table ];

end

68
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Feature extraction helper function: mfcc.m [39]
function [ CC , FBE , frames ] = mfcc( speech , fs , Tw , Ts ,
alpha , window , R, M, N, L )
%% PRELIMINARIES
if( max(abs( speech )) <=1 ), speech = speech * 2^15; end;
Nw = round( 1E -3* Tw*fs ); % frame duration ( samples )
Ns = round( 1E -3* Ts*fs ); % frame shift ( samples )
nfft = 2^ nextpow2 ( Nw ); % length of FFT analysis
K = nfft /2+1; % length of the unique part
of the FFT
hz2mel = @( hz )( 1127* log (1+ hz /700) ); % Hertz to
mel warping function
mel2hz = @( mel )( 700* exp(mel /1127) -700 ); % mel to
Hertz warping function
dctm = @( N, M )( sqrt (2.0/M) * cos( repmat ([0:N-1]. ' ,1 ,
M).* repmat (pi *([1:M] -0.5)/M,N ,1) ) );
ceplifter = @( N, L )( 1+0.5* L*sin(pi *[0:N -1]/L) );
%% FEATURE EXTRACTION
speech = filter ( [1 -alpha], 1, speech );
frames = speech ;
MAG = abs( fft(frames ,nfft ,1) );
H = trifbank ( M, K, R, fs , hz2mel , mel2hz );
FBE = H * MAG (1:K ,:);
DCT = dctm( N, M );
CC = DCT * log( FBE );
lifter = ceplifter ( N, L )
CC = diag( lifter ) * CC;

69
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Feature extraction helper function: trifbank.m [39]
function [ H, f, c ] = trifbank ( M, K, R, fs , h2w , w2h )
f_min = 0; % filter coefficients start at this
frequency (Hz)
f_low = R(1); % lower cutoff frequency (Hz) for
the filterbank
f_high = R(2); % upper cutoff frequency (Hz) for
the filterbank
f_max = 0.5* fs; % filter coefficients end at this
frequency (Hz)
f = linspace ( f_min , f_max , K ); % frequency range (Hz),
size 1xK
fw = h2w( f );
c = w2h( h2w( f_low )+[0:M +1]*(( h2w( f_high )-h2w( f_low))/(M
+1)) );
cw = h2w( c );

H = zeros( M, K ); % zero otherwise

for m = 1:M
k = f >=c(m)&f <=c(m+1); % up -slope
H(m,k) = (f(k)-c(m))/(c(m+1) -c(m));
k = f >=c(m+1)&f <=c(m+2); % down -slope
H(m,k) = (c(m+2) -f(k))/(c(m+2) -c(m+1));

end

70
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Feature extraction helper function: spectral_entropy.m [39]
function specEntropy = spectral_entropy (data , fs , cutoff )
npoints = size(data ,1);
padfactor = 1;

%% Calculate the FFT of the signal

nfft = 4096;
datafft = fft(data , nfft);
ps = real( datafft .* conj( datafft ));

f = fs /2* linspace (0,1, nfft /2);

cutoffidx = length (find(f <= cutoff ));
f = f(1: cutoffidx );
ps_norm = ps (1: cutoffidx )/sum(ps (1: cutoffidx ));

% Estimate spectral entropy

estimate =0;
for i = 1: length ( ps_norm )
if ps_norm (i)~=0
logps=log( ps_norm (i));
else
logps =0;
end
estimate = estimate - ps_norm (i)*logps;
end

% Scale the spectral entropy estimate to a 0 - 1 range

specEntropy = estimate /log( length ( ps_norm ));
specEntropy = ( specEntropy - 0.5) ./(1.5 - specEntropy );

71
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Feature extraction helper function: signal_entropy.m [39]
function [estimate ,nbias ,sigma , descriptor ]= signal_entropy (x,
descriptor ,approach ,base)
if nargin <1
disp('Usage: [ESTIMATE ,NBIAS ,SIGMA , DESCRIPTOR ] = ENTROPY (
X)')
disp(' [ESTIMATE ,NBIAS ,SIGMA , DESCRIPTOR ] = ENTROPY (
X, DESCRIPTOR )')
disp(' [ESTIMATE ,NBIAS ,SIGMA , DESCRIPTOR ] = ENTROPY (
X, DESCRIPTOR , APPROACH )')
disp(' [ESTIMATE ,NBIAS ,SIGMA , DESCRIPTOR ] = ENTROPY (
X, DESCRIPTOR ,APPROACH ,BASE)')
disp('Where: DESCRIPTOR = [LOWERBOUND ,UPPERBOUND ,NCELL]')
return
end

[NRowX ,NColX ]= size(x);

if NRowX ~=1
error('Invalid dimension of X');
end;

if nargin >4
error('Too many arguments ');
end;

if nargin ==1
[h, descriptor ]= histogram_signalEntropy (x);
end;
% h = h/ length (x);

if nargin >=2
[h, descriptor ]= histogram_signalEntropy (x, descriptor );
end;

72
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 if nargin <3
approach ='unbiased ';
end;

if nargin <4
base=exp (1);
end;

lowerbound = descriptor (1);

upperbound = descriptor (2);
ncell= descriptor (3);

estimate =0;
sigma =0;
count =0;

for n=1: ncell

if h(n)~=0
logf=log(h(n));
else
logf =0;
end;
count = count +h(n);
estimate =estimate -h(n)*logf;
sigma = sigma +h(n)*logf ^2;
end;

% biased estimate
estimate = estimate / count ;
sigma =sqrt( (sigma /count - estimate ^2) /( count -1) );
estimate = estimate +log( count)+log (( upperbound - lowerbound )/
ncell );

73
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 nbias =-(ncell -1) /(2* count );

% conversion to unbiased estimate

if approach (1) == 'u'
estimate =estimate - nbias;
nbias =0;
end;

% conversion to minimum mse estimate

if approach (1) == 'm'
estimate =estimate - nbias;
nbias =0;
lambda = estimate ^2/( estimate ^2+ sigma ^2);
nbias =(1 - lambda )* estimate ;
estimate = lambda * estimate ;
sigma = lambda *sigma;
end;

% base transformation
estimate = estimate /log(base);
nbias = nbias /log(base);
sigma = sigma /log(base);

74
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Feature extraction helper function: dominant_frequency_features.m [39]
function [maxfreq , maxval , maxratio ] =
dominant_frequency_features (data , fs , cutoff , plotflag )
[~, nchannels ] = size(data);

% nfft = 2^ nextpow2 ( npoints * padfactor );

nfft = 4096;
f = fs /2* linspace (0,1, nfft /2);
cutoffidx = length (find(f <= cutoff ));
f = f(1: cutoffidx );

% calculate the power spectrum using FFT method

datafft = fft(data ,nfft);
ps = real( datafft .* conj( datafft ));

% keep only the non - redundant portion

ps = ps (1: cutoffidx ,:);
for ich = 1: nchannels
ps(:, ich) = ps(:, ich)/sum(ps(:, ich));
end

if( plotflag )
figure ,
subplot (211) , plot(data)
subplot (212) , plot(f,ps);
axis ([0 cutoff 0 max(ps)]);
pause
end

% locate max value below cutoff

[maxval , maxind ] = max(ps);
maxfreq = f( maxind );

75
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % calculate peak energy by summing energy from maxfreq -delta
to maxfreq + delta
% then normalize by total energy below cutoff
delta = 5; % Hz
maxratio = zeros (1, nchannels );
for ich = 1: nchannels
maxrange = f >= maxfreq (ich)-delta & f<= maxfreq (ich)+delta
;
maxratio (ich) = sum(ps(maxrange ,ich)) / sum(ps(:, ich));
end

76
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Correlation Analysis
% Feature correlation coefficients and P_values
%remap reference labels to [1,-1]
feature_table_remap = feature_table ;
keySet = {'Background ', 'Drone '};
valueSet = { -1 ,1};
labelMap = containers .Map(keySet , valueSet );

for iwin = 1: height ( feature_table_remap );

current_class = feature_table_remap .class{iwin };
feature_table_remap .class{iwin} = labelMap ( current_class
);

end

feature_table_remap . class = table2array ( cell2table (

feature_table_remap {:," class "}));
[corr_coef , P_value ] = corrcoef ( feature_table_remap . Variables
);

corr_coef_table = array2table ( corr_coef );

corr_coef_table . Properties . VariableNames = {'meanValue ', '
medianValue ', 'standardDeviation ','meanAbsoluteDeviation '
, 'quantile25 ','quantile75 ','signalIQR ','sampleSkewness ',
'sampleKurtosis ','signalEntropy ','spectralEntropy ','
dominantFrequencyValue ','dominantFrequencyMagnitude ','
dominantFrequencyRatio ','MFCC1 ','MFCC2 ','MFCC3 ','MFCC4 ','
MFCC5 ','MFCC6 ','MFCC7 ','MFCC8 ','MFCC9 ','MFCC10 ','MFCC11 ',
'MFCC12 ','MFCC13 ','class '};
writetable ( corr_coef_table ,'corr_coef_table .xlsx ');

P_value_table = array2table ( P_value );

77
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 P_value_table . Properties . VariableNames = {'meanValue ', '
medianValue ', 'standardDeviation ','meanAbsoluteDeviation '
, 'quantile25 ','quantile75 ','signalIQR ','sampleSkewness ',
'sampleKurtosis ','signalEntropy ','spectralEntropy ','
dominantFrequencyValue ','dominantFrequencyMagnitude ','
dominantFrequencyRatio ','MFCC1 ','MFCC2 ','MFCC3 ','MFCC4 ','
MFCC5 ','MFCC6 ','MFCC7 ','MFCC8 ','MFCC9 ','MFCC10 ','MFCC11 ',
'MFCC12 ','MFCC13 ','class '};
writetable ( P_value_table ,'P_value_table .xlsx ');

xvalues = {'meanValue ', 'medianValue ', 'standardDeviation ','

meanAbsoluteDeviation ', 'quantile25 ','quantile75 ','
signalIQR ','sampleSkewness ','sampleKurtosis ','
signalEntropy ','spectralEntropy ','dominantFrequencyValue '
,'dominantFrequencyMagnitude ','dominantFrequencyRatio ','
MFCC1 ','MFCC2 ','MFCC3 ','MFCC4 ','MFCC5 ','MFCC6 ','MFCC7 ','
MFCC8 ','MFCC9 ','MFCC10 ','MFCC11 ','MFCC12 ','MFCC13 ','class
'};
yvalues = {'meanValue ', 'medianValue ', 'standardDeviation ','
meanAbsoluteDeviation ', 'quantile25 ','quantile75 ','
signalIQR ','sampleSkewness ','sampleKurtosis ','
signalEntropy ','spectralEntropy ','dominantFrequencyValue '
,'dominantFrequencyMagnitude ','dominantFrequencyRatio ','
MFCC1 ','MFCC2 ','MFCC3 ','MFCC4 ','MFCC5 ','MFCC6 ','MFCC7 ','
MFCC8 ','MFCC9 ','MFCC10 ','MFCC11 ','MFCC12 ','MFCC13 ','class
'};
h = heatmap (xvalues , yvalues , abs( corr_coef ));
h.Title = 'Correlation Coefficient Map ';
h. FontSize = 5

78
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Neighborhood Component Analysis
runNCA = true;
if ~ runNCA && exist ('SelectedFeatures .mat ')%#ok
load('SelectedFeatures .mat ')
else
rng (1);
mdl = fscnca ( table2array ( feature_table (: ,1:27)),
table2array ( feature_table (: ,28)), 'Lambda ', 0.005 , '
Verbose ', 0);
selected_feature_indx = find(mdl. FeatureWeights > 0.1);
stem(mdl. FeatureWeights ,'bo');
xlabel ('Features ')
ylabel ('Weighted Response ')
save('SelectedFeatures ', 'selected_feature_indx ');
end

79
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Support Vector Machine with Quadratic Kernel - MATLAB Generated
function [ trainedClassifier , validationAccuracy ] =
trainQuadraticSVMClassifier ( trainingData )
%% Input :
% trainingData : A table containing the same predictor and
response columns as those imported into the app.

%% Output :
% trainedClassifier : A struct containing the trained
classifier . The struct contains various fields with
information about the trained classifier .

% trainedClassifier . predictFcn : A function to make

predictions on new data.

% validationAccuracy : A double containing the accuracy as a

percentage . In the app , the Models pane displays this
overall accuracy score for each model.

% Extract predictors and response

% This code processes the data into the right shape for
training the model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable . class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Train the classifier

80
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % This code specifies all the classifier options and trains
the classifier .
classificationSVM = fitcsvm (predictors ,response ,'
KernelFunction ', 'polynomial ','PolynomialOrder ', 2,'
KernelScale ', 'auto ','BoxConstraint ', 1,'Standardize ',
true ,'ClassNames ', {'Background '; 'Drone '});

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames );
svmPredictFcn = @(x) predict ( classificationSVM , x);
trainedClassifier . predictFcn = @(x) svmPredictFcn (
predictorExtractionFcn (x));

% Add additional fields to the result struct

trainedClassifier . RequiredVariables = {'MFCC1 ', 'MFCC10 ', '
MFCC11 ', 'MFCC13 ', 'MFCC2 ', 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', '
MFCC6 ', 'MFCC7 ', 'MFCC8 ', 'MFCC9 ', '
dominantFrequencyValue '};
trainedClassifier . ClassificationSVM = classificationSVM ;
trainedClassifier . About = 'This struct is a trained model
exported from Classification Learner R2022a .';
trainedClassifier . HowToPredict = sprintf ('To make
predictions on a new table , T, use: \n yfit = c.
predictFcn (T) \ nreplacing ''c'' with the name of the
variable that is this struct , e.g. ''trainedModel ''. \n \
nThe table , T, must contain the variables returned by: \n
c. RequiredVariables \ nVariable formats (e.g. matrix /
vector , datatype ) must match the original training data.
\ nAdditional variables are ignored . \n \nFor more
information , see <a href =" matlab : helpview ( fullfile (
docroot , ''stats '', ''stats.map ''), ''
appclassification_exportmodeltoworkspace '')">How to
predict using an exported model </a>.');

81
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % Extract predictors and response
% This code processes the data into the right shape for
training the
% model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable .class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Perform cross - validation

partitionedModel = crossval ( trainedClassifier .
ClassificationSVM , 'KFold ', 5);

% Compute validation predictions

[ validationPredictions , validationScores ] = kfoldPredict (
partitionedModel );

% Compute validation accuracy

validationAccuracy = 1 - kfoldLoss ( partitionedModel , '
LossFun ', 'ClassifError ');

82
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Support Vector Machine with Medium Gaussian Kernel - MATLAB Generated
function [ trainedClassifier , validationAccuracy ] =
trainMedGaussianSVMClassifier ( trainingData )

%% Input :
% trainingData : A table containing the same predictor and
response columns as those imported into the app.

%% Output :
% trainedClassifier : A struct containing the trained
classifier . The struct contains various fields with
information about the trained classifier .

% trainedClassifier . predictFcn : A function to make

predictions on new data.

% validationAccuracy : A double containing the accuracy as a

percentage . In the app , the Models pane displays this
overall accuracy score for each model.

% Extract predictors and response

% This code processes the data into the right shape for
training the model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable . class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Train the classifier

83
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % This code specifies all the classifier options and trains
the classifier .
classificationSVM = fitcsvm (predictors ,response ,'
KernelFunction ', 'gaussian ','PolynomialOrder ', [],'
KernelScale ', 3.6 , 'BoxConstraint ', 1,'Standardize ', true ,
'ClassNames ', {'Background '; 'Drone '});

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames );
svmPredictFcn = @(x) predict ( classificationSVM , x);
trainedClassifier . predictFcn = @(x) svmPredictFcn (
predictorExtractionFcn (x));

% Add additional fields to the result struct

trainedClassifier . RequiredVariables = {'MFCC1 ', 'MFCC10 ', '
MFCC11 ', 'MFCC13 ', 'MFCC2 ', 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', '
MFCC6 ', 'MFCC7 ', 'MFCC8 ', 'MFCC9 ', '
dominantFrequencyValue '};
trainedClassifier . ClassificationSVM = classificationSVM ;
trainedClassifier . About = 'This struct is a trained model
exported from Classification Learner R2022a .';
trainedClassifier . HowToPredict = sprintf ('To make
predictions on a new table , T, use: \n yfit = c.
predictFcn (T) \ nreplacing ''c'' with the name of the
variable that is this struct , e.g. ''trainedModel ''. \n \
nThe table , T, must contain the variables returned by: \n
c. RequiredVariables \ nVariable formats (e.g. matrix /
vector , datatype ) must match the original training data.
\ nAdditional variables are ignored . \n \nFor more
information , see <a href =" matlab : helpview ( fullfile (
docroot , ''stats '', ''stats.map ''), ''
appclassification_exportmodeltoworkspace '')">How to
predict using an exported model </a>.');

84
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % Extract predictors and response
% This code processes the data into the right shape for
training the
% model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable .class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Perform cross - validation

partitionedModel = crossval ( trainedClassifier .
ClassificationSVM , 'KFold ', 5);

% Compute validation predictions

[ validationPredictions , validationScores ] = kfoldPredict (
partitionedModel );

% Compute validation accuracy

validationAccuracy = 1 - kfoldLoss ( partitionedModel , '
LossFun ', 'ClassifError ');

85
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Weighted k-Nearest Neighbors - MATLAB Generated
function [ trainedClassifier , validationAccuracy ] =
trainWeightedKNNClassifier ( trainingData )

%% Input :
% trainingData : A table containing the same predictor and
response columns as those imported into the app.

%% Output :
% trainedClassifier : A struct containing the trained
classifier . The struct contains various fields with
information about the trained classifier .

% trainedClassifier . predictFcn : A function to make

predictions on new data.

% validationAccuracy : A double containing the accuracy as a

percentage . In the app , the Models pane displays this
overall accuracy score for each model.

% Extract predictors and response

% This code processes the data into the right shape for
training the model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable . class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Train the classifier

86
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % This code specifies all the classifier options and trains
the classifier .
classificationKNN = fitcknn (predictors ,response ,'Distance ',
'Euclidean ','Exponent ', [],'NumNeighbors ', 10,'
DistanceWeight ', 'SquaredInverse ','Standardize ', true ,'
ClassNames ', {'Background '; 'Drone '});

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames );
knnPredictFcn = @(x) predict ( classificationKNN , x);
trainedClassifier . predictFcn = @(x) knnPredictFcn (
predictorExtractionFcn (x));

% Add additional fields to the result struct

trainedClassifier . RequiredVariables = {'MFCC1 ', 'MFCC10 ', '
MFCC11 ', 'MFCC13 ', 'MFCC2 ', 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', '
MFCC6 ', 'MFCC7 ', 'MFCC8 ', 'MFCC9 ', '
dominantFrequencyValue '};
trainedClassifier . ClassificationKNN = classificationKNN ;
trainedClassifier . About = 'This struct is a trained model
exported from Classification Learner R2022a .';
trainedClassifier . HowToPredict = sprintf ('To make
predictions on a new table , T, use: \n yfit = c.
predictFcn (T) \ nreplacing ''c'' with the name of the
variable that is this struct , e.g. ''trainedModel ''. \n \
nThe table , T, must contain the variables returned by: \n
c. RequiredVariables \ nVariable formats (e.g. matrix /
vector , datatype ) must match the original training data.
\ nAdditional variables are ignored . \n \nFor more
information , see <a href =" matlab : helpview ( fullfile (
docroot , ''stats '', ''stats.map ''), ''
appclassification_exportmodeltoworkspace '')">How to
predict using an exported model </a>.');

87
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % Extract predictors and response
% This code processes the data into the right shape for
training the
% model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable .class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Perform cross - validation

partitionedModel = crossval ( trainedClassifier .
ClassificationKNN , 'KFold ', 5);

% Compute validation predictions

[ validationPredictions , validationScores ] = kfoldPredict (
partitionedModel );

% Compute validation accuracy

validationAccuracy = 1 - kfoldLoss ( partitionedModel , '
LossFun ', 'ClassifError ');

88
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Ensemble Bagged Trees - MATLAB Generated
function [ trainedClassifier , validationAccuracy ] =
trainBaggedTreeEnsembleClassifier ( trainingData )
%% Input :
% trainingData : A table containing the same predictor and
response columns as those imported into the app.

%% Output :
% trainedClassifier : A struct containing the trained
classifier . The struct contains various fields with
information about the trained classifier .

% trainedClassifier . predictFcn : A function to make

predictions on new data.

% validationAccuracy : A double containing the accuracy as a

percentage . In the app , the Models pane displays this
overall accuracy score for each model.

% Extract predictors and response

% This code processes the data into the right shape for
training the
% model.
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable . class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Train the classifier

89
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % This code specifies all the classifier options and trains
the classifier .
template = templateTree ('MaxNumSplits ', 17757 , '
NumVariablesToSample ', 'all ');
classificationEnsemble = fitcensemble (predictors ,response ,'
Method ','Bag ','NumLearningCycles ', 30,'Learners ',
template ,'ClassNames ', {'Background '; 'Drone '});

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames );
ensemblePredictFcn = @(x) predict ( classificationEnsemble , x)
;
trainedClassifier . predictFcn = @(x) ensemblePredictFcn (
predictorExtractionFcn (x));

% Add additional fields to the result struct

trainedClassifier . RequiredVariables = {'MFCC1 ', 'MFCC10 ', '
MFCC11 ', 'MFCC13 ', 'MFCC2 ', 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', '
MFCC6 ', 'MFCC7 ', 'MFCC8 ', 'MFCC9 ', '
dominantFrequencyValue '};
trainedClassifier . ClassificationEnsemble =
classificationEnsemble ;
trainedClassifier . About = 'This struct is a trained model
exported from Classification Learner R2022a .';
trainedClassifier . HowToPredict = sprintf ('To make
predictions on a new table , T, use: \n yfit = c.
predictFcn (T) \ nreplacing ''c'' with the name of the
variable that is this struct , e.g. ''trainedModel ''. \n \
nThe table , T, must contain the variables returned by: \n
c. RequiredVariables \ nVariable formats (e.g. matrix /
vector , datatype ) must match the original training data.
\ nAdditional variables are ignored . \n \nFor more
information , see <a href =" matlab : helpview ( fullfile (

90
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 docroot , ''stats '', ''stats.map ''), ''
appclassification_exportmodeltoworkspace '')">How to
predict using an exported model </a>.');

% Extract predictors and response

% This code processes the data into the right shape for
training the
% model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable .class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Perform cross - validation

partitionedModel = crossval ( trainedClassifier .
ClassificationEnsemble , 'KFold ', 5);

% Compute validation predictions

[ validationPredictions , validationScores ] = kfoldPredict (
partitionedModel );

% Compute validation accuracy

validationAccuracy = 1 - kfoldLoss ( partitionedModel , '
LossFun ', 'ClassifError ');

91
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Ensemble Subspace k-Nearest Neighbors - MATLAB Generated
function [ trainedClassifier , validationAccuracy ] =
trainSubspaceKNNClassifier ( trainingData )
%% Input :
% trainingData : A table containing the same predictor and
response columns as those imported into the app.

%% Output :
% trainedClassifier : A struct containing the trained
classifier . The struct contains various fields with
information about the trained classifier .

% trainedClassifier . predictFcn : A function to make

predictions on new data.
%
% validationAccuracy : A double containing the accuracy as a
percentage . In the app , the Models pane displays this
overall accuracy score for each model.

% Extract predictors and response

% This code processes the data into the right shape for
training the
% model.
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable . class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Train the classifier

92
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % This code specifies all the classifier options and trains
the classifier .
subspaceDimension = max (1, min (7, width( predictors ) - 1));
classificationEnsemble = fitcensemble (predictors ,response ,'
Method ', 'Subspace ','NumLearningCycles ', 30,'Learners ', '
knn ','NPredToSample ', subspaceDimension ,'ClassNames ', {'
Background '; 'Drone '});

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames );
ensemblePredictFcn = @(x) predict ( classificationEnsemble , x)
;
trainedClassifier . predictFcn = @(x) ensemblePredictFcn (
predictorExtractionFcn (x));

% Add additional fields to the result struct

trainedClassifier . RequiredVariables = {'MFCC1 ', 'MFCC10 ', '
MFCC11 ', 'MFCC13 ', 'MFCC2 ', 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', '
MFCC6 ', 'MFCC7 ', 'MFCC8 ', 'MFCC9 ', '
dominantFrequencyValue '};
trainedClassifier . ClassificationEnsemble =
classificationEnsemble ;
trainedClassifier . About = 'This struct is a trained model
exported from Classification Learner R2022a .';
trainedClassifier . HowToPredict = sprintf ('To make
predictions on a new table , T, use: \n yfit = c.
predictFcn (T) \ nreplacing ''c'' with the name of the
variable that is this struct , e.g. ''trainedModel ''. \n \
nThe table , T, must contain the variables returned by: \n
c. RequiredVariables \ nVariable formats (e.g. matrix /
vector , datatype ) must match the original training data.
\ nAdditional variables are ignored . \n \nFor more
information , see <a href =" matlab : helpview ( fullfile (

93
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 docroot , ''stats '', ''stats.map ''), ''
appclassification_exportmodeltoworkspace '')">How to
predict using an exported model </a>.');

% Extract predictors and response

% This code processes the data into the right shape for
training the
% model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable .class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Perform cross - validation

partitionedModel = crossval ( trainedClassifier .
ClassificationEnsemble , 'KFold ', 5);

% Compute validation predictions

[ validationPredictions , validationScores ] = kfoldPredict (
partitionedModel );

% Compute validation accuracy

validationAccuracy = 1 - kfoldLoss ( partitionedModel , '
LossFun ', 'ClassifError ');

94
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Wide Neural Network - MATLAB Generated
function [ trainedClassifier , validationAccuracy ] =
trainWideNeuralNetworkClassifier ( trainingData )
%% Input :
% trainingData : A table containing the same predictor and
response columns as those imported into the app.

%% Output :
% trainedClassifier : A struct containing the trained
classifier . The struct contains various fields with
information about the trained classifier .
%
% trainedClassifier . predictFcn : A function to make
predictions on new data.
%
% validationAccuracy : A double containing the accuracy as a
percentage . In the app , the Models pane displays this
overall accuracy score for each model.

% Extract predictors and response

% This code processes the data into the right shape for
training the
% model.
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable . class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Train the classifier

95
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 % This code specifies all the classifier options and trains
the classifier .
classificationNeuralNetwork = fitcnet ( predictors , response ,
'LayerSizes ', 100 , 'Activations ', 'relu ', 'Lambda ', 0, '
IterationLimit ', 1000 , 'Standardize ', true ,'ClassNames ',
{'Background '; 'Drone '});

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames );
neuralNetworkPredictFcn = @(x) predict (
classificationNeuralNetwork , x);
trainedClassifier . predictFcn = @(x) neuralNetworkPredictFcn (
predictorExtractionFcn (x));

% Add additional fields to the result struct

trainedClassifier . RequiredVariables = {'MFCC1 ', 'MFCC10 ', '
MFCC11 ', 'MFCC13 ', 'MFCC2 ', 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', '
MFCC6 ', 'MFCC7 ', 'MFCC8 ', 'MFCC9 ', '
dominantFrequencyValue '};
trainedClassifier . ClassificationNeuralNetwork =
classificationNeuralNetwork ;
trainedClassifier . About = 'This struct is a trained model
exported from Classification Learner R2022a .';
trainedClassifier . HowToPredict = sprintf ('To make
predictions on a new table , T, use: \n yfit = c.
predictFcn (T) \ nreplacing ''c'' with the name of the
variable that is this struct , e.g. ''trainedModel ''. \n \
nThe table , T, must contain the variables returned by: \n
c. RequiredVariables \ nVariable formats (e.g. matrix /
vector , datatype ) must match the original training data.
\ nAdditional variables are ignored . \n \nFor more
information , see <a href =" matlab : helpview ( fullfile (
docroot , ''stats '', ''stats.map ''), ''

96
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 appclassification_exportmodeltoworkspace '')">How to
predict using an exported model </a>.');

% Extract predictors and response

% This code processes the data into the right shape for
training the
% model .
inputTable = trainingData ;
predictorNames = {'dominantFrequencyValue ', 'MFCC1 ', 'MFCC2 '
, 'MFCC3 ', 'MFCC4 ', 'MFCC5 ', 'MFCC6 ', 'MFCC7 ', 'MFCC8 ', '
MFCC9 ', 'MFCC10 ', 'MFCC11 ', 'MFCC13 '};
predictors = inputTable (:, predictorNames );
response = inputTable .class;
isCategoricalPredictor = [false , false , false , false , false ,
false , false , false , false , false , false , false , false ];

% Perform cross - validation

partitionedModel = crossval ( trainedClassifier .
ClassificationNeuralNetwork , 'KFold ', 5);

% Compute validation predictions

[ validationPredictions , validationScores ] = kfoldPredict (
partitionedModel );

% Compute validation accuracy

validationAccuracy = 1 - kfoldLoss ( partitionedModel , '
LossFun ', 'ClassifError ');

97
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

98
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 APPENDIX C:
Feature Tables

It was not practical to include a feature table with all 22,488 samples. Instead, included here
is a preview of the first 10 rows of both the training and test features used in classification.

Table C.1. Training Set Feature Table

Dominant Frequency Value MFCC1 MFCC2 MFCC3 MFCC4 MFCC5 MFCC6 MFCC7 MFCC8 MFCC9 MFCC10 MFCC11 MFCC13 Class
12.21299463 78.04469891 6.426757768 11.20454424 2.214013691 9.983761737 2.781114169 2.922437323 4.265902102 -0.686158383 6.762974394 -4.943545179 -2.727286396 Drone
12.21299463 75.37337228 9.291414579 13.60785786 -2.081741282 11.68691505 0.553862749 0.887054704 6.448254981 1.153166183 10.08975352 -9.693230948 -4.186160459 Drone
36.63898388 75.9211968 9.148906433 12.50637186 -1.233445849 12.33633422 -1.669419785 2.882833139 4.675862282 1.985749936 7.57262639 -7.626704737 -4.176820418 Drone
12.21299463 77.14498289 7.873766262 12.13540431 -2.441035299 8.690541082 0.018107895 1.151322139 5.683095741 1.619489181 8.495250316 -8.077486506 -2.469775698 Drone
12.21299463 76.13025519 9.005377897 13.11453421 -1.509339193 11.64856049 -0.377112155 -1.399567694 4.410699259 0.734953906 7.521551064 -8.711937433 -3.489724051 Drone
12.21299463 38.03901178 6.750768912 8.795125109 8.722387082 6.983330723 5.659603488 3.967171064 5.162664106 1.059026809 -0.052062267 -2.290959473 -2.140881281 Background
24.42598925 38.85366025 6.387106093 8.654459215 7.332731709 7.32262718 5.677214424 4.471850161 1.793466775 1.240794824 -0.479494245 -0.491414242 -3.620408531 Background
12.21299463 41.66939023 7.780746616 5.79442678 5.501082069 4.40853154 4.321111033 2.369041641 2.58403963 0.800685689 0.389062107 -0.065229683 -0.299726365 Background
12.21299463 35.95156506 6.761048973 11.18880419 8.995755919 7.52994977 4.450985683 3.193499882 1.172094377 -0.945861203 -2.992019912 -3.056883183 -3.698964188 Background
12.21299463 36.78039395 7.514512438 11.65565464 8.641368851 7.147982954 5.502015382 2.152037149 3.451757782 -0.712641085 -1.65845391 -2.293805415 -1.939017779 Background

Table C.2. Test Set Feature Table

Dominant Frequency Value MFCC1 MFCC2 MFCC3 MFCC4 MFCC5 MFCC6 MFCC7 MFCC8 MFCC9 MFCC10 MFCC11 MFCC13 Class
24.42598925 88.40290799 -0.618971613 4.598211777 9.403783151 -1.527554941 0.519110483 -0.477862293 3.754537895 12.38277446 6.669867019 -7.592512047 -3.655396072 Drone
24.42598925 88.12000556 0.437054421 1.872884005 9.041108509 -0.257279588 1.179681799 -3.582637877 5.535200563 8.173171672 11.40878061 1.73036947 -0.527857766 Drone
12.21299463 87.20156094 0.304624034 -0.199183151 8.963690653 1.150744573 0.314358851 -0.414415639 -1.219877871 4.086770669 6.644101121 8.486010988 4.169765046 Drone
12.21299463 86.66598539 0.299931494 -0.190043112 9.939091093 0.721954807 -0.837855948 -1.014789108 -2.662595029 5.544241053 6.531171596 8.500698023 2.494179823 Drone
12.21299463 85.52418392 0.542221404 1.234422035 9.234974075 -3.852828505 -1.491240555 -2.665892397 3.056200911 1.206333338 6.310223114 4.489334383 3.526100654 Drone
12.21299463 36.83434187 6.064736536 9.035071276 6.953081292 6.75350462 5.402842985 2.372294245 0.971463218 -0.357830247 -0.555291597 -1.801682451 -1.270067448 Background
12.21299463 37.97660116 5.676761239 7.451045336 5.622374857 5.551732843 4.198248124 3.172243331 2.402643275 0.221308243 -0.93529019 -0.359703564 -0.647210065 Background
12.21299463 38.65127488 6.601783729 7.330769296 6.339524037 5.069241475 3.242611603 2.841726206 0.902567197 1.042648242 -1.062183157 -1.424832792 -2.321813108 Background
12.21299463 36.73540238 5.652384324 8.57492433 6.718557544 6.107313419 4.893229168 2.721078906 1.189958262 0.271245648 -1.272035403 -1.780447394 -0.830401892 Background
12.21299463 36.61153728 5.994177223 9.420277986 7.090925276 6.819165343 4.095053785 4.128869443 1.607289428 -0.006106527 -0.859262643 -2.367814979 -1.956530804 Background

99
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

100
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 List of References

[1] S. Pettyjohn, “Drones are transforming the battlefield in ukraine but in evolution-
ary fashion,” Mar, 2024. Available: https://warontherocks.com/2024/03/drones-are-
transforming-the-battlefield-in-ukraine-but-in-an-evolutionary-fashion/

[2] V. Cotovio and C. Sebastian and A. Goodwin, “Ukraine’s ai-enabled drones are try-
ing to disrupt russia’s industry. so far, it’s working,” Apr, 2024. Available: https:
//www.cnn.com/2024/04/01/energy/ukrainian-drones-disrupting-russian-energy-
industry-intl-cmd/index.html

[3] P. Satam, “Chinese citizen flies drone over js izumo, uss ronald reagan aircraft carri-
ers in japan,” The Aviationist, May 2024. Available: https://theaviationist.com/2024/
05/18/drone-yokosuka-naval-base/

[4] Q. Abdul-Zahra and J. Gambrell and G. Moulson, “Enemy drone that killed us
troops in jordan was mistaken for a us drone, preliminary report suggests,” Jan,
2024. Available: https://www.usnews.com/news/politics/articles/2024-01-29/
preliminary-probe-suggests-enemy-drone-that-killed-us-troops-in-jordan-was-
mistaken-for-a-us-drone

[5] T. Kosuke, “Japan’s defense ministry: Js izumo drone video posted online is highly
likely authentic,” The Diplomat, May 2024. Available: https://thediplomat.com/
2024/05/japans-defense-ministry-js-izumo-drone-video-posted-online-is-highly-
likely-authentic/

[6] U.S. Border Patrol Official Release, “Human smugglers now using drones to surveil
usbp,” Mar, 2023. Available: https://www.cbp.gov/newsroom/local-media-release/
human-smugglers-now-using-drones-surveil-usbp

[7] H. Kolamuna et al., “Droneprint: acoustic signatures for open-set drone detection
and identification with online data,” Proc. ACM Interact. Mob. Wearable Ubiquitous
Technol., mar 2021. Available: https://dl.acm.org/doi/10.1145/3448115

[8] “Scottsdale, arizona police department leverage dedrone to protect pga’s phoenix
open tour from drone threats,” Dedrone, 2024. Accessed May 21, 2024. Available:
https://www.dedrone.com/customers/pga-phoenix-open

[9] V. Matic, V. Kosjer, A. Lebl, B. Pavic, and J. Radivojevic, “Methods for drone
detection and jamming,” Information Society of Serbia, 2020. Available: https:
//www.eventiotic.com/eventiotic/files/Papers/URL/f07e8f39-5c16-420e-b0e3-
5eb5b5ab1ba0.pdf

101
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 [10] Robin Radar Systems, “Counterthe drone threat with counter-uas technology,” ,.
Available: https://www.robinradar.com/press/blog/10-counter-drone-technologies-
to-detect-and-stop-drones-today
[11] J. Ivancic, G. Karunasiri, and F. Alves, “Directional resonant mems acoustic sensor
and associated acoustic vector sensor,” Sensors, Oct. 2023. Available: https://doi.org/
10.3390/s23198217
[12] I. A. Soule, “Gunshot direction of arrival determination using bio-inspired mems
sensors,” M.S. thesis, Naval Postgraduate School, Monterey, CA, 2022.
[13] J. Park, “Operation of mems underwater directional sound sensor in neutral
bouyancy configuration,” M.S. thesis, Naval Postgraduate School, Monterey, 2019.
[14] F. Alves, R. Rabelo, and G. Karunasiri, “Dual band mems directional acoustic sen-
sor for near resonance operation,” Sensors, jul 2022. Available: https://doi.org/10.
3390/s22155635
[15] D. Wilmott, F. Alves, and G. Karunasiri, “Bio-inspired miniature direction find-
ing acoustic sensor,” Scientific Reports, Jul. Available: https://doi.org/10.1038/
srep29957
[16] J. Ivancic and F. Alves, “Directional multi-resonant micro-electromechanical system
acoustic sensor for low frequency detection,” Sensors, May 2024. Available: https:
//doi.org/10.3390/s24092908
[17] S. Badillo et al., “An introduction to machine learning,” Clinical Pharmacology and
Therapeutics, vol. 107, no. 4, Apr. 2020. Available: https://doi.org/10.1002/cpt.1796
[18] B. Mahesh, “Machine learning algorithms - a review,” International Journal of Sci-
ence and Research, vol. 9, no. 1, Jan. 2020. Available: https://www.researchgate.net/
publication/344717762_Machine_Learning_Algorithms_-A_Review
[19] A. Ng, A. Maas, and S. Tandon, “Stanford unsupervised feature learning and deep
learning tutorial,” Stanford Tutorial, Sep. 2013. Available: http://ufldl.stanford.edu/
tutorial/
[20] I. Sarker, “Machine learning: algorithms, real-world applications and research di-
rections,” Springer Nature Computer Science, vol. 160, no. 2, Mar. 2021. Available:
https://link.springer.com/article/10.1007/s42979-021-00592-x
[21] A. Bagasta, Z. Rustam, J. Pandelaki, and W. Nugroho, “Comparison of cubic SVM
with Gaussian SVM: classification of infarction for detecting ischemic stroke,” in
2019 IOP Conference Series: Materials Science and Engineering, 2019. Available:
https://iopscience.iop.org/article/10.1088/1757-899X/546/5/052016/meta

102
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 [22] A. Rozantsev, V. Lepetit, and P. Fua, “Detecting flying objects using a single moving
camera,” IEEE Trans Pattern Anal. Mach. Intell., vol. 39, no. 5, 2017. Available:
https://doi.org/10.1109/TPAMI.2016.2564408

[23] D. Lee, W. La, and H. Kim, “Drone detection and identification system using arti-
ficial intelligence,” in 2018 International Conference on Information and Commu-
nication Technology Convergence (ICTC), 2018. Available: https://doi.org/10.1109/
ICTC.2018.8539442

[24] R. Yoshihashi, T. Trinh, R. Kawakami, S. You, M. Iida, and T. Naemura, “Dif-

ferentiating objects by motion: joint detection and tracking of small flying ob-
jects,” Computer vision and pattern recognition, vol. 39, no. 5, 2018. Available:
https://doi.org/10.48550/arXiv.1709.04666

[25] B. Taha and A. Shoufan, “Machine learning-based drone detection and classifi-
cation: state-of-the-art in research,” IEEE Access, vol. 7, 2019. Available: https:
//doi.org/10.1109/ACCESS.2019.2942944

[26] W. Zhang, G. Li, and C. Baker, “Radar recognition of multiple micro-drones based
on their micro-doppler signatures via dictionary learning,” The Institution of Engi-
neering and Technology, jul 2020. Available: https://doi.org/10.1049/iet-rsn.2019.
0225

[27] P. Molchanov, K. Egiazarian, J. Astola, R. Harmanny, and J. de Wit, “Classification

of small UAVs and birds by micro-Doppler signatures,” in 2013 European Radar
Conference, 2013. Available: https://ieeexplore.ieee.org/document/6689141

[28] N. Mohajerin, J. Histon, R. Dizaji, and S. Waslander, “Feature extraction and radar
track classification for detecting UAVs in civilian airspace,” in 2014 IEEE Radar
Conference, 2014. Available: https://doi.org/10.1109/RADAR.2014.6875676

[29] G. Mendis, T. Randeny, J. Wei, and A. Madanayake, “Deep learning based doppler
radar for micro UAS detection and classification,” in 2016 IEEE Military Commu-
nications Conference, 2016. Available: https://doi.org/10.1109/MILCOM.2016.
7795448

[30] A. Flemming, “Feasibility of detecting and classifying small unmanned aerial sys-
tem threats using acoustic data,” M.S. thesis, Naval Postgraduate School, Monterey,
CA, 2019.

[31] A. Bernardini, F. Mangiatordi, E. Pallotti, L. Capodiferro, and F. U. Bordoni, “Drone

detection by acoustic signature identification,” Society for Imaging Science and
Technology, jan 2017. Available: https://doi.org/10.2352/ISSN.2470-1173.2017.10.
IMAWM-168

103
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 [32] S. Al-Emadi, A. Al-Ali, A. Mohammad, and A. Al-Ali, “Audio based drone detec-
tion and identification using deep learning,” in 15th International Wireless Commu-
nications Mobile Computing Conference, 2019. Available: https://doi.org/10.1109/
IWCMC.2019.8766732

[33] R. Parhi and R. Nowak, “The role of neural network activation functions,” IEEE
Signal Processing Letters, vol. 27, pp. 1779–1783, 2020. Available: https://doi.org/
10.1109/LSP.2020.3027517

[34] J. Yang et al., “Parameter selection of gaussian kernel svm based on local density of
training set,” Inverse Problems in Science and Engineering, vol. 29, no. 4, July 2021.
Available: https://doi.org/10.1080/17415977.2020.1797716

[35] V. Vapnik, “An overview of statistical learning theory,” IEEE Transactions on Neural
Networks, vol. 10, no. 5, Sep. 1999. Available: https://doi.org/10.1109/72.788640

[36] S. Keerthi and C. Lin, “Asymptotic behaviors of support vector machines with Gaus-
sian kernel,” in 2003 Neural Computation, 2003. Available: https://direct.mit.edu/
neco/article/15/7/1667/6751/Asymptotic-Behaviors-of-Support-Vector-Machines

[37] J. Mercer, “Functions of positive and negative type, and their connection with the
theory of integral equations,” in Philosophical Transactions of the Royal Society
of London. Series A, Containing Papers of a Mathematical of Physical Character,
R. Society, Ed. London, United Kingdom: Royal Society, 1909, pp. 415–446. Avail-
able: https://doi.org/10.1098/rspa.1909.0075

[38] B. Scholkopf et al., “Comparing support vector machines with gaussian kernels to
radial basis function classifiers,” IEEE transactions on Signal Processing, vol. 45,
no. 11, Nov. 1997. Available: https://doi.org/10.1109/78.650102

[39] B. Suhm, Mathworks, USA. 2019. Heart Sound Classifier, MATLAB Central File
Exchange. Accessed February 12, 2024. Available: https://www.mathworks.com/
matlabcentral/fileexchange/65286-heart-sound-classifier

[40] M. Djeddou and T. Touhami, “Classification and modeling of acoustic gunshot

signatures,” Arabian Journal for Science and Engineering, jul 2013. Available:
https://www.researchgate.net/publication/256600891_Classification_and_Modeling_
of_Acoustic_Gunshot_Signatures

[41] M. Pawar and R. Kokate, “Convolution neural network based automatic speech emo-
tion recognition using mel-frequency cepstrum coefficients,” Multimedia Tools and
Applications, Feb. 2021. Available: https://doi.org/10.1007/s11042-020-10329-2

104
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 [42] S. Salman, J. Mir, M. Farooq, A. Malik, and R. Haleemdeen, “Machine learning
inspired efficient audio drone detection using acoustic features,” in International
Bhurban Conference on Applied Sciences and Technologies, 2021. Available: https:
//doi.org/10.1109/IBCAST51254.2021.9393232
[43] Z. Abdul and A. Al-Talabani, “Mel frequency cepstral coefficient and its appli-
cations: a review,” IEEE Access, Nov. 2022. Available: https://doi.org/10.1109/
ACCESS.2022.3223444
[44] S. Ali, S. Tanweer, S. Khalid, and N. Rao, “Mel frequency cepstral coefficient: a
review,” ICIDSSD, Feb. 2021. Available: https://eudl.eu/doi/10.4108/eai.27-2-2020.
2303173
[45] X. Zhao and D. Wang, “Analyzing noise robustness of mfcc and gfcc features in
speaker identification,” IEEE International Conference on Acoustics, Speech and
Signal Processing, May 2013. Available: https://doi.org/10.1109/ICASSP.2013.
6639061
[46] U. Ayvaz, H. Guruler, F. Khan, N. Ahmed, T. Whangbo, and A. Bobomirzaevich,
“Automatic speaker recognition using mel-frequency cepstral coefficients through
machine learning,” Computers, Materials Continua, Nov. 2021. Available: https:
//doi.org/10.32604/cmc.2022.023278
[47] L. Luzi, E. Gonzalez, P. Bruillard, M. Prowant, and J. Skorpik, “Acoustic firearm
discharge detection and classification in an enclosed environment,” Mathematics
Faculty Publications, May 2016. Available: https://doi.org/10.1121/1.4948994
[48] N. Malan and S. Sharma, “Feature selection using regularized neighbourhood com-
ponent analysis to enhance classification performance of motor imagery signals,”
Computers in Biology and Medicine, Feb. 2019. Available: https://doi.org/10.1016/j.
compbiomed.2019.02.009
[49] W. Yang, K. Wang, and W. Zuo, “Neighborhood component feature selection for
high-dimensional data,” Journal of Computers, Jan. 2012. Available: https://doi.org/
10.4304/jcp.7.1.161-168
[50] S. Kumar and I. Chong, “Correlation analysis to identiful the effective data in ma-
chine learning: prediction of depressive disorder and emotion states,” Interna-
tional Journal of Environmental Research and Public Health, Dec. 2018. Available:
https://doi.org/10.3390/ijerph15122907
[51] A. Almazroey and S. Jarraya, “Abnormal events and behavior detection in crowd
scenes based on deep learning and neighborhood component analysis feature selec-
tion,” in International Conference on Artificial Intelligence and Computer Vision,
2020. Available: https://doi.org/10.1007/978-3-030-44289-7_25

105
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 [52] N. Kavya et al., “Feature selection using neighborhood component analysis with
support vector machine for classification of breast mammograms,” in International
Conference on Communication, Computing and Electronics Systems, 2020. Avail-
able: https://doi.org/10.1007/978-981-15-2612-1_24

[53] M. Ismail, R. Alabdullatif, and O. Bchir, “Alzheimer’s disease detection using

neighborhood components analysis and feature selection,” International Journal
of Advanced Computer Science and Applications, Nov. 2020. Available: https:
//doi.org/10.14569/IJACSA.2020.0111002

[54] N. Aslan, G. Koca, M. Kobat, and S. Dogan, “Multi-classification deep cnn model
for diagnosing covid-19 using iterative neighborhood component analysis and iter-
ative relieff feature selection techniques with x-ray images,” Chemometr Intell Lab
Syst., Mar. 2022. Available: https://doi.org/10.1016/j.chemolab.2022.104539

[55] K. Nicodemus and J. Malley, “Predictor correlation impacts machine learning al-
gorithms: implications for genomic studies,” Bioinformatics, May 2009. Avail-
able: https://www.researchgate.net/publication/24445041_Predictor_Correlation_
Impacts_Machine_Learning_Algorithms_Implications_for_Genomic_Studies

[56] T. Assegie, S. Sushma, B. Bhavya, and S. Padmashree, “Correlation analysis for de-
termining effective data in machine learning: detection of heart failure,” SN Com-
puter Science, Apr. 2021. Available: https://doi.org/10.1007/s42979-021-00617-5

[57] “Cross-validaiton,” MathWorks, 2024. Accessed April 15, 2024. Available: https:
//www.mathworks.com/discovery/cross-validation.html

[58] “Support vector machine,” MathWorks, 2024. Accessed April 17, 2024. Available:
https://www.mathworks.com/discovery/support-vector-machine.html

[59] “Choosing the right classification model,” MathWorks, 2024. Accessed May 5,
2024. Available: https://www.mathworks.com/campaigns/offers/next/choosing-the-
best-machine-learning-classification-model-and-avoiding-overfitting.html

[60] “Classification using nearest neighbors,” MathWorks, 2024. Accessed April 17,
2024. Available: https://www.mathworks.com/help/stats/classification-using-
nearest-neighbors.html

[61] “Classification ensembles,” MathWorks, 2024. Accessed April 17, 2024. Available:
https://www.mathworks.com/help/stats/classification-ensembles.html

[62] “Treebagger,” MathWorks, 2024. Accessed April 17, 2024. Available: https://www.
mathworks.com/help/stats/treebagger.html

106
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 [63] “Decision trees,” MathWorks, 2024. Accessed May 5, 2024. Available: https://www.
mathworks.com/help/stats/decision-trees.html

[64] “fitcnet,” MathWorks, 2024. Accessed April 17, 2024. Available: https://
www.mathworks.com/help/stats/fitcnet.html#mw_a9fa4524-aecf-4d9b-95ef-
58914f7780f3

[65] “Ensemble algorithms,” MathWorks, 2024. Accessed May 13, 2024. Available:
https://www.mathworks.com/help/stats/ensemble-algorithms.html#bsw8avb

[66] H. Yang, K. Lee, Y. Choo, and K. Kim, “Underwater acoustic research trends with
machine learning: Passive sonar applications,” Journal of Ocean Engineering and
Technology, vol. 34, no. 3, pp. 227–236, 2020. Available: https://doi.org/10.26748/
KSOE.2020.017

[67] X. Luo, L. Chen, H. Zhou, and H. Cao, “A survey of underwater acoustic tar-
get recognition methods based on machine learning,” Journal of Marine Science
and Engineering, vol. 11, no. 2, p. 384, 2023. Available: https://doi.org/10.3390/
jmse11020384

107
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 THIS PAGE INTENTIONALLY LEFT BLANK

108
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 Initial Distribution List

1. Defense Technical Information Center

Fort Belvoir, Virginia

2. Dudley Knox Library

Naval Postgraduate School
Monterey, California

109
_________________________________________________________
NAVAL POSTGRADUATE SCHOOL | MONTEREY, CALIFORNIA | WWW.NPS.EDU
 DUDLEY KNOX LIBRARY
NAVAL POSTGRADUATE SCHOOL
WWW.NPS.EDU

_________________________________________________________
WHERE SCIENCE MEETS THE ART OF WARFARE