Aquacultural Engineering 103 (2023) 102350

Contents lists available at ScienceDirect

Aquacultural Engineering
journal homepage: www.elsevier.com/locate/aque

Review

Research progress of computer vision technology in abnormal fish detection

Chunhong Liu a, b, c, d, e, *, Zhiyong Wang a, d, e, Yachao Li a, d, e, Zhenzuo Zhang a, d, e, Jiawei Li a, d, e,
Chen Xu a, d, e, Rongxiang Du a, d, e, Daoliang Li a, b, c, d, e, Qingling Duan a, b, c, d, e
a
National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China
b
Key Laboratory of Smart Farming Technologies for Aquatic Animal and Livestock, Ministry of Agriculture and Rural Affairs China Agriculture University, Beijing
100083, China
c
Beijing Engineering and Technology Research Center for Internet of Things in Agriculture, China
d
College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China
e
Laizhou Jinyiyuan Aquatic Products Co., Ltd., Shandong 261418, China

A R T I C L E I N F O A B S T R A C T

Keywords: Abnormal fish detection helps producers adjust breeding strategies in a timely manner to prevent the occurrence
Computer vision technology of diseases and improve aquaculture efficiency and quality. The rapid development of computer vision tech­
Deep learning nology provides a noninvasive method for abnormal fish detection, which can be used to identify and classify
Abnormal physiological activities
abnormal fish. This paper provides an overview of the research progress of computer vision techniques in
Abnormal trajectories
detecting abnormal fish over the past two decades. For the first time, the abnormal fish detection task is divided
Abnormal surface features
into three external manifestations: abnormal physiological activities, abnormal trajectories, and abnormal sur­
face features of fish. The traditional methods and deep learning methods in computer vision technology are
further summarized for their application approaches in these three research types, and the commonly used
classical algorithm models in abnormal fish detection are introduced comprehensively. In addition, this paper
summarizes several common methods for obtaining public datasets in aquaculture and evaluation indicators of
model accuracy, emphasizing two methods for researchers to collect experimental on-site data. Finally, based on
the above work, this paper analyzes several challenges in abnormal fish detection, proposes feasible strategies for
each challenge, and notes the importance of improving models to effectively integrate and analyze data from
multiple platforms. This paper provides some reference value for research on abnormal fish.

In the past, abnormal fish detection relied on visual inspection by

1. Introduction humans or judgements based on the experience of professionals, but this
method has many flaws, such as low efficiency, low accuracy, and high
Fish are an important food sources for humans, and the aquaculture subjectivity (Beitinger, 1990; Takahara et al., 2012; Wang et al., 2021).
industry is an important component of agricultural economies world­ To overcome the abovementioned challenges, acoustic (Føre et al.,
wide (FAO and F.A.A.D, 2018; Lauria et al., 2018; Sarà et al., 2018). 2017), sensor (Martos-Sitcha et al., 2019) and computer vision tech­
With the growth of population and economic development, the demand nologies (Davidson et al., 2011) have been successively applied in
for fish is constantly increasing, and the scale of aquaculture is also abnormal fish detection. Acoustic methods require placing a small
increasing, bringing new opportunities and challenges to fish farming acoustic tag inside the fish’s abdomen or on its exterior, and the received
(Asche, 2016; Yang et al., 2020b). However, during the fish farming signals can be used to assess the fish’s activity status. However, acoustic
process, due to the complex and uncontrollable underwater environ­ methods are susceptible to noise interference and are relatively expen­
ment, some abnormal situations may occur in fish growth, such as dis­ sive, which limits their application in abnormal fish behavior detection
eases, pollution, and parasites, which leads to a decrease in fish quantity, (Peck et al., 2021). The sensor method requires embedding sensors into
quality, fish welfare, causing considerable losses to the aquaculture in­ the fish’s skin or body, which can intuitively obtain important infor­
dustry (Zhao et al., 2021). Therefore, quickly and accurately detecting mation such as the fish’s respiratory rate, swimming speed, and blood
and identifying abnormal fish has become a hot issue in the fish farming glucose level (Cooke et al., 2016). However, due to the invasive nature
industry. of the sensor methods on fish, they will, to some extent, affect the health

* Corresponding author at: National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China.
E-mail address: [email protected] (C. Liu).

https://doi.org/10.1016/j.aquaeng.2023.102350
Received 13 May 2023; Received in revised form 28 July 2023; Accepted 30 July 2023
Available online 1 August 2023
0144-8609/© 2023 Elsevier B.V. All rights reserved.
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

Nomenclature YOLO You Only Look Once.

Models Evaluation Parameters

CBAM Convolutional Block Attention Module. ACC Accuracy.
CNN Convolutional Neural Network. AUC Area Under Curve.
DCNN Dynamic Convolution Neural Network. FN False Negative.
Faster R-CNN Faster Region-based Convolutional Network. FP False Positive.
GANs Generative Adversarial Networks. IOU Intersection Over Union.
GCN Graph Convolutional Networks. MSE Mean Square Error.
GMM Generalized Method of Moments. RMSE Root Mean Squared Error.
HSV Hue, Saturation, Value. ROC Receiver Operating Characteristic.
KM Kuhn Munkres. TN True Negative.
LSTM Long Short-term Memory. TP True Positive.
ORB Oriented Fast and Rotated Brief. Others
PCA Principal Components Analysis. CV Computer Vision.
ResNet Residual Neural Network. DL Deep Learning.
RNN Recurrent Neural Network. F4K Fish4–Knowledge.
RPN Region Proposal Network. LCF LifeCLEF.
SGD Stochastic Gradient Descent. NOAA National Oceanic and Atmospheric Administration.
SiamRPN Siamese Region Proposal Network. ROV Remotely Operated Vehicles.
SSD Single Shot MultiBox Detector. TL Transfer Learning.
SVM Support Vector Machine.

of the fish and data accuracy, especially for small-sized fish. Addition­ vision technology provides new solutions for many bottleneck problems
ally, it is very difficult to implant devices. Therefore, the low cost and in abnormal fish detection (Saberioon et al., 2017). This has made the
noninvasive characteristics of computer vision technology have obvious use of computer vision technology for abnormal fish detection a new
advantages in abnormal fish detection (Dell et al., 2014). research hotspot (Niu et al., 2018).
With the development of computer vision technology and the pro­ Abnormal fish detection involves various computer vision technol­
posal of intelligent aquaculture, many researchers have applied com­ ogies, such as image processing, feature extraction, object detection, and
puter vision technology to aquaculture. By using computers to analyze species classification. The development of these technologies has pro­
and process images or videos, some shortcomings of traditional methods vided strong support for detecting fish abnormalities (Xia et al., 2018).
can be overcome. Currently, applying computer vision technology in Currently, computer vision technology has formed a relatively complete
aquaculture is very extensive. In recent years, relevant review literature technical route in intelligent aquaculture, with classic methods available
on applying computer vision technology in aquaculture is shown in for each link (Anas et al., 2020). However, due to the complexity of
Table 1. Through analyzing fish feeding behavior (Hu et al., 2022), abnormal fish research, this field still faces many challenges, such as
estimating fish quality (Hao et al., 2022; Saberioon and Císař, 2018), data quality, types of fish abnormalities, underwater environmental
counting fish fry (Zhang et al., 2020; Zhao et al., 2022b), and classifying diversity, and real-time detection issues (Saleh et al., 2022), which have
species (Hu et al., 2012; Rauf et al., 2019),using computer vision tech­ posed higher demands for the application of computer vision technology
nology can achieve faster and more accurate measurements. Due to the in abnormal fish detection. Therefore, continuous optimization and
advantages of automation, high efficiency, and high accuracy, computer improvement of computer vision technology are needed, combined with
existing advanced technologies, to develop more intelligent, accurate,
and stable abnormal fish detection systems to meet practical application
Table 1 needs.
Review articles on the application of computer vision technology in aquaculture. The rapid development of deep learning has promoted the progress
Review article Author Year Related areas of computer vision technology, especially in recent years, and the
emergence of numerous visual models has helped solve many problems
Survey of fish behavior analysis by Niu et al. 2018 Fish behavior
computer vision (2018) analysis in aquaculture (Yang et al., 2020b). Deep learning is a machine learning
Computer vision models in intelligent Yang et al. 2020 Fish behavior method based on neural networks that can effectively extract high-level
aquaculture with emphasis on fish (2020a) analysis target features for recognition and classification. It has strong perfor­
detection and behavior analysis: A mance and robustness and extensive application prospects in detecting
review
Application of computer vision in fish An et al. 2020 Fish intelligent
fish abnormalities (Kaur et al., 2023; Sun et al., 2020).
intelligent feeding system: A review (2020) feeding Research papers on detecting abnormal fish in the last 20 years are
Automatic recognition methods of fish Li et al. 2020 Fish intelligent summarized in this paper, referring to Fig. 1, which shows the number of
feeding behavior in aquaculture: A (2020a) feeding publications and the highest accuracy achieved each year using tradi­
review
tional methods and deep learning methods in computer vision technol­
Overview of smart aquaculture system: Vo et al. 2021 Aquaculture
Focusing on applications of machine (2021) ogy for abnormal fish research from 2003 to 2022. In recent years, with
learning and computer vision the development of deep learning, research on abnormal fish detection
Application of machine learning in Zhao et al. 2021 Aquaculture has become increasingly prevalent. However, there is currently no sys­
intelligent fish aquaculture: A review (2021) tematic review of the application of computer vision technology in
A review of computer vision Li et al. 2021 Fish tracking
abnormal fish detection. Li et al. (2021a) provided a good review of
technologies for fish tracking (2021b)
Recent advances of target tracking Mei et al. 2022 Fish tracking using machine vision, acoustics, and sensors to detect fish stress
applications in aquaculture with (2022) behavior. This paper focuses on computer vision technology and
emphasis on fish abnormal fish, summarizing previous work to identify three criteria for

2
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

30 100%

Year's Highest Classifica�on Accuracy

90%

Cumula�ve Number of Thesis

25
80%

70%
20
60%

15 50%

40%
10
30%

20%
5
10%

0 0%
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022

Thesis Tradi�onal Methods in Computer Vision Deep Learning Methods in Computer Vision

Fig. 1. An overview of the development trend and performance of using traditional methods and deep learning methods in computer vision technology for abnormal
fish detection from 2003 to 2022. The bar chart on the left vertical axis shows the number of related research papers each year, and the line chart on the right vertical
axis shows the highest accuracy achieved in each year’s research.

judging abnormal fish and several commonly used algorithm models. technology in abnormal fish detection in the past 20 years. The structure
This study provides a valuable reference for future researchers in of the article is as follows. Section 1 introduces the background of fish
abnormal fish detection. farming, the demand for abnormal fish detection, and the current
This paper reviews the research progress of computer vision technology status. Section 2 summarizes the three criteria for judging

Fig. 2. Structure of the paper.

3
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

whether fish are abnormal in aquaculture, including abnormal physio­ The breathing frequency gradually decreases, and finally, the body sinks
logical activity, abnormal trajectory, and abnormal body surface fea­ to the bottom of the water, leading to breathing cessation. In addition to
tures. Section 3 summarizes the application of traditional methods in oxygen content, the water temperature can also cause changes in fish
computer vision technology in abnormal fish detection. Section 4 sum­ physiological activities. Each species of fish has its own lethal temper­
marizes the current mainstream deep learning models and discusses the ature. Studies have shown that mystus gulio exhibit increased breathing
research progress of deep learning methods in abnormal fish detection in and surface emergence frequency when exposed to higher temperatures,
computer vision technology. Section 5 provides two methods for as well as rapid movement, inversion, restlessness, and other phenom­
obtaining relevant datasets and some commonly used public datasets. It ena (Murugaian et al., 2008).
summarizes the evaluation indicators of model accuracy and some de­ Social pressure in fish groups is also a cause of abnormal physio­
tails of data collection in the field. Section 6 summarizes some chal­ logical activities in fish. Some fish have complex social systems and
lenges in the abnormal fish detection process and strategies for these hierarchical divisions among them, and this social pressure can have
challenges, providing new ideas for developing abnormal fish detection. inhibitory and fearful effects on fish, manifested as appetite suppression,
Section 7 is a summary of the entire work. The structure of this paper is aggression and swimming ability (Bose et al., 2019). Studies have shown
shown in Fig. 2. This review can provide helpful assistance for abnormal that when the cost of caring for offspring is very high, the plainfin
fish detection research. midshipman fish (Porichthys notatus) will adopt cost-reducing strate­
gies, such as self-mutilation and taking over nests, to increase their
2. Basis for abnormal fish detection in aquaculture chances of survival (Bose et al., 2016). Light intensity, growth differ­
ences, and hunger levels can also lead to fish killing each other, which is
Classifying abnormal fish has always been a relatively vague a common physiological phenomenon in the grouper (Epinephelus) ju­
concept, as it is usually caused by factors such as water quality, disease, venile period (Smith and Reay, 1991). Automatically monitoring the
and competition pressure within the fish group. Therefore, abnormal above abnormal physiological activities can promptly notify breeders to
fish behavior can be detected to determine if there are any problems. take corresponding measures to avoid losses in fish fry quantity and
Wang et al. (2020) noted that abnormal fish behavior is usually a improve economic benefits.
symptom of fish illness or stress. This statement can only summarize Fish feeding behavior and inbreeding behavior are important di­
some abnormal fish behaviors and cannot serve as a comprehensive rections for research on abnormal physiological activities of fish. Pre­
basis for categorizing all abnormal behaviors of fish. Therefore, some dation and mating are two factors that promote abnormal behavior. In
researchers have divided abnormal fish behavior into biological stress research, recognizing fish feeding behavior can be used to determine
and nonbiological stress based on the type of pressure, i.e., whether it is whether the fish have feeding needs (Zhang et al., 2022), providing a
caused by the environment or other organisms. This method can basis for aquaculture managers to control the amount of feed to a certain
comprehensively explain the reasons for various abnormal behaviors (Li extent. However, inbreeding behavior between fish is time-consuming
et al., 2021a), but it is not consistent with the purpose of this article, and has a small reproductive capacity but a high economic value (Du
which focuses on using computer vision technology to study abnormal et al., 2022). Additionally, fish groups are usually fragile during the
behaviors in fish rather than the reasons for their occurrence. breeding process and are easily affected by the environment, so timely
To better address the issues related to the abnormal fish classification observation and protection by fish farmers are required during the
mentioned above, this article proposes three criteria for judging whether spawning period. Identifying these special behaviors is a necessary
fish are abnormal, namely, abnormal physiological activities, trajec­ method for achieving abnormal fish detection in intelligent aquaculture.
tories, and surface features. In the wild underwater environment, The physiological activities of abnormal fish are shown in Fig. 3.
different water quality factors such as dissolved oxygen, temperature,
salinity, and ammonia nitrogen are constantly changing. When they 2.2. Abnormal trajectory of fish
exceed normal ranges, fish often exhibit abnormal physiological activ­
ities (Bao et al., 2018), such as changes in tail beat frequency, jumping, Tracking fish trajectories can effectively determine whether they are
biting, and flipping. In addition, in a fish group, mutual killing behavior abnormal. In trajectory recognition, water quality deterioration can
may occur during the juvenile stage or during feeding, and detecting alter fish movement. Xu et al. (2020) analyzed the effect of ammonia
abnormal fish group trajectories can prevent this phenomenon (Wang concentration on goldfish trajectory by drawing the behavioral goldfish
et al., 2022a). When affected by diseases such as fish ulcerative syn­ trajectory in different ammonia concentration environments. The results
drome, tail fin rot, bacterial gill disease, and parasitic disease (Sikder showed that as the ammonia concentration increased, goldfish activity
et al., 2021), the skin of the fish may have spots or changes in color, and decreased rapidly. In addition, the competitive behavior among indi­
in severe cases, may cause the fish to flip over or die. vidual fish can also affect the fish trajectory. During the feeding process,
Based on the above analysis, this section focuses on reviewing 14 researchers found that the fish activity level was higher when there was
articles that reflect abnormal fish behavior in aquaculture. Through the a feeding requirement, and in situations of insufficient bait, they will
summary and analysis of these articles, the above classification method engage in fights and competition for resources (Zhang et al., 2022).
can effectively summarize the main research directions of current Tracking technology can be used to track the trajectory of fish groups
abnormal fish detection work and provide a reference for future and determine the fish feeding status to control the feeding amount.
research. Studies have shown that porphyry seabream (Sparidentex hasta) with
poor health status can also exhibit abnormal trajectories (Zhao et al.,
2.1. Abnormal physiological activities of fish 2022a). In individual cases, fish may even flip over during the struggle
for flow, which leads to a weak trajectory of sick or dead fish. Therefore,
Adverse changes in aquatic environments can directly lead to abnormal fish detection can be achieved by extracting fish trajectories.
abnormal physiological activities of fish. In underwater environments Abnormal fish trajectories are shown in Fig. 4.
with low dissolved oxygen content, researchers have observed that the
escape speed and lateralization advantage of sebastes caurinus signifi­ 2.3. Abnormal surface features of fish
cantly decrease (Mattiasen et al., 2020). Under conditions of severe
hypoxia, Ishibashi et al. (2002) observed that the breathing frequency of Distinguishing abnormal fish based on changes in their body surface
nile tilapia (Oreochromis niloticus) would first reach the highest level, is another important method for identifying abnormal fish caused by fish
accompanied by a sharp increase in plasma cortisol and glucose levels, diseases and fighting among fish groups. In fish diseases, diseased fish
followed by a significant increase in lactate levels in muscles and liver. often show symptoms such as irritability, anorexia, respiratory distress,

4
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

Fig. 3. Abnormal physiological activities of fish. (a) represents cannibalistic fish behavior, (b) represents fish side-flipping behavior, and (c) represents chasing
behavior between fish (Du et al., 2022; Wang et al., 2022a; Zhao et al., 2022a).

Fig. 4. Abnormal fish trajectories. (a) shows the trajectories of fish under high water temperature, (b) shows the trajectories of fish after being attacked, and (c)
shows the trajectories of fish in a state of hunger (Beyan, 2015).

Fig. 5. Seventeen manifestations of abnormal fish body surfaces (Waleed et al., 2019).

5
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

and changes in body surface color. One typical symptom of fish fin then find the optimal hyperplane in that space to separate the data
nematode disease is the appearance of white spots on the skin (Peyghan points of different classes as much as possible. In classification problems,
et al., 2021). These typical characteristics of diseased fish provide a basis SVM finds an optimal hyperplane that maximizes the distance between
for abnormal fish detection. Moreover, studies have shown that wounds different class samples and the hyperplane. In regression problems, SVM
on the surface of fish and fish lice also increase the risk of fish infection finds an optimal hyperplane that minimizes the distance between the
(Gupta et al., 2022; Zhu and Li, 2007). Therefore, timely detection of sample points and the hyperplane.
abnormal body surfaces in fish can greatly reduce fish mortality and In abnormal fish detection, SVM algorithm can be used for the
minimize losses in aquaculture. The seventeen manifestations of classification of normal fish and abnormal fish. SVM algorithm has good
abnormal fish body surfaces are shown in Fig. 5. generalization performance and interpretability, and can effectively
handle nonlinear problems in high-dimensional space. However, the
3. Abnormal fish detection based on traditional methods in limitation of SVM algorithm is the long training time when facing large-
computer vision technology scale datasets, and it is also likely to face the problem of dimensionality
catastrophe when dealing with high-dimensional data.
Before the rise of deep learning methods, traditional methods in
computer vision technology were dominant in aquaculture (Zion, 2012). 3.1.2. K-Means clustering
Computer vision technology refers to the detection and recognition of K-Means clustering is a commonly used unsupervised learning al­
abnormal fish through image processing such as feature extraction and gorithm (MacQueen, 1967). Its basic idea is to divide n data points into k
classification (Niu et al., 2018). This method usually includes three main clusters, so that the similarity within the cluster is as high as possible,
steps: image preprocessing, feature extraction, and classification (Sze­ and the similarity between different clusters is as low as possible. The
liski, 2010). In the preprocessing stage, techniques such as image two main steps of the algorithm are initialization and iterative update. In
enhancement, filtering, and binarization are usually used to improve the initialization phase, k initial points are selected as centroids, and
image quality and reduce noise interference. In the feature extraction each data point is assigned to the cluster where its nearest centroid is
stage, visual features such as color, texture, and shape are typically used, located. In the iterative update phase, the new centroids of each cluster
and mathematical models are used to calculate and analyze features are calculated, and each data point is reassigned to the cluster where its
related to abnormal fish. In the classification stage, methods such as nearest centroid is located, and this process is repeated until the cen­
support vector machines and neural networks are commonly used to troids no longer change or the predetermined iteration times are
train and classify extracted features, to achieve abnormal fish detection reached.
and recognition (Yasruddin et al., 2022). The basic architecture of using In abnormal fish detection, K-means clustering algorithm is often
computer vision technology for abnormal fish detection is shown in used for cluster analysis and feature extraction of fish image data to
Fig. 6. This section briefly introduces six traditional methods in com­ achieve abnormal fish detection and classification. For example, image
puter vision technology used for abnormal fish detection, among which data can be extracted for features such as color and texture, and K-means
SVM, K-means, and PCA are classification methods, while optical flow, clustering algorithm can be used for image clustering to detect abnormal
frame difference, and background subtraction methods are motion ob­ fish. However, in practical applications, K-means clustering algorithm
ject detection methods. This section summarizes and analyzes a large still needs to be combined with other algorithms and techniques to
number of relevant references and discusses abnormal physiological improve its accuracy and stability.
activities, abnormal trajectories, and abnormal surface features,
providing a reference and guidance for subsequent research. 3.1.3. Principal component analysis
Principal component analysis (PCA) is a commonly used data
3.1. Traditional models for abnormal fish detection dimensionality reduction algorithm. It maps high-dimensional data to a
low-dimensional space, preserving the main information while mini­
3.1.1. Support vector machine mizing information loss (Jolliffe and Cadima, 2016). The basic idea is to
Support vector machine (SVM) is a supervised learning algorithm map the original data to a new coordinate system, where the variance of
widely used in classification and regression problems (Cortes and Vap­ the data is maximized, thereby preserving the main features of the data.
nik, 1995). Its basic idea is to map data to a high-dimensional space and In the mapping process, the basis vectors of the new coordinate system

Fig. 6. Basic architecture for abnormal fish detection based on computer vision.

6
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

are the eigenvectors of the original data after feature value decompo­ be performed for specific scenes to achieve better detection results.
sition, and the mapped data is the projection of the original data in the
new coordinate system. 3.2. Detection methods for abnormal physiological activities of fish
PCA algorithm has applications in many fields, such as image pro­
cessing, signal processing, and pattern recognition. In abnormal fish Abnormal physiological activities of fish can be discerned through
detection, PCA algorithm can effectively distinguish normal fish from observable indicators, such as swimming speed, swimming patterns, tail
abnormal fish by feature extraction and dimensionality reduction of fish beat frequency, and respiratory rhythm. Traditional methods in com­
images, thereby improving the accuracy of detection. puter vision technology can effectively capture and analyze these
features.
3.1.4. Optical flow Changes in the environment and physical health can lead to signifi­
Optical flow is a method for estimating object motion by calculating cant changes in fish swimming speed. Analyzing and measuring the
the distance and direction of each pixel’s movement in the image swimming speed of individual fish or fish groups can effectively deter­
sequence over time (Sun et al., 2010). The basic idea is to use the mine their health and activity levels, thus providing timely alerts for
brightness changes between pixels to reflect the object’s motion over abnormal situations. To detect explosive swimming behavior in fish
time. By comparing the position and gray value of each pixel in two caused by environmental stimuli, researchers used the difference
adjacent images, the movement direction and velocity of the point on method of images to represent the feature values of sudden acceleration
the image can be calculated. In practical applications, typical motion in fry of farmed-raised bluefin tuna (Thunnus thynnus) and used the
characteristics of different types of fish can be obtained through statis­ obtained feature values for discrimination between normal and
tical analysis of the optical flow field, thus further improving the accu­ abnormal behavior. This enabled the detection of individual fish fry
racy and robustness of abnormal fish detection. using video scenes (Abe et al., 2018). Although the abovementioned
However, it should be noted that the calculation process of optical method can accurately measure the speed of individual fish, in actual
flow is complex, requiring a lot of computation for image sequences. aquaculture environments, fish in the breeding pond tend to be densely
Therefore, when processing large-scale image data, issues of computing packed, so detecting abnormal swimming speed in fish groups becomes
resources and time need to be considered. Additionally, due to the high more important. Yu et al. (2021) used the Harris corner detection
requirements for image quality and lighting conditions, image pre­ method to extract feature points for special behavior, and then used the
processing and correction are necessary for obtaining more accurate Lucas Kanade optical flow method to obtain the speed of cyprinus carpio
results in practical applications. with a body length of 6–7 cm in sub-images. This ultimately enabled
judgement of whether the fish group’s speed was abnormal, with an
3.1.5. Temporal difference average precision of 96.02%.
Temporal difference is a commonly used video detection technique The swimming mode, activity status, and tail-beat frequency of fish
that detects the presence of motion objects in a video by comparing the can intuitively reflect their abnormal conditions. Employing appropriate
differences between adjacent frame pixels (Singla, 2014). Its main idea computer vision algorithms to analyze video images, precise detection of
is to compare the current frame image with one or more previous frames fish’s abnormal activities can be achieved. In order to analyze fish
at the pixel level. If the difference between the pixel values at the same swimming mode, Serra-Toro et al. (2010) used the background sub­
position in two frames exceeds the set threshold, the pixel point is traction method to detect each fish in each video frame and calculated
considered to be in the motion area of the target. This method is simple, fish relative motion from the centroids of the fish detected in consecu­
fast, and robust to factors such as illumination and background changes, tive frames, thereby extracting the abnormal swimming states of fish
making it widely used in real-time video monitoring, motion object from these sequences of discrete movements. However, the accuracy of
tracking, and other fields. the above method will be significantly affected when the background
In abnormal fish detection, the temporal difference algorithm can be image is too complex. To solve this problem, Gaussian mixture models
used to detect the movement of fish schools, such as the speed and di­ (GMM) are used to process fish images, which can be used for back­
rection of fish school movement. By setting an appropriate threshold, ground modeling and motion foreground extraction, so as to achieve
the moving area of the fish school can be separated from the stationary automatic judgment of image preprocessing and morphological pro­
area, making it easier to detect anomalies. In addition, the temporal cessing, and then achieve the purpose of detecting fish activity states
difference can also be used to detect the interaction behavior between (Lin et al., 2016). Fish movement is a continuous process, and contin­
fish schools and other moving objects, such as being chased by predators uous tracking of key fish is required to accurately detect and capture
or attacking each other. abnormal fish. Therefore, particle filtering is used for fish tracking.
Based on tracking, researchers estimate the tail-beat frequency of each
3.1.6. Background subtraction fish according to the movement of the tail tip, thereby monitoring
Background subtraction is a commonly used computer vision method whether the fish is abnormal (Terayama et al., 2017).
for detecting moving objects in images or videos (Maddalena and Pet­ The collective behavior, respiratory rhythm, and other details in fish
rosino, 2008). It is mainly based on the comparison between foreground movement process can serve as criteria for judging whether fish are
objects and the background. By statistically analyzing the pixels in the abnormal, holding significant value in detecting the health status of fish.
image sequence, it determines whether the pixel is a foreground pixel, For some details in the fish movement process, such as acceleration,
and then performs object detection through clustering, contour extrac­ total distance traveled, and angle changes, Kane et al. (2004) designed a
tion, and other algorithms on the foreground pixels. video analysis system to obtain behavior endpoints carrying different
In abnormal fish detection, background subtraction can be used to fish information by digitizing video data, transforming them into points
detect moving fish that are different from the background, such as tar­ in a coordinate system, and tracking fish. The above method provides a
gets with different swimming directions, sizes, and colors compared to framework for similar research, but the defect is that there is no rigorous
other fish. However, the accuracy and robustness of background sub­ algorithm to support it, and the subjectivity of the method is strong. To
traction depend largely on the accuracy of the background model and its overcome the above problems, Zheng et al. (2014) used different com­
ability to adapt to changes in illumination. If there are significant puter vision algorithms of image processing methods such as support
changes in the background, such as the shaking of water plants or vector machine, adaptive boosting and mathematical morphology to
changes in lighting, the effectiveness of background subtraction may be design a remote experimental platform, which achieved real-time
affected. Therefore, when using background subtraction for abnormal detection of fish respiratory rhythm. However, in some special envi­
fish detection, reasonable parameter settings and model training need to ronments, it is necessary to extract the distribution status and movement

7
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

characteristics of fish groups. Zhao et al. (2016) successfully improved changes that occur during their movement, while surface features of fish
the kinetic model of fish detection in the entire shallow area by using focus more on obtaining and processing information on the fish’s skin.
optical flow, entropy, and statistical methods. By combining dispersion Abnormal surface features of fish often result from diseases, skin damage
measurement with changes in behavioral characteristics, the existing caused by biting and fighting, etc. Capturing the information in a timely
model was corrected, greatly improving the accuracy of the model in manner can help improve the survival rate of fish.
fish group abnormal detection. Fish diseases often lead to various abnormal changes in the surface of
fish. Each type of fish disease has different external manifestations.
3.3. Detection methods for abnormal trajectories of fish Therefore, to achieve differentiation of different types of fish diseases,
Miao-jun et al. (2013) combined rough set theory and fuzzy c-means
Abnormal trajectories detection typically involves extracting fish clustering algorithm, and finally effectively solved the problem of
swimming trajectories from videos and using appropriate algorithms to automated classification of fish diseases. Since accurate detection of fish
train and determine whether the trajectories are abnormal. This enables diseases is the basis for achieving disease classification, Paul (2015) used
early diagnosis and monitoring of fish, helping to prevent and control principal component analysis and k-means clustering method to com­
the spread of fish diseases. In trajectory detection, a method using a plete the recognition and segmentation of diseased areas in fish images,
floating grid can be used to extract fish movement trajectories. Based on and used HSV and morphological operations in the diseased area to
these trajectories, a neural network mechanism can quickly determine ultimately improve disease recognition accuracy and calculation of
the frequency of pattern changes, ultimately achieving the extraction of disease area.
abnormal trajectories (Ma et al., 2010). In contrast, Beyan and Fisher The aforementioned methods can achieve the detection and classi­
(2012) designed a rule-based trajectory filtering mechanism that uses 21 fication of fish diseases to a certain extent, but the diagnosis of fish
defined filters to continuously reject normal trajectories until the diseases is a more cumbersome task, requiring the selection of appro­
remaining part of the last filter is all abnormal trajectories. This method priate algorithms in the processing of images, diagnosis of diseases, and
greatly reproduces the trajectory activity of fish, but filter selection classification of diseases in order to build an accurate and fast disease
depends largely on subjective judgment and the method is relatively detection system. To achieve real-time detection of fish diseases, re­
cumbersome and difficult to automate. To address these issues, Beyan searchers combined principal component analysis with an accelerated
and Fisher (2013) used principal component analysis to extract features segment test feature detector based on machine learning algorithms and
from trajectories, and then performed clustering and outlier detection on finally developed a high-precision real-time fish disease diagnosis sys­
the extracted feature subset to detect fish abnormal trajectories using tem with an accuracy rate of 86% (Malik et al., 2017). However, the
multiple features in unconstrained underwater videos. above accuracy rate does not meet the needs of actual production.
Continuous advancements to fish detection and tracking algorithms Therefore, in order to further improve the accuracy of fish disease
in videos have improved the accuracy of extracting abnormal fish tra­ detection and classification, Adl (2019) used the ORB algorithm to
jectories, making progress in detection and tracking technology the extract features from preprocessed images, and classified diseased fish
foundation for studying fish abnormal trajectories. To further enhance and normal fish using a logistics regression machine learning algorithm
the accuracy of feature extraction in detection and tracking algorithms, combined with ant colony optimization algorithm, yielding a model
Tian and Chen (2017) analyzed fish kinematic features in videos and accuracy rate of 92.8%.
constructed a new feature called spatiotemporal local kinematic pattern In addition to causing changes in the physical features of fish, fish
(STLKP) by extracting spatiotemporal features. The statistical informa­ diseases can also lead to damage to their epidermis through mutual
tion of the proposed STLKP feature was integrated into the traditional biting, and fish with damaged epidermis generally have a higher inci­
SVM classifier, achieving the classification of abnormal trajectories. In dence of disease, so timely monitoring and response measures are
addition to improving the feature extraction accuracy, appropriate necessary. To detect the extent of damage to fish, Sheng et al. (2016)
detection and tracking algorithms can be selected to track fish. Zhao proposed two methods for detecting the surface damage rate of fish
et al. (2019) used an Otsu adaptive segmentation algorithm based on based on the k-means clustering algorithm and the Otsu threshold al­
background subtraction to extract fish targets, estimated motion states gorithm. The study showed that the use of neural networks, genetic al­
using a Kalman filter, and ultimately achieved multiple fish tracking of gorithms, and other methods can improve model performance,
red porgy (Pagrus major) using the inter-frame relationship matrix. robustness, and accuracy. During fish fights, changes in fish gills often
Although the fish tracking technology has formed a relatively com­ occur, and Sweidan et al. (2015) used edge detection and wavelet
plete research system, there are still some problems that need to be transform methods to extract the shape features of fish gills and ach­
solved urgently. For example, during the process of tracking fish schools, ieved classification of abnormal fish gill shapes through principal
there will be mutual occlusion between fish, which can affect the ac­ component analysis and support vector machine methods.
curacy of tracking and even cause loss of target. To solve the above
problems, Spampinato et al. (2008) used an underwater fish behavior 3.5. Summary
monitoring method that combines three-dimensional motion trajectory
synthesis and integrated learning. They used the Kuhn Munkres (KM) Table 2 provides details of the literature on abnormal fish detection
algorithm to match the target points of fish bodies, updated the current based on traditional methods in computer vision technology. The
state with a Kalman filter, and found the best tracking position as the advantage of these methods is that they are simple to use, require fewer
tracking result, thereby achieving tracking of occluded targets. How­ computing resources, and have low requirements for data volume and
ever, due to the high complexity of three-dimensional imaging, the ac­ quality. However, there are some problems in the process of imple­
curacy of past related research has generally been low. In the study, Xiao menting abnormal fish classification using traditional methods. First,
et al. (2016) used the principle of planar mirror imaging and camera since fish have different appearance features under different angles and
imaging to find the relationship between mirror-targets and lighting conditions, traditional methods have difficulty accurately
direct-targets, and calculated the three-dimensional coordinates of fish extracting fish features. Second, traditional methods require the manual
under occlusion conditions, thereby achieving occlusion tracking of fish design of feature extraction algorithms, which requires considerable
using Kalman filtering. professional knowledge and experience, and is time-consuming. Finally,
traditional methods have limitations in complex fish image classification
3.4. Detection methods for abnormal surface features of fish tasks and it is difficult to achieve high-precision classification. There­
fore, it is necessary to explore new methods to improve the shortcomings
Physiological activities and trajectory features of fish are dynamic of traditional methods and thus improve the accuracy and speed of

8
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

Table 2
Research on abnormal fish detection based on traditional methods in computer vision technology.
Type Manifestation References Year Scenes Models/Algorithms/ Results Comparisons with
Methods other methods

Abnormal Active state Lin et al. (2016) 2016 Lab Gaussian mixture model Easy to measure the No
physiological abnormal condition
activity
Special behaviors Zhao et al. 2016 Lab Optical flow、Entropy and Detection accuracy： No
(2016) Statistics 97.20 ± 1.23%
Tail Beat Frequency Terayama et al. 2017 Pond Particle filter Average errors:0.126 Hz No
(2017)
Swimming speed Yu et al. (2021) 2021 Pond Harris corner detection、Lucas- Detection accuracy：96.2 SGM and the kinetic
Kanade optical flow energy model
Abnormal trajectories Trajectories Ma et al. (2010) 2010 Lab Floating-grid、Neural network Effectively distinguish No
mechanism trajectories
Trajectories Xiao et al. 2016 Lab Kalman filter Enables tracking in turbid No
(2016) environments
Trajectories Zhao et al. 2019 Pond Otsu adaptive segmentation、 Improved performance and Temporal difference
(2019) Kalman filter robustness
Abnormal body Fish disease Paul (2015) 2015 Lab K-means clustering Greater precision No
surface features
Fish disease Sikder et al. 2021 Pond C-means fuzzy Detection accuracy： K-means clustering
(2021) logic clustering、Gabor’s 97.90%
Filters
Injured rate of fish Sheng et al. 2016 Pond K-means clustering Greater precision Otsu’s threshold
surface (2016) algorithm

abnormal fish detection. abnormal fish detection, among which Faster R-CNN and YOLO are
object detection algorithms, LSTM and SiamRPN+ + are mainly used for
4. Abnormal fish detection based on deep learning methods in tracking in abnormal fish detection, and ResNet and MobileNet are
computer vision technology commonly used in algorithm improvement. In most aquaculture tasks,
combining multiple models is often necessary to achieve better results,
To address the problems brought by traditional methods in computer and this is also the case in the abnormal fish detection process.
vision technology, deep learning algorithms have been introduced into
abnormal fish detection. Deep learning is a machine learning method
based on multilayer neural networks that can learn high-level data 4.1. Deep learning models for abnormal fish detection
representations (Saleh et al., 2022). By adjusting the connections and
weights between multiple neurons layers, deep learning methods can 4.1.1. Faster R-CNN
automatically learn multiple abstract feature hierarchies from input Faster R-CNN is a deep learning-based method for object detection in
data, effectively extracting features from various forms of data such as the field of computer vision, proposed in 2015 (Ren et al., 2015). The
images, audio, and text (Hao et al., 2016). Among them, convolutional algorithm employs two modules to perform object detection: Region
neural networks (CNNs) play an important role in object detection. The Proposal Network (RPN) and Fast R-CNN. RPN is the core module of the
basic structure of a CNN consists of an input layer, convolutional layer, Faster R-CNN algorithm, which performs classification and regression
pooling layer, fully connected layer, and output layer, as shown in Fig. 7. on each position of the image using a sliding window approach to obtain
The use of the CNN model structure has completely changed the appli­ a series of candidate regions. Anchors are fixed-size and aspect-ratio
cation of computer vision technology in aquaculture, and researchers bounding boxes in the original image. By adjusting anchors of
have shifted from manual feature extraction to automatic feature different sizes and aspect ratios, RPN can generate multiple candidate
extraction, resulting in a significant improvement in detection accuracy boxes of different sizes. After obtaining candidate boxes, the Faster
and speed. The difference between traditional methods and deep R-CNN module performs classification and position regression on them
learning methods in computer vision technology is shown in Fig. 8. to determine the presence of objects in each box and refine the position
In recent years, deep learning methods in computer vision have been of the object box.
widely studied and applied in intelligent aquaculture (Sun et al., 2020). The advantage of Faster R-CNN algorithm is that it can simulta­
The development of various deep learning models has provided possi­ neously generate candidate boxes and perform object detection, which
bilities for further improving fish detection and tracking accuracy. The results in faster detection speed and higher accuracy compared to
timeline of classic models and lightweight networks is shown in Fig. 9. traditional object detection methods. Furthermore, since deep learning
This section briefly introduces six important algorithm models in methods are used, the algorithm has good robustness in processing
large-scale data. The Faster R-CNN algorithm has been widely applied in

Fig. 7. Schematic diagram of CNN architecture.

9
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

Fig. 8. The main differences between traditional methods and deep learning methods in computer vision technology. The upper part shows manual feature
extraction for detecting parasitic diseases, while the lower part shows automatic detection and classification of fish skin diseases using deep learning models.

Fig. 9. Summary of deep learning algorithm models from 2014 to 2023. The upper half shows classic algorithms in the fields of object detection and tracking, while
the lower half shows common lightweight model structures.

detecting various abnormal fish behaviors, with high detection accuracy 4.1.3. LSTM
and real-time detection capability, which can effectively assist fisher­ Long short-term memory (LSTM) is a type of recurrent neural
men in management and fishing. network that is designed to address the challenge of modeling long se­
quences faced by traditional recurrent neural networks (Hochreiter and
4.1.2. YOLO Schmidhuber, 1997). Compared to traditional RNNs, LSTM has better
You only look once (YOLO) is a deep learning-based object detection memory performance and can better capture the temporal dependencies
model (Redmon and Farhadi, 2018). Compared to traditional object of long sequences. It has been widely used in natural language pro­
detection models such as RCNN series and Single Shot MultiBox Detector cessing, speech recognition, image description generation, and other
(SSD), YOLO has faster detection speed and higher accuracy. The basic fields.
idea of the YOLO model is to divide the image into grids and predict In abnormal fish detection, the LSTM model can be used to model the
whether there is an object, the class of the object, and its position in each movement trajectory of fish in water and then determine whether the
grid. Its core is to use CNN to extract image features and use multi-layer behavior of the fish is normal. For example, the position coordinates of a
fully connected layers to predict object position and class. Compared to fish over a period of time can be inputted, processed through multiple
traditional object detection methods, YOLO can perform detection and LSTM units, and finally output the abnormal movement state of the fish.
localization in a single network, reducing redundant computation and The LSTM model has broad application prospects in abnormal fish
speeding up the detection process. Currently, the YOLO series networks detection, effectively solving the problem of modeling long sequences
have developed to the YOLOv8. and providing more accurate, efficient, and intelligent detection
In aquaculture, by training the YOLO model with a large number of methods for the fish farming industry.
abnormal fish samples, the model can quickly and accurately detect
abnormal fish in images, improving detection efficiency and accuracy. 4.1.4. SiamRPN+ +
Moreover, the YOLO model can achieve real-time detection, which can SiamRPN+ + is an algorithm in the field of target tracking, and its
be applied in the real-time monitoring system of fish farms to promptly core is based on the RPN-based tracker (Cao et al., 2021). Compared to
detect abnormal situations and take measures, thus improving aqua­ traditional classifier-based trackers, SiamRPN+ + algorithm has faster
culture efficiency and reducing losses. speed and higher accuracy. The main contribution of this algorithm is to
propose an adaptive branch structure that can adaptively select the best
branch under different target scales. At the same time, the algorithm also

10
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

introduces a loop feedback mechanism between the template and search MobileNetV3 network to reduce the number of parameters. They
branches, enabling the tracker to better adapt to target changes. improved the dead fish detection accuracy through deformable convo­
In abnormal fish detection, the SiamRPN+ + algorithm can predict lution and achieved improvement on the YOLOv4 model. The final dead
the location of the abnormal fish in the next frame based on the detec­ fish detection accuracy reached 95.47%. However, the method’s limi­
tion results of the previous frame, and make fine adjustments to more tation is that it can only distinguish abnormal turning over behavior
accurately identify the abnormal fish. In addition, the SiamRPN+ + after death from normal turning over behavior during fish swimming by
algorithm can also track abnormal fish, even in cases of occlusion or detecting static images. To address the above issue, an improved neural
scene changes, thus improving the robustness of the detection. network model was used to detect and track the turning over behavior of
the cultured porphyry seabream (Sparidentex hasta). The approach
4.1.5. ResNet improved the YOLOv5s network by merging multi-level features and
ResNet is a classic deep neural network model mainly used for image adding feature mapping in the object detection part. In the object
classification and object detection tasks (He et al., 2015). It mitigates the tracking part, the SiamRPN+ + single-object tracking algorithm was
problem of gradient vanishing in deep neural networks by introducing iteratively implemented to track multiple abnormal fish, which effec­
residual blocks, allowing the network to learn features at a deeper level. tively judged the presence of abnormal turning over behavior of fish
In ResNet, each residual block contains multiple convolutional layers (Wang et al., 2022b).
and identity mapping. Identity mapping refers to directly passing the Feeding behavior and reproductive behavior are special behaviors of
input data to the output of the residual block, enabling the direct fish, and if not intervened positively in aquaculture, they will lead to
learning of the features of the residual part. abnormal fish. Researchers can better control feeding amounts by
An important contribution of ResNet is solving the problem of detecting feeding behavior, thus saving aquaculture costs (Zheng et al.,
gradient vanishing during the training process of deep neural networks, 2023). Detection of reproductive behavior can better track high-quality
allowing the network to be well-trained to a deeper level. In addition, breeding fish and provide better breeding conditions through artificial
ResNet is widely used in various computer vision tasks, including object intervention during their breeding period, thereby increasing breeding
detection, image classification, and image segmentation. ResNet is often success rates (Du et al., 2023). To detect special fish behaviors, re­
used as the base network structure and by adding some adaptive or searchers use the movement characteristics of fish groups and the
specific layers on top of it, it can effectively extract fish feature infor­ swimming postures of individuals in the fish group to construct fish
mation, thereby achieving more accurate abnormal detection. group images. Then, the graph convolutional neural network (GCN)
model is used to train the dataset, achieving a detection accuracy of
4.1.6. MobileNet 97.3% (Huang et al., 2022). However, different behaviors will have
MobileNet is a lightweight CNN proposed by Google in 2017 certain differences. For precise identification of fish feeding behavior,
(Howard et al., 2017). Compared with traditional CNN, MobileNet uses Zhang et al. (2022) introduced SENet into the original MobileNetV2
depthwise separable convolution to reduce the size and computational feature extraction network and constructed a feature extraction and
complexity of the model. Depthwise separable convolution consists of weighting network that integrates MobileNetV2 and SENet, achieving a
two steps: depthwise convolution and pointwise convolution. First, a detection accuracy of 97.76%. The above research has achieved good
conventional convolution is performed on the depth, and then pointwise detection of the special behaviors of fish, but they focus on processing
convolution is performed on each depth. This method can significantly the image features and ignore the time information contained in the
reduce the number of parameters of the model, thus improving the behavior. Therefore, Du et al. (2022) applied the ResNet50-LSTM
running speed of the model on mobile devices. Currently, the MobileNet method in the study of fish behavior, using ResNet50 to obtain the
series network has developed to MobileNetV3 version. features of different dimensions of data and using the LSTM network to
When applying deep learning models on mobile devices, the size and extract behavioral information in time series. Finally, the detection ac­
computational complexity of the model are crucial. MobileNet reduces curacy of fish behavior during the breeding period reached 98.52%.
the size of the model by using depthwise separable convolution while Due to the influence of group pressure, water environment, food and
maintaining good accuracy, making it very suitable for deployment on other factors, local abnormal behaviors often occur in a certain area
mobile devices. An important application of MobileNet is image classi­ among fish groups, such as local attacking and fighting behavior, which
fication, but it can also be applied to object detection, semantic seg­ is one of the important factors causing losses in aquaculture. To address
mentation, and other tasks. Based on this, MobileNet can be used for the identification of local abnormal fish behavior, Zhao et al. (2018)
abnormal fish detection in scenarios with limited resources such as designed a fish local abnormal behavior detection and localization
embedded devices. In addition, since the morphology of fish is variable method based on the revised motion impact map and recursive neural
and different types of fish need to be classified and detected, MobileNet network (RNN) by comprehensively considering factors such as speed,
can improve the recognition rate of different types of fish by fine-tuning direction, distance, visibility, and information transmission mode.
the model parameters during the training process. However, the above research does not have a complete deep learning
model as a basis, and its accuracy still needs to be further improved. In
4.2. Detection methods for abnormal physiological activities of fish another study aimed at effectively identifying self-killing behavior of
fish in fish groups, Wang et al. (2022a) used a multi-head attention
The application of deep learning technology has facilitated the mechanism in the YOLOv5 backbone network, improved the feature
detection of complex fish activities, such as turning over behavior, fusion performance based on the idea of BIfpn, and used the up-sampling
fighting behavior, feeding behavior, and breeding behavior. Further­ operator Carafe to replace the original up-sampling method, solving
more, the powerful feature extraction capability of CNN allows re­ problems such as small targets, severe occlusion, and target blur in fish
searchers to study fish physiological activities from both local and group images, making the accuracy of detecting fish self-killing behavior reach
behavior perspectives. 97%.
The vertical turning over of fish can be divided into normal turning In the context of fish detection, diverse research objectives necessi­
during swimming and abnormal turning after sickness or even death. tate the selection of appropriate research subjects and the pre-
Accurately detecting abnormal vertical turning over behavior can help determination of target fish based on the specific study characteristics.
take timely measures for sick or dead fish, avoiding water quality Whether it involves detecting fish group activity or individual fish ac­
deterioration and the spread of fish diseases. To identify the phenome­ tivity, a well-defined research plan is crucial. For detecting fish feeding
non of dead fish symbolized by fish turning over, Zhao et al. (2022a) status and group stimulation status, fish groups need to be selected as
replaced the feature extraction backbone network with a lightweight research objects. To achieve the detection of group behavior, Han et al.

11
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

(2020) used the Lucas Kanade optical flow method to extract optical great difficulties in tracking multiple fish, and there is an urgent need to
flow energy maps, sent the spatiotemporal fusion images to CNN for improve the quality of underwater images and adjust and optimize
training, learning, and testing. By combining the spatial information of traditional deep learning algorithms to overcome the challenges of un­
fish group distribution with the time information reflected in motion derwater fish tracking. To tackle the obstruction issue in fish tracking,
energy, they effectively identified different behavioral states of fish Mao et al. (2016) systematically discussed the basic techniques of
groups. However, some abnormal behaviors do not spread to the group obstruction tracking from the aspects of optical imaging, geometry,
level, so to detect the abnormal behavior of individual fish, Li et al. digital image and video processing, and studied the matching models of
(2022b) improved the model by adding a bi-directional feature pyramid fish obstruction, non-fish obstruction, and both obstruction at the same
network, coordinate attention block, and spatial pyramid pooling to the time. Finally, they achieved unobstructed tracking of three-dimensional
YOLOv5 network. They overcame the target occlusion problem in un­ fish targets. However, the above research has certain subjectivity and
derwater fish detection, and the model achieved a detection accuracy of still has a lot of room for improvement. Li et al. (2022a) proposed a
96.69%. video tracking model called CMFTNet, mainly by sharing the features of
multiple fish detection and tracking tasks through joint detection and
4.3. Detection methods for abnormal trajectories of fish embedding paradigms, and solved the problem of mutual obstruction of
fish schools using anchor-free method. They finally achieved
The advent of deep learning algorithms has brought new research high-precision multi-target tracking of fish. In addition to the mutual
ideas for obtaining abnormal fish trajectories. There are usually three obstruction problem between fish, the turbid underwater environment is
angles to explore for obtaining fish trajectories: object detection, object also an important factor restricting fish tracking. To extract fish trajec­
tracking, and trajectory generation, and researchers have different al­ tories in turbid water, researchers used a multi-scale Retinex algorithm
gorithm choices and improvements for these three parts. In order to to enhance underwater turbid images, reduce the impact of turbid un­
roughly track the trajectory of fish schools, Xu and Cheng (2017) derwater environment, and combine YOLO object detection and optical
improved the accuracy of fish detection by using data augmentation and flow algorithm to track the movement of fish in each frame of the video,
iterative CNN training methods, and can achieve fish school trajectory basically achieving the fish tracking task (Wageeh et al., 2021).
tracking at different time spans. However, fish school tracking often has
large deviations, and individual fish tracking must be accurately ach­ 4.4. Detection methods for abnormal surface features of fish
ieved, so Li et al. (2020b) used the Faster R-CNN model to detect fish in
the image, and then used intersection-over-union, center distance, and Most abnormal surface features in fish are caused by diseases.
probability similarity as criteria to associate the prediction results Employing deep learning models for detecting diseased fish can sub­
through greedy algorithms and Hungarian algorithms, achieving the stantially enhance fish farming efficiency. However, due to the difficulty
drawing and classification of individual fish trajectories, and the model in collecting fish disease datasets and the poor image quality under­
accuracy reached 80.94%. In the fish tracking process, complex under­ water, which can affect the training of models, transfer learning
water environments and occlusion between fish often lead to target loss methods are needed to improve the efficiency of abnormal surface
and other problems. To address these challenges, researchers have used feature detection. To monitor the occurrence of ulcers and white spots
background subtraction and selected appropriate reference images to on the surface of fish, Yasruddin et al. (2022) explored the feasibility of
detect fish in murky underwater environments, and used genetic algo­ using computer vision and dynamic convolution neural network
rithms and the good performance of LSTM as fish trajectory predictors in (DCNN) models for fish disease detection and trained images using the
murky underwater videos, proving the feasibility of using genetic al­ Faster-RCNN model, which produced good results. In terms of surface
gorithms and LSTM for trajectory prediction (Palconit et al., 2020). feature detection, both shrimp and fish are limited by the lack of
The development of underwater fish tracking technology provides diseased images, so many scholars have introduced the concept of
more possibilities for the study of abnormal trajectories, with im­ transfer learning to recognize diseased shrimp and classify two types of
provements mainly focused on fish detection algorithms and tracking shrimp diseases, which are white spot disease and black gill disease.
algorithms. In terms of detection algorithms, in order to improve the After comparison, the model with the highest accuracy was Mobile­
accuracy of fish tracking, Xia et al. (2019) used cross-entropy loss NetV1, with an accuracy of up to 95% (Ashraf and Atia, 2021). Given the
functions to distinguish targets and backgrounds of different domains in ideal effect of transfer learning in fish disease detection, it can be
deep convolutional neural networks, and optimized the loss function considered to use transfer learning and improved models for fish disease
through stochastic gradient descent (SGD), realizing fish target posi­ detection. Chen et al. (2022) achieved transfer learning through the
tioning and behavior tracking based on multi-domain deep convolu­ ImageNet dataset and were able to classify abnormal fish, with the
tional neural networks. However, the above research does not meet the ResNet-50, DensNet-201, InceptionV3, and Inception-ResNetV2 models
detection accuracy requirements of practical production environments. participating in training. Among them, the ResNet-50 algorithm can
In order to further improve the accuracy of target detection, Gong et al. achieve an accuracy rate of 96.43% for underwater image classification
(2022) used YOLOv4 network as the basic model, added convolutional of abnormal grouper (Epinephelus).
block attention module (CBAM) attention mechanism on it, and com­ In addition to detecting fish that have already developed diseases,
bined the improved target detector with the sorting tracker to achieve computer vision methods can also be employed for the prevention of fish
underwater fish tracking. The algorithm has high accuracy and can diseases. Fish lice often attach to the surface of fish before they become
achieve a tracking speed of 78.9fps. Regarding tracking algorithms, re­ diseased, and the incidence of disease increases when wounds appear on
searchers used master-slave cameras to capture images, and used LSTM the skin surface. Therefore, if these features are detected and measures
networks to predict the three-dimensional state of fish at each moment. are taken before the fish becomes ill, the incidence of disease can be
They established cross-view associations through motion continuity and greatly reduced, thus avoiding the large-scale spread of fish diseases. To
external constraint clues to achieve 3D tracking of fish (Wang et al., identify fish with lice and wounds on the surface, Gupta et al. (2022)
2017). Currently, there are still major problems in model innovation for designed a convolutional neural network based on two-dimensional
fish tracking, mainly due to the fact that current algorithms do not images for lice and wound detection in fish, which can classify
achieve high-precision processing of video data, and fish tracking is abnormal and normal fish and has high recognition accuracy. The above
easily affected by the underwater environment, making it easy to lose methods can perform well in detecting normal-sized targets, but it is
targets. Therefore, research on fish tracking will be a hot research di­ difficult to achieve accurate recognition of small targets. Li et al. (2023)
rection in future aquaculture. modified YOLOv4 PANet by adding additional detection layers and used
The obstruction caused by fish and the turbidity of the water pose the trained lice model weights for pre-training weights, which can better

12
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

detect small targets with a detection accuracy of 95.41%. approximately 700,000 underwater video clips that detect underwater
Due to the difficulty of collecting surface feature data for fish dis­ live fish near coral reefs, and the dataset can be downloaded from
eases and the lack of large, similar datasets available for use, research in http://groups.inf.ed.ac.uk/f4k/. The LCF-14 dataset includes 1000
this area remains relatively limited. However, as algorithm models videos of 10 fish, showing about 20,000 fish labels in the videos (Salman
continue to advance and data collection devices improve, new oppor­ et al., 2016), and the dataset can be downloaded from https://www.
tunities will be brought to the study of abnormal body features in fish. imageclef.org/2014/lifeclef/fish. The LCF-15 dataset consists of 93
underwater videos of 15 fish, providing 9000 video annotations and 20,
4.5. Summary 000 labeled images, and the dataset can be downloaded from
http://www.imageclef.org/lifeclef/2015/fish. The SeaCLEF2016 data­
Table 3 provides specific details on the literature related to abnormal set consists of 20 low-resolution videos and 20,000 images of 15 coral
fish detection using deep learning methods in computer vision. reef species. The main issue with this dataset is the imbalance of fish and
Compared with traditional methods in computer vision technology, the changes in lighting conditions, and the dataset can be downloaded from
advantage of deep learning algorithms is that they can automatically https://www.imageclef.org/lifeclef/2016/sea. The Labeled Fishes in
extract features, reducing the need for manually designing feature the Wild dataset was developed by the National Oceanic and Atmo­
extraction algorithms, and allowing models to be trained on large data spheric Administration (NOAA) and contains 829 training images, 100
quantities, thus improving classification accuracy. In addition, the validation images, 211 test samples, and 3167 non-fish images (Cutter
powerful fitting ability of deep learning algorithms allows them to et al., 2015). The photo quality is high, and the classification of fish is
handle complex fish image classification tasks, providing new solutions clear and labeled accordingly. The dataset can be found at
for abnormal fish detection. http://swfscdata.nmfs.noaa.gov/labeled-fishes-in-the-wild. The specific
information of public datasets is shown in Table 4.
5. Dataset and evaluation metrics Collecting experimental on-site data has become an important means
for studying abnormal fish. To ascertain the behavioral patterns of fish
5.1. Dataset in aquatic environments, researchers use on-site data collection methods
to obtain images. Yang et al. (2021) captured images of fish feeding
The dataset is the fundamental part of abnormal fish detection, behavior in the water by installing underwater cameras at the diagonal
directly affecting the training effectiveness and prediction accuracy of corners of the fish pond to capture the entire fish pool, ultimately col­
the model (Saberioon et al., 2017). A good dataset should possess the lecting and annotating 15,150 fish images. Additionally, data can be
following characteristics. First, it should be sufficiently large with collected using remotely operated vehicles (ROVs) equipped with
enough data samples, but not too large, as this will increase training cameras. Betancourt et al. (2020) proposed an integrated ROV solution
time and computational resource consumption. Second, the dataset that utilizes computer vision technology to inspect underwater net cages
should have a good balance, with roughly equal numbers of samples for in fish farms. After the ROV completes data collection, the visual algo­
each class, to avoid overfitting or underfitting the model. Third, the rithm mentioned in the text can be directly used Fig. 10 illustrates the
dataset should include different types of fish and different abnormal two methods for collecting experimental data on-site.
situations to improve the model’s generalization ability. Finally, the During fish detection, occlusion often poses a challenge, and an
samples in the dataset should be real, accurate, and representative, effective equipment deployment strategy can partially address this issue.
without too much noise or incorrect labelling. Based on these, the For instance, the use of cameras with reflective mirrors or adopting a
currently reliable and commonly used methods for obtaining datasets master-slave camera-based video capture method can be beneficial. Mao
fall into two categories: downloading public datasets and collecting et al. (2015) obtained a theoretical fish occlusion and non-fish occlusion
them through experiments. model through the principle of planar mirror imaging and realized
In the research of abnormal fish detection, several commonly used real-time 3D fish tracking. However, this method is limited by the
public datasets include Fish4-Knowledge (F4K), LifeCLEF (LCF) series, shooting angle of the camera, making subsequent processing work more
SeaCLEF2016, and Labeled Fishes in the Wild. The F4K dataset contains cumbersome. Wang designed a tracking method based on a master-slave

Table 3
Research on abnormal fish detection based on deep learning methods in computer vision technology.
Type Manifestation References Year Scenes Models/Algorithms/ Results Comparisons with other methods
Methods

Abnormal Dead fish Zhao et al. 2022 Pond Improved YOLOv4 Detection accuracy： Faster R-CNN、YOLOv3、
physiological (2022a) 95.47 YOLOv4、
activity
Fish turn over Wang et al. 2022 Pond Improved YOLOv5、 Detection accuracy： YOLOv5
(2022b) SiamRPN+ + 99.4%
Cannibalism Wang et al. 2022 Pond Improved YOLOv5 Detection accuracy： YOLOv5
(2022a) 97%
Feeding action Zhang et al. 2022 Pond MobileNetV2、SENet Detection accuracy： EfficientNet_B0、ShuffleNetV2、
(2022) 97.76% AlexNet and EfficientNetV2
Chase Du et al. 2022 Pond Resnet50、LSTM Detection accuracy： Resnet18-LSTM、
(2022) 98.52% Resnet101-LSTM
Abnormal Trajectories Li et al. 2020 Lab Faster R-CNN、 Detection accuracy： Greedy algorithm
trajectories (2020b) Hungarian algorithm 76.7%
Trajectories Palconit et al. 2020 Lab LSTM Mean Absolute Genetic Algorithm
(2020) Percentage Error:
3.33–17.74%
Abnormal body Fish disease Chen et al. 2022 Ocean Transfer learning、 Detection accuracy： ResNet-50
surface features (2022) InceptionV3 98.94%
Wound and lice Gupta et al. 2022 Pond CNN Detection accuracy： VGG-16、VGG-19
detection (2022) 96.7%
Wound and lice Li et al. (2023) 2023 Lab Transfer learning、 Total average accuracy YOLOv3
detection YOLOv4 (mAP):95.41%

13
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

Table 4 ment, the evaluation parameters must be adjusted according to the

Specific information on public datasets. actual situation. The following are various metrics used to evaluate the
Dataset Species Numbers of Resolution Characteristics accuracy of the model.
videos/
images (1) Accuracy (ACC) refers to the ratio of the correctly classified
Fish4–Knowledge 3000 700,000 320 × 240 The quantity of samples to the total number of classified samples. This standard is
videos with pix fish categories is commonly used to evaluate the overall performance of the model.
10 min each not balancing in The formula is shown as follows:
clip these video
datasets, and the TP + TN
ground truth is ACC = (1)
TP + TN + FP + FN
noisy in
coordinate.
LifeCLEF2014 10 1000 videos 640 × 480 It is a smaller
pix、 dataset, including (2) Precision can be used to indicate the ratio of correctly classified
320 × 240 algae attachment abnormal fish to all recognized abnormal fish (TP + FP). The
pix data and turbid formula is shown as follows:
water data.
LifeCLEF2015 15 93 videos 640 × 480 It presents a more TP
Precision = (2)
and 2000 pix、 noisy, blurry TP + FP
images with 320 × 240 environment and
class label pix poor
illumination.
(3) Recall, also known as sensitivity, is the ratio of the correctly
SeaCLEF2016 15 Training set 640 × 480 The main
consisting 20 pix、 challenge of identified fish to the total number of fish present. The formula is
videos and 320 × 240 datasets is the shown as follows:
20,000 pix change in
images illumination and TP
Recall = (3)
sample,73 unbal- ance of the TP + FN
test videos number of fish.
Labeled Fishes in 7 929 training 720 × 480 The existing
the Wild image, 211 pix challenge (4) Mean squared error (MSE) is the expected value of the square
frame testing contains small
difference between the predicted value and the true value. It
and 3167 particles in water,
negative rocks or crevices equals zero when the predicted value perfectly matches the true
sample that hide some value, indicating a perfect model. The larger the value, the
fish and the greater the error. The formula is shown as follows:
dataset is of good
quality, clearly 1∑ n

divided. MSE = ̂ i )2
(Yi − Y (4)
n i=1

camera structure, tracking the 2D main view, locating the slave view where n represents the number of observations, Yi represents the
through a focus detector, and then reconstructing the 3D trajectory true value, Ŷ i represents the predicted value, and i is an integer
(Wang et al., 2016). However, this method has the disadvantage of being ranging from 1 to n.
relatively complex to deploy and the data volume is usually large. (5) Root Mean Square Error (RMSE) is the square root of the ratio of
Therefore, researchers can choose an appropriate device deployment the sum of the squared deviations between predicted and true
scheme based on actual circumstances. values to the number of observations n. Essentially, it is taking the
square root of the Mean Squared Error to make the scale more
intuitive. The formula is shown as follows:
5.2. Evaluation metrics
√̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅
1∑ n
After the dataset is trained and the results are obtained, evaluation RMSE = (Yi − Y ̂ i )2 (5)
n i=1
metrics need to be set to assess the accuracy of the model. Among them,
True Positive (TP), True Negative (TN), False Positive (FP), and False
Negative (FN) are commonly used evaluation metrics in the field of
(6) Intersection over union (IOU) is the overlap between the candi­
image processing. In the detection and classification tasks of abnormal
date region and the ground truth region, where a larger overlap
fish, accuracy and precision are the main evaluation metrics. However,
leads to a higher IOU value. In an ideal scenario, the two regions
due to the differences in model, data, and system operating environ­

Fig. 10. Two methods of collecting experimental data in the field. (a) involves data collection using deployed master-slave cameras., and (b) is data collection using
underwater robots.

14
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

completely overlap. IOU is often used in non-maximum sup­ data on-site, but the number of these datasets and the types of abnormal
pression to eliminate multiple predicted boxes of the same object fish are usually very limited, which greatly limits the research results.
based on their detection box confidence, retaining only the one (2) The generalization ability of deep convolutional networks is
with the highest confidence. The formula is shown as follows: weak, which limits the wide use of abnormal fish detection models in
practical aquaculture environments. Generalization ability refers to the
A∩B
IOU = (6) gap between the model’s performance on the training dataset and un­
A∪B
seen data. The sign of overfitting is a large gap between the training
where A represents the ground truth box and B represents the predicted accuracy and validation accuracy, indicating that the model memorizes
box by the model. only the training data rather than learning their features.
As deep learning algorithms continue to advance, their accuracy has (3) The selection of deep learning architecture and parameters is
rapidly improved in various scenarios. However, the choice of evalua­ crucial and difficult. The excessive number of existing deep learning
tion metrics can also limit research. For example, in the classification of architectures makes it difficult for researchers to quickly select the most
abnormal fish, the accuracy of the algorithm may vary due to the quality suitable model for their task, and the model requires many parameters to
and distribution of the training and testing samples. To address this be set, such as the number of layers, weight initialization values, and
issue, receiver operating characteristic (ROC) (Krupinski, 2017) and learning rate. However, the control of these variables requires expert
area under the curve (AUC) (Janssens and Martens, 2020) have been judgement, which poses a great challenge to researchers’ ability.
widely used as standard measures of model performance in binary (4) Underwater environments pose challenges for collecting fish
classification settings. These metrics can assist researchers in selecting data, and fish may also exhibit issues such as occlusion, changes in
the optimal feature set and optimizing the performance of the classifier. shape, and variations in distance and scale. Underwater environments
However, the root of the problem still lies in the differences in the differ from those on land and often exhibit phenomena such as murky
datasets. Therefore, constructing a high-quality public dataset of un­ water, which can result in unclear fish images. Furthermore, various fish
derwater fish is key to reducing the limitations of evaluation metrics. activities can pose challenges for tasks such as detecting and tracking
abnormal fish. The issues inherent in fish data are illustrated in Fig. 11.
6. Challenges and strategies
6.2. Strategies for challenges
6.1. Challenges in abnormal fish detection
The challenges mentioned above have created bottlenecks for using
The rapid improvement of deep learning algorithms has promoted computer vision technology in abnormal fish detection, and timely ad­
the development of computer vision technology, which has provided justments must be made to seek solutions. To promote the smooth
new strategies for many previously unsolvable problems in aquaculture. progress of research work, this paper summarizes the state-of-the-art
However, as research continues to deepen, many challenges have been work in computer vision technology and reads a large number of ref­
exposed in abnormal fish detection through computer vision methods. In erences to think about corresponding strategies for the above problems,
this section, we summarize several key issues in abnormal fish detection as follows.
using computer vision methods as follows: (1) The solutions for limited abnormal fish data include data
(1) The dataset provided to train the model is very limited, especially augmentation, transfer learning, active learning, and the use of gener­
due to the complexity of the underwater environment, and it is difficult ative adversarial networks. Data augmentation technology uses tradi­
to collect data for abnormal fish. Moreover, constructing a large dataset tional image processing methods to expand images. Transfer learning
requires many resources, such as hiring personnel to collect, organize, (TL) reduces the quantity of training data required by transferring
and label data, which generally takes a long time. Therefore, it has knowledge from similar domains (Shin et al., 2016). Active learning
become common for researchers to conduct experiments by collecting reduces the number of sample annotations amount to save costs and
provide possibilities for the preparation of large datasets (Kong et al.,

Fig. 11. Issues with fish data.

15
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

2022). Generative adversarial networks (GANs) can be used to obtain making. Common methods include random search (Bergstra and Bengio,
relevant information about data without annotated information and 2012) and Bayesian hyperparameter optimization (Snoek et al., 2012).
then synthesize similar images with high-quality sample images. The In ideal conditions, the decisions made by these methods can even
above methods provide solutions for the lack of datasets and support surpass expert-level optimization.
abnormal fish detection from the data perspective. (4) To address the various problems caused by underwater condi­
(2) To address the issue of weak network generalization ability, re­ tions and fish movements, model improvements and multimodal tech­
searchers need to effectively monitor model overfitting. This can be niques can be considered. Model improvement is the most common
done by visualizing the training and validation accuracy at each training method, achieved by adjusting the model’s structure, parameters, and so
stage to determine the relationship between training accuracy and on. For example, introducing deformable convolutions and lightweight
validation accuracy. Common methods to improve the generalization network structures (Zhao et al., 2022a) can improve the detection ac­
ability of the model include regularization, which adds a regularization curacy and speed of the original model for abnormal fish. In addition,
term to the model’s loss function to prevent parameter sizes from developing multimodal deep learning techniques has broad prospects in
becoming too large and prevent overfitting. In addition, transfer abnormal fish detection. This method can learn features from multiple
learning can be used to obtain more data to enable the model to fully sources, such as analyzing audio and video simultaneously during the
understand the recognition problem. By improving the generalization feature learning process. Abnormal fish can be reflected from different
ability of the model, well-performing models can be used more quickly perspectives, and if multimodal deep learning techniques can be utilized
in practical production. to consider factors such as motion, trajectory, appearance, and sound,
(3) To address the difficulty of selecting deep learning architecture abnormal fish detection accuracy can be greatly improved Fig. 12 shows
and parameters, in addition to relying on expert experience, some the solutions to the four challenges presented in the text.
hyperparameter optimization algorithms can be used to assist decision-

Fig. 12. Challenges and strategies in abnormal fish detection.

16
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

7. Conclusion Adl, A., 2019. Detection Of Freshwater Fish White Spots Disease Using The Machine
Learning LR Classifier And ACO, 〈https://sc.panda321.com/scholar?hl=zh-CN&as
_sdt=0%2C5&q=Detection+Of+Freshwater+Fish+White+Spots+Disease+Using+
With the advancement of computer vision technology, especially the The+Machine+Learning+LR+Classifier+And+ACO&btnG=〉.
rapid development of deep learning methods in recent years, a new An, D., Hao, J., Wei, Y., Wang, Y., Yu, X., 2020. Application of computer vision in fish
solution has emerged for abnormal fish detection in aquaculture, which intelligent feeding system—a review. Aquac. Res. 52, 423–437. https://doi.org/
10.1111/are.14907.
has attracted the attention of more researchers. This article summarizes Anas, O., Wageeh, Y., Mohamed, H.E.-D., Fadl, A., ElMasry, N., Nabil, A.M., Atia, A.,
the research progress of computer vision technology in abnormal fish 2020. Detecting Abnormal Fish Behavior Using Motion Trajectories In Ubiquitous
detection in the past 20 years and, for the first time, divides abnormal Environments, FNC/MobiSPC. http://10.1016/j.procs.2020.07.023.
Asche, F., 2016. Aquaculture: Opportunities and Challenges, 〈http://e15initiative.org/
fish detection into three categories: abnormal physiological activities, wp-content/uploads/2015/09/E15-Fisheries-Asche-Final.pdf〉.
abnormal trajectories, and abnormal surface features of fish in aqua­ Ashraf, A., Atia, A., 2021. Comparative Study Between Transfer Learning Models to
culture. By citing a large number of references, this article analyzes the Detect Shrimp Diseases. 2021 16th International Conference on Computer
Engineering and Systems (ICCES), 1–6.http://doi.org/10.1109/ICCES54031.202
research ideas and progress of abnormal fish detection tasks from two 1.9686116.
perspectives: traditional methods and deep learning methods in com­ Bao, Y.J., Ji, C.Y., Zhang, B., Gu, J., 2018. Representation of freshwater aquaculture fish
puter vision technology. The selection and improvement of deep behavior in low dissolved oxygen condition based on 3D computer vision. MPLB 32.
https://doi.org/10.1142/S0217984918400900.
learning models have been the focus of research in abnormal fish Beitinger, T.L., 1990. Behavioral reactions for the assessment of stress in fishes. J. Gt
detection in recent years. Although developing target detection models Lakes Res. 16, 495–528.
has been rapid due to the difficulty of collecting diseased fish data, Bergstra, J., Bengio, Y., 2012. Random search for hyper-parameter optimization.
J. Mach. Learn. Res. 13, 281–305. https://doi.org/10.5555/2503308.2188395.
research on the abnormal surface features of fish is still relatively slow.
Betancourt, J.M., Coral, W., Colorado, J.D., 2020. An integrated ROV solution for
After sorting out the past research content, this article provides several underwater net-cage inspection in fish farms using computer vision. SN Appl. Sci. 2.
methods for obtaining public fish datasets and collecting experimental https://doi.org/10.1007/s42452-020-03623-z.
data. To address the existing problems, we considered corresponding Beyan, C., 2015. Detection of unusual fish trajectories from underwater videos, 〈https
://www.semanticscholar.org/paper/Detection-of-unusual-fish-trajectories-from-
solutions, which can provide a reference for future research. videos-Beyan/32842e2c9477768141e84bfbafba2913a644ab4b〉.
Computer vision technology still has great potential in aquaculture. Beyan, C., Fisher, R.B., 2012. A filtering mechanism for normal fish trajectories.
To fully leverage the role of computer vision technology, it is necessary Proceedings of the 21st International Conference on Pattern Recognition
(ICPR2012), 2286–2289, 〈https://sc.panda321.com/scholar?hl=zh-CN&as_sdt=0%
to explore solutions to the problems faced, apply transfer learning, 2C5&q=A+Filtering+Mechanism+for+Normal+Fish+Trajectories&btnG=〉.
reinforcement learning, active learning, generative adversarial net­ Beyan, C., Fisher, R.B., 2013. Detecting abnormal fish trajectories using clustered and
works, and multimodal deep learning methods to relevant research, and labeled data. 2013 IEEE International Conference on Image Processing, 1476–1480.
http://doi.org/10.1109/ICIP.2013.6738303.
make efforts to explore and improve the model to adapt to the aqua­ Bose, A.P.H., McClelland, G.B., Balshine, S., 2016. Cannibalism, competition, and costly
culture environment. In addition, improving the model to effectively care in the plainfin midshipman fish, Porichthys notatus. Behav. Ecol. 27, 628–636.
fuse and analyze data from multiple platforms will become an important https://doi.org/10.1093/BEHECO/ARV203.
Bose, A.P.H., Lau, M.J., Cogliati, K.M., Neff, B.D., Balshine, S., 2019. Cannibalism of
direction in the field of abnormal fish detection. Hopefully, this review young is related to low paternity and nest take-overs in an intertidal fish. Anim.
can provide valuable clues for abnormal fish detection. Behav. 153, 41–48. https://doi.org/10.1016/j.anbehav.2019.04.018.
Cao, Z., Fu, C., Ye, J., Li, B., Li, Y., 2021. SiamAPN++: Siamese Attentional Aggregation
Network for Real-Time UAV Tracking. 2021 IEEE/RSJ International Conference on
CRediT authorship contribution statement
Intelligent Robots and Systems (IROS), 3086–3092.http://doi.org/10.1109/IRO
S51168.2021.9636309.
Chunhong Liu: Conceptualization, Methodology, Software, Visual­ Chen, J.C., Chen, T.-L., Wang, H.-L., Chang, P., 2022. Underwater abnormal
ization, Funding acquisition. Zhiyong Wang: Conceptualization, classification system based on deep learning: a case study on aquaculture fish farm in
Taiwan. Aquacult Eng. 99. https://doi.org/10.1016/j.aquaeng.2022.102290.
Methodology, Software, Visualization, Writing – original draft. Yachao Cooke, S.J., Brownscombe, J.W., Raby, G.D., Broell, F., Hinch, S.G., Clark, T.D.,
Li: Data curation, Investigation. Zhenzuo Zhang: Data curation, Semmens, J.M., 2016. Remote bioenergetics measurements in wild fish:
Investigation. Jiawei Li: Data curation, Investigation. Chen Xu: Data opportunities and challenges. Comp. Biochem. Physiol. Part A Mol. Integr. Physiol.
202, 23–37. https://doi.org/10.1016/j.cbpa.2016.03.022.
curation, Investigation. Rongxiang Du: Data curation, Investigation. Cortes, C., Vapnik, V.N., 1995. Support-vector networks. MLear 20, 273–297. https://
Daoliang Li: Writing – review & editing, Supervision. Qingling Duan: doi.org/10.1023/A:1022627411411.
Conceptualization, Supervision. Cutter, G.R., Stierhoff, K., Zeng, J., 2015. Automated detection of rockfish in
unconstrained underwater videos using haar cascades and a new image dataset:
labeled fishes in the wild. 2015 IEEE Winter Appl. Comput. Vis. Workshops 57–62.
Declaration of Competing Interest https://doi.org/10.1109/WACVW.2015.11.
Davidson, J., Good, C., Welsh, C., Summerfelt, S.T., 2011. Abnormal swimming behavior
and increased deformities in rainbow trout Oncorhynchus mykiss cultured in low
The authors declare that they have no known competing financial exchange water recirculating aquaculture systems. Aquacult. Eng. 45, 109–117.
interests or personal relationships that could have appeared to influence https://doi.org/10.1016/J.AQUAENG.2011.08.005.
the work reported in this paper. Dell, A.I., Bender, J.A., Branson, K., Couzin, I.D., de Polavieja, G.G., Noldus, L., Pérez-
Escudero, A., Perona, P., Straw, A.D., Wikelski, M., Brose, U., 2014. Automated
image-based tracking and its application in ecology. Trends Ecol. Evol. 29 (7),
Data Availability 417–428. https://doi.org/10.1016/j.tree.2014.05.004.
Du, L., Lu, Z., Li, D., 2022. Broodstock breeding behaviour recognition based on
Resnet50-LSTM with CBAM attention mechanism. Comput. Electron. Agric. 202,
Data will be made available on request.
107404 https://doi.org/10.1016/j.compag.2022.107404.
Du, L., Lu, Z., Li, D., 2023. A novel automatic detection method for breeding behavior of
Acknowledgements broodstock based on improved YOLOv5. Comput. Electron. Agric. 206, 107639
https://doi.org/10.1016/j.compag.2023.107639.
FAO, F.A.A.D, 2018. The State of World Fisheries and Aquaculture 2018 - Meeting the
This work was supported by the Special Project for Technology Sustainable Development Goals. The State of World Fisheries and Aquaculture.
Innovation and Application Developing of Chongqing (CSTB2022TIAD- http://doi.org/10.18356/8d6ea4b6-en.
ZXX0053) and the Beijing Digital Agriculture Innovation Consortium Føre, M., Føre, M., Frank, K., Dempster, T., Alfredsen, J.A., Høy, E., 2017. Biomonitoring
using tagged sentinel fish and acoustic telemetry in commercial salmon aquaculture:
Project (BAIC10-2022). a feasibility study. Aquacult. Eng. 78, 163–172. https://doi.org/10.1016/J.
AQUAENG.2017.07.004.
References Gong, L., Hu, Z., Zhou, X., 2022. A Few Samples Underwater Fish Tracking Method Based
on Semi-supervised and Attention Mechanism. 2022 6th International Conference on
Robotics, Control and Automation (ICRCA), 18–22.http://doi.org/10.1109/icrca550
Abe, K., Tanaka, M., Habe, H., Taniguchi, Y., Iguchi, N., 2018. Video Scene Detection of
33.2022.9828911.
Burst Swimming by Fry of Farmed-raised Bluefin Tuna. 2018 4th International
Conference on Frontiers of Signal Processing (ICFSP), 105–109.http://doi.org/10.11
09/ICFSP.2018.8552079.

17
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

Gupta, A., Bringsdal, E., Knausgård, K.M., Goodwin, M., 2022. Accurate wound and lice MacQueen, J., 1967. Some methods for classification and analysis of multivariate
detection in atlantic salmon fish using a convolutional neural network. Fishes 7. observations, 〈https://sc.panda321.com/scholar?hl=zh-CN&as_sdt=0%
https://doi.org/10.3390/fishes7060345. 2C5&q=Some+methods+for+classification+and+analysis+of+multivariate+obs
Han, F., Zhu, J., Liu, B., Zhang, B., Xie, F., 2020. Fish shoals behavior detection based on ervations&btnG=〉.
convolutional neural network and spatiotemporal information. IEEE Access 8, Maddalena, L., Petrosino, A., 2008. A self-organizing approach to background
126907–126926. https://doi.org/10.1109/ACCESS.2020.3008698. subtraction for visual surveillance applications. ITIP 17, 1168–1177. https://doi.
Hao, X., Zhang, G., Ma, S., 2016. Deep learning. Int. J. Semant. Comput. 10, 417. https:// org/10.1109/TIP.2008.924285.
doi.org/10.1142/S1793351×16500045. Malik, S., Kumar, T., Sahoo, A.K., 2017. A Novel Approach to Fish Disease Diagnostic
Hao, Y., Yin, H., Li, D., 2022. A novel method of fish tail fin removal for mass estimation System based on Machine Learning. http://10.14738/AIVP.51.2809.
using computer vision. Comput. Electron. Agric. 193, 106601 https://doi.org/ Mao, J., Xiao, G., Sheng, W., Liu, X., 2015. A 3D occlusion tracking Model of the
10.1016/j.compag.2021.106601. underwater fish targets. 2015 IEEE International Conference on Electro/Information
He, K., Zhang, X., Ren, S., Sun, J., 2015. Deep Residual Learning for Image Recognition. Technology (EIT), 082–086.http://doi.org/10.1109/EIT.2015.7293324.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Mao, J., Xiao, G., Sheng, W., Qu, Z., Liu, Y., 2016. Research on realizing the 3D occlusion
770–778.http://doi.org/10.1109/cvpr.2016.90. tracking location method of fish’s school target. Neurocomputing 214, 61–79.
Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Comput. 9, https://doi.org/10.1016/j.neucom.2016.05.067.
1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. Martos-Sitcha, J.A., Sosa, J., Ramos-Valido, D., Bravo, F.J., Carmona-Duarte, C.,
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, Gomes, H.L., Calduch-Giner, J.A., Cabruja, E., Vega, A., Ferrer, M.A., Lozano, M.,
M., Adam, H., 2017. MobileNets: Efficient Convolutional Neural Networks for Montiel-Nelson, J.A., Afonso, J.M., Pérez-Sánchez, J., 2019. Ultra-low power sensor
Mobile Vision Applications. ArXiv abs/1704.04861, 〈https://www.semanticscholar. devices for monitoring physical activity and respiratory frequency in farmed fish.
org/paper/MobileNets%3A-Efficient-Convolutional-Neural-Networks-Howard-Zh Front. Physiol. 10, 667. https://doi.org/10.3389/fphys.2019.00667.
u/3647d6d0f151dc05626449ee09cc7bce55be497e〉. Mattiasen, E., Kashef, N.S., Stafford, D.M., Logan, C.A., Sogard, S.M., Bjorkstedt, E.P.,
Hu, J., Li, D., Duan, Q., Han, Y., Chen, G., Si, X., 2012. Fish species classification by color, Hamilton, S.L., 2020. Effects of hypoxia on the behavior and physiology of kelp
texture and multi-class support vector machine using computer vision. Comput. forest fishes. Glob. Change Biol. 26, 3498–3511. https://doi.org/10.1111/
Electron. Agric. 88, 133–140. https://doi.org/10.1016/J.COMPAG.2012.07.008. gcb.15076.
Hu, W.-C., Chen, L.-B., Huang, B.-K., Lin, H., 2022. A computer vision-based intelligent Mei, Y., Sun, B., Li, D., Yu, H., Qin, H., Liu, H., Yan, N., Chen, Y., 2022. Recent advances
fish feeding system using deep learning techniques for aquaculture. IEEE Sens. J. 22, of target tracking applications in aquaculture with emphasis on fish. Comput.
7185–7194. https://doi.org/10.1109/jsen.2022.3151777. Electron. Agric. 201, 107335 https://doi.org/10.1016/j.compag.2022.107335.
Huang, J., Yu, X., Chen, X., An, D., Zhou, Y., Wei, Y., 2022. Recognizing fish behavior in Miao-jun, X., Jian-ke, Z., Hui, L., 2013. A Method for Fish Diseases Diagnosis Based on
aquaculture with graph convolutional network. Aquacult. Eng. 98. https://doi.org/ Rough Set and FCM Clustering Algorithm. 2013 Third International Conference on
10.1016/j.aquaeng.2022.102246. Intelligent System Design and Engineering Applications, 99–103.http://doi.org/10
Ishibashi, Y., Ekawa, H., Hirata, H., Kumai, H., 2002. Stress response and energy .1109/ISDEA.2012.31.
metabolism in various tissues of Nile tilapia Oreochromis niloticus exposed to Murugaian, P., Ramamurthy, V., Karmegam, N., 2008. Effect of Temperature on the
hypoxic conditions. Fish. Sci. 68, 1374–1383. https://doi.org/10.1046/J.1444- Behavioural and Physiological Responses of Catfish, Mystus gulio (Hamilton), 〈htt
2906.2002.00577.X. ps://sc.panda321.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=+Effect+of+Tempe
Janssens, A.C.J.W., Martens, F.K., 2020. Reflection on modern methods: revisiting the rature+on+the+Behavioural+and+Physiological+Responses+of+Catfish%2C+
area under the ROC curve. Int. J. Epidemiol. 49, 1397–1403. https://doi.org/ Mystus+gulio&btnG=〉.
10.1093/ije/dyz274. Niu, B., Li, G., Peng, F., Wu, J., Zhang, L., Li, Z., 2018. Survey of fish behavior analysis by
Jolliffe, I.T., Cadima, J., 2016. Principal component analysis: a review and recent computer vision. J. Aquac. Res. Dev. 9, 1–15. https://doi.org/10.4172/2155-
developments. Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci. 374, 20150202 https:// 9546.1000534.
doi.org/10.1098/rsta.2015.0202. Palconit, M.G.B., Almero, V.J.D., Rosales, M.A., Sybingco, E., Bandala, A.A., Vicerra, R.R.
Kane, A.S., Salierno, J.D., Gipson, G., Molteno, T.C.A., Hunter, C., 2004. A video-based P., Dadios, E.P., 2020. Towards Tracking: Investigation of Genetic Algorithm and
movement analysis system to quantify behavioral stress responses of fish. Water Res. LSTM as Fish Trajectory Predictors in Turbid Water. 2020 IEEE REGION 10
38 (18), 3993–4001. https://doi.org/10.1016/J.WATRES.2004.06.028. CONFERENCE (TENCON), 744–749.http://doi.org/10.1109/TENCON50793.2020.
Kaur, G., Adhikari, N., Krishnapriya, S., Wawale, S.G., Malik, R.Q., Zamani, A.S., Perez- 9293730.
Falcon, J., Osei-Owusu, J., 2023. Recent advancements in deep learning frameworks Paul, R., 2015. Image Processing Technique To Detect Fish Disease.〈http://https://www.
for precision fish farming opportunities, challenges, and applications. J. Food Qual. cscjournals.org/manuscript/Journals/IJCSS/Volume9/Issue2/IJCSS-1013.pdf〉.
2023. https://doi.org/10.1155/2023/4399512. Peck, M.R., Tapilatu, R.F., Kurniati, E., Rosado, C., 2021. Rapid coral reef assessment
Kong, Q., Du, R., Duan, Q., Zhang, Y., Chen, Y., Li, D., Xu, C., Li, W., Liu, C., 2022. using 3D modelling and acoustics: acoustic indices correlate to fish abundance,
A recurrent network based on active learning for the assessment of fish feeding diversity and environmental indicators in West Papua, Indonesia. PeerJ 9, e10761.
status. Comput. Electron Agric. 198. https://doi.org/10.1016/j. https://doi.org/10.7717/peerj.10761.
compag.2022.106979. Peyghan, R., Rezaie, A., Hassani, F., Abiavi, T., 2021. Histopathological study of gill and
Krupinski, E.A., 2017. Receiver Operating Characteristic (ROC) Analysis. http://10 fish treatment in Ichthyophthiriasis of white skirt tetra (Gymnocorymbus ternetzi),
.14786/FLR.V5I2.250. 〈https://sc.panda321.com/scholar?hl=zh-cn&q=Histopathological+study+of+gill
Lauria, V., Das, I., Hazra, S., Cazcarro, I., Arto, I., Kay, S., Ofori-Danson, P.K., Ahmed, M., +and+fish+treatment+in+Ichthyophthiriasis+of+white+skirt+tetra〉.
Hossain, M.A.R., Barangé, M., Fernandes, J.A., 2018. Importance of fisheries for food Rauf, H.T., Lali, M.I.U., Zahoor, S., Shah, S.J.H., Rehman, A.U., Bukhari, S.A.C., 2019.
security across three climate change vulnerable deltas. Sci. Total Environ. 640–641, Visual features based automated identification of fish species using deep
1566–1577. https://doi.org/10.1016/j.scitotenv.2018.06.011. convolutional neural networks. Comput. Electron. Agric. 167. https://doi.org/
Li, D., Wang, Z., Wu, S., Miao, Z., Du, L., Duan, Y., 2020a. Automatic recognition 10.1016/j.compag.2019.105075.
methods of fish feeding behavior in aquaculture: a review. Aquaculture 528, Redmon, J., Farhadi, A., 2018. YOLOv3: An Incremental Improvement. ArXiv abs/
735508. https://doi.org/10.1016/j.aquaculture.2020.735508. 1804.02767, 〈https://arxiv.org/abs/1804.02767〉.
Li, D., Wang, G., Du, L., Zheng, Y., Wang, Z., 2021a. Recent advances in intelligent Ren, S., He, K., Girshick, R.B., Sun, J., 2015. Faster R-CNN: towards real-time object
recognition methods for fish stress behavior. Aquacult. Eng. 96. https://doi.org/ detection with region proposal networks. ITPAM 39, 1137–1149. https://doi.org/
10.1016/j.aquaeng.2021.102222. 10.1109/TPAMI.2016.2577031.
Li, J., Lian, Z., Wu, Z., Zeng, L., Mu, L., Yuan, Y., Bai, H., Guo, Z., Mai, K., Tu, X., Ye, J., Saberioon, M., Císař, P., 2018. Automated within tank fish mass estimation using
2023. Artificial intelligence–based method for the rapid detection of fish parasites infrared reflection system. Comput. Electron. Agric. 150, 484–492. https://doi.org/
(Ichthyophthirius multifiliis, Gyrodactylus kobayashii, and Argulus japonicus. 10.1016/j.compag.2018.05.025.
Aquaculture 563. https://doi.org/10.1016/j.aquaculture.2022.738790. Saberioon, M., Gholizadeh, A., Císař, P., Pautsina, A., Urban, J., 2017. Application of
Li, W., Li, F., Li, Z., 2022a. CMFTNet: Multiple fish tracking based on counterpoised machine vision systems in aquaculture with emphasis on fish: state-of-the-art and
JointNet. Comput. Electron. Agric. 198, 107018 https://doi.org/10.1016/j. key issues. Rev. Aquac. 9, 369–387. https://doi.org/10.1111/RAQ.12143.
compag.2022.107018. Saleh, A., Sheaves, M., Azghadi, M.R., 2022. Computer Vision and Deep Learning for Fish
Li, X., Liu, M., Zhang, S., Zheng, R., 2020b. Fish Trajectory Extraction Based on Object Classification in Underwater Habitats: A Survey. ArXiv abs/2203.06951, 977–999.
Detection. 2020 39th Chinese Control Conference (CCC), 6584–6588.http://doi. http://doi.org/10.1111/faf.12666.
org/10.23919/CCC50068.2020.9188642. Salman, A., Jalal, A., Shafait, F., Mian, A.S., Shortis, M., Seager, J., Harvey, E.S., 2016.
Li, X., Hao, Y., Zhang, P., Akhter, M., Li, D., 2022b. A novel automatic detection method Fish species classification in unconstrained underwater environments based on deep
for abnormal behavior of single fish using image fusion. Comput. Electron. Agric. learning. Limnol. Oceanogr. Methods 14, 570–585. https://doi.org/10.1002/
203, 107435 https://doi.org/10.1016/j.compag.2022.107435. lom3.10113.
Li, Z., Li, W., Li, F., Yuan, M., 2021b. A Review of Computer Vision Technologies for Fish Sarà, G., Gouhier, T.C., Brigolin, D., Porporato, E.M.D., Mangano, M.C., Mirto, S.,
Tracking. ArXiv abs/2110.02551, 〈https://www.semanticscholar.org/paper/A-Re Mazzola, A., Pastres, R., 2018. Predicting shifting sustainability trade-offs in marine
view-of-Computer-Vision-Technologies-for-Fish-Li-Li/3521bddd675b0bab138e86c3 finfish aquaculture under climate change. Glob. Change Biol. 24, 3654–3665.
41f03959a5de51e6〉. https://doi.org/10.1111/gcb.14296.
Lin, C.R., Chen, Y., Lin, X., Yuan, F., Zhu, Y., 2016. Water Monitoring System Based on Serra-Toro, C., Montoliu, R., Traver, V.J., Hurtado-Melgar, I.M., Núñez-Redó, M.,
Recognition of Fish Behavior. http://10.1007/978–981-10–0740-8_47. Cascales, P., 2010. Assessing Water Quality by Video Monitoring Fish Swimming
Ma, H., Tsai, T.-F., Liu, C.-C., 2010. Real-time monitoring of water quality using temporal Behavior. 2010 20th International Conference on Pattern Recognition, 428–431.
trajectory of live fish. Expert Syst. Appl. 37, 5158–5171. https://doi.org/10.1016/j. http://doi.org/10.1109/ICPR.2010.113.
eswa.2009.12.076. Sheng, D.B., Kim, S.B., Nguyen, T.H., Kim, D.H., Gao, T.S., Kim, H.-K., 2016. Fish injured
rate measurement using color image segmentation method based on K-means

18
C. Liu et al. Aquacultural Engineering 103 (2023) 102350

clustering algorithm and Otsu’s threshold algorithm. Power Syst. Eng. 20, 32–37. Xia, X., Zhao, Y., Hu, Z., Wang, Z., Yu, C., Bai, Y., 2019. Fish Behavior Tracking
https://doi.org/10.9726/KSPSE.2016.20.4.032. Algorithm Based on Multi-Domain Deep Convolutional Neural Network. Proceedings
Shin, H.-C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D.J., of the 2019 4th International Conference on Multimedia Systems and Signal
Summers, R.M., 2016. Deep convolutional neural networks for computer-aided Processing.http://doi.org/10.1145/3330393.3330422.
detection: CNN architectures, dataset characteristics and transfer learning. IEEE Xiao, G., Fan, W.-k, Mao, J., Cheng, Z., Zhong, D., Li, Y., 2016. Research of the Fish
Trans. Med. Imaging 35, 1285–1298. https://doi.org/10.1109/TMI.2016.2528162. Tracking Method with Occlusion Based on Monocular Stereo Vision. 2016
Sikder, J., Sarek, K.I., Das, U.K., 2021. Fish Disease Detection System: A Case Study of International Conference on Information System and Artificial Intelligence (ISAI),
Freshwater Fishes of Bangladesh.http://10.14569/ijacsa.2021.01206100. 581–589.http://doi.org/10.1109/ISAI.2016.0129.
Singla, N., 2014. Motion Detection Based on Frame Difference Method, 〈https://sc.panda Xu, W., Zhu, Z., Ge, F., Han, Z., Li, J., 2020. Analysis of behavior trajectory based on deep
321.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=Motion+Detection+Based+on+ learning in ammonia environment for fish. Sensors 20. https://doi.org/10.3390/
Frame+Difference+Method&btnG=〉. s20164425.
Smith, C., Reay, P.J., 1991. Cannibalism in teleost fish. Rev. Fish. Biol. Fish. 1, 41–64. Xu, Z., Cheng, X.E., 2017. Zebrafish tracking using convolutional neural networks. Sci.
https://doi.org/10.1007/BF00042661. Rep. 7, 42815. https://doi.org/10.1038/srep42815.
Snoek, J., Larochelle, H., Adams, R.P., 2012. Practical Bayesian Optimization of Machine Yang, L., Liu, Y., Yu, H., Fang, X., Song, L., Li, D., Chen, Y., 2020a. Computer vision
Learning Algorithms. ArXiv abs/1206.2944, 〈https://arxiv.org/pdf/1206.2944.pdf〉. models in intelligent aquaculture with emphasis on fish detection and behavior
Spampinato, C., Chen-Burger, Y.-H.J., Nadarajan, G.D., Fisher, R.B., 2008. Detecting, analysis: a review. Arch. Comput. Methods Eng. 28, 2785–2816. https://doi.org/
Tracking and Counting Fish in Low Quality Unconstrained Underwater Videos, 10.1007/s11831-020-09486-2.
International Conference on Computer Vision Theory and Applications.http://1 Yang, L., Yu, H., Cheng, Y., Mei, S., Duan, Y., Li, D., Chen, Y., 2021. A dual attention
0.5220/0001077705140519. network based on efficientNet-B2 for short-term fish school feeding behavior
Sun, D., Roth, S., Black, M.J., 2010. Secrets of optical flow estimation and their analysis in aquaculture. Comput. Electron. Agric. 187, 106316 https://doi.org/
principles. 2010 IEEE Computer Society Conference on Computer Vision and Pattern 10.1016/j.compag.2021.106316.
Recognition, 2432–2439.http://doi.org/10.1109/CVPR.2010.5539939. Yang, X., Zhang, S., Liu, J., Gao, Q., Dong, S., Zhou, C., 2020b. Deep learning for smart
Sun, M., Yang, X., Xie, Y., 2020. Deep Learning in Aquaculture: A Review, 〈https://sc.pan fish farming: applications, opportunities and challenges. ArXiv abs/2004.11848,
da321.com/scholar?hl=zh-cn&q=Deep+Learning+in+Aquaculture%3A+A+Revie 66–90.http://doi.org/10.1111/RAQ.12464.
w〉. Yasruddin, M.L., Hakim Ismail, M.A., Husin, Z., Tan, W.K., 2022. Feasibility study of fish
Sweidan, A.H., El-Bendary, N., Hegazy, O.M., Hassanien, A.E., Snás̃el, V., 2015. Water disease detection using computer vision and deep convolutional neural network
Pollution Detection System Based on Fish Gills as a Biomarker, International (DCNN) algorithm. 2022 IEEE 18th Int. Colloq. Signal Process. Appl. (CSPA)
Conference on Intelligent Cloud Computing.http://10.1016/J.PROCS.2015.09.004. 272–276. https://doi.org/10.1109/CSPA55076.2022.9782020.
Szeliski, R., 2010. Computer Vision - Algorithms and Applications, Texts in Computer Yu, X., Wang, Y., An, D., Wei, Y., 2021. Identification methodology of special behaviors
Science.http://10.1007/978–1-84882–935-0. for fish school based on spatial behavior characteristics. Comput. Electron. Agric.
Takahara, T., Minamoto, T., Yamanaka, H., Doi, H., Kawabata, Zi, 2012. Estimation of 185, 106169 https://doi.org/10.1016/j.compag.2021.106169.
fish biomass using environmental DNA. PLoS One 7, e35868. https://doi.org/ Zhang, L., Li, W., Liu, C., Zhou, X., Duan, Q., 2020. Automatic fish counting method
10.1371/journal.pone.0035868. using image density grading and local regression. Comput. Electron. Agric. 179,
Terayama, K., Hioki, H., Sakagami, M.-A., 2017. Measuring tail beat frequency and coast 105844 https://doi.org/10.1016/j.compag.2020.105844.
phase in school of fish for collective motion analysis, International Conference on Zhang, L., Wang, J., Li, B., Liu, Y., Zhang, H., Duan, Q., 2022. A MobileNetV2-SENet-
Graphic and Image Processing.http://10.1117/12.2266447. based method for identifying fish school feeding behavior. Aquacult Eng. 99.
Tian, J., Chen, L., 2017. Abnormal motion detection in video using statistics of https://doi.org/10.1016/j.aquaeng.2022.102288.
spatiotemporal local kinematics pattern. 2017 IEEE International Conference on Zhao, J., Gu, Z., Shi, M., Lu, H., Li, J., Shen, M., Ye, Z., Zhu, S., 2016. Spatial behavioral
Image Processing (ICIP), 2065–2068. http://doi.org/10.1109/ICIP.2017.8296645. characteristics and statistics-based kinetic energy modeling in special behaviors
Vo, T.T.E., Ko, H., Huh, J.-H., Kim, Y., 2021. Overview of smart aquaculture system: detection of a shoal of fish in a recirculating aquaculture system. Comput. Electron.
focusing on applications of machine learning and computer vision. Electronics 10. Agric. 127, 271–280. https://doi.org/10.1016/j.compag.2016.06.025.
https://doi.org/10.3390/electronics10222882. Zhao, J., Wei-jun, B., Fengdeng, Z., Zhu, S., Liu, Y., Lu, H., Shen, M., Ye, Z., 2018.
Wageeh, Y., El-Din Mohamed, H., Fadl, A., Anas, O., ElMasry, N., Nabil, A., Atia, A., Modified motion influence map and recurrent neural network-based monitoring of
2021. YOLO fish detection with Euclidean tracking in fish farms. J. Ambient Intell. the local unusual behaviors for fish school in intensive aquaculture. Aquaculture
Humaniz. Comput. 12, 5–12. https://doi.org/10.1007/s12652-020-02847-6. 493, 165–175. https://doi.org/10.1016/J.AQUACULTURE.2018.04.064.
Waleed, A., Medhat, H., Esmail, M., Osama, K., Samy, R., Ghanim, T.M., 2019. Automatic Zhao, S., Zhang, S., Liu, J., Wang, H., Zhu, J., Li, D., Zhao, R., 2021. Application of
Recognition of Fish Diseases in Fish Farms. 2019 14th International Conference on machine learning in intelligent fish aquaculture: a review. Aquaculture 540. https://
Computer Engineering and Systems (ICCES), 201–206.http://doi.org/10.1109/ICCE doi.org/10.1016/J.AQUACULTURE.2021.736724.
S48960.2019.9068141. Zhao, S., Zhang, S., Lu, J., Wang, H., Feng, Y., Shi, C., Li, D., Zhao, R., 2022a.
Wang, C., Li, Z., Wang, T., Xu, X., Zhang, X., Li, D., 2021. Intelligent fish farm—the future A lightweight dead fish detection method based on deformable convolution and
of aquaculture. Aquac. Int 29, 2681–2711. https://doi.org/10.1007/s10499-021- YOLOV4. Comput. Electron. Agric. 198, 107098 https://doi.org/10.1016/j.
00773-8. compag.2022.107098.
Wang, H., Zhang, S., Zhao, S., Lu, J., Wang, Y., Li, D., Zhao, R., 2022a. Fast detection of Zhao, X., Yan, S., Gao, Q., 2019. An algorithm for tracking multiple fish based on
cannibalism behavior of juvenile fish based on deep learning. Comput. Electron. biological water quality monitoring. IEEE Access 7, 15018–15026. https://doi.org/
Agric. 198, 107033 https://doi.org/10.1016/j.compag.2022.107033. 10.1109/ACCESS.2019.2895072.
Wang, H., Zhang, S., Zhao, S., Wang, Q., Li, D., Zhao, R., 2022b. Real-time detection and Zhao, Y., Li, W., Li, Y., Qi, Y., Li, Z., Yue, J., 2022b. LFCNet: a lightweight fish counting
tracking of fish abnormal behavior based on improved YOLOV5 and SiamRPN++. model based on density map regression. Comput. Electron. Agric. 203, 107496
Comput. Electron. Agric. 192, 106512 https://doi.org/10.1016/j. https://doi.org/10.1016/j.compag.2022.107496.
compag.2021.106512. Zheng, H.Y., Liu, R., Zhang, R., Hu, Y.Q., 2014. A method for real-time measurement of
Wang, J.-H., Lee, S.-K., Lai, Y.-C., Lin, C.-C., Wang, T.-Y., Lin, Y.-R., Hsu, T.-H., respiratory rhythms in medaka (Oryzias latipes) using computer vision for water
Huang, C.-W., Chiang, C.-P., 2020. Anomalous Behaviors Detection for Underwater quality monitoring. Ecotoxicol. Environ. Saf. 100, 76–86. https://doi.org/10.1016/j.
Fish Using AI Techniques. IEEE Access 8, 224372–224382. https://doi.org/10.1109/ ecoenv.2013.11.016.
ACCESS.2020.3043712. Zheng, K., Yang, R., Li, R., Guo, P., Yang, L., Qin, H., 2023. A spatiotemporal attention
Wang, S., Liu, X., Zhao, J., Liu, Y., Chen, Y.Q., 2016. 3D tracking swimming fish school network-based analysis of golden pompano school feeding behavior in an
using a master view tracking first strategy. 2016 IEEE International Conference on aquaculture vessel. Comput. Electron. Agric. 205, 107610 https://doi.org/10.1016/
Bioinformatics and Biomedicine (BIBM), 516–519.http://doi.org/10.1109/BIBM. j.compag.2022.107610.
2016.7822572. Zhu, W., Li, D., 2007. A CBR System for Fish Disease Diagnosis, Conference on Control
Wang, S., Zhao, J., Liu, X., Qian, Z., Liu, Y., Chen, Y.Q., 2017. 3D tracking swimming fish Technology and Applications.http://10.1007/978–0-387–77253-0_97.
school with learned kinematic model using LSTM network. 2017 IEEE International Zion, B., 2012. Review: the use of computer vision technologies in aquaculture - a
Conference on Acoustics, Speech and Signal Processing (ICASSP), 1068–1072.http:// review. Comput. Electron. Agric. 88, 125–132. https://doi.org/10.1016/J.
doi.org/10.1109/ICASSP.2017.7952320. COMPAG.2012.07.010.
Xia, C., Fu, L., Liu, Z., Liu, H., Chen, L., Liu, Y., 2018. Aquatic toxic analysis by
monitoring fish behavior using computer vision: a recent progress. J. Toxicol. 2018,
2591924 https://doi.org/10.1155/2018/2591924.

19