Flame and Smoke Detection in Substation Based on

Wavelet Analysis and Convolution Neural Network

ZHANG Xu, WANG Wanguo LI Xinrui, LIU Bin TIAN Yuan
Shandong Luneng Intelligent Shandong Luneng Intelligent State Grid Shandong Electric Power
Technology Company Technology Company Company Overhaul Company,Jinan,
Limited,Jinan,250101,China; Limited,Jinan,250101,China; 250000,China;
[email protected] The First High School of Changsha， [email protected]
[email protected] Hunan,Changsha,410005,China;
[email protected]
[email protected]

ABSTRACT fire occurrence and exterminate the fire in the bud.With the
development of digital image processing technology in computer
In this paper, a fire detection method based on color features, field, image-based fire detection technology is more and more
wavelet analysis, and convolution neural network is proposed. widely used in the field of fire detection. Different smoke and
Firstly, the candidate region of flame is extracted by color flame detection algorithms are constantly proposed [1-4].In smoke
segmentation method, and then the candidate region of smoke is image processing, Shu Xueming and others[5] proposed a fire
generated by the background fuzzy model based on wavelet detection method based on particle imaging system, light
analysis. Then the candidate region is filtered by the trained CNN scattering system and smoke channel system .Xie Zhenping[6]uses
model, and the position of flame and smoke in a picture is located. Bayesian decision-making method to detect video smoke . Gomez-
Finally, a large number of fire pictures in different scenes are used Ro-driguez F et al. [7] extract the moving region by optical flow
to test the algorithm. The results show that this method can detect method, and then detect the smoke by using wavelet method
the location of flame and smoke accurately and quickly from combined with the motion characteristics of smoke.Yuan [8-9]
images or videos, and can be applied to fire detection tasks in uses cumulative motion direction model to detect smoke. However,
substation scenarios. estimating the direction of motion by simple pixel summation
method will lead to inaccurate estimation of direction.Then Yuan
CCS Concepts uses the double mapping structure to extract shape invariant
Computing methodologies → Object detection features and combines them with AdaBoost classifier to detect
video smog in [10].In [11], the author uses the pixel-level
Keywords classification algorithm based on histogram to detect flame and
fire detection; substation; smoke detection; color characteristics smoke. The video sequences containing flame and smoke are used
as training samples, and the regions with reduced high frequency
1. INTRODUCTION energy components in images are recognized as smoke by wavelet
There are inflammable and explosive devices such as transformers, transform. Phillips [12] et al. applied color histogram model based
capacitors and high voltage switches in substations, which have on Gauss distribution to flame detection. In [13], the author
strong electromagnetic interference and are prone to fire. Fire detects the flame by video analysis in wavelet domain, detects the
protection in substations directly affects whether the substation flame flicker by one-dimensional time wavelet transform, and
can operate safely.However, the traditional fire detection detects the color change of the flame moving region by two-
technology mainly uses sensors to identify the flame and dimensional space wavelet transform.
temperature. Each sensor can only detect the local space around
the control points, which is difficult to play a role in special According to the high randomness of fire detection in complex
occasions such as open space. At the same time, there will be false and changeable environment, a new fire detection method based
alarm or missed alarm. Because these methods can only sense the on color feature, wavelet analysis and convolution neural network
occurrence of fire when the fire spreads to a certain extent, it will is proposed in this paper. The candidate area of flame is generated
inevitably cause some losses. In order to reduce the loss caused by by the color feature of flame, and the candidate area of smoke is
fire, it is necessary to detect flame and smoke in the early stage of generated by the background fuzzy feature. Candidate areas are
filtered by carefully designed small CNN models, and the
Permission to make digital or hard copies of all or part of this work for positions of flame and smoke are detected. The experimental
personal or classroom use is granted without fee provided that copies are results show that the method can detect flame and smoke in
not made or distributed for profit or commercial advantage and that different scenes through real-time monitoring of specified scenes,
copies bear this notice and the full citation on the first page. Copyrights and achieve the purpose of early detection of fire and accurate
for components of this work owned by others than ACM must be location of fire.
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior This paper is organized as follows.The next section analyzes the
specific permission and/or a fee. image features of flame and smoke and introduces the generation
Request permissions from [email protected].
method of flame and smoke candidate region.Section III describes
ICIAI 2019, March 15–18, 2019, Suzhou, China
© 2019 Association for Computing Machinery.
the proposed CNN architecture. In Section IV, results of this
ACM ISBN 978-1-4503-6128-6/19/03…$15.00 method are described. this paper is concluded in the last section.
https://doi.org/10.1145/3319921.3319962

248
2. GENERATION OF CANDIDATE AREAS unsatisfactory. Therefore, it is necessary to transform the image
into HSV space to extract the flame area.
FOR FLAME AND SMOKE
2.1 Flame Detection Model The flame detection proposed in this paper is aimed at the early
Flames usually appear red. As we know, RGB color model has stage of fire, when the flame area is small, it is R > G > B in RGB
less computational complexity than other color models. However, space and saturation in HSI color space. The smoke image is
HIV and HSV color models are often used in flame image transformed from RGB space to HSI space, and the color model of
recognition, because their description of color is more suitable for flame image in HSI color space is obtained, which is expressed as
human perception of the color of the objective world. Firstly, the follows:
flame images in different scenarios are sampled, and then the RGB S  1  3  (min(min( r , g ), b) /( r  g  b))
（2）
spatial flame color feature model is established by analyzing the R, sTh
G, B mean values of the flame area.Figure 1 shows a partially S  (255  r ) 
rTh
sampled image .Table 1 shows the R, G, B statistical mean values
of flame images in each sample of Figure 1. In the above formula, H represents the H component in HSI space,
S represents the S component in HSI space. By adjusting the
parameters rTh and sTh , the number of flame candidate areas
can be changed. The smaller the ratio of rTh to sTh , the more
flame candidate areas can be obtained, and the more false alarms
can be obtained. In order to detect all flame regions, rTh = 200
and sTh = 5 are set in this paper.

2.2 Background Fuzzy Model

Current smoke detection methods are mostly based on smoke,
temperature, light and composite detectors. Because these
detectors rely on by-products of combustion process to detect
Figure 1. Part of sampling images smoke, they can only effectively detect smoke near flame and
smoke source. For large space or outdoor places, the reliability of
Table 1. Sampling values of flame image in RGB space detection is low.
In order to improve the real-time and reliability of smoke
detection, image processing technology should be used in smoke
Image detection research. From the perspective of image processing
sequence B G R
technology, smoke can be detected by using the characteristics of
number background ambiguity, diffusivity, principal direction angle and so
on. After synthetically comparing the difficulty and accuracy of
1 98 230 250 various methods, we decided to use the background ambiguity of
2 56 145 200 smoke to detect smoke.
3 88 188 230 Mean Background Model is a commonly used background
ambiguity model, which regards moving objects as noise and
4 38 129 213
eliminates them by means of cumulative averaging. The
5 49 130 247 background image is obtained by averaging the sequence images
6 108 219 219 of moving objects running for a period of time. It can be defined
by the expression：
7 76 234 240
N
1
 img ( x, y)
8 71 225 251
Background ( x, y )  i （3）
9 60 164 253 N i 1

Mean background model is relatively simple to implement. It

Table 1 shows that the value of R channel is larger in most flame needs to read every frame of image, and then sum and average the
regions and the value of B channel is the smallest. In this paper, corresponding pixels of each frame. In this paper, a three-
the RGB color feature model of flame is expressed as follows: dimensional matrix mat ( x, y, k ) is used to store the gray value
(i ) R  G  B of each frame image, where is a fixed value, which can be the
（1） whole video or a sequence image for a period of time.
(ii ) R  mean( R), G  mean(G ), B  mean( B)
2.3 Smoke Detection Model
The mean (R), mean (G) and mean (B) in the above formula
Smoke can partially occlude other objects in general. In spatial
represent the mean of RGB channels of all the pixels in a picture,
domain, the background becomes blurred, and in frequency
RGB color space is often used in display system, which is not
domain, the high frequency signal attenuates. According to the
suitable for image segmentation and analysis. Only using RGB
characteristics of smoke, we can use two-dimensional discrete
color model to segment flame will produce a large number of false
wavelet transform to extract the texture features of smoke image
flame areas. The effect of flame segmentation is very

249
and distinguish the texture blurring characteristics, so as to detect architecture inspired by Hubel's research work [15]. Leunun [16]
the existence of smoke. successfully applied them to digital classification and built Letnet
neural network to recognize handwritten numerals. In
The main feature of wavelet transform is that it can fully highlight convolutional neural networks, each layer acts as a filter to extract
some aspects of the image characteristics, and can localize the the specific features in the original image. The shallow feature
time (space) frequency analysis. The signal is refined at multi- map of CNN is mainly used to extract those relatively obvious
scale by scaling translation operation, and the result of time image features.
subdivision at high frequency and frequency subdivision at low
frequency is achieved. Through color segmentation of the original image in RGB space
and HSV space, the candidate areas of flame can be extracted. The
An image can be decomposed into four parts after two- background ambiguity method based on wavelet analysis can be
dimensional discrete wavelet transform: a low-frequency (cA) used to extract candidate areas of smoke. Because of the
component sub-image and three high-frequency component sub- complexity of image background, the extracted candidate regions
images. Among them, three high-frequency component sub- may contain some false-detection image blocks. The candidate
images contain texture information in horizontal direction (HL), regions of flame and smoke can be filtered out by carefully
vertical direction (LH) and diagonal direction (HH). When there is designed CNN classifier. At the same time, in order to achieve
smoke in the image, the energy value of these three high- real-time speed of detection algorithm, a small CNN network is
frequency component sub-images is usually reduced. designed. The network consists of three convolution layers, three
Suppose w n ( x, y ) represents a composite image consisting of pooling layers and two full connection layers. The size of the
model is 391k. Among them, the convolution kernel size of the
the sum of the energy values of three high frequency component first two convolutional layers is 3×3, and the convolution kernel
subimages:
size of the third convolutional layer is 2 ×2.Each convolution
w n ( x, y )  LH n ( x, y )  HLh ( x, y )  HH n ( x, y )
2 2 2
（4） layer is followed by a pooling layer and a PReLu activation
function. The output feature dimension of the first fully connected
The composite image is decomposed into sub-blocks of size layer is 128, and the output feature dimension of the second fully
( K1, K 2) . The energy ei (l1 , l2 ) of the sub-block i is: connected layer is 3, and the final softmax layer can calculate the
probability of the input image blocks belonging to flame, smoke,

 w ( x, y )
and background, respectively.
e i (l1 , l2 )  n （5）
( x , y )Ri 3.2 Training Process
The purpose of designing CNN classifier is to filter candidate
In the formula above, Ri is the i sub-block with the size of (K1, areas of flame and smoke, so as to filter out false candidate targets.
There are three types of training samples: flame samples, smoke
K2) in the composite image wn ( x, y ) , and (l1, l2 ) represents the
samples and background. The production process of training
corresponding position of the energy sub-block in the composite samples is as follows: 1. Marking the flame and smoke areas of
image. In this paper, the size of the block is 4x4. the collected fire images with image annotation tools; 2. Cutting
The local high frequency energy of the current image after wavelet any size image blocks randomly from the fire images according to
transform is compared with that of the background image after the annotation information and scaling them to 24*24 size; 3.
wavelet transform. If the energy value of the sub-block at (l1 , l2 ) Background samples were randomly clipped from images without
flame or smoke, and scaled to 24 x 24 size. The training set
decreases, it means that the texture or edge of the current image is contains 60,000 images, and the number ratio of flame samples,
no longer as sharp as that of the background image, and there may smoke samples and background samples is 1:2:3. In the training
be smoke in this area. process, 60% of the images are used as training set, 20% as
In order to improve the recognition rate, two thresholds of 0 < T1 verification set and 20% as test set. We use the SGD method to
< T2 < 1 are set. In this paper, T1 = 0.6 and T2 = 0 are selected. If train the network. The batch size is 256, and the weights of the
smoke in the image results in the reduction of high frequency CNN network are randomly initialized. The initial learning rate
energy of the wavelet, it should satisfy the following requirements: was 0.01 and the momentum was 0.9. At the same time, in order to
prevent the CNN network from over-fitting, Dropout layer is
ei (l1 , l2 )  T 2  ei (l1 , l2 )  T1 ei (l1 , l2 ) （6） added after the two full connection layers, and dropout_ratio is 0.5.
We trained the network for roughly 1000 cycles.
represents the sub-block energy value of the composite image
at.If the energy value of the sub-block satisfies the condition of 4. EXPERIMENTAL SIMULATION
Eq.(6), all the pixel values in the sub-block are replaced by 0.The
original image is restored by inverse wavelet transform. At this
4.1 Fire Data Set
time, the original image is missing the pixel value of the identified Because pyrotechnic experiments are not allowed in substations, it
smoke area. Then, by constructing the binary image of the image, is difficult to obtain ideal experimental video data. In view of the
all connected areas are found as candidate areas of the smoke area. experiment of flame detection, our team has set up a flame data set,
and collected 3000 flame pictures in different environments and
3. CNN FOR VIDEO FIRE AND SMOKE scenarios, including some substation flame pictures, forest flame
pictures, grassland flame pictures, urban flame pictures and so on.
CLASSIFICATION For the smoke detection experiment, this paper tests on eight
3.1 CNN Structure smoke videos. Four of the videos contain smoke, which is used to
The concept of convolutional neural network was first proposed test the recognition accuracy of the algorithm. The other four
by Fukushima [14]. He built a hierarchical neural network

250
videos do not contain smoke, which is used to test the false Table 3.Comparison of video frames in which smoke is first
detection rate of the algorithm. detected
4.2 Experimental Results Total The
For flame detection, Prec (precision)、 PTPR ( true positive rate), Video number of proposed [3] [8]
sequence frames method
and PFPR ( false positive rate) are used to quantify the performance
of the algorithm. 1 1127 272 312 339
2 2889 77 93 120
The experiment is compared with [17-19]. [17] uses the Gauss
mixture model to detect the flame foreground, extracts color 3 347 200 -- 230
features from a large number of forest flame samples and finally 4 933 66 74 110
detects the flame. In [18], color features are extracted from a large
number of flame images, motion features are obtained by
cumulative geometric independent component analysis (C-GICA),
and flame recognition is achieved by back propagation neural Table 4.Comparisons of frames misdetected in smokeless video
network (BP).[19] extracts brightness and color features based on
visual saliency, uses principal component analysis to reduce Total The
dimensionality, and combines dynamic and static features through Video number of proposed [3] [8]
sequence frames method
linear weighting. The final experimental results are as follows:
Table 2. Results of various flame detection algorithms 5 4536 3 12 20

6 1000 0 0 1
processin
Flame g 7 3000 0 0 3
Prec PTPR PFPR
detection
method Time(ms) 8 700 0 2 6

The For the smoke video in Table 3, the proposed algorithm can detect
proposed 0.882 0.791 0.002 80 smoke earlier than the algorithms in[3] and [8]. For video 1, this
method
algorithm detects smoke in frame 272,[3] detects smoke in frame
[17] 0.855 0.786 0.008 120 312, [8] detects smoke in frame 339; for video 3, this algorithm
[18] 0.832 0.769 0.03 100 detects smoke in frame 200, [3] detects smoke in frame 230, and
[19] 0.866 0.790 0.004 200 [2] has not detected smoke. For the four smokeless videos in Table
4, the error detection of this algorithm is the least. For video 5,
there are 3 frame errors in this algorithm, 12 frame errors in [3]
It can be seen from Table 2 that compared with the other three and 20 frame errors in [4]. For video 6-8, there is no false alarm in
flame detection methods, the flame detection algorithm of this this algorithm, while the algorithms in [4] have false alarm in
paper can not only ensure the accuracy of flame detection, but also video 6-8. Through comparison, it is found that the proposed
the average detection time is the least, which can meet the real- algorithm can detect smoke earlier, and the false detection rate is
time requirements of substation flame detection. The following is low. Figure 3 below is a screenshot of smoke detection results in
the actual detection effect of the partial flame picture. this paper.

Figure 2. Flame image detection results Figure 3. Screenshots of test results in this paper
For smoke detection, this paper has been tested on 8 segments of
video, and compared with the smoke detection algorithms in [3] 5. CONCLUSION
and [8]. The experimental results are shown in Table 3 and Table This paper proposes a fire detection method based on color feature,
4. wavelet analysis, and convolutional neural network for substation
environment. Firstly, the candidate regions of flame and smoke are
extracted by color segmentation method and background blur
model respectively, and then the candidate regions are screened by
CNN classifier, and the positions of flame and smoke are quickly
and accurately located. The experimental results show that the

251
proposed algorithm can detect flames and smoke in similar “ Development and Application of Electric Power Patrol Robot
substation scenarios, which can meet the requirements of rapid Based on Artificial Intelligence Technology ”,Thanks very much.
and accurate substation fire detection.
7. REFERENCE
6. ACKNOWLEDGMENTS
This work was financially supported by the technology project of
State Grid Corporation of China,The specific project name is
[1] LIAN Qiusheng,LI Dang. A smoke detection algorithm [11] Krstinic D, Stipanicev D, Jakovcevic T. Histogram-based
combined multiple properties.OPTICAL TECHNIQUE, smoke segmentation in forest firedetection system.
2009,35(4):523-528. Information Technology and Control 38, 237–244.
[2] Verstockt S , Vanoosthuyse A , Hoecke S V , et al. Multi- [12] Iii W P , Shah M , Lobo N D V . Flame recognition in
sensor Fire Detection by Fusing Visual and Non-visual Flame video[J]. Pattern Recognition Letters, 2000, 23(1):319-327.
Features[C]// International Conference on Image & Signal
[13] Töreyin B U,Dedeoglu Y, Güdükbay U,et al. Computer
Processing. Springer-Verlag, 2010.
vision based method for real-time fire and flame detection.[J].
[3] Toreyin B U, Dedeoglu Y, Gudukbay U, Cetin A E. Pattern Recognition Letter, 2006, 27(1):49-58.
Computer vision based method for real-time fire and flame
[14] Fukushima K . Neocognitron: A self-organizing neural
detection [J]. Pattern Recognition Letter, 2006, 27(1):49-58.
network model for a mechanism of pattern recognition
[4] Çelik T, ÖzkaramanlıH, Demirel H. Fire and smoke unaffected by shift in position[J]. Biological Cybernetics,
detection without sensors: image processing based approach 1980, 36(4):193-202.
[C]// European Signal Processing Conference. IEEE, 2015. [15] Hubel D H , Wiesel T N . Ferrier Lecture: Functional
[5] SHU Xueming.A new method of laser sheet imaging-based Architecture of Macaque Monkey Visual Cortex[J].
fire smoke detection[J].J. Fire Sciences,2006,24(2):95-104. Proceedings of the Royal Society of London. Series B,
[6] XIE Zhenping, WANG Tao, LIU Yuan. New video smoke Containing papers of a Biological character. Royal Society
detection method using Bayesian decision. Computer (Great Britain), 1977, 198(1130):1-59.
Engineering and Applications, 2014, 50(3):173-176. [16] Lecun Y L , Bottou L , Bengio Y , et al. Gradient-Based
[7] Gomez-Rodriguez F , Arrue B C , Ollero A , et al. SPIE Learning Applied to Document Recognition[J]. Proceedings
Proceedings [SPIE AeroSense 2003 - Orlando, FL (Monday of the IEEE, 1998, 86(11):2278-2324.
21 April 2003)] Automatic Target Recognition XIII - [J]. [17] Celik T ,Demirel H .Fire detection in video sequences using a
2003, 5094:404-411. generic color model [J].Fire Safety Journal ,2009 ,44(2):147-
[8] Yuan F N.A fast accumulative motion orientation model 158 .
based on integral image for video smoke detection[J].Pattern [18] RONG Jianzhong,ZHOU Dechuang,YAO Wei,et al .Fire
Recognition Letters,2008,29(7):925-932. flame detection based on GICA and target tracking
[9] YUAN Feiniu,ZHANG Yongming,LIU Shixing,et al.Video [J] .Optics & Laser Technology,2013,47(4):283-291 .
Smoke Detection Based on Accumulation and Main Motion [19] DU Jing,YAN Yunyang,GAO Shangbing,et al .Video flame
Orientation.[J]Journal of Image andGraphics,2008,13(4):808- detection [J] .Journal of Computer-Aided Design &
813. Computer Graphics ,2014,26(3):479-485 .
[10] Yuan F N.A double mapping framework for extraction of
shape-invariant features based on multi-scale partitions with
AdaBoost for video smoke detection[J].Pattern
Recognition,2012,45(12):4326-4336.

252