Building and Environment 45 (2010) 1113–1122

Contents lists available at ScienceDirect

Building and Environment

journal homepage: www.elsevier.com/locate/buildenv

Multi-feature fusion based fast video flame detection

Juan Chen a, c, Yaping He b, Jian Wang a, *
a
State Key Laboratory of Fire Science, University of Science and Technology of China, Hefei 230027, PR China
b
School of Engineering, University of Western Sydney, Australia
c
Zhejiang Key Laboratory of Safety Engineering and Technology Research, Zhejiang Institute of Safety Science and Technology, Hangzhou 310012, PR China

a r t i c l e i n f o a b s t r a c t

Article history: A video flame detection method based on the multi-feature fusion is presented in this paper. The
Received 26 June 2009 temporal and spatial characteristics of flames, such as ordinary flame movement and color clues, a flame
Received in revised form flickering detection algorithm is incorporated into the scheme to detect fires in color video sequences. An
22 October 2009
improved Gaussian mixture model method is firstly adopted to extract moving foreground objects from
Accepted 25 October 2009
the still background of detection scenes; secondly, detected moving objects are then categorized into
candidate and non-candidate flame regions by using a flame color filtering algorithm; finally, a flame
Keywords:
flicker identification algorithm based on statistical frequency counting is used to distinguish true flames
Motion detection
Gaussian mixture model from fire-like objects in video images. Testing results show that the proposed algorithms are effective,
Color mapping robust and efficient. The processing rate of the flame detection method can achieve 24 fps with image
Flickering analysis size of 320  240 pixels on a PC with an AMD 2.04 GHz processor.
Ó 2009 Elsevier Ltd. All rights reserved.

1. Introduction incorporated in the existing surveillance systems with a minimal

addition to total fire protection cost.
Fire detection is an important part of building fire safety systems. Early methods of visual fire detection rely extensively on spectra
Early and reliable detection and warning are crucial for building analysis using expensive spectroscopy equipments [6]. The cost of
occupants to evacuate to safety during fire emergencies [1,2], other such equipments hinders the wide application of this technology.
fire safety measures, such as fire suppression system and smoke To by-pass the spectroscopy and utilize the ordinary black-and-
control unit, may also rely on fire detections systems to activate [3]. white or color surveillance cameras, effective and efficient digital
The conventional fire detection technology is usually seen in the image processing algorithms are required. Healey et al. [7] pre-
forms of smoke or heat detectors. These detectors are mainly based sented a pioneering vision-based fire detection technique which
on sampling of soot particles, product gas, temperature or radiant makes use of the color information to distinguish fire or non-fire
heat [4]. Their applications are, as a result, confined to enclosure fire regions in the captured images. Cheng et al. [8] demonstrated the
detections. For large space and out-door scenes this kind of tech- principle of the visual fire detection with the aid of liquid crystal
nology suffers a dramatic reduction in detection capability because light valve to adjust the valid integral range of CCD cameras, and
the variations in detectable parameters become so weak when the used the high intensity of flame image to extract flame pixels.
detection distance becomes large. A new technology of video fire Methods simply based on criteria of flame color or intensity
detection has attracted considerable attention with the develop- have low reliability and may produce false alarms in case of the
ment of CCTV (Closed Circuit Television) surveillance system in presence of non-fire objects which resemble the color of fires, e.g.,
recent years [6–10,12–14,17–22]. This new technology can provide a reflection of the Sun, a torch or a moving flag. To improve the
more reliable information and be more cost-effective. The video fire robustness and reliability of the detection algorithm, the analysis of
detection technology uses CCD cameras to capture images of the ordinary motion feature is incorporated to identify other fire aliases
observed scene, which provides abundant and intuitive information in addition to chromatic features. Yamagishi and Yamaguchi [9]
for fire detection using image processing algorithms. This tech- introduced a neural network based flame detection method by
nology can be easily incorporated into the intelligent building adopting the spatiotemporal features of the flame contour as
systems [5]. Furthermore, this video based fire detect system can be inputs. The contour of the flame is extracted in their method on the
basis of statistical criteria from images in HSV color space.
The contour is then mapped into a polar coordinate to convert the
* Corresponding author. Tel.: þ86 5513606463; fax: þ86 5513601669. image sequences into a fluctuating time series. This time series
E-mail address: [email protected] (J. Wang). signal is transferred into Fourier domain to obtain the frequency

0360-1323/$ – see front matter Ó 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.buildenv.2009.10.017
1114 J. Chen et al. / Building and Environment 45 (2010) 1113–1122

spectrum which is at last passed to a trained neural network to movement of turbulent open flames. This method represents
identify fire flames. These procedures give good results but the a fusion of flickering and flow movement analyses. It delivers
computational complexity is too high to be used for real-time interesting and promising results and may be applied with ordinary
sequences. Phillips et al. [6] used the color predicate information video cameras for video fire detection. An example of the applica-
and temporal variation to recognize flame in video sequences. tion of video surveillance flame detection in conjunction with a fire
A manually labeled training set is employed to create a look-up suppression system can be found in Chen et al. [22], though the
table in the preprocessing phase. By using the look-up table, fire flame identification algorithm was not fully described in this
colored regions are extracted in the video image first. Then, particular publication.
a temporal variation of pixels based on a small subset of images is In the current study, the focus is given to a multi-feature fusion
also used to determine which parts of these pixels are actually fire based flame detection method to improve the speed and the reli-
pixels. The shape of a fire region is also represented in terms of the ability of video fire detection. Most of the algorithms discussed
spatial frequency content of the region contour using its Fourier above face some typical problems in robustness and flexibility, as
coefficients by Liu and Ahuja [10]. The temporal changes in these well as efficiency. Typically, the challenges are imposed by the
coefficients are treated as the temporal signatures of the fire region. changes in lighting conditions, image quality, scene complexity and
However, the problem is that spatial quantization errors for small processor performance. These challenges are addressed by a multi-
regions are likely to introduce considerable noise in the Fourier feature fusion method in the current study. In addition to the
domain. To avoid this problem, a threshold is introduced to elimi- ordinary flame motion and color analysis, flame flickering is also
nate regions of small size and also exclude elongated narrow analyzed in this method to enhance the robustness and efficiency
regions. Consequently, the algorithm described in ref. [10] is limited and to reduce both false alarm and failure rates.
to detection of developed flames that occupy significant portions of Full descriptions of the method are presented in Section
video image frames. Phillips et al.’s [6] method and a motion ‘‘Method’’ of this paper. The applications of the method to a number
detection method by Wren et al. [11] were merged and improved of recorded fire and non-fire scenes and the results are presented in
by Celik et al. [12]. The resulting statistical color model was Section ‘‘Testing and analysis’’ followed by ‘‘Concluding remarks’’.
reported to have high reliability and processing speed. However,
the demonstrated applications were limited to images of low 2. Method
resolution (176  144).
An alternative method can be found in Chen et al. [13]. In this The proposed method consists of three major steps:
method, the existence of fire is detected through analyzing flame
color characteristics first and then consolidating the results by (1) Motion detection and elimination of static background;
examining the status of the flame spread. In the analysis of flame (2) Color discrimination and elimination of non-fire colored
color, the RGB model is combined with the HSI model. The fire moving objects;
pixels in an image are simply deduced by examining the chromatic (3) Flickering test and elimination of non-flickering fire colored
features, namely the intensity and saturation of the R component. objects.
The identified fire region in the images is then further verified by
calculating the number of total fire-pixel changes from every two The first step is achieved by applying an improved fast Gaussian
consecutive frames. This algorithm also has a poor reliability. To mixture model (GMM) to analyze the RGB color space of image
detect car fires in expressway tunnels, Ono et al. [14] used features pixels. The second step is achieved by further filtering non-fire
of experimental fires in images which have high red intensity. In characteristic colors and the third by detecting the flickering
this method, the potential flame zones are identified by means of motions in the flame image. Detailed discussions of these steps are
calculating red intensity differences between the background presented in the following subsections.
image and each new coming frame. A set of fire feature parameters
is extracted from the potential fire zones and then fed into a pre- 2.1. Motion detection
designed artificial neural network (ANN) to confirm and complete
the detection. Wider applications of this method have not been Natural fire flames are seen as dynamic objects in the video
reported in the literature. images due to the reason that the majority of objects in images of
One of the characteristics of fire flames is the flicker, or the a fixed surveillance camera are static in observed scenes. This
flickering motion of the flames around the edges of the flaming feature can be utilized to distinguish a true fire flame from other
region [15,16]. To improve the reliability of detection and reduce fire-like objects or sources of fire-like images. In order to eliminate
the false alarms due to ordinary moving objects with fire-like the disturbance of the fire-like background objects (stationary
colors, some periodicity analyses are employed in detection algo- scene), the first step is to use a Gaussian mixture model (GMM) [23]
rithm. Toreyin et al. [17] and Dedeoglu et al. [18] employed to distinguish the foreground moving objects from the still back-
temporal wavelet analysis to extract the quasi-periodic behavior of ground. The Gaussian mixture model is an effective and simple
flickers and use spatial wavelet analysis to capture spatial color adaptive background modeling algorithm which tracks the history
variations of flames. Moreover, they proposed a method using of each pixel with a mixture of K Gaussian distributions respectively
three-state hidden Markov models to model flame flicker in visible in a video sequence. These K distributions are classified into back-
video images [19]. The two properties of fires, flickering and ground and foreground subsets. A pixel that fits its distributions of
maximal luminance, have been used in the method. Marbach et al. background subset is considered as background, otherwise it is
[20] proposed to realize automatic real-time video fire detection. considered as foreground. This method can solve the problem of
A temporal accumulation of time derivative images was used to multi-modal background and adapt the parameters of each mixture
select a best candidate fire region. From this candidate region, model automatically with gradual changes of illumination envi-
characteristic fire features are extracted and combined to compute ronment. However, the Gaussian mixture model suffers from a slow
the fire indicator, whose pattern are analyzed to determine speed of calculation because it is based on pixel-level which
whether fires are present or not. Schultze et al. [21] used not only contains certain degree of space-time redundancy. In order to
sonagram and spectrogram to investigate the flickering property, achieve a real-time detection of video moving objects, an improved
but also a simplified flow analysis to trace the characteristic upward algorithm is introduced.
 J. Chen et al. / Building and Environment 45 (2010) 1113–1122 1115

2.1.1. Gaussian mixture model weighted sum is more than a threshold value T are chosen to
The color feature vector for a given pixel with coordinates (x, y) approximate the background subset, while the remaining K–B
of an image at any time t in RGB color space can be expressed as: distributions are the foreground subset, where B is estimated as:
!  T " #
B
X
X xy;t ¼ Rxy;t ; Gxy;t ; Bxy;t (1)
Bxy;t ¼ arg minB wxy;k;t > T (5)
A series of observations of the color vector for a pixel (x, y) along k¼1
! !
a time sequence are recorded as f X xy;1 ; .; X xy;t g, which can be seen
The threshold T is a minimum portion which should be
as a stochastic process independent of other pixels, and has its own
accounted for the background. If T is high, the background model is
probability density function. A new observation from a new image is
composed of a multi-modal distribution and could represent
considered to be a background pixel if it is well described by this very
complex background information.
density function. However, pixel values often have complex distri-
For newly observation value of a pixel, it may match one of the
butions, making the efficient estimation and adaptation of proba-
major components of the K distributions. The matching Gaussian
bility density functions a difficult problem. Hence, a pixel in time !
distribution can be determined if the distance between X xy;t and
series is modeled by a mixture of K Gaussian distributions [23], where !
u xy;k;t is less than a threshold of l multiplied by the standard
K is the number of Gaussian distributions. With a larger K, the system
deviations. Every new observation of a given pixel is checked
can model more complex scenes at the cost of substantially increased
against the existing K Gaussian components according to the
computational expenses. Taking the calculation complexity into
following equation to determine which of the previous distribution
account, K generally ranges from 3 to 5 [23]. The probability of
is satisfied firstly.
observing a given color vector for a pixel is then expressed as
K !  8 ! 
!  X ! <  ! 
P X xy;t ¼ wxy;k;t h X xy;t ; u xy;k;t ; Sxy;k;t (2) 1; if  X xy;t  u xy;k;t <lsxy;k;t
Mxy;k;t ¼ (6)
k¼1 : 0; otherwise
where wxy,k,t, is the weight, which is regarded as the probability
where Mxy, k,t is a matching function of the kth Gaussian distribution
that an observation value belongs to the kth Gaussian component
! at time t to update parameters in the following step; the modera-
(the sum of these K weights is one), u xy;k;t and Sxy, k,t are mean
tion factor l is a positive number and is usually set in the range of
vector and covariance matrix of the kth Gaussian distribution in the
2.5–3 [24] to achieve a confidence level higher than 0.95. If the
mixture at time t, respectively. h is the Gaussian probability density !
observation value X xy;k;t matches with one of these B background
function, defined as:
distributions, then the pixel (x, y) is marked as a background pixel
! 
h X xy;t ; !
u xy;k;t ; Sxy;k;t for the present frame, otherwise it is considered to be a foreground
pixel.
! ! P1 ! !
1 1=2ð X xy;t  u xy;k;t Þ
T
ð X xy;t  u xy;k;t Þ In practice, the sunlight change during daytime, weather
¼ n 1
e xy;k;t ð3Þ
ð2pÞ jSxy;k;t j
2 2 conditions and switching light could change illuminations in the
scene. Therefore, it is necessary to update parameters of GMM to
where n is the dimension of color feature vector, which as a result adapt for the changed illumination or other natural effects. Thus,
equals to 3. according to the current match result for each frame, the corre-
To avoid time-consuming matrix calculation, the red, green, and sponding Gaussian mixture model for each pixel will be updated
blue color channels of the each pixel are assumed to be indepen- depending on the current observation value. The method for
dent and have the same variance, which means updating the first matched distribution is as follows: at time t, if
h i there is a match in a mixture of K Gaussian distributions, the mean
Sxy;k;t ¼ diag s2R ; s2G ; s2B ¼ s2xy;k;t I (4) vector and covariance are adjusted according to the current
!
observation X xy;t , but remain the same for non-match distributions.
where I is the identity matrix and s is the standard deviation. Meanwhile, the weights of the K distributions of time t are adjusted
It is known that a larger weight of Gaussian distribution, w, by the current matching function. A simple adaptation of corre-
means more observations matching the Gaussian distribution over sponding pixel’s model parameters can be formulated as in the
a period of time; a smaller standard deviation s stands for a more following equation:

8
>
>w ¼ ð1  aÞwxy;k;t þ aMxy;k;t
> xy;k;tþ1
>   !
>
>! !
< u xy;k;tþ1 ¼ 1  rxy;k;t u xy;k;t þ rxy;k;t X xy;t
>
  ! T  !  (7)
>
> s2xy;k;tþ1 ¼ 1  rxy;k;t s2xy;k;t þ rxy;k;t X xy;t  !
u xy;k;tþ1
!
X xy;t  u xy;k;tþ1
>
>  
>
> 2
>
: rxy;k;t ¼ ah Xxy;t ; uxy;k;t ; s I xy;k;t

stable Gaussian distribution. Therefore, the K Gaussian distribu- where a refers to the learning rate of the background updating,
tions describing the history of a pixel are ordered by the descending with a values between 0 and 1. If there is no match in the K
!
value of w/s first to determine its status, i.e., whether the pixel distributions for the current observation X xy;t , the Gaussian
belongs to the background subset or the foreground subset. Typi- component with small w/s is replaced by a new distribution with
!
cally, the moving objects will be represented by some distributions its mean vector set by X xy;t , and with an initially large variance and
with small weights. Therefore, the first B distributions whose small prior weight.
1116 J. Chen et al. / Building and Environment 45 (2010) 1113–1122

2.1.2. Fast algorithm for Gaussian mixture model

The background subtraction method discussed above is based j! !
X A;t  u A;k;t j < l1 sA;k;t (10)
on establishing a GMM for each pixel, which has the advantages of
flexibility and accuracy. However, as large portions of image frames where the moderation factor l1 is a positive number smaller than l,
are usually occupied by still background, the information obtained then the mean and the standard deviation of the Gaussian distri-
from applying GMM to the background pixels is highly redundant. bution are updated using the following equations instead of Eq. (7):
The pixels which are associated with potential flame images are ! !
well correlated in spatial and temporal domains and these corre- u A;k;tþ1 ¼ u A;k;t (11)
lations can be utilized to reduce the amount of data processing task. and
In other words, it is unnecessary to apply GMM to every pixel in
 
a frame. s2A;k;tþ1 ¼ 1  rA;k;t s2A;k;t (12)
Based on the correlation of pixels in temporal and spatial
domains, the pixels in a frame can be classified into active pixels The above discussed algorithm removed redundant information
and inactive pixels and only the active pixels instead of all the pixels in both temporal and spatial domains, and greatly improves the
in the image are processed [25,26]. For example, Fig. 1 represents an efficiency of motion detection over the existing method employed
image with a resolution of 6  6, where the grey cells represent by Phillips et al. [6] and Celik et al. [12].
active pixels and the white inactive ones. If only the history of each
active pixel is modeled by a GMM respectively, then the total 2.1.3. Motion detection
number of GMM is reduced by one half. In summary, the improved Gaussian mixture model is estab-
In the current study, the active and inactive pixels are prescribed lished for each active pixel in the RGB color space, whereby each
in alternate fashion along the rows and columns of the pixel matrix active pixel is classified as part of the foreground moving object or
as shown in Fig. 1. The background or foreground statuses of all the background static object by calculating the corresponding
active pixels are judged according to the method described in model match, while inactive pixels are classified as background or
Section ‘‘Gaussian Mixture Model’’. The statuses of all inactive pixels foreground according to the classification of their neighboring
are determined according to the statuses of their four neighboring active pixels. The result of background subtraction is a binary map
active pixels. For instance, the pixel m is an inactive pixel in Fig. 1, (BM) corresponding to the following formula:
which will be classified as a background or foreground pixel in
accordance with its four adjacent pixels (A, B, C and D). If there are !
0; if X xy;t 4background
more than two pixels judged as background in A, B, C and D, the BMðx; y; tÞ ¼ ! (13)
inactive pixel m is then judged as background, and vice versa. If 1; if X xy;t 4foreground
there are equal number of foreground and background neighboring !
where X xy;t is the color feature vector in coordinates (x, y) at time t.
pixels, the status of m is judged to be background if the condition as See Eq. (1).
expressed by Eq. (8) is satisfied or otherwise foreground. Fig. 2 shows the results of moving object detection using the
improved Gaussian mixture model. Snapshot Fig. 2(a) was taken
j! !
X m;t  u m;t j < Tm (8) with an out-door flame and a walking man. Snapshot Fig. 2(c) is
! a scene of a camp fire beside a road with street lights and moving
where Tm is a threshold, X m;t is the observation value of pixel m at vehicles. The moving objects, including the walking man, the flames
!
time t and u m;t is the average of the mean vectors of the first and vehicle headlights, are segmented accurately, and the static
Gaussian distribution descended by w/s for the four neighboring background objects that have fire-like colors, such as street lamps,
active pixels A, B, C and D, i.e., are eliminated as shown in Fig. 2(b) and Fig. 2(d) respectively.
! ! ! ! ! 
u m;t ¼ u A;t þ u B;t þ u C;t þ u D;t 4 (9)
2.2. Fire color feature extraction
Redundant information also exists in the temporal domain and
this redundancy is carried forward during the updating of GMM. Flames have distinctive visual features. Except for combustions
The following algorithm is introduced to minimize this redun- involving very high temperature or certain types of fuel materials,
!
dancy. Assume the observation X A;t of active pixel A matches its where flame color can be blue, most diffusion fire flames display
!
Kth Gaussian distribution at time t; if the distance between X A;t bright white color in the core region. The color changes from white
!
and its mean u A;k;t satisfies the following condition: to yellow then to red as the temperature decreases from the core
region towards the flame edges [10]. Further flame image analysis
was conducted in the current study to quantify the description of
flame color features. Over 3000 images were collected from various
sites under 50 different illumination backgrounds. Nearly 117 k
flame pixels were identified manually from the images collected.
The RGB values of these pixels formed a flame color database.
Presented in Fig. 3 is a two-dimensional representation, or
mapping, of the flame color RGB distribution. The abscissa in this
figure is the G component value of a flame pixel and the ordinate
are the corresponding R, G and B component values of the same
pixel [27]. The mapping of G component naturally collapses on to
a straight line, whilst the R and B components are scattered in two
regions separated by the G component line. For most flame images,
the R values are greater than the G values which in turn are greater
than the B values. In addition, it can be seen from Fig. 3 that the
Fig. 1. Active and inactive pixels in an image. majority of the red dots are above a threshold value, say 110. Based
 J. Chen et al. / Building and Environment 45 (2010) 1113–1122 1117

Fig. 2. Results of motion detection. (a) and (c) are original images, (b) and (d) are the binary maps BM of (a) and (c) respectively, which are extracted using background subtraction.

on these observations, the following two rules [13] are adopted for images captured by video cameras. As can be seen from this figure,
flame color identification in the current work: there is a correlation between the blue color component and the
saturation. The saturation of the fire pixels are almost all in
Rule1: Rðx; y; tÞ  RT (14) a bounded region enclosed by curves Y1 and Y2. As a result, the
third rule is introduced to further consolidate the determination of
Rule 2 : Rðx; y; tÞ  Gðx; y; tÞ  Bðx; y; tÞ (15) fire colored pixels:

where R(x, y, t), G(x, y, t), B(x, y, t), denotes the R, G, B components of Rule 3 : Y2  Sðx; y; tÞ  Y1 (16)
a given pixel coordinates (x, y) at time t, and RT is the global
threshold value of R component It has been found through exper- where S(x, y, t) represents the value of saturation [28], Y1 and Y2 are
iment that the optimum threshold values of R component is in the upper and lower boundaries obtained by curve fitting as shown
a range from 115 to 135, which is in agreement with reference [13]. in Fig. 4. The expressions for Y1 and Y2 are as follows,
Furthermore, the background illumination may adversely affect
the video images, resulting in spurious fire-liker regions which 100  0:48Bðx; y; tÞ; if Bðx; y; tÞ  117
Y1 ¼ (17)
must be eliminated by the detection scheme. This elimination is
79:27  0:311Bðx; y; tÞ; if Bðx; y; tÞ < 117
achieved by introducing a color saturation constraint. Presented in
Fig. 4 is a plot of the color saturation vs blue color scale of true flame Y2 ¼ 2:0147 þ 90:59435 eBðx;y;tÞ=77:6027 (18)

Fig. 3. Two-dimensional RGB color component mapping of flame pixels. Fig. 4. The saturation against the blue component of the fire pixels.
1118 J. Chen et al. / Building and Environment 45 (2010) 1113–1122

To save computation time, a look-up table for the discrete B retained in Fig. 5(d) after the combined motion detection and color
value domain [0, 255] is generated from Eqs. (17) and (18) and is filtering process. Further processing is required to eliminate the
used in the processing scheme to perform Rule 3 check. images of fire-like objects.
In order to eliminate foreground moving objects which do not
correspond to flames, the above described rules are implemented
2.3. Flame flicker feature extraction
for pixels which are judged as pixels of moving objects based on
background subtraction in the binary map BM. The pixels which
It has been observed that burning fires exhibit flickering, or
have ordinary motion and fire-like color characteristics are
pulsation, movement with apparent random oscillations at flame
considered as potential flame pixels. Through color discrimination,
edge, forming intermittent flame regions [15,16,30]. This flickering
a binary Color Map (CM) is generated. CM(x, y, t) indicates whether
motion is caused by turbulent entrainment flows of the
a pixel at a given location and time (x, y, t) is classified as potential
surrounding air. According to Hamins et al. [30], the dynamic
fire pixel (binary value of 1) or non-potential fire pixel (binary value
frequency range of flame flicker is around 10 Hz no matter what the
of 0). The decision-making formula is as follows.
fuels are. More observations by Yang and Wang [31] indicate that
8 the contours and chrominance or brightness of flames generally
>
> !
>
< 1;
if ðBMðx; y; tÞ ¼ 1Þ& X xy;t 4 oscillate with a frequency range of 0.5–20 Hz.
 The characteristic oscillation frequency of the flickering motion
CMðx; y; tÞ ¼ ðRule 1 & Rule 2 & Rule 3Þ ð19Þ
>
> offers an opportunity to differentiate fire flames from other moving
>
: 0; otherwise
light sources by performing a frequency analysis. However, accord-
ing to the Shannon sampling theorem, the sampling frequency of
Signal noises existing in the RGB values may sometimes cause a signal should be greater than twice the signal frequency. Unfor-
incorrect registrations in CM. In order to remove these noises, tunately, the video capture rate is generally at 25 Hz (25 frames per
morphological operations of erosion and dilation are applied [29]. second) in most surveillance systems. This sampling frequency can
To merge the same object together as far as possible, the adjacent only reveal any parameter change rate of less than 12.5 Hz, and
pixels of CM are linked appropriately. The foreground pixels can cannot meet the sampling requirement. In addition, transforming
then be labeled into connected regions by an eight-connectivity signals from the time domain to the frequency domain is time-
component labeling algorithm [29] while any connected compo- consuming and will affect the efficiency of the detection algorithm.
nent with less than 10 pixels are removed. A different approach is employed in the current study to resolve
Fig. 5 displays the results of the combined motion detection and this problem. The approach is described below.
color mapping for the examples presented in Fig. 2. The potential With the passage of time, the brightness and color of a pixel at
fire candidates are encircled with rectangular frames. (x, y) that registers a flickering motion of a flame vary significantly
In Fig. 5(b), the moving pedestrian image in Fig. 5(a) is filtered so that the corresponding CM will undergo transformations, dis-
out from the corresponding binary image presented in Fig. 2(b). The appearing into background and reappearing in the foreground. This
effectiveness of color mapping is demonstrated. However, both the transformation or oscillation is repeated several times per second.
moving flame and the moving vehicle headlights in Fig. 5(c) are Let SUM be a matrix that register the oscillation frequency of pixels.

Fig. 5. Results of fire recognition with motion detection and fire color criteria. (a) and (c) are original images, (b) and (d) are the binary maps CM of (a) and (c) respectively, which
are extracted by using motion and color filtering.
 J. Chen et al. / Building and Environment 45 (2010) 1113–1122 1119

Fig. 6. Final flame recognitions based on flame flicker analysis. (a) and (c) are original images, (b) and (d) are the corresponding binary maps of (a) and (c) after motion detection,
color filtering and flickering analysis.

If the brightness value of a pixel at (x, y) changes between two laboratory experiments and by comparing the oscillation registra-
consecutive image frames, SUM(x, y, t) is added by 1, or 0 otherwise. tion counters for flickering motion pixels with that for non-flick-
To eliminate the impact of system noise, a global threshold TI is ering pixels in 14 different video clips. It was found that the
introduced for registering the change of brightness, DI(x, y, t). The
above scheme is formulated in Eqs. (20)–(22) below.

SUMðx; y; t  1Þ þ 1; ifðDI  TI Þ
SUMðx; y; tÞ ¼ (20)
SUMðx; y; t  1Þ þ 0; ifðDI < TI Þ
where

DIðx; y; tÞ ¼ jIðx; y; tÞ  Iðx; y; tÞj (21)

and
1
Iðx; y; tÞ ¼ ½Rðx; y; tÞ þ Gðx; y; tÞ þ Bðx; y; tÞ (22)
3
The brightness I can also be equal to 0.30R þ 0.59G þ 0.11B,
according to the transformation from the RGB color space into the
YUV color space. But for computation performance, equally
weighted average of the three colors is used to reduce floating point
computation.
The value of the threshold TI in Eq. (20) was determined by
averaging the changes in brightness of the image pixels in recorded
video sequences. The threshold TI was set to a value slightly greater
than the average change by a fraction of 10% to minimize of the
effect of noise.
A pixel is regarded as part of the flickering flame picture if its
oscillation registration counter exceeds a threshold SUM0 in a given
period:

ðSUMðx; y; tÞ  SUMðx; y; t  NÞÞ > SUM0 (23)

where N is the counting period. The threshold SUM0 is also related
to the counting period. Its value was determined in image Fig. 7. Flow chart of the proposed fire detection algorithm.
1120 J. Chen et al. / Building and Environment 45 (2010) 1113–1122

Fig. 8. Detection result of highway scene.

oscillation counters for flickering pixels registered much larger

values than those for non-flickering pixels. The value of SUM0 was Nis0 =NiRECT > r (24)
then set in between the two group average values.
The flicker analysis is performed to the candidate flame regions then the region is regarded as a flame region; otherwise, it is
in the binary map CM [see the rectangles in Fig. 5(b and d)] after the regarded as a pseudo-flame region.
color filtering described in Section ‘‘Fire Color Feature Extraction’’. Fig. 6 depicts the results of the combined motion detection,
The number, Nis0, of the pixels that satisfy Eq. (23), as well as the color filtering and flickering analysis to two original video
total number, NiRECT, of the object pixels within the candidature sequences. It can be seen by comparing Fig. 6(d) with Fig. 5(d) that
flame region, is counted. If the ratio of the two numbers for disturbances of vehicle headlights were removed after flickering
a candidate flame region exceeds a threshold r, which is deter- analysis.
mined by real video fire clips, i.e.,

Fig. 9. Testing images of eight different scenes. Successful detections of flames are marked by green frames around the identified objects and successful omissions of non-fire
moving light sources are shown in (d) and (e).
 J. Chen et al. / Building and Environment 45 (2010) 1113–1122 1121

2.4. Flow chart of the algorithm that the testing of the algorithm was conducted on video clips
obtained from independent sources, which were taken under
The flow chart of the proposed video flame detection algorithm limited environmental conditions. Other environmental conditions,
is presented in Fig. 7. such as wind or ambient air movement and heat release rate of the
fires, were not known in those incidences. Further laboratory real-
3. Testing and analysis time experiments with wider a range of recoded conditions are
required to achieve more quantifiable performance assessment.
The proposed algorithm has been implemented using Cþþ and The video flame detection technology relies on the flame visual
tested on a PC with an AMD 2.04 GHz processor and 1 GB memory. characteristics of fires. This limits its applications to flaming fire
The algorithm was applied to a set of video clips from a variety of detection only. It can be used to enhance fire detection, but not to
scenes including different environmental background and illumi- replace other type of detectors with which non-flaming and/or
nation conditions. The free available video clips were collected non-visible fires can be detected. Further work is needed to extend
from http://202.117.58.58/files/videoForFlasm.rar and http://signal. the video flame detection technology to detect flames with unusual
ee.bilkent.edu.tr/VisiFire/Demo/. The resolution of the video color characteristics, such as those produced by hydrogen or metal
images are 320  240. fuels.
Fig. 8 contains multiple images of a fire beside a highway at The detection algorithm presented in this paper can be
nighttime. In the original video clip, the ignition occurs at frame No. combined with a location determination algorithm to generate
635, and the flame can be identified by visual inspection of frame information for automatic fire suppression and fire fighting oper-
No. 643. Fig. 8 presents frames Nos. 645, 680, 694 and 704. The ations. It is possible to extend the technology such that not only the
detection of the flame by the proposed method is highlighted by location but also the size of the detected fires can be determined.
the green rectangle inside the pictures. It can be seen from these
results that the proposed algorithm was able to detect the fire in
a very short time, less than 2 s from the start of the fire. The Acknowledgements
detection was not disturbed by the cyclist, street lamps and vehicle
headlights. The authors are indebted to Mr Ma Jian and Dr Jeffrey Zou for
Presented in Fig. 9 are test images of eight different scenes. The their helpful discussions. The support from the Ministry of Science
fire in Fig. 9(a) is against the high-brightness of snow and there are and Technology of the People’s Republic of China (2006BAK06B07)
some pedestrians and brown grassland in the background. Fig. 9(b) is acknowledged.
depicts a fire in a corridor with a strong floor reflection. The fire in
Fig. 9(c) is located on a lawn. In all of these scenes the fires appear at
long ranges which result in small observable flame images. References
Nevertheless, the fires were successfully detected. Fig. 9(f–h)
demonstrate the successful detection of close range fires against [1] Proulx G. Evacuation time, the SFPE handbook of fire protection engineering.
not much dissimilar background colors. The steady and moving 4th ed. Quincy, MA: National Fire Protection Association; 2008. p. 3.355–3.372.
[2] Tavares RM, Galea ER. Evacuation modelling analysis within the operational
light sources in Fig. 9(d and e) were successfully eliminated by the research context: a combined approach for improving enclosure designs.
detection algorithm. Building and Environment 2009;44(5):1005–16.
The test results have shown the adaptability and robustness of [3] Custer RLP, Meacham BJ, Schifiliti RP. Design of detection systems, the SFPE
handbook of fire protection engineering. 4th ed. Quincy, MA: National Fire
the proposed detection algorithm under different scenes and illu- Protection Association; 2008. p. 4.1–4.44.
mination conditions. The processing rate of the flame detection [4] Cote AE, editor. Fire protection handbook. 19th ed. Quincy, Mass: National Fire
method achieved 24 fps in the sample applications. Protection Association; 2003.
[5] Wong J, LI H. Development of a conceptual model for the selection of intel-
ligent building systems. Building and Environment 2006;41(8):1106–23.
4. Concluding remarks [6] Phillips III W, Shah M, da Vitoria Lobo N. Flame recognition in video. Pattern
Recognition Letters 2000;23(1–3):319–27.
[7] Healey G, Slater D, Lin T, Drda B, Goedeke AD. A system for real-time fire
In this study, a video flame detection method has been developed
detection. In: Proceedings of conference on computer vision and pattern
based on the multi-feature fusion. The method incorporates three recognition (CVPR ’93); 1993. p. 605–6.
features, namely, motion detection, statistical color filtering and [8] Cheng XF, Wu JH, Yuan X, Zhou H. Principles for a video fire detection system.
statistical flickering analysis, into one scheme. The method is able to Fire Safety Journal 1999;33(1):57–69.
[9] Yamagishi H, Yamaguchi J. Fire flame detection algorithm using a color
achieve high detection reliability and fast processing speed for camera. In: Proceedings of 1999 International symposium on micro-
reasonably high-resolution images. Firstly, a real-time adaptive mechatronics and human science; 1999. p. 255–60.
background subtraction method based on a Gaussian mixture model [10] Liu CB, Ahuja N. Vision based fire detection. In: Proceedings of the 17th
International conference on pattern recognition (ICPR ’04); 2004. p. 134–7.
is applied to aid the segmentation of moving objects which may be [11] Wren C, Azarbayejani A, Darrell T, Pentland A. Pfinder: real-time tracking of
potential fire flames. Secondly, the images are further subjected to the human body. IEEE Transactions on Pattern Analysis and Machine Intelli-
color mapping and flickering counting to distinguish true fire flames gence 1997;19(7):780–5.
[12] Celik T, Demirel H, Ozkaramanli H, Uyguroglu M. Fire detection using statis-
from fire-like objects. The improved Gaussian mixture model and tical color model in video sequences. Journal of Visual Communication and
the simple flickering identification method significantly enhanced Image Representation 2007;18(2):176–85.
the efficiency of the data processing. In particular flickering identi- [13] Chen TH, Wu PH, Chiou YC. An early fire-detection method based on image
processing. In: Proceedings of IEEE International Conference on image pro-
fication method is based on the statistics of the change frequencies cessing (ICIP ’04); 2004. p. 1707–10.
in pixel brightness in video sequences. This method overcomes the [14] Ono T, Ishii H, Kawamura K, Miura H, Momma E, Fujisawa T, et al. Application
difficulty and complexity associated with the traditional trans- of neural network to analyses of CCD colour TV-camera image for the detec-
tion of car fires in expressway tunnels. Fire Safety Journal 2006;41(4):279–84.
formation and analysis in the frequency domain.
[15] McCaffrey BJ. Fire plume dynamics – a review. In: Proceedings of large-scale
The experimental results, which were obtained under a variety fire phenomenology conference; 1984. p. 1–10.
of experimental environments, demonstrate that the multi-feature [16] Detriche P, Lanore JC. An acoustic study of pulsation characteristics of fires.
fusion video flame detection algorithm has good adaptability, Fire Technology 1980;16(3):204–11.
[17] Toreyin BU, Dedeoglu Y, Gudukbay U, Cetin AE. Computer vision based method
robustness and efficiency. Moreover, this method can be sued for for real-time fire and flame detection. Pattern Recognition Letters
real-time surveillance and flame detection. It should be pointed out 2006;27(1):49–58.
1122 J. Chen et al. / Building and Environment 45 (2010) 1113–1122

[18] Dedeoglu Y, Toreyin BU, Gudukbay U, Cetin AE. Real-time fire and flame K: the number of Gaussian distributions in a mixture model
detection in video. In: Proc. IEEE International Conference on acoustics, Mxy,k ,t: matching function of the kth Gaussian distribution of a pixel with coordinates
speech, and signal processing (ICASSP ’05); 2005. p. 669–72. (x, y)at time t
[19] Toreyin BU, Dedeoglu Y, Cetin AE. Flame detection in video using hidden N: a given integer in order to track for N consecutive frames of the changes in
markov models. In: Proc. 2005 International conference on image processing brightness
(ICIP 2005); 2005. p. 2457–60. Nis0: the number of the pixels that satisfy S(x, y, t) >S0 within the identified candidate
[20] Marbach G, Loepfe M, Brupbacher T. An image processing technique for fire rectangular region i
detection in video images. Fire Safety Journal 2006;41(4):285–9. NiRECT: the number of the object pixels within the identified candidate rectangular
[21] Schultze T, Kempka T, Willms I. Audio-video fire-detection of open fires. Fire region i
!
Safety Journal 2006;41(4):311–4. Pð X xy;t Þ: the probability of observing a given color vector value for a pixel (x, y)
[22] Chen T, Yuan HY, Su GF, Fan WC. Automatic fire searching and suppression R(x, y, t): the red component of a given pixel with coordinates (x, y) at time t
system for large spaces. Fire Safety Journal 2004;39(4):297–307. RT: a threshold of R component
[23] Stauffer C, Grimson WEL. Learning patterns of activity using real-time SUM(x, y, t): counter that registers the oscillation frequency of a pixel with coordi-
tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence nates (x, y) at time t
2000;22(8):747–57. SUM0: threshold of the oscillation frequency of flames after tracking for N consec-
[24] Huang WQ, Wang YM. Tracking and detecting multi-objects based on adaptive utive frames
mixture models and features of region. Computer Measurement & Control S(x, y, t): the saturation of a given pixel with coordinates (x, y) at time t
2003;11(9):648–50. 654 [in Chinese]. T: the minimum portion accounted for the background
[25] Gianluca B, Massimo B. Background estimation with Gaussian distribution for TI: a global threshold for registering the change of brightness
image segmentation, a fast approach. In: Proceedings of the 2005 IEEE Inter- Tm: threshold that determines the match of inactive pixels between their obser-
national workshop on measurement systems for homeland security, contra- vations and mean vector
!
band detection and personal safety workshop (IMS 2005); 2005. p. 29–30. u xy;k;t : mean vector of the kth Gaussian for a given pixel with coordinates (x, y) in
[26] Meng F, Wang CR. Research of fast moving object detection in multimode the mixture at time t
!
background. Electronic Measurement Technology 2007;30(6):34–5. 79 [in u A;k;t : mean vector with the highest priority of active pixel A at time t
Chinese]. wxy,k,t: weight of the kth Gaussian in a mixture model for a given pixel with coor-
[27] Cheng X, Wang DC, Yin DL. Image type fire flame detecting principle. Fire dinates (x, y) at time
!
Safety Science 2005;14(4):239–45 [in Chinese]. X xy;t : color feature vector for a given pixel with coordinates (x, y) at time t
[28] Gonzalez RC, Woods RE. Digital image processing. Beijing: Publishing House of x: x coordinate
!
Electronics Industry; 2003 [in Chinese]. u : the average mean of a inactive pixel m at time t
!m;t
[29] Gonzalez RC, Woods RE, Eddins SL. Digital image processing using MATLAB. X : the observation of a inactive pixel m at time t
!m;t
Beijing: Publishing House of Electronics Industry; 2005 [in Chinese]. X A;t : the observation of a active pixel A at time t
[30] Hamins A, Yang JC, Kashiwagi T. An experimental investigation of the pulsa- y: y coordinate
tion frequency of flames. In: Proceedings of the 24th (International) sympo- a: learning rate of background updating
sium on combustion; 1992. p. 1695–702. DI(x,y,t): the change in brightness of a pixel at (x, y) between consecutive frame at
[31] Yang J, Wang RS. A survey on fire detection and application based on video time t
image analysis. Video Engineering 2006;8:92–6 [in Chinese]. l: the moderation factor that determines the matching distance between the
observation values and corresponding mean for the updating scheme in the
GMM. See Eq. (6)
Nomenclature l1: the moderation factor that determines the matching distance between the
observation values and corresponding mean for simplified updating scheme. See
Bxy,t: background subset of theK Gaussian distributions for a pixel (x, y) at time t Eq. (9)
B(x, y, t): blue component of a given pixel with coordinates (x, y) at time t h: Gaussian probability density distribution function
BM(x, y, t): Background Map, a binary map corresponding background subtraction r: the threshold of area ratio for a candidate flame region
CM(x, y, t): Color Map, a binary map corresponding color filtering Sxy,k ,t: covariance matrix of the kth Gaussian for a given pixel with coordinates (x, y)
G(x, y, t): green component of a given pixel with coordinates (x, y) at time t in the mixture at time t
I: identity matrix sxy,k,t: standard deviation for a given pixel with coordinates (x, y) at time
I(x, y, t): brightness of pixel (x, y) at time t sA,k,t: standard deviation of a active pixel A at time t