Complex & Intelligent Systems

https://doi.org/10.1007/s40747-022-00792-9

ORIGINAL ARTICLE

Multimodal medical image fusion with convolution sparse

representation and mutual information correlation in NSST domain
Peng Guo1,2 · Guoqi Xie2 · Renfa Li2 · Hui Hu1

Received: 9 June 2021 / Accepted: 27 May 2022

© The Author(s) 2022

Abstract
Multimodal medical image is an effective method to solve a series of clinical problems, such as clinical diagnosis and
postoperative treatment. In this study, a medical image fusion method based on convolutional sparse representation (CSR)
and mutual information correlation is proposed. In this method, the source image is decomposed into one high-frequency
and one low-frequency sub-band by non-subsampled shearlet transform. For the high-frequency sub-band, CSR is used for
high-frequency coefficient fusion. For the low-frequency sub-band, different fusion strategies are used for different regions
by mutual information correlation analysis. Analysis of two kinds of medical image fusion problems, namely, CT–MRI and
MRI–SPECT, reveals that the performance of this method is robust in terms of five common objective metrics. Compared
with the other six advanced medical image fusion methods, the experimental results show that the proposed method achieves
better results in subjective vision and objective evaluation metrics.

Keywords Medical image fusion · NSST · Convolution sparse representation · Mutual information correlation

Introduction attention of researchers, because it adopts a similar multires-

olution processing mechanism to the human visual system.
In recent years, medical imaging has become an indispens- These methods include pyramid-based decomposition [3],
able means in clinical diagnosis, surgery, and radiotherapy. wavelet-based decomposition [4] and multiscale geomet-
However, single-modality medical images only focus on a ric analysis decomposition [5]. These methods exist two
certain type of morphological features. For example, com- common problems: first, it is difficult to determine the
puted tomography (CT) images reflect structural information decomposition level; Second, the fusion strategy is difficult
about bone but are insensitive to soft tissues with a similar to choose.
density. Although magnetic resonance imaging (MRI) has a Decomposition level is a key problem to be solved. When
strong ability to display soft tissues, it is poor in showing bone the decomposition level is low, it cannot extract enough
lesions and calcified lesions. Therefore, image fusion can spatial details from the source map, whereas when the
obtain complementary information from different modalities decomposition level is high, the fusion of high-frequency
of medical images and help clinicians perform postoperative sub-band is more sensitive to noise and registration [6–8].
detection and tumor and bone growth monitoring [1, 2]. The general solution is simply to decompose an image into
Among many medical image fusion methods, a kind one high-frequency sub-band and one low-frequency sub-
of method based on multiscale transform has attracted the band. The high-frequency sub-bands contain more details
and edge information, whereas the low-frequency sub-band
contains the contour and structure information of images.
In recent years, multiscale decomposition methods based
B Guoqi Xie
[email protected] on NSCT and NSST are popular because of their mul-
tiscale, multidirectional, and shift invariant. In particular,
1 School of Computer and Communication, Hunan Institute of NSST has attracted more attention because of its superior
Engineering, Xiangtan 411104, People’s Republic of China
computational efficiency to NSCT. Compared with pyramid-
2 College of Computer Science and Electronic Engineering, based methods, such as Gaussian pyramid decomposition,
Hunan University, Changsha 410082, People’s Republic of
China

123
 Complex & Intelligent Systems

Laplacian pyramid decomposition [9], and gradient pyra- time. To obtain better fusion effect, different fusion strategies
mid transformation [10], the method based on NSST can be are selected according to different sub-bands in this study,
decomposed from multiple directions, thus obtaining more namely, maximum fusion is used for the high-frequency
image details. Compared with wavelet methods, such as sub-band and details of the low-frequency sub-band, and
discrete wave and dual tree complex wavelet, the method weighted average fusion is used for the similar structural
based on NSST can represent the curve and edge details information in the low-frequency sub-band.
of image well. Compared with multiscale geometric trans- This study focuses on the determination of decomposition
formations, such as contourlet transform (COT) [11] and scale and the fusion strategy of different frequency bands in
shearlet transform (ST) [12], the method based on NSST does NSST decomposition. To avoid the influence of high noise
not produce pseudo Gibbs phenomenon due to frequency and registration on the fusion of high-frequency sub-band
aliasing. However, most of the existing NSST decomposi- when the NSST decomposition scale is too high, this study
tion methods have a higher decomposition levels, which not only carries out one-level decomposition of NSST, that is,
only increases the amount of calculation, but also makes one high-frequency sub-band and one low-frequency sub-
high-frequency sub-bands susceptible to noise. To preserve band. How to use mutual information correlation analysis
the structural information of the image as much as possi- to mine the detailed information in the low-frequency sub-
ble and to extract additional salient details, a new multiscale band is one of the research objectives of this study. It has
decomposition method is proposed in this study. Unaffected been explained above that it is inappropriate for all sub-bands
by the scale parameters of the general multiscale decom- to adopt the same fusion strategy. Another objective of this
position, this method uses NSST to decompose the image study is to study which fusion strategy should be adopted for
only in two scales, namely, one high-frequency sub-band and high-frequency sub-band, and similar and dissimilar regions
one low-frequency sub-band. In addition to using convolu- of low-frequency sub-band.
tion sparse representation to enhance the detailed information The main innovation points of this study include the fol-
of high-frequency sub-band, correlation analysis should be lowing three aspects:
used to extract the detailed information of low-frequency
sub-band due to the rich detailed information contained in
low-frequency sub-band. Sparse representation seeks to rep- 1. The convolutional sparse representation (CSR) model
resent image features with as few sparse vectors as possible, is used to process the high-frequency sub-band, which
which is widely used in image reconstruction and denoising. increases the detailed features and reduces the block
The improvement of convolution sparse representation is that effect caused by NSST decomposition, as well as the
the sparse coefficients of local image blocks are replaced by redundant information of different source graphs.
global sparse coefficients. 2. Mutual information correlation is used to extract detail
The fusion strategy is important for the quality of fused information of low-frequency sub-bands. Given that
image. In multiscale decomposition, a common strategy is only two-scale decomposition is conducted, the low-
to measure the activity of the decomposition coefficients frequency sub-band contains abundant details. The
first and then fuse them in accordance with the mean or mutual information correlation analysis can find the
maximum value. For example, in [13, 14], high- and the low- regions containing detailed information from the low-
frequency sub-bands adopt the maximum scheme for fusion. frequency sub-band.
However, low-frequency sub-bands provides structure infor- 3. Two different fusion strategies are used for low-
mation similar to the source image, whereas high-frequency frequency sub-band. The structural information of simi-
sub-bands contain important details, thus, the same fusion larity is fused using the weighted average scheme, where
scheme cannot consider the similarity and importance of the the weight takes the product of the correlation analysis
image simultaneously. In [15], a weighted average fusion coefficient and the regional energy sum. The Laplacian
strategy is adopted for similar regions of images, in this energy gradient was used to measure the activity of the
strategy, weight is calculated using the Siamese network. dissimilar regions to reflect the contrast changes of the
However, the definition of similar regions by this method regions.
directly affects the effect of the final image fusion. Recently,
principal component analysis (PCA) [16], sparse representa-
tion [17, 18], smallest univalue segment assimilating nucleus The remaining sections of this paper are organized as fol-
(SUSAN) [19], and pulse coupled neural network (PCNN) lows: the next section describes related work about NSST and
[20, 21] have been used to enhance the salient information of CSR. The following section explains the methods in detail.
fused images and measure the activity of decomposition coef- In the next section, a comparative experiment is simulated,
ficients. However, these methods have their own problems and the corresponding results are analyzed. The last section
either in the selection of sparse dictionaries or in the training summarizes the study.

123
Complex & Intelligent Systems

is defined as

minx0 s.t. y − Dx2 < ε, (1)

x

Here, y ∈ R n is the stacked vector representation of the

√ √
image patch n × n. D ∈ R n×m is an overcomplete dic-
tionary, and x ∈ R m is the sparse coefficient to be solved.
However, the disadvantage of this model is that the sparse
coefficient is obtained through the calculation of overlap-
ping patches, thus, the global sparse coefficient of the whole
image cannot be obtained. To improve fusion performance,
Fig. 1 Two-level NSST decomposition Wohlberg [29] proposed the convolution form of SR. Liu
[30] integrated morphological component analysis (MCA)
and CSR into a unified optimization framework, which could
Related work realize multicomponent and global SR of source images
simultaneously. CSR is given by the following equation:
NSST
 2
1   
M M
 
Non-subsampled contourlet and non-subsampled shearlet min Y − dm × X m  + λ X m 1 . (2)
xm 2  
waves are two popular multiscale geometric decomposition m1 2 m1
methods, because they are multiscale, multidirectional, and
shift invariant. Given that the non-subsampled shearlet wave Here, Y is the whole image, which is modeled as the
does not limit its direction and does not need to reverse sum of the convolution map between M local dictionary fil-
the directional filter bank, its computational efficiency is ters and global coefficients. the global single value and shift
higher than that of NSCT. NSST consists of two processes: invariance features of CSR are conducive to extracting more
non-subsampled pyramid scale decomposition (NSPFs) and detailed information and enhancing its robustness.
shift-invariant shearlet filter banks (SFBS). Another difficulty in SR is the learning of overcomplete
Figure 1 shows the framework of two-level NSST decom- dictionaries. In SR, the more fully the dictionary repre-
position. The input image is decomposed into a high- sents all the details of the image, the more likely the result
frequency sub-band and a low-frequency sub-band after of the reconstruction can restore the source image. The
the first-level scale decomposition by NSPF, and then the design of dictionaries usually adopts two methods: one
low-frequency sub-band is decomposed into the second- is based on known transformation basis, such as discrete
level high-frequency sub-band and low-frequency sub-band. cosine transforms and wavelet basis. However, as data and
Therefore, the input image decomposed by L-level NSST application range change, the performance of such fixed
will be transformed into L high-frequency sub-bands and one dictionaries degrades considerably. The second is a learning-
low-frequency sub-band. At each scale, multiple directions of based approach. K-SVD and its improved dictionary learning
sub-bands can be obtained by SFBs. Moreover, given that the method are widely used in medical image fusion [31, 32]. The
traditional subsampling decomposition may bring frequency adaptive K-SVD dictionary is constantly updated through
overlap, the pseudo-Gibbs phenomenon easily occurs. Thus, iterative training, and it is updated alternately with sparse
non-subsampled decomposition is adopted in NSST [22]. coding. The disadvantage is that the dictionary training time
is long. In multimodal medical image fusion, the structure
of medical images from different sensing devices is more
Convolutional sparse representation complex, and data are more redundant. Therefore, dictionary
learning based on joint block clustering is a better choice. By
The idea of SR comes from the learning process of image clustering similar patches of all source images, a complete
structures by the receptor field of simple cells in the visual and compact dictionary can be formed.
cortex V1 area. Given its simple representation, SR has been
widely used in image denoising [23], feature extraction [24],
and super-resolution [25, 26]. Yang [27] and Yin [28] et al. Proposed framework
applied SR to image fusion. The main difficulties in SR image
fusion are sparse model selection and overcomplete dictio- As shown in Fig. 2, based on the general framework of
nary learning. The common sparse model is based on a single image fusion in multiscale transformation domain, the pro-
image component and local patch, and the mathematical form posed method mainly consists of three stages: multiscale

123
 Complex & Intelligent Systems

Fig. 2 Multimodal medical image fusion framework based on NSST decomposition

decomposition, low-frequency and high-frequency sub-band the source image. Thus, the sparse coefficient fusion rule of
fusion, and NSST reconstruction. For simplicity, two source high-frequency sub-band is defined as
diagrams are used for illustration. First, NSST is applied F
X m, 1 :N (x, y)
to source image Ia and Ib . After first-level decomposition,     
 A   B 
a high-frequency sub-band with significant details and a
A
X m, (x, y) if X (x, y)  ≥ X (x, y) 
 1:N m, 1:N 1 m, 1:N 1 .
low-frequency sub-band with structure information can be X m, 1:N (x, y) otherwise
B

obtained. Second, a CSR-based maximum fusion strategy is (3)

used for the fusion of high-frequency coefficients. For the
low-frequency sub-band, mutual information is used for the The fused sparse coefficients are reconstructed by Eq. (4).
correlation analysis, and then local energy coefficients are The fused high-frequency sub-band is defined as
used for the weighted summation of similar parts, whereas
domain energy gradient and maximum fusion are used for 
M 
M

dissimilar parts. Finally, the image is reconstructed by NSST HF  dm × X mF + θ X mF . (4)

m1 m1
inverse solution.

Low-frequency sub-band fusion

High-frequency sub-band fusion
Given the low decomposition scale, the low-frequency sub-
Since the high-frequency sub-band contains the details of the band still has abundant important information. To extract
image, the fusion of the high-frequency sub-band is mainly such information further, normalized mutual information
to fuse the salient features of the high-frequency sub-band. (NMI) is used for correlation analysis. For the area with
The advantage of convolution sparse representation is that it high correlation, the fusion strategy of local energy weighted
can describe these features with fewer sparse coefficients. summation is adopted to preserve energy as much as pos-
The sparse coefficients of the high-frequency sub-band sible. For the area with low correlation, the neighborhood
of each source image are obtained by Eq. (2). Set X m k indi- energy gradient (NEG) is adopted to highlight the contrast
cates the sparse coefficient of the high-frequency sub-band edge information of the source map as much as possible.
1:N (x, y) indicates the content of the
of the k image, X m, k The basis of pixel-level image fusion is that the input
location (x, y); it is a N dimensional vector. Here, the norm images are linear and complementary. By correlation analysis
1:N (x, y) is used to measure the activity level of
k
L 1 of X m, of low-frequency sub-band, the salient features of the source

123
Complex & Intelligent Systems

maps can be preserved. Mutual information is often used in indicates the weight of the local pixel. Given that the low-
multimodal image registration, which is a statistical corre- frequency sub-band image is relatively smooth, the weight
lation method based on gray value. The greater the mutual can be directly represented in 22N −d , where d is the dis-
information between two images, the higher the correlation tance from the field pixel to the center point, if N  1, then
between the two images. The mutual information quantity of normalized W L is defined as
an image can be calculated by Kullback–Leibler Divergence,
and the mathematical form is as follows: ⎡ ⎤
 121
P(x, y) 1⎢ ⎥
MI(x, y)  P(x, y) log , (5) WL  ⎣ 2 4 2 ⎦.
x y
P(x)P(y) 16
121

H (x)  − P(x) log2 P(x), (6)
x
 For the low-frequency sub-band with high correlation, the
H (y)  − P(y) log2 P(y). (7) coefficient blocks are fused by the strategy of the weighted
y sum of energy of the center pixel and are defined as

P(x) and P(y) represent the probability distribution of

random variables X and Y , P(x, y) represents the joint distri- L F  w1 L A + w2 L B ,
bution, H (x, y) represents the joint entropy of X and Y , and (10)
EA EB
the joint entropy reflects the correlation of random variables w1  , w2  .
EA + EB EA + EB
X and Y . Here, X represents the image Ia and Y represents
the image Ib . The joint entropy of the two can be calculated
using the joint histogram. For the low-frequency sub-band with low correlation, that
The mutual information result is easily affected by the is, when the NMI of the local region is less than the threshold
clustering result with numerous clusters, thus, the NMI maps T , the fusion of the coefficient blocks adopts the strategy of
the mutual information to the interval range of [0, 1] and is maximum energy gradient. NEG is essentially the Sum of
defined as Laplace energy, which is a parameter characterizing image
2M I (x, y) edge features. NEG reflects the contrast change of the neigh-
NMI(x, y)  . (8) bor window and the edge information of the image block and
H (x) + H (y)
is defined as
After a 3 × 3 sliding window partitioning, the correla-
tion of the local region of the low-frequency sub-band can be
obtained. Here, the mutual information T of the whole image 
N 
N
NEG(x, y)  LEG(x + i, y + j)2 , (11)
is regarded as the threshold of correlation. When the local
i−N j−N
region NMI is greater than the threshold T , the image blocks
are correlated. Therefore, weighted average scheme is used.

In medical image fusion, the intensities of different source LEG(x + i, y + j)2  [L(x, y) − L(m, n)]2 . (12)
images at the same location may vary magnificently, because (m, n)∈
the source images are captured with different imaging mech-
anisms. Therefore, the weight matrix cannot be evaluated
with a simple average. Here, we use the center pixel energy Here, Ω indicates a neighbor window, after the activity
to calculate the matrix, the center pixel energy is an adap- is measured by NEG, the maximum value is taken for the
tive weight calculate method based on region energy, and the fusion of low-frequency coefficients, and the equation is as
mathematical form is defined by the following equation: follows:


N 
N 
E m (x, y)  W L (i + N + 1, j + N + 1) L A ifNEG(x, y)1 ≥ NEG(x, y)2
LF  . (13)
i−N j−N L B otherwise
× L m (x + i, y + j)2 . (9)

Here, N is the radius of the local region (2N + 1, 2N + 1), The detailed description of the algorithm is shown in Algo-
L m is the low-frequency coefficients of m image, and W L rithm 1.

123
 Complex & Intelligent Systems

Algorithm 1 Proposed Medical Image Fusion Algorithm NSCT decomposition in the PC–LLE method, the high-
Input: source images: A and B. frequency sub-bands are fused by the phase consistency rule.
The image of Interest–Laplacian filter (IOI–LLF) method
Part 1: NSST decomposition
uses local LLF to decompose the source image into resid-
01: For each source image S = [A, B];
ual and Ground images and further decomposes the residual
02: Perform NSST decomposition on S to obtain {H , L }; image based on IOI. The CNN–CP method uses a trained
03: End; Siamese convolution network to fuse the pixel activity infor-
Part 2: Fusion of high-frequency sub-band mation of the source image and generate a weight map. The
04: For each source image S = [A, B]; first two methods are based on SR, whereas the last four
05: Calculate the convolution sparse coefficient by Eq. (2); methods are all multiscale decomposition methods.
06: End;
07: Convolution sparse coefficient fusion by Eq. (3); Objective evaluation metrics
08: The fused high-frequency sub-band are obtained by
Eq. (4); To evaluate the performance of the various methods, five
Part 3: Fusion of low-frequency sub-band
widely recognized object metrics, namely, entropy (EN) [39],
structural similarity (Q e ) [40], mutual information (MI) [41],
09: For each 3*3 sliding window AB
gradient (Q /F ) [42] and the human eye visual perception
10: Calculate the correlation of low-frequency sub-band
(VIF) [43], are used in the experiment. EN can reflect the
by the mutual information Eq. (5);
amount of information contained in the fused image; Q e rep-
11: The low-frequency sub-band is divided into two resents the degree of similarity between the fused and source
different regions according to the correlation threshold T; images; MI is a mutual information indicator used to mea-
AB
12: if NMI > T, then fusion is performed by Eq. (10); sure the information contained in the fused image; Q /F is
13: if NMI < T, then fusion is performed by Eq. (13); a quality metric based on gradient, which is mainly used to
14: End; measure the edge information of fused images; VIF is the
Part 4: NSST reconstruction information ratio between the fused image and the source
15: Perform inverse NSST on {H , L } to obtain F; image and is used to evaluate the human visualization per-
formance of the fused image.
Output: the fused image F.

Experimental settings

In this experiment, 10 groups of CT–MRI images and 10

groups of MRI–SPECT images are used in fusion perfor-
Experiment and analysis
mance tests. As shown in Fig. 3, the first row (a) shows two
sets of CT–MRI images. The second row (b) shows two sets
Comparison algorithms
of MRI–SPECT images. these images are from the Whole
Brain Atlas provided by Harvard Medical School; each image
Nine medical image fusion methods, which have been
has a resolution size of 256 × 256. All experiments are
proposed in recent years, are compared with the pro-
programmed by MATLAB 2014a, and the simulation exper-
posed method. These methods are based on SR or multi-
iment environment is Intel(R) Core(TM) I7-8565U CPU @
scale transformation and include LP–ASR [33], SR–NSCT
1.80 GHz and 8.00 GB RAM.
[34], parameter-adaptive and pulse-coupled neural network
(PA–PCNN) [13], PC–LLE [14], IOI–LLF [35], CNN–CP
[15], CoF–MLE–NSST [36], PSO–NSST [37] and PCN- Experimental results
N–NSST [38]. The LP–ASR method is based on Laplacian
Pyramid decomposition and adaptive SR, and the sparse Figure 4 shows the results of two groups of CT–MRI images
coefficient fusion scheme is used to reduce the noise of the obtained by different fusion methods. LP–ASR and SR–N-
high-frequency components. The SR–NSC method incorpo- SCT lose part of the energy of CT images and reduce the
rates NSCT into the SR fusion framework, and different contrast of fusion images; PC–LLE and CNN–CP lose part of
fusion strategies are used for low- and high-frequency the information in the MRI source images, whereas noise-like
coefficients. The PA–PCNN method first performs NSST artifacts exist in IOI–LLF fusion images. The fusion images
decomposition on the source images, and then a PA–PCNN of PA–PCNN, PCNN–NSST and the proposed method have
model is used in the high-frequency sub-band fusion. After better contrast and edge detail information.

123
Complex & Intelligent Systems

Fig. 3 Experimental source

images: a CT–MRI,
b MRI–SPECT

Figure 5 shows the results of two sets of MRI–SPECT To compared the computational costs of different fusion
images obtained by different fusion methods. Among them, methods, the total time of 10 groups of CT–MRI fusion
LP–ASR and SR–NSCT lose part of MRI information, and images is first calculated and then divided by 10 to obtain
local image distortion exists. The IOI–LLF, PC–LLE and the average running time. The calculation was repeated 10
CNN–CP methods have complete texture information, but times, and the results and standard deviations are shown in
some SPECT functional information is missing. The fusion Table 3. The proposed method is inferior to the LP–ASR
effect of PA–PCNN, PCNN–NSST and the proposed method and PC–LLE methods but superior to the other four meth-
is better subjectively. ods. Particularly, the performance of the proposed method
To evaluate the performance of each fusion method objec- is similar to that of PA–PCNN, but the computational effi-
tively, Tables 1 and 2, respectively, show the average scores ciency is higher, because the iteration process of PCNN is
of the CT–MRI and MRI–SPCET fusion results. The higher time consuming. Here, the IOI–LLF method has the lowest
the index value, the better the fusion performance, where the calculation efficiency, because IOI takes too much time.
highest score is indicated in bold, and the lowest score is The proposed algorithm’s performance was also evaluated
indicated by subscript. In addition, the performance of the by changing the value of parameters used in the proposed
proposed method is compared with that of several recent method, such as NSST decomposition level, and the direc-
NSST methods. Among them, COF–MLE–NSST method tions number. These values are obtained over 20 pairs of
uses co-occurrence filter to measure the activity of low- multi-modality medical images, and the average outcomes
frequency sub-band coefficient, PSO–NSST method uses are shown in Table 4. From Table 4, it can be analyzed that as
particle swarm optimization algorithm to optimize the mem- the decomposition level and directions are increasing, the val-
AB
bership function of low-frequency sub-band fuzzy logic ues of En and MI are also increased. The values of Q e , Q /F
system, and PCNN–NSST method uses PCNN to fuse high- and VIF are optimal when Level  3. In general, with the
frequency sub-band. Compared with the other nine methods, increase of level, the value of each metrics increases slightly.
AB
the proposed method ranks first in Q e , MI, and Q /F for
CT–MRI and MRI–SPECT images, indicating that it pre-
serves most of the structure information in the source images
and keeps the edge of the source image and structure well. At Conclusions
the same time, because the method based on transformation
domain is accompanied by the loss of a certain amount of In this study, we propose a multimodal medical image fusion
information, the ranking of EN and VIF is not the highest, method based on NSST and mutual information correlation
but the ranking is still relatively high, indicating that the pro- analysis. Based on NSST scale decomposition, this method
posed method has good robustness. The proposed method is uses CSR to enhance the high-frequency detail informa-
inferior to the PA–PCNN method in VIF, because the latter tion and uses mutual information correlation to mine the
adopts a neuron perceptron similar to that of humans. detail information of low-frequency sub-band. Then, dif-
ferent fusion strategies are adopted for different areas of

123
 Complex & Intelligent Systems

Fig. 4 Fusion results of two sets

of CT–MRI images using
different algorithms. a3–a12,
b3–b12, respectively, denote the
fused images of the LP–ASR,
SR–NSCT, IOI–LLF, PC–LLE,
CNN–CP, PA–PCNN,
CoF–MLE–NSST, PSO–NSST,
PCNN–NSST and the proposed
algorithms

123
Complex & Intelligent Systems

Fig. 5 Fusion results of two sets

of MRI–SPECT images using
different algorithms. c3–c12,
d3–d12, respectively, denote the
fused images of the LP–ASR,
SR–NSCT, IOI–LLF, PC–LLE,
CNN–CP, PA–PCNN,
CoF–MLE–NSST, PSO–NSST,
PCNN–NSST and the proposed
algorithms

low-frequency sub-band according to correlation. To achieve following limitations: first, the setting of the threshold of the
this goal, two new activity level measurement methods based low-frequency sub-band correlation analysis has a certain
on the domain energy gradient and central pixel energy sum influence on the final fusion effect. If the threshold is set too
are designed. By comparing with other advanced methods small, then the extraction of detail information is insufficient;
and numerous experiments, the effectiveness of the pro- if the threshold is set too large, then meaningless details in
posed method is proven. However, the method still has the the MRI image are introduced into the fused image, causing

123
 Complex & Intelligent Systems

Table 1 Objective evaluation of

Methods EN Qe MI AB VIF
CT–MRI fusion images
Q F

LP–ASR 4.1466 0.7505 2.0982 0.8246 0.4502

SR–NSCT 4.3242 0.7626 1.9384 0.8344 0.4428
IOI–LLF 5.1825 0.7259 1.7656 0.7872 0.4375
PC–LLE 5.0340 0.7384 2.1178 0.8495 0.3827
CNN–CP 4.8099 0.7631 1.6211 0.7901 0.4438
PA–PCNN 5.1479 0.7754 2.0137 0.8024 0.4679
CoF–MLE–NSST 5.0987 0.7698 2.0592 0.8499 0.4685
PSO–NSST 5.0901 0.7724 2.1105 0.8450 0.4606
PCNN–NSST 5.0424 0.7805 2.1837 0.8503 0.4688
Proposed 5.0790 0.7827 2.2035 0.8507 0.4615

Table 2 Objective evaluation of

Methods EN Qe MI AB VIF
MRI–SPECT fusion images
Q F

LP–ASR 4.5285 0.7811 2.6251 0.4954 0.6053

SR–NSCT 4.6179 0.7930 2.7989 0.6366 0.5986
IOI–LLF 4.7326 0.7685 2.6814 0.5507 0.6311
PC–LLE 4.6901 0.7724 2.5413 0.6544 0.6257
CNN–CP 5.1703 0.8022 2.7422 0.5979 0.6365
PA–PCNN 5.0224 0.8152 2.6907 0.6618 0.6389
CoF–MLE–NSST 5.0702 0.8029 2.7147 0.6689 0.6210
PSO–NSST 5.1126 0.8133 2.7194 0.6732 0.6255
PCNN–NSST 5.0677 0.8158 2.7867 0.6767 0.6378
Proposed 4.9790 0.8190 2.8068 0.6835 0.6290

Table 3 Running time for

different methods Methods LP–ASR SR–NSCT IOI–LLF PC–LLE CNN–CP PA–PCNN Proposed

Average 0.286 6.351 73.25 0.495 14.12 8.68 2.97

Standard 0.001 0.02 0.37 0.006 0.18 0.05 0.02
devia-
tion

Table 4 Fusion quality metrics

analysis based on different levels Quality Data sets Level  1 Level  2 Level  3 Level  4 PLevel  5
and directions metrics Directions Directions Directions Directions Directions 
 16  16,16  16,16,8  16,16,8,8 16,16,8,8,4

EN CT–MRI 5.0790 5.0896 5.1201 5.1576 5.1589

MRI–SPECT 4.9790 4.9815 4.9924 4.9935 4.9936
Qe CT–MRI 0.7827 0.7830 0.7845 0.7843 0.7841
MRI–SPECT 0.8190 0.8194 0.8196 0.8122 0.8037
MI CT–MRI 2.2035 2.2046 2.2049 2.2051 2.2049
MRI–SPECT 2.8068 2.8072 2.8073 2.8073 2.8074
AB CT–MRI 0.8507 0.8509 0.8510 0.8509 0.8508
Q F
MRI–SPECT 0.6835 0.6836 0.6838 0.6838 0.6837
VIF CT–MRI 0.4615 0.4618 0.4619 0.4617 0.4615
MRI–SPECT 0.6290 0.6291 0.6295 0.6297 0.6298

123
Complex & Intelligent Systems

artifacts. In this study, the mutual information of the whole 10. Chen G et al (2019) Weighted sparse representation and gradient
source image is used as the threshold value for the corre- domain guided filter pyramid image fusion based on low-light-level
dual-channel camera. IEEE Photon J 99:1
lation analysis of low-frequency sub-bands; this strategy is 11. Li GX, Wang K (2007) Color image fusion algorithm using the
not an optimal scheme. In addition, Table 3 shows that this contourlet transform. Acta Electron Sin 35:112
method is not as fast as some fusion methods, because the 12. Miao QG, Cheng S, Xu PF et al (2011) A novel algorithm of image
local mutual information correlation is calculated by sliding fusion using shearlets. Opt Commun 284(6):1540–1547
13. Yin M, Liu X, Liu Y et al (2018) Medical image fusion
window, resulting in low calculation efficiency. In the future, with parameter-adaptive pulse coupled neural network in non-
we will devote ourselves to the research of a more effec- subsampled shearlet transform domain. IEEE Trans Instrum Meas
tive threshold determination scheme by combining the prior 68(1):49–64
information of source images. 14. Zhu Z, Zheng M, Qi G et al (2019) A phase congruency and
local laplacian energy based multi-modality medical image fusion
method in NSCT domain. IEEE Access 7:20811–20824
Acknowledgements This project is supported by the Provincial Natu-
15. Wang K, Zheng M, Wei H et al (2020) Multi-modality medical
ral Science Foundation of Hunan, China (Grant No. 2020JJ6021), the
image fusion using convolutional neural network and contrast pyra-
Research Foundation of Education Bureau of Hunan Province, China
mid. Sensors 20(8):2169
(Grant No. 21A0451, No. 19C0483), Construct Program of the Key
16. Shahdoosti HR, Ghassemian H (2016) Combining the spectral PCA
Discipline in Hunan Province: Control Science and Engineering.
and spatial PCA fusion methods by an optimal filter. Inf Fusion
27:150–160
Open Access This article is licensed under a Creative Commons
17. Liu Y, Liu S, Wang Z (2015) A general framework for image
Attribution 4.0 International License, which permits use, sharing, adap-
fusion based on multi-scale transform and sparse representation.
tation, distribution and reproduction in any medium or format, as
Inf Fusion 24:147–164
long as you give appropriate credit to the original author(s) and the
18. Wang K, Qi G, Zhu Z, Chai Y (2017) A novel geometric dictio-
source, provide a link to the Creative Commons licence, and indi-
nary construction approach for sparse representation based image
cate if changes were made. The images or other third party material
fusion. Entropy 19:306
in this article are included in the article’s Creative Commons licence,
19. Garaigordobil A, Ansola R, Veguería E et al (2019) Overhang
unless indicated otherwise in a credit line to the material. If material
constraint for topology optimization of self-supported compliant
is not included in the article’s Creative Commons licence and your
mechanisms considering additive manufacturing. Comput Aided
intended use is not permitted by statutory regulation or exceeds the
Design 109:33–48
permitted use, you will need to obtain permission directly from the copy-
20. Subashini MM, Sahoo SK (2014) Pulse coupled neural networks
right holder. To view a copy of this licence, visit http://creativecomm
and its applications. Expert Syst Appl 41(8):3965–3974
ons.org/licenses/by/4.0/.
21. Wang M, Shang X (2020) An improved simplified PCNN model
for salient region detection. Vis Comput 10–12:1–13
22. Easley G, Labate D, Lim W-Q (2008) Sparse directional image
representations using the discrete shearlet transform. Appl Comput
Harmon Anal 25(1):25–46
References 23. Kim M, Han DK, Ko H (2016) Joint patch clustering-based dictio-
nary learning for multimodal image fusion. Inf Fusion 27:198–214
1. Ganasala P, Kumar V (2016) Feature-motivated simplified adaptive 24. Liu H, Liu Y, Sun F (2015) Robust exemplar extraction using struc-
PCNN-based medical image fusion algorithm in NSST domain. J tured sparse coding. IEEE Trans Neural Netw 26:1816–1821
Digit Imaging 29:73–85 25. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-
2. Qi G, Wang J, Zhang Q, Zeng F, Zhu Z (2017) An integrated dic- resolution via sparse representation. IEEE Trans Image Process
tionary learning entropy-based medical image fusion framework. 19(11):2861–2873
FutureInternet 9(4):61 26. Dong W et al (2011) Image deblurring and super-resolution by
3. Petrovic V, Xydeas C (2004) Gradient-based multiresolution image adaptive sparse domain selection and adaptive regularization. IEEE
fusion. IEEE Trans Image Process 13:228–237 Trans Image Process 20(7):1838–1857
4. Sundar K, Jahnavi M, Lakshmisaritha K (2017) Multi-sensor image 27. Yang B, Li S (2010) Multifocus image fusion and restoration with
fusion based on empirical wavelet transform. In: 2017 international sparse representation. IEEE Trans Instrum Meas 59(4):884–892
conference on electrical, electronics, communication, computer, 28. Yin H, Li S, Fang L (2013) Simultaneous image fusion and super-
and optimization techniques (ICEECCOT). IEEE, pp 93–97 resolution using sparserepresentation. Inf Fusion 14:229–240
5. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense 29. Wohlberg B (2015) Efficient algorithms for convolutional sparse
SIFT. Inf Fusion 23:139–155 representations. IEEE Trans Image Process 25(1):301–315
6. Xia J, Lu Y, Tan L et al (2021) intelligent fusion of infrared and 30. Liu Y, Chen X et al (2019) Medical image fusion via convolutional
visible image data based on convolutional sparse representation and sparsity based morphological component analysis. IEEE Signal
improved pulse-coupled neural network. Comput Mater Continua Process Lett 26(3):485–489
67(1):613–624 31. Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm
7. Yuan G, Ma S, Liu J et al (2021) Fusion of medical images based on for designing overcomplete dictionaries for sparse representation.
salient features extraction by PSO optimized fuzzy logic in NSST IEEE Trans Signal Process 54:4311–4322
domain. Biomed Signal Process Control 69(12):102852 32. Dong W et al (2013) Sparse representation based image interpo-
8. Ouerghi H, Mourali O, Zagrouba E (2020) Multi-modal image lation with nonlocal autoregressive modeling. IEEE Trans Image
fusion based on weight local features and novel sum-modified- Process 22(4):1382–1394
Laplacian in non-subsampled Shearlet transform domain. In: 33. Wang Z, Cuia Z, Zhu Y (2020) Multi-modal medical image fusion
International symposium on visual computing by Laplacian pyramid and adaptive sparse representation. Comput
9. Shen J, Zhao Y, Yan S, Li X (2014) Exposure fusion using boosting Biol Med 123:103823
Laplacian pyramid. IEEE Trans Cybern 44:1579–1590

123
 Complex & Intelligent Systems

34. Li Y, Sun Y, Huang X et al (2018) An image fusion method based 42. Petrović V (2007) Subjective tests for image fusion evaluation and
on sparse representation and sum modified-Laplacian in NSCT objective metric validation. Inf Fusion 8:208–216
domain. Entropy 20(7):522 43. Bovik HA (2006) Image information and visual quality. IEEE Trans
35. Jiao D, Li W, Xiao B (2017) Anatomical-functional image fusion Image Process 15(2):430
by information of interest in local Laplacian filtering domain. IEEE
Trans Image Process 12:1–1
36. Diwakar M, Singh P, Shankar A (2021) Multi-modal medical image
Publisher’s Note Springer Nature remains neutral with regard to juris-
fusion framework using co-occurrence filter and local extrema in
dictional claims in published maps and institutional affiliations.
NSST domain. Biomed Signal Process Control 68(12):102788
37. Yuan GA et al (2021) Fusion of medical images based on salient
features extraction by PSO optimized fuzzy logic in NSST domain.
Biomedical Signal Process Control 69:102852
38. Wei T et al (2020) Multimodal medical image fusion algorithm in
the era of big data. Neural Comput Appl 3:1–21
39. Cvejic N, Canagarajah C, Bull D (2006) Image fusion metric
based on mutual information and Tsallis entropy. Electron Lett
42:626–627
40. Zhang X-L, Li X-F, Li J (2014) Validation and correlation analysis
of metrics for evaluating performance of image fusion. Acta Autom
Sin 40(2):306–315
41. Qu G, Zhang D, Yan P (2002) Information measure for performance
of image fusion. Electron Lett 38:313–315

123