Neural Computing and Applications (2023) 35:24109–24121

https://doi.org/10.1007/s00521-023-09005-x (0123456789().,-volV)(0123456789().,-volV)

ORIGINAL ARTICLE

Diagnosis of breast cancer molecular subtypes using machine learning

models on unimodal and multimodal datasets
Samta Rani1 • Tanvir Ahmad1 • Sarfaraz Masood1 • Chandni Saxena2

Received: 27 April 2023 / Accepted: 22 August 2023 / Published online: 19 September 2023
 The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2023

Abstract
Breast cancer is a significant global health concern, with millions of cases and deaths each year. Accurate diagnosis is
critical for timely treatment and medication. Machine learning techniques have shown promising results in detecting breast
cancer. Previous studies have primarily used single-modality data for breast cancer diagnosis. Hence, this work aims to
mobilize the benefits of multimodal data over unimodality samples. This study proposes a custom deep learning-based
model pipeline that works over this multimodal data. This work has been separated into three phases. Phase 1 and Phase 2
under the unimodal category examine gene expression data and histopathological images separately. The Cancer Genome
Atlas makes these datasets available. In Phase 3, the proposed pipeline operates on both data types’ samples for each
patient in the multimodal category. This study investigates how data pre-processing (cleaning, transformation, reduction)
and cascaded filtering affect model performance. Precision, recall, f1-score, and accuracy assessed the models, whereas L2
regularization, exponentially weighted moving average, and transfer learning minimized over-fitting. A custom deep neural
network and support vector machine obtained 86% accuracy in Phase 1, whereas the VGG16 model reached 80.21%
accuracy in Phase 2. In Phase 3, the curated multimodal dataset was applied to a custom deep learning pipeline (VGG16
backbone with hyper-tuned machine learning models as head classifiers) to achieve 94% accuracy, demonstrating the
importance of multimodal data over unimodal in breast cancer subtype classification. These findings highlight the
importance of multimodal data for breast cancer diagnosis and subtype prediction.

Keywords Machine learning  Unimodal data  Multimodal data  Breast cancer molecular subtypes  Deep neural network

1 Introduction currently affected by this condition worldwide [1]. The

survival rates vary from country to country due to their
Breast cancer is a serious global health issue that has a financial condition or lack of availability of programs for
substantial impact on morbidity and mortality rates across cancer diagnosis [2–4]. In a new study, experts from the
many nations and millions of women of all ages are ‘‘International Agency for Research on Cancer (IARC)‘‘
and partner institutions assessed the impact of breast cancer
in 2040 and provided a global picture of the disease’s
& Samta Rani
[email protected] burden in 2020. By 2040, experts estimate that there will be
more than 3 million in new cases of breast cancer annually
Tanvir Ahmad
[email protected] (a 40% increase) and more than 1 million fatalities annu-
ally (a 50% increase) [5, 6]. The reason behind this is the
Sarfaraz Masood
[email protected] link between advanced clinical stages of breast cancer and
wait times longer than three months between the onset of
Chandni Saxena
[email protected] symptoms and the beginning of therapy, as per the majority
of research conducted in the developed world [7]. To
1
Department of Computer Engineering, Jamia Millia Islamia reduce the mortality rate and future burden of breast can-
University, New Delhi, India cer, significant improvements in early diagnosis are
2
The Chinese University of Hong Kong, Sha Tin, required.
Hong Kong, SAR, China

123
24110 Neural Computing and Applications (2023) 35:24109–24121

Using artificial intelligence (AI) in the current cancer 3. For an effective analysis, we use textual or quantitative
screening procedure makes acquiring results much more gene expression data in the first phase, histopatholog-
straightforward and convenient. Among the benefits of AI ical image data in the second phase, and integrated
approaches in breast cancer screening include quicker and data, or multimodal data, in the third phase. For each of
more accurate outcomes [8]. By enabling early identifica- the three phases, we use different ML models and also
tion and treatment of breast cancer, the use of AI and fine-tune these models to overcome the overfitting
machine learning (ML) has boosted the patient’s chance of issue.
survival [9, 10]. 4. Finally, the deep learning based pipeline applied over
Histopathological images (biopsy report), text data, CT the multimodal data samples used in this work
scans (computerized tomography scan), X-rays, gene generates a fine-grained classification output as non-
expressions (genetic activity), copy number variations cancerous cells as Normal and cancer subtypes based
(genetic alterations), and signals are a few of the different on molecules as Basal, Her2-enriched, Luminal-A,
data modalities used to detect breast cancer. Each modality Luminal-B.
can be used to identify, track, and treat the illness while
also offering particular insights into the biology of breast
cancer. Histopathological images, for example, allow us to
2 Related work
examine the structure and behaviour of breast cancer cells
at a microscopic molecular level. CT scans and X-rays
This section provides a summary of relevant research that
generate detailed images of the breast tissue, while gene
has utilized unimodal data (e.g. imaging data or gene
expressions and copy number variations provide informa-
expression data only) and multimodal data (e.g. combining
tion about the genetic makeup of the cancer cells. Magnetic
two or more data related to breast cancer) for the diagnosis
resonance imaging (MRI) signals, for example, can be used
of breast cancer. In Table 1, we also present a brief sum-
to point out the variations in the breast tissue that may
mary of recent research in this area.
signify the presence of cancer. With the integration of
different modalities, machine learning (ML) models can be
2.1 Previous research using unimodal data
trained to make diagnostic and prognostic models more
reliable and accurate. This brings AI closer to clinical
Studies using unimodal data have applied various machine
practice.
learning techniques to images or gene expression data to
The majority of existing research uses ML and deep
detect anomalies indicative of breast cancer. Kurt et al.
learning models that rely on unimodal data, which can only
[12] analyze unimodal gene expression data (METABRIC)
project a limited number of features, overlooking the larger
based on clinical and RNA-seq collected from The Cancer
clinical environment, unavoidably limiting their potential.
Genome Atlas (TCGA)1 and the Molecular Taxonomy of
Therefore, it becomes challenging to consistently link
Breast Cancer International Collaboration. In the frame-
patient characteristics to specific cancer subtypes. Inte-
work of ensemble modeling, the authors assessed six
gration of numerous data modalities offers chances to
diagnostic techniques and identified an mRNA diagnostic
improve diagnostic and prognostic models’ accuracy and
panel that can distinguish patients from healthy individuals
resilience, bringing AI closer to clinical practice [11].
and classify breast cancer into genetic subgroups. Kim
Therefore, we propose an approach that leverages mul-
et al. [13] examine the reproducibility and accuracy of
timodal data to improve the performance of machine
machine learning models for a biomedical classification
learning-based models to predict breast cancer molecular
task (including breast cancer). The authors train multiple
subtypes. This work aims to achieve the following
machine learning models (logistic regression, random for-
objectives:
est, SVM, neural nets) on gene expression data from
1. To show the efficacy of multimodal data samples over TGCA. The paper also suggests using ensemble methods
unimodal samples for breast cancer detection, a dataset and evaluating reproducibility. Phan et al. [14] employs
was curated from TCGA-BRCA such that the benefits convolutional neural networks (CNNs) to predict breast
of multimodalities can be highlighted. cancer gene expression signatures from histopathological
2. This work also proposes a custom DNN pipeline for images. The authors use four pre-trained models (VGG16,
processing unimodal and multimodal data samples ResNet50, ResNet101, and Xception) using pathological
consisting of gene expression data and histological images from a recurring breast cancer patient in the first
images for the same patient IDs. This includes phase of transfer learning and the ‘‘The Cancer Genome
comprehensive pre-processing for sample
enhancement. 1
https://portal.gdc.cancer.gov/projects/TCGA-BRCA.

123
Neural Computing and Applications (2023) 35:24109–24121 24111

Table 1 Summary of related work based on modalities

References Modal Type Data Types Methods Results

[12] Unimodal Gene expressions Ensemble method Acc = 82:55%

[13] Unimodal Gene expressions Random Forest method F1-score = 0.663
[14] Unimodal Histopathological images ResNet101 Acc = 91%
[15] Unimodal Histopathological images Inception_Resnet_V2 Acc = 97:63%
[16] Multimodal Gene expressions and Histopathological images Hybrid model method Acc = 88:07%
[17] Multimodal Genes, Histopathological images ImageCCA Joint Analysis
[18] Multimodal Genomic, pathological images Multiple Kernel Learning AUC = 82:8%
[19] Multimodal CT scans, MRI, Positron Emission Tomography (PET) and X-ray ResNet18 Acc = 96:00%
[20] Multimodal mRNA expression, CNV and miRNA Graph Neural Network Acc = 80:05%
[11] Multimodal Histology, Genomics, Radiology and, Electronic health records AI methods Review
[21] Multimodal Histopathology images, Gene Expression Codebook Learning Joint Analysis
[22] Multimodal Histopathology images, Gene Expression Deep Learning methods Review

Atlas Breast Invasive Carcinoma (TCGA-BRCA) dataset’’ [19] compile 4.5 million images into a single dataset.
in the second phase of the transfer learning. The study ResNet-50, ResNet-18, and VGG16 train to classify these
shows that ResNet101 can achieve reasonably high accu- images by the imaging modalities used to record them (CT,
racy (up to 0.91) in predicting breast cancer subtypes from MRI, PET, and X-ray) across several body locations. This
images without annotations. Xie et al. [15] propose using study demonstrates that it is possible to train Deep
CNNs to analyze breast cancer histopathology images to Learning (DL) Convolutional Neural Networks (CNNs)
classify them into cancer subtypes (e.g. Luminal-A, Basal). with multimodal datasets containing millions of images.
The study shows an accuracy of 97.6% using CNN Such models are applicable to real-world applications with
(Inception_ResNet_V2). hyper-scale volumes of image data. Li et al. [20] propose a
unique end-to-end multi-omics GNN framework that
2.2 Previous research using multimodal data includes inter-omics and intra-omics relationships derived
from recognized biological knowledge. For precise cate-
Recent years have witnessed major growth advancements gorization, the proposed approach integrates both learned
in the evolution of technology based on deep learning for graph features and global genome features. With the inte-
fusing multimodal data. As an example, Liu et al. [16] gration of multimodal data, Lipkova et al. [11] establish
suggest a hybrid model based on deep learning that uses approaches for AI interpretability and exploratory paths. In
multiple types of data to predict breast cancer subtypes. oncology, a variety of techniques, including radiography,
The authors combine patient imaging and gene data histology, genetics, and electronic medical records, are
modalities to create the multimodal fusion framework. used to describe the patient’s condition. They use radiog-
They use the TCGA-BRCA dataset to predict four breast raphy, histology, genetics, and oncology-specific electronic
cancer subtypes based on their genetic characteristics. Ash medical records to characterize the diagnosis. Their find-
et al. [17] utilize histopathology images to characterize ings support the use of multimodal AI to address problems
complex features such as tumor stage, to link these features including classifying cancer subtypes, predicting survival
to high-dimensional genetic markers. For analyzing com- rates, and recommending personalized therapies. Popovici
parable histology images and gene expression data, the et al. [21] present a novel approach for a joint analysis of
authors apply autoencoder and correlation analysis meth- histopathology images and gene expression data. They
ods. They intend to identify subsets of genes for whom jointly cluster the image features and gene expression data
levels of expressions are associated with subsets of mor- to find patterns in the integrated data. The clustering results
phological characteristics from the accompanying image. identify patient categories with distinct imaging and
Sun et al. [18] propose a model based on multiple kernel genetic characteristics. The use of full-color data and the
learning which employs genetic data and histopathological inclusion of other types of attributes can easily be added to
image data effectively. The model executes feature fusion, their approach. A recent survey [22] describes a variety of
which is ingrained in breast cancer classification, using the deep learning techniques for cancer diagnosis that use
aforementioned heterogeneous characteristics. By inte- several data modalities, including images, gene expression
grating 102 medical imaging datasets, Macfadyen et al. data, and clinical data. The authors present a thorough

123
24112 Neural Computing and Applications (2023) 35:24109–24121

review of contemporary deep learning approaches for integration of data obtained from phases one and two to
multimodal data fusion (early, late, and join) in cancer construct a multimodal dataset. We examine various clas-
diagnosis, including CNNs, autoencoders, and recurrent sification models to identify or categorize the subtypes of
neural networks (RNNs). breast cancer based on molecules. Furthermore, using both
The accumulation of intratumor is one of the primary unimodal and multimodal datasets, this study presents a
causes of treatment and recovery failure in breast cancer, complete analysis of the basic aspects of learning.
which is a heterogeneous disease constituted of diverse
genetic subtypes. Thus, the identification of cancer sub- 3.1 Phase 1: Gene expression data analysis
types becomes essential for accurate diagnosis and treat- and ML model exploration
ment. In particular, we emphasize the breast cancer
subtypes based on molecules in our work. In the first stage of the unimodal data, the gene expression
It may be observed from the review presented, that numeric dataset is utilized as the foundation. This section
almost all of the works, largely revolve around exploring gives a summary of the description of the dataset, the
custom machine learning models for some specific uni- workflow, and the models, as well as a proposed method-
modal sample type only and hence are limited by the ology for identifying breast cancer subtypes.
model’s performance over multimodal scenarios. The
• Workflow
favour of multimodal data based on the model’s perfor-
Figure 2 outlines the workflow used for the deploy-
mances is clearly shown in Table 1. As a result, multi-
ment of classification models trained for breast cancer
modal data not only improve prediction accuracy but also
subtype prediction using gene expression data. The data
enable cross-validation and measurement of complex
preprocessing stage involves three distinct steps, which
characteristics that are difficult (or even impossible) to
are as follows:
accurately represent with only a single data source. Com-
Data Cleaning: Handle missing data by ignoring
pared to multimodal data models, unimodal data models
tuples and removing samples with the majority of 0
provide fewer traits and information for precise disease
values to robust and highly accurate model.
detection. Even in the real-world scenario, the task of
Data Transformation: The dataset values already
cancer detection usually surrounds exploring inputs from
use Log2 normalization as primitive processing in view
multiple modalities for reaching any definite conclusions.
of the fact that log transformation reduces data
Hence in this work, we also aim to explore avenues of
variability and makes data conform precisely to the
working with multi-modalities for breast cancer molecular
normal distribution.
subtype detection.
Data Reduction: We explored various dimension-
ality reduction methods (Principal Component Analy-
sis, Chi-Square, Linear Discriminant Analysis, Singular
3 Material and methods
Value Decomposition) and it was observed that Prin-
cipal Component Analysis (PCA) outperformed com-
In this paper, we consider two publicly available breast
pared to the others methods. By selecting the ideal
cancer datasets on the TCGA website to diagnose its 4
number of features (components), PCA can compress
molecular subtypes: Basal, Her2-enriched, Luminal-A, and
your data while keeping the majority of the information
Luminal-B with 1 non-cancerous type: Normal [23, 24].
content.
The first dataset is related to the gene expression levels of
The 831 remaining samples are partitioned into
patients, whereas the second dataset consists of
training, testing, and validation sets, and it is observed
histopathological images of patients. Unimodal datasets are
that Support Vector Machine (SVM), Stochastic Gra-
analyzed independently, while multimodal datasets involve
dient Descent (SGD), K-Nearest Neighbor (KNN), and
the integration of both gene expression data and
the suggested Deep Neural Network (DNN) exhibit
histopathological images. We present our work structured
superior performance compared to other classification
into three distinctive phases, which are depicted in Fig. 1.
models in terms of precision, recall, F1-score, and
The initial phase entails exploring different machine
accuracy.
learning models alongside the proposed model for classi-
• Gene Expression Dataset
fying breast cancer molecular subtypes through gene
The initial step involved collecting a dataset of 1,098
expression data. The second phase outlines the preparation
samples, each with 20,211 genes, from the National
of the image dataset for analysis, which involves imple-
Cancer Institute website (TCGA-BRCA project).2
menting deep learning classification techniques and pre-
processing methods for the pathological images of breast
2
cancer. The third phase of this study involves the https://portal.gdc.cancer.gov/projects/TCGA-BRCA.

123
Neural Computing and Applications (2023) 35:24109–24121 24113

Fig. 1 Distribution of work overview in three phases (Phase 1: Based on gene expression data, Phase 2: Based on histopathological image data,
Phase 3: Based on multimodal data)

Fig. 2 Workflow followed in case of gene expression data

Table 2 Configuration of gene expression dataset

Breast cancer subtypes Number of Samples

Normal 32
Basal 148
Her2-enriched 71
Luminal-A 412
Luminal-B 168
Total 831

Fig. 3 Structure of deep neural network model

Table 2 shows the distribution of 831 samples per class
after implementing the pre-processing steps. employ the following additional steps to prevent and
mitigate the effects of overfitting:
• Models Applied
To perform cancer subtype prediction, we employ – Choose the L2 regularization model, with the
various classification models—Naive Bayes (NB), following fundamental equation, as regularisation
Random Forest (RF), K-Nearest Neighbor (KNN), is a typical technique to avoid overfitting issues.
X
Decision Tree (DT), Stochastic Gradient Descent Rðwe Þ ¼ kwe k22 ¼ jw2ei j ð1Þ
(SGD), Support Vector Machine (SVM), and Deep i
Neural Network (DNN) [25, 26]. The structure of a
we represents the weight of each layer’s network
DNN as illustrated in Fig. 3 is derived from prior
nodes. Excessive weight restriction avoids using this
research [16] on the TCGA-BRCA gene expression
regularization technique.
dataset. To continually reduce abstract features and
– Apply the Exponential Weighted Moving Average
maintain suitable network parameters, we build the
model (EWMA) in addition to the L2 regularisation
DNN using the inverted pyramid approach [16]. To
to further improve the ability of DNN. EWMA can
overcome overfitting due to limited training samples,
restrict overfitting to some degree and improve
we utilize a 5-layer DNN. The structure with each
network performance on test data [16].
layer’s activation function is shown in Fig. 3. We also

123
24114 Neural Computing and Applications (2023) 35:24109–24121

3.2 Phase 2: Histopathological image data samples. These patches or samples are then separated
analysis and ML model exploration into sets for training, testing, and validation.
Step 6: In the last step, we use a transfer learning-
Using histopathology scans of patients with the same based VGG16 model to diagnose or classify the
patient-id as in phase 1, we create an image dataset for molecular subtypes of breast cancer.
testing a deep learning model that classifies the dataset into
five classes(4 classes based on molecular subtypes and 1
class for Normal samples). The description of this phase is • Preparing the Dataset
explained in the following subsections: We apply the following actions step by step to
prepare the image dataset (five histopathological
• Workflow images from TCGA-BRCA) for exploration.
Figure 4 illustrates the process flow of phase 2,
which employs a deep learning model and a distinct
data type to aid in diagnosing breast cancer subtypes. 1. Cut each image into patches of size 1024 by 1024 and
Each step is explained below: give their labels the same as those belonging to the
• Step 1: Initially, we collect five large-sized histopatho- original large image.
logical images, each of which belongs to one class. 2. Then applies the first filter, the canny filter, to all the
Step 2: We have partitioned these five images into image patches to reduce the effect of noise. This filter
equally sized (1024  1024) image patches. Table 3 is based on a Gaussian derivative, which is then used to
provides an account of the number of image patches in figure out how strong the gradients are [28].
each class. 3. Apply the convolutional filter to all image patches
Step 3: During the pre-processing stage, we use two obtained after the second step. During the convolu-
pairs of cascaded filters to eliminate noise and analyze tional filter, the brightness of a pixel is calculated based
their influence on the performance of the model for the on a weighted average of the luminance of its
image dataset. The first pair consists of Sobel and neighboring pixels [29].
Hough filters, whereas the second pair consists of 4. In the last step, a projective image transformation
Canny and Convolutional filters. approach is used to enhance the filtered image patches.
Step 4: After subjecting the image patches to After ensuing each step, Table 3 displays the configuration
filtering, we apply two transformation methods, Affine of image patches. Step 1 generates a collection of 1465
and Projective [27], to enhance the number of image image patches, which is subsequently reduced to 863 image
patches. By utilizing the projective transformation patches after applying both filters. To increase the number
method, we achieve noteworthy outcomes. Figure 5 of image patches, a projective transformation is applied.
depicts the fundamental transformation style for the Finally, we obtained a total of 1726 samples in our curated
projective method, which demonstrates how the visual image dataset.
objects alter as the observer’s viewpoint shifts. • Model Applied
Step 5: After applying the transformation methods, Image classification heavily relies on deep learning
now there are 1726 image patches obtained as total methods [30]. CNN-based models, in particular, have

Fig. 4 Process flow for exploring and deploying model on pathological image data

123
Neural Computing and Applications (2023) 35:24109–24121 24115

Table 3 Configuration of
Breast Cancer subtype Before Image Filtration After Image Filtration Projective Enhancement
histopathological image patches
Normal 378 235 470
Basal 441 264 528
Her2-enriched 266 165 330
Luminal-A 240 83 166
Luminal-B 140 116 232
Total 1465 863 1726

with a unique patient-id. Breast cancer patients with more

features will have access to more detailed information
about their health. The following subsections describe the
workflow, dataset, and models used to explore this phase.
• Workflow
The dataset is first pre-processed and then split into
the train, test, and valid sets following the 8:2 rule.
Fig. 5 Projective transformation for sample image Subsequently, various machine learning models are
trained on the training set. Finally, the trained model
shown remarkable performance on various image undergoes a series of tests to predict the breast cancer
modalities, owing to the concept of transfer learning. In molecular subtypes.
this study, we employ the hyper-tuned (HPT-VGG16) • Multimodal Dataset
model to classify breast cancer subtypes. For the preparation of a multimodal dataset, a
Transfer Learning: By utilizing transfer learning feature-level fusion strategy is employed. Using the
[31, 32], model pre-training is employed to learn new appropriate feature normalization, transformation, and
samples effectively. This technique is particularly use- reduction processes, feature-level fusion integrates the
ful in mitigating the issue of insufficient data and pre- feature sets emerging from various biometric data
venting overfitting during this phase. Furthermore, fine- sources into a single feature set [35]. The number of
tuning approaches can be utilized to expedite the samples in a multimodal dataset is contingent upon the
model’s convergence time. To extract reusable features number of image patches present in the image data.
from ImageNet’s 1.2 million annotated images, we When combining gene expression and image data, it is
perform data pre-training. The trained model is then imperative to take into account the patient-id. The
applied to the pathological images. configuration of samples for each class in a multimodal
VGG16 Model: Convolutional neural network dataset is presented in Table 4.
VGG-16 consists of 16 layers. The model is trained via In Fig. 6, we illustrate the steps involved in gener-
transfer learning on the ImageNet dataset, which con- ating a multimodal dataset. Initially, gene expression
tains more than 14 million high-resolution images with data is processed using a neural network to extract
1000 distinct labels [33]. The design is characterized by features, while image data is processed using a CNN-
its simplicity, featuring a max-pooling layer followed based model [34]. The extracted features are then
by two consecutive blocks of two convolution layers,
and three consecutive blocks of three convolution lay-
ers, culminating in three dense layers. It should be
noted, however, that the depths of the final three con- Table 4 Configuration of multimodal dataset
volution layers may differ across different designs. Breast Cancer Subtypes Number of Samples

Normal 470
3.3 Phase 3: Multimodal data analysis and ML Basal 528
model exploration Her2-enriched 330
Luminal-A 166
By integrating the datasets from the first and second pha- Luminal-B 232
ses, we curated a multimodal dataset for ML model Total 1726
exploration. This dataset contains both gene expression and
histopathological image features [34], for a single patient

123
24116 Neural Computing and Applications (2023) 35:24109–24121

Fig. 6 Process flow to prepare multimodal dataset using gene expression data and histopathological image data

concatenated or combined in the subsequent step to 4 Results and discussion

form the final multimodal dataset.
• Models Applied This section explains the performance metrics and their
The curated multimodal dataset is passed to the values by conducting a comparative study of the results
pipeline with VGG16 as the backbone and numerous obtained in all three phases with those of closely related
machine learning models, such as SVM with multiple studies. The suggested technique demonstrates the positive
variants, ANN, MLP, DNN, and SGD as the model and negative aspects of the methods by highlighting the
head for yielding notable results. The structure of a results of the comparison study.
DNN is illustrated in Fig. 3. Notably, the multi-layer
perceptron model [36], displays exceptional perfor- 4.1 Evaluation parameters
mance among all the models considered. In Fig. 7, we
present the MLP model’s architecture utilized for In evaluating the models, precision, recall, F1-score, and
classification. The first layer serves as the input layer, accuracy are utilized as performance metrics [37]. We also
followed by two hidden layers of size 55 units. The employ learning curves to represent the models’ perfor-
output layer comprises five units, representing the five mance graphically. The formulas for these metrics are
classes. discussed below:
TrueP þ TrueN
Accuracy ¼ ð2Þ
TrueP þ FalseP þ TrueN þ FalseN
TrueP
Recall ¼ ð3Þ
TrueP þ FalseN
TrueP
Precision ¼ ð4Þ
TrueP þ FalseP
Precision  Recall
F1  score ¼ 2  ð5Þ
Precision þ Recall
where P is positive and N is negative.

4.2 Performance of models based on unimodal

gene expression dataset

Table 5 outlines the evaluation results of DNN, KNN,

SVM, and SGD models, with the classification accuracy
ranging from 75 to 86%. The two models that achieve the
highest accuracy are DNN and SVM, both at 86%, while
the KNN model attains the lowest accuracy of 75%. In
Fig. 7 Architecture for multi-layered perceptron model contrast to these outcomes, all classification models do not

123
Neural Computing and Applications (2023) 35:24109–24121 24117

Table 5 Performance of models

Model Precision Recall F1-score Accuracy
using gene expression data
Macro Average Macro Average Macro Average

KNN 0.73 0.77 0.55 0.75 0.58 0.71 75%

SGD 0.90 0.85 0.71 0.85 0.76 0.82 84%
DNN 0.93 0.87 0.72 0.86 0.77 0.83 86%
SVM 0.93 0.87 0.72 0.86 0.78 0.85 86%
Bold font represent the highest values

perform well when no data pre-processing steps are used, image pre-processing. Table 6 presents the accuracy values
with accuracy ranging from 61 to 75%. for both sets: the first set (Sobel and Hough filters with
Both macro and weighted averages are considered for Affine Transformation) and the second set (Canny and
analyzing all classes. The macro average computes preci- Convolution filters with Projective Transformation) based
sion, recall, and F1-score for each label, without account- on different epoch values. The first set achieves a maxi-
ing for the proportion of each label in the dataset, and mum accuracy of 73:37% over 150 epochs, while the
returns the average. Conversely, the weighted average second set attains a maximum accuracy of 80:21% over
computes precision, recall, and F1-score for each label in 300 epochs. Notably, accuracy does not increase beyond
the dataset, and returns the average [38]. 300 epochs for both sets, remaining constant in the range of
Machine learning curves aid in comparing different 100 to 300 epochs. These results are far superior to those
methods, selecting model parameters during design, opti- obtained without employing any filters or enhancing
mizing convergence by modifying optimization, and esti- methods on image samples as described in section 3.2 of
mating the training volume [39]. Figure 8 depicts the this work, with the accuracy ranging between 57% and
learning curve [40], for the support vector machine model, 70%.
showcasing the relationship between the training score and Figure 9 represents the graph between accuracy and
cross-validation score. Notably, this model attains the epochs for two sets of pre-processing methods. We can
highest accuracy. The learning curve employs training easily interpret that, using the second set, the HPT-VGG16
examples on the x-axis and score values on the y-axis. The model perform well in comparison to the first set across all
SVM classifier utilizes RBF (radial basis function) as the epochs.
kernel value while discussing the hyperparameters.
4.4 Performance of models based on multimodal
4.3 Performance of models based on unimodal dataset
histopathological image dataset
In this phase, we evaluate different classification models
Based on epochs [41], in this phase, we compare two dif- using the multimodal dataset. Table 7 presents the perfor-
ferent sets of methods using the HPT-VGG16 model during mance metric values for the DNN, KNN, SGD, MLP, and
SVM models, with precision, recall, F1-score, and accu-
racy utilized as performance metrics along with their macro
and weighted average values. The accuracy of all other
models ranges from 81% to 90%, while the multi-layered
perceptron model achieved the highest accuracy of 94%.
Not only accuracy but also the MLP model achieved the
highest macro and average recall values of 0.92 and 0.93
respectively which signifies very low false negatives.
To represent the performance of the multi-layered per-
ceptron model in this phase, accuracy and loss curves are
employed. These curves provide insight into how the
learning performance varies throughout the number of
epochs and are crucial in estimating training and validation
loss [40]. Figure 10 displays the accuracy and loss curves
for phase 3.
On the y-axis are displayed the accuracy and loss values,
Fig. 8 Learning curve for support vector machine model on gene
expression data
while the x-axis displays the number of epochs. In fine-

123
24118 Neural Computing and Applications (2023) 35:24109–24121

Table 6 Performance of
Epochs Sobel?Hough ?Affine Canny?Convolutional ?Projective
VGG16 model using
histopathological image data in 100 63.81% 72.18%
terms of accuracy
150 73.37% 75.83%
200 72.82% 78.01%
250 73.01% 79.41%
300 71.64% 80.21%
350 73.00% 78.99%
Bold font represent the highest values

is promising since it indicates a low false negative rate (the

probability of incorrectly labeling an individual as healthy
when in fact they have cancer). Integrating diverse forms
of data enhances the medical domain’s understanding of
individuals’ health conditions. The integration of data
modalities increases the feature count, which facilitates the
training of classification models and the prediction of
breast cancer molecular subtypes. Consequently, adminis-
tering appropriate medication or treatment at the appro-
priate time can save countless lives.
Table 9 compares our work and that of others on diverse
types of data obtained from the TCGA-BRCA site to
identify breast cancer subtypes using different methods.
Fig. 9 VGG16 performance based on epochs and pre-processing
Liu et al. [16], utilize the same type of data from the
methods
TCGA-BRCA and their model achieves 85:06% accuracy
tuning the performance of the MLP classifier, we optimize on gene expression data, which is less than our proposed
multiple parameters, resulting in the ‘‘solver‘‘ parameter set DNN model. By applying complex cascading filters and a
to ‘‘adam’’, an initial learning rate of 0.01, and a maximum transformation strategy during data pre-processing on the
of 200 epochs, which produce the best accuracy and loss suggested image dataset, the VGG16 model achieved an
results. accuracy of 80:21%, which is markedly superior to the
CNN model’s accuracy of 77:77%. Our curated multi-
4.5 Discussion modal dataset achieved an accuracy of 94% with the MLP
model, which is significantly superior to the accuracy
Table 8 displays the impact of hyperparameter values uti- achieved by their multimodal fusion model [16]. Various
lized during model training. Notably, the accuracy of 94% authors implement machine learning models such as
achieved for multimodal datasets surpasses the accuracy GPMKL, Ensemble, and MDNMD using a variety of data
attained for unimodal datasets. When compared to the type combinations for creating multimodal datasets.
recall metric values of unimodal data, the importance of However, their outcomes are inferior to this effort
multimodal data with recall metric values of 0.93 and 0.94 [12, 18, 42].
in predicting life-threatening breast cancer becomes evi- Overall, this study analyses unimodal and multimodal
dent. In the context of cancer detection, a high recall value datasets to predict breast cancer subtypes using a variety of
machine learning models. Due to its inverted pyramid

Table 7 Performance of models

Model Precision Recall F1-score Accuracy (%)
using multimodal data
Macro Average Macro Average Macro Average

SGD 0.80 0.83 0.79 0.81 0.79 0.81 81

KNN 0.83 0.84 0.80 0.82 0.80 0.82 82
SVM 0.94 0.91 0.83 0.89 0.87 0.88 89
DNN 0.88 0.90 0.89 0.90 0.88 0.90 90
MLP 0.93 0.94 0.92 0.93 0.93 0.94 94
Bold font represent the highest value

123
Neural Computing and Applications (2023) 35:24109–24121 24119

size, which may exacerbate the overfitting issue. However,

this study makes use of L2 regularization, EWMA, and
transfer learning techniques to address this issue as much
as possible.

5 Conclusion

The condition of a breast cancer patient is defined by a

broad range of modalities, including imaging, histology,
genetics, and electronic medical records. In order to start
the patient’s therapy as soon as possible, it may not be
enough to predict breast cancer using data from a single
modality. For enhanced decision-making, multimodal data
fusion aims to extract and combine complementary con-
textual information from several modalities. This study
aimed at harnessing the benefits of multimodal data sam-
ples for fine-grained breast cancer detection. The work has
been conducted in three phases that utilize various machine
learning models to detect and classify breast cancer sub-
types using textual or quantitative gene expression data in
phase 1, histopathological images in phase 2, and finally
the integrated or multimodal data in phase 3. Results show
that unimodal data may not be sufficient for an efficient
diagnosis of the disease. Instead, models trained on mul-
timodal data yield better results than those trained on
unimodal data, as most machine learning models struggle
to establish a consistent correlation between patient char-
acteristics and specific cancer subtypes using unimodal
Fig. 10 Training and Validation curve for MLP model on multimodal
data.
data a accuracy curve, b loss curve Overall, multimodal data lead to an accuracy of 94%
compared to unimodal datasets, which attained 86% and
technique and maintained network settings, the curated 80.21% accuracy across all phases. In comparison to uni-
DNN provides higher performance metric values in the modal datasets, the performance in terms of macro and
case of the gene expression dataset. The performance of the average recall values for the multimodal dataset is also
VGG16 for the phase 2 histopathology image dataset is remarkable as high recall values are a good sign specially
enhanced by the application of cascading filters (Canny & for cancer detection works. The high value of recall indi-
Convolutional) and projective augmentation techniques. cates that very less chances of identifying a subject as
MLP is a general model with strong multimodal perfor- healthy even though he/she had cancer which is very
mance, low computational complexity, and good scalabil- important.
ity properties [43]. These assertions about the MLP are also The study effectively handled large images by slicing
supported by the findings of phase 3’s multimodal dataset- them into identically sized image patches and employing
based results. This study’s major limitation is its sample filters to eliminate noise. The sample size of this study is

Table 8 Summary of overall performance with hyperparameter values

Dataset Model best Hyperparameter Accuracy
performed (%)

Unimodal (Gene Expression DNN, SVM Weight of L2 regularization (0.05), Decay Coefficient of EWMA 86
Data) (0.99)
Unimodal (Image Data) HPT-VGG16 Canny & Convolutional Filters ? Projective Transformation 80.21
Multimodal Data MLP Solver(Adam), initial learning rate(0.01) 94

123
24120 Neural Computing and Applications (2023) 35:24109–24121

Table 9 Overall comparison of results: Type1, Type2, Type3 are the types of data and Acc1 is the accuracy corresponds to Model1, Acc2 is the
accuracy corresponds to Model2, Acc3 is the accuracy corresponds to Model3
Paper Data Type Model Result
Type1 Type2 Type3 Model1 Model2 Model3 Acc1 (%) Acc2 Acc3

[42] Gene Expression CNV Clinical MDNNMD – – 82.6 – –

[12] Clinical RNA-seq – Ensemble – – 86 – –
[18] Genomic Pathological images – GPMKL – – 82.8 – –
[16] Gene Expression Histology Images – DNN CNN Fusion model 85.06 72.77% 88.07%
Ours Gene Expression Pathological Images – DNN VGG16 MLP 86 80.21% 94%
Bold font represent the highest accuracy values

one of its main limitations, which may make the overfitting Current and future burden of breast cancer: global statistics for
problem worse. However, to overcome this issue as much 2020 and 2040. Breast 66:15–23
6. Kuhl CK (2023) What the future holds for the screening, diag-
as feasible, this study employs the Exponential Weighted nosis, and treatment of breast cancer. Radiological Society of
Moving Average model (EWMA) in addition to the L2 North America
regularization, and transfer learning approaches. This lim- 7. Unger-Saldaña K (2014) Challenges to the early diagnosis and
itation can also be resolved using oversampling techniques. treatment of breast cancer in developing countries. World J Clin
Oncol 5(3):465
As an extension to the present work, we plan to gather 8. Dileep G, Gyani SGG (2022) Artificial intelligence in breast
diverse medical information (CT scans, X-rays, copy cancer screening and diagnosis. Cureus 14(10):e30318
number variation, signals etc.) beyond numeric and 9. Nassif AB, Talib MA, Nasir Q, Afadar Y, Elgendy O (2022)
histopathological images for breast cancer diagnosis to Breast cancer detection using artificial intelligence techniques: a
systematic literature review. Artif Intell Med 102276
validate various machine learning and deep learning 10. Yersal O, Barutca S (2014) Biological subtypes of breast cancer:
models. prognostic and therapeutic implications. World J Clin Oncol
5(3):412
11. Lipkova J, Chen RJ, Chen B, Lu MY, Barbieri M, Shao D,
Funding There is no funding involved in writing this paper. Vaidya AJ, Chen C, Zhuang L, Williamson DF et al (2022)
Artificial intelligence for multimodal data integration in oncol-
Data availability The dataset analyzed during this study can be found ogy. Cancer Cell 40(10):1095–1110
in (https://portal.gdc.cancer.gov/projects/TCGA-BRCA). 12. Kurt F, Agaoglu M, Arga KY (2022) Precision oncology: an
ensembled machine learning approach to identify a candidate
MRNA panel for stratification of patients with breast cancer.
Declarations OMICS 26(9):504–511
13. Kim AA, Zaim SR, Subbian V (2020) Assessing reproducibility
and veracity across machine learning techniques in biomedicine:
Conflict of interest There is no potential conflict of interest in our
a case study using TCGA data. Int J Med Inform 141:104148
paper. And all authors have seen the manuscript and approved it to
14. Phan NN, Huang C-C, Tseng L-M, Chuang EY (2021) Predicting
submit to your journal.
breast cancer gene expression signature by applying deep con-
volutional neural networks from unannotated pathological ima-
ges. Front Oncol 11:769447
References 15. Xie J, Liu R, Luttrell J IV, Zhang C (2019) Deep learning based
analysis of histopathological images of breast cancer. Front Genet
1. DeSantis CE, Bray F, Ferlay J, Lortet-Tieulent J, Anderson BO, 10:80
Jemal A (2015) International variation in female breast cancer 16. Liu T, Huang J, Liao T, Pu R, Liu S, Peng Y (2022) A hybrid
incidence and mortality rates international variation in female deep learning model for predicting molecular subtypes of human
breast cancer rates. Cancer Epidemiol Biomark Prevent breast cancer using multimodal data. Irbm 43(1):62–74
24(10):1495–1506 17. Ash JT, Darnell G, Munro D, Engelhardt BE (2021) Joint analysis
2. Goldin A, Venditti JM, Macdonald JS, Muggia FM, Henney JE, of expression levels and histological images identifies genes
Devita Jr VT (1981) Current results of the screening program at associated with tissue morphology. Nat Commun 12(1):1609
the division of cancer treatment, national cancer institute. Eur J 18. Sun D, Li A, Tang B, Wang M (2018) Integrating genomic data
Cancer (1965) 17(2), 129–142 and pathological images to effectively predict breast cancer
3. Momenimovahed Z, Salehiniya H (2019) Epidemiological char- clinical outcome. Comput Methods Programs Biomed 161:45–53
acteristics of and risk factors for breast cancer in the world. 19. MacFadyen C, Duraiswamy A, Harris-Birtill D (2023) Classifi-
Breast Cancer Targets Ther 151–164 cation of hyper-scale multimodal imaging datasets. medRxiv
4. Siegel RL, Miller KD, Wagle NS, Jemal A (2023) Cancer 2023-01
statistics. CA Cancer J Clin 73(1):17–48 20. Li B, Nabavi S (2023) A multimodal graph neural network
5. Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne framework for cancer molecular subtype classification. arXiv
M, Vignat J, Gralow JR, Cardoso F, Siesling S et al (2022) preprint arXiv:2302.12838

123
Neural Computing and Applications (2023) 35:24109–24121 24121

21. Popovici V, Budinska E, Čápková L, Schwarz D, Dušek L, Feit J, 33. Tammina S (2019) Transfer learning using VGG-16 with deep
Jaggi R (2016) Joint analysis of histopathology image features convolutional neural network for classifying images. Int J Sci Res
and gene expression in breast cancer. BMC Bioinform 17(1):1–9 Publ (IJSRP) 9(10):143–150
22. M’Sabah CEL, Bouziane A, Ferdi Y (2021) A survey on deep 34. Khalid S, Khalil T, Nasreen S (2014) A survey of feature
learning methods for cancer diagnosis using multimodal data selection and feature extraction techniques in machine learning.
fusion. In: 2021 international conference on e-health and bio- In: 2014 science and information conference. IEEE, pp 372–378
engineering (EHB), pp 1–4. IEEE 35. Ross AA, Govindarajan R (2005) Feature level fusion of hand
23. Hou Y (2020) Breast cancer pathological image classification and face biometrics. In: Biometric technology for human iden-
based on deep learning. J Xray Sci Technol 28(4):727–738 tification II, vol 5779. SPIE, pp 196–204
24. Zhu Z, Albadawy E, Saha A, Zhang J, Harowicz MR, Mazur- 36. Taud H, Mas J (2018) Multilayer perceptron (MLP). Geomatic
owski MA (2018) Breast cancer molecular subtype classification approaches for modeling land change scenarios, pp 451–455
using deep features: preliminary results. In: Medical imaging 37. Tohka J, Van Gils M (2021) Evaluation of machine learning
2018: computer-aided diagnosis, vol 10575. SPIE, pp 651–656 algorithms for health and wellness applications: a tutorial.
25. Kotsiantis SB, Zaharakis I, Pintelas P et al (2007) Supervised Comput Biol Med 132:104324
machine learning: a review of classification techniques. Emerging 38. Takahashi K, Yamamoto K, Kuchiba A, Koyama T (2022)
Artif Intell Appl Comput Eng 160(1):3–24 Confidence interval for micro-averaged f 1 and macro-averaged f
26. Dwivedi AK (2018) Artificial neural network model for effective 1 scores. Appl Intell 52(5):4961–4972
cancer classification using microarray gene expression data. 39. Mohr F, van Rijn JN (2022) Learning curves for decision making
Neural Comput Appl 29:1545–1554 in supervised machine learning—a survey. arXiv preprint arXiv:
27. Redzuwan R, Radzi NAM, Din NM, Mustafa I (2015) Affine 2201.12150
versus projective transformation for sift and ransac image 40. Viering T, Loog M (2022) The shape of learning curves: a
matching methods. In: 2015 IEEE international conference on review. IEEE Trans Pattern Anal Mach Intell
signal and image processing applications (ICSIPA). IEEE, 41. Brownlee J (2018) What is the difference between a batch and an
pp 447–451 epoch in a neural network. Mach Learn Mastery 20
28. Rana R, Verma A (2014) Comparison and enhancement of digital 42. Sun D, Wang M, Li A (2018) A multimodal deep neural network
image by using canny filter and sobel filter. IOSR J Comput Eng for human breast cancer prognosis prediction by integrating
16(1):06–10 multi-dimensional data. IEEE/ACM Trans Comput Biol Bioinf
29. Jeong W-K, Pfister H, Fatica M (2011) Medical image processing 16(3):841–850
using GPU-accelerated ITK image filters. In: GPU computing 43. Liu H, Dai Z, So D, Le QV (2021) Pay attention to MLPS. Adv
gems emerald edition. Elsevier, New York, pp. 737–749 Neural Inf Process Syst 34:9204–9215
30. Suzuki K (2017) Overview of deep learning in medical imaging.
Radiol Phys Technol 10(3):257–273 Publisher’s Note Springer Nature remains neutral with regard to
31. Alzubaidi L, Fadhel MA, Al-Shamma O, Zhang J, Santamarı́a J, jurisdictional claims in published maps and institutional affiliations.
Duan Y, Oleiwi SR (2020) Towards a better understanding of
transfer learning for medical imaging: a case study. Appl Sci
Springer Nature or its licensor (e.g. a society or other partner) holds
10(13):4523
exclusive rights to this article under a publishing agreement with the
32. Boumaraf S, Liu X, Zheng Z, Ma X, Ferkous C (2021) A new
author(s) or other rightsholder(s); author self-archiving of the
transfer learning based approach to magnification dependent and
accepted manuscript version of this article is solely governed by the
independent classification of breast cancer in histopathological
terms of such publishing agreement and applicable law.
images. Biomed Signal Process Control 63:102192

123