2024 IEEE Cloud Summit

Short-term Lung Cancer Survival Prediction:

Combining Linear Regression and Convolutional
Neural Network
Xuandi Wang Kevin Zeng Mikhail Y. Shalaginov
BASIS International School Hangzhou Academy for Advanced Research and Department of Materials Sciences and
Hangzhou, Zhejiang, China Development Engineering, MIT
[email protected] 14th floor, One Broadway, Cambridge, MA 77 Massachusetts Ave, Cambridge, MA
02142, USA 02139, USA
[email protected] [email protected]

Shuang Xiang Lin *Tingying Helen Zeng

Department of Radiology Division of Career Education
The Second Affiliated Hospital, Academy for Advanced Research and Development
Zhejiang University School of Medicine 14th floor, One Broadway, Cambridge, MA 02142, USA
Hangzhou, Zhejiang, [email protected] [email protected]
*Corresponding authors

Abstract— Lung cancer, one of the most common and deadly and treatment planning important for doctors to decide on
cancers, could be a great financial and mental burden to patients remedies.
and their families. Thus, it is important to develop a model that
2024 IEEE Cloud Summit | 979-8-3503-7006-5/24/$31.00 ©2024 IEEE | DOI: 10.1109/CLOUD-SUMMIT61220.2024.00014

assesses the patients’ conditions and provide insightful feedback Curing cancer in general is a process that requires
for doctors to refer to. One such feedback that proves valuable significant amount of financial investment. For lung cancer,
is survival prediction, the assessment of the chances to stay alive important treatment steps such as surgery and chemotherapy
in a given period of time. In this study, we propose to use linear will contribute greatly to the total cost. Specifically, surgery
regression to analyze clinical data and Convolutional Neural costs $15,000 to over $30,000, and chemotherapy $1,000 to
Network (CNN) models to analyze Computer Tomography (CT) $12,000 per month[1] . The great amount of financial stress it
scans data to complete the task of lung cancer survival places on families likewise urges for remedy plan that can help
prediction in short-term timespan, within 6 months. The linear patients to effectively allocate their financial assets.
regression and CNN model achieved an Area Under the
Receiver Operating Characteristic Curve (AUROC) of 0.654 One factor that aids doctors for treatment decisions is the
and 0.572, respectively, while the combination of the two models survival prediction. By acknowledging survival prediction,
yields excellent AUROC of 0.789. The improvement in doctors can assign medical resources accordingly. Survival
combined results provides valuable insights to the feasibility of prediction also helps families to decide on treatment plans. For
implementing hybrid models on cancer survival prediction. patients with better survival probability and stable medical
conditions, families could pursue a long-term medication to
Keywords—Survival Prediction, Machine Learning, Linear battle cancer. On the other hand, for patients having
Regression, Lung Cancer, Convolutional Neural Network. comparably lower survival rate, the patient could either pursue
a more radical treatment method, or simply choose to give up
on remedying. This is especially prominent in analyzing
I. INTRODUCTION survival probability at a small timescale, such as 6 months and
up to 1 year. This makes short-term survival prediction a
Lung cancer, mainly caused by smoking or exposure to
crucial element of prognosis.
secondhand smoke, is one of the deadliest cancers [1]. It can be
classified into two types: small cell lung cancer (SCLC) and As artificial intelligence becomes more mature for medical
non-small cell lung cancer (NSCLC). NSCLC could further fields, AI-assisted cancer treatments are an obvious
be classified as adenocarcinoma, squamous cell carcinoma, application. Thus, it is logical to employ an AI-assisted
and large cell carcinoma. This classification is mainly based approach in the determination of survival prediction. While
on the behavior and orientation of the cancerous cells. healthcare professionals could only determine survival rate by
Nevertheless, all types of lung cancer are dangerous and experience and similar cases[2], machine learning (ML) could
require extensive care and effective treatment. observe deeper, underlying trends in provided data, and thus
being a great tool for cancer prediction. Our goal is to develop
Major symptoms of lung cancer include coughing with
a machine learning model that could effectively determine the
blood, short breaths, and chest pain. Professional treatment
survival time for patients.
should be applied as soon as possible to reduce such burden.
In some cases, patients would choose euthanasia over
medication processes to relieve pain. This makes prognosis

979-8-3503-7006-5/24/$31.00 ©2024 IEEE 42

DOI 10.1109/Cloud-Summit61220.2024.00014
Authorized licensed use limited to: ANNA UNIVERSITY. Downloaded on August 21,2024 at 03:59:46 UTC from IEEE Xplore. Restrictions apply.
 II. LITERATURE SURVEY Patient
age
Clinical.T Clinical.N Clinical.M
ID Stage Stage Stage
Survival prediction is a problem that many have attempted Lung1- 83.8001
to solve using machine learning or other techniques.[3] 2 0 0
002
Overall.
One of the earliest studies concerned with survival Histology gender Survival.time deadstatus.event
Stage
prediction is from Sir David Cox, who developed the well- squamous
known Cox Proportional Hazards model (CPH) [4]. CPH I cell male 155 1
explores the relationship between individual predictor carcinoma
variables and the gross hazard function, and specifically, that Table 1: Sample clinical data corresponding to the patient in Figure 1.
individual predictor variables are multiplicatively related to
the hazard function.[5] The study provided valuable insights to In the sample clinical data shown in Table 1, Clinical T, N,
future explorations in the topic. M stage refers to the subdivided stage determined according
to the TNM guidelines[13], which is then considered
In many studies, ML and deep learning was employed as holistically to determine the overall stage. Column
a major method. Christoph Haarburger et al. used “deadstatus.event” refers to the current status of the patients
Convolutional Neural Networks (CNNs) trained on image with respect to the date of data collection, with 1 indicating
data and achieved promising results. [6] Shreyesh Doppalapudi death and 0 otherwise.
et al. furthered the usage of machine learning and compared
different types of ML models. The study shows that Artificial Ultimately, the data columns “age”, “Overall Stage”,
Neural Networks, Convolutional Neural Networks and “Histology”, “gender”, and “Survival.time” are used as
Recurrent Neural Networks outperforms traditional ML training and testing data, with “PatientID” as index reference.
models.[7] Qanita Bani Baker et al. compared the performance
For the convenience of machine learning model training,
of various different models such as Random Forest (RF) and
CT scans were expected to be
the Cox-PH model, a variation of the CPH model. [8]
of the same size. Patient data
Other studies explored numerous techniques in the task of with not enough CT scans
survival prediction. M. Berkan Sesen et al.[9], Lu Zhong et al.[5], were filtered, and patient data
and Fattaneh Jabbari et al.[10] employed the Bayes Theory in with too much CT scans were
survival prediction models. James A. Bartholomai et al. truncated to meet the
experimented with linear regression, Gradient Boosted expectation. In short, all CT
Machines, RF, etc. scans data have 100 scans in
total, with each scan resized to
All these studies have contributed to the further 224*224 pixel. The size of the
explorations on lung cancer survival prediction and the dataset after the filtering is
creation of new models. 154 patients (n = 154).
A few of the patients have
III. MATERIALS AND METHODS missing essential data, that is,
data to be used for model
A. Dataset training and testing, and thus
The data used in this study is from the Cancer Imaging were removed, along with the
Archive (TCIA) [11] collection “NSCLC-Radiomics” [12]. The corresponding CT scan data.
dataset contains 422 patients subjects of data (n = 422), of The size of the dataset after
which includes clinical data, such as age and sex, and imaging the filtering is 130 patients (n
data, namely computer topography (CT) scans. The dataset is = 130).
public and can be used for related scientific studies. Patients with
Figure 2: Flow chart
deathstatus.event data equals showing the filtering of
0 were also filtered, since data.
censoring time is not of great
value to survival period
prediction. The final size of used dataset is 114 patients (n =
114).
The dataset contains patients with age ranging from 42 to
88, with median 68. 20% of the patients are in stage I cancer,
7.7% in stage II, and 72.3% in stage III. 34.6% of the patients
have lung cancer type “Large Cell Carcinoma”, 36.2% have
“Squamous Cell Carcinoma”, 14.6% for both
Figure 1: Sample cross-sectional CT scan. “Adenocarcinoma” and “NOS” (Not Otherwise Specified).
74.6% of the patients are male, and 25.4% female. The
survival period of all patients ranges from 25 to 3632, with a
median of 544 (unit = days). The survival period of patients

43

Authorized licensed use limited to: ANNA UNIVERSITY. Downloaded on August 21,2024 at 03:59:46 UTC from IEEE Xplore. Restrictions apply.
with deathstatus.event value = 1 ranges from 25 to 3494, with The model will then compare the actual weight with the
a median of 446 (unit = days). predicted weight, and give the next predicted weight within
the interval
ሺߚ௣௥௘ௗ ǡ ሺߚ௔௖௧ െ ߚ௣௥௘ௗ ሻ ‫ ߩ כ‬൅ ߚ௣௥௘ௗ ሻ,
B. Linear Regression Model
where ߚ௣௥௘ௗ is the previous predicted weight, ߚ௔௖௧ is the
The simplest of all regression models is linear
previous actual weight, and ߩ <1 is the learning rate.
regression[14]. Linear regression is modeled according to Eq. 1:
The model repeats the procedure until ‫ ݍ‬ൌ ݇, with k being
ௗ the number of iterations specified before model training.
‫ ݕ‬ൌ  ෍ ߚ௜ ‫ݔ‬௜ ൅ ߝǡ ሺͳሻ
௜ୀଵ
‫ ݕ‬is the predicted output, or survival period in this case,
modeled by ݀ variables, with ߚ௜ being the relative weight of
variables, or predictors, ‫ݔ‬௜ , and ߝ being the intercept or the
error term.
This model is simple and versatile to adapt to. Linear
regression does not require much computing resource to train,
thus being a cost-effective and efficient model. Other complex
models such as polynomial regression are more costly since
they contain more terms, especially at higher dimensions.
Linear regression’s rigidness gives it resistance to
overfitting and its versatility avoids underfitting. In general, a
high degree polynomial regression is prone to overfitting
because of its sensitive pattern detection in the training data[15].
These characteristics of a linear regression model make it
especially effective in handling clinical data. Being in entirely
numerical form, clinical data is best analyzed with regression Figure 3: Flow chart showing training progress of a linear regression model.
models, and having multiple variables and data columns
makes linear regression a good choice that avoids overfitting. If ߩ is too high, it may create great flunctuations in weight
prediction; and if ߩ is too little, it will result in reduction in
The process of fitting is as such: the model will initialize
training efficiency. The value of ߩ should be determined after
itself by first making a guess of the weights. Then the model
doing various tests with different values of ߩ, selecting the
evaluates itself by taking ݀ numbers of rows of data (݀ is the
number of predictor variables, or dimensions), and calculating best. Similarly, the value of ݇ should also be determined with
the actual weight using matrix multiplication (shown in Eq. 2): experiments. In this study, we renew the best results after
each training epoch to capture the best version of the model
࢞ࢼ ൌ ࢟ possible, thus it is not incorrect to say the higher the value of
֝ ݇ the better. However, large ݇ value will bring additional
ࢼ ࢟
࢞ burden to computational capabilities, and thus it is advised to
ᇩᇭᇭᇭᇭ
‫ݔ‬ଵǡ௤ ᇭᇭᇭᇪ
‫ݔ‬ଶǡ௤ ᇭᇭᇭ
ǥ ‫ݔ‬ᇭᇭᇭ
ௗǡ௤ ฑߚଵ ᇩᇭᇭᇪᇭ
‫ݕ‬௤ keep the value of ݇ from getting too large. Normally ݇ should
‫ݔ‬ଵǡ௤ାଵ ‫ݔ‬ଶǡ௤ାଵ ǥ ‫ݔ‬ௗǡ௤ାଵ ߚଶ ‫ݕ‬௤ାଵ be about 3-4 times the number of data to ensure the best
൦ ‫ڭ‬ ‫ڭ‬ ‫ڰ‬ ‫ڭ‬ ൪൦ ൪ ൌ ൦ ‫ ڭ‬൪
‫ڭ‬ output.
‫ݔ‬ଵǡ௤ାௗିଵ ‫ݔ‬ଶǡ௤ାௗିଵ ǥ ‫ݔ‬ௗǡ௤ାௗିଵ ߚௗ ‫ݕ‬௤ାௗିଵ
֝
࢞ି૚ ࢞ࢼ ൌ ࢞ି૚ ࢟ ֜ ࢼ ൌ ࢞ି૚ ࢟ C. CNN Model
֝ The image data is being trained on a machine learning
ߚଵ ‫ݔ‬ଵǡ௤ ‫ݔ‬ଶǡ௤ ǥ ‫ݔ‬௡ǡ௤ ିଵ ‫ݕ‬௤
CNN classification model. CNN models are typically used for
ߚ ‫ݔ‬ ଵǡ௤ାଵ ‫ݔ‬ଶǡ௤ାଵ ǥ ‫ݔ‬௡ǡ௤ାଵ ‫ݕ‬௤ାଵ
൦ ଶ൪ ൌ ൦ ൪ ൦ ‫ ڭ‬൪ ሺʹሻ image classification for their convolutional characteristics that
‫ڭ‬ ‫ڭ‬ ‫ڭ‬ ‫ڰ‬ ‫ڭ‬
are meant to deal with data of high dimensionality.
ߚௗ ‫ݔ‬ଵǡ௤ାௗିଵ ‫ݔ‬ଶǡ௤ାௗିଵ ǥ ‫ݔ‬௡ǡ௤ାௗିଵ ‫ݕ‬௤ାௗିଵ
The CNN model is constructed with the Keras
‫ ݍ‬signifies the current iteration and ‫݆݀݋݉ݍ‬, where j is application[16]. The CNN model features a binary
the number of rows the dataset contains, is thus the index of classification with class 0 and 1 representing alive and dead in
row data. ‫ ݍ‬does not need to be less than ݆, with the extra ‫ ݍ‬െ each period (set to 6 months = 180 days). Image data is passed
݆ number of rows of data be taken from the start of the dataset. in 3D format, as volumetric representations of every patient’s
Directly after initialization, ‫ ݍ‬ൌ Ͳ. lung CT. Such data type requires significant computer
resources, thus only a relatively simple model was built. In
short, there are 3 convolutional layers, each followed by a

44

Authorized licensed use limited to: ANNA UNIVERSITY. Downloaded on August 21,2024 at 03:59:46 UTC from IEEE Xplore. Restrictions apply.
MaxPooling layer with ReLU activation. Two dense layers of
8 and 2 neurons each are passed afterwards. The CNN model The ROC curve shows an AUC of 0.654, indicating
structure is shown in Figure 4. limited effectiveness in lung cancer survival prediction. It was
hypothesized that linear regression model does not take
Figure 4: Flowchart showing the structure of CNN model
advantage in complex situations. Trends in clinical patient
data are unlikely to take a simple linear pattern, limiting the
abilities of a linear regression model to accurately and
precisely predict.
The CNN model is evaluated next. It was trained solely on
the CT scans.

D. Combined Model
The results from the previous models are analyzed and
used to train the hybrid model. Both the linear regression and
CNN model outputs numerical value between 0 to 1 that
represent survival probability within 6 months after diagnosis
and treatment plans. Using these results, we constructed
another linear regression model to fit the two outputs to
produce the ultimate prediction. This model has the same
structure and logic as the first linear regression model.
Training parameters of the hybrid model are set to ߩ ൌ
ͲǤ͵ǡ ݇ ൌ ͷͲͲ.
Figure 6&7: Training graph for CNN model. Accuracy (above) and Loss
(below) is plotted against epoch number
IV. RESULTS AND DISCUSSION
The training graphs for CNN model did show a fast-
The linear regression model and CNN model are first peaking accuracy and a steep sloped loss graph. Both graphs
trained separately. Results are analyzed with Scikit Learn indicate success in training. However, ROC of the CNN did
application[17]. not turn out as good (see Fig. 8). With relatively high training
Figure 5 shows the results of linear regression model doing accuracy and low AUC, overfitting might have occurred.
lung cancer survival prediction based solely on clinical data.
Figure 5: ROC (blue) of the linear regression model with ߩ ൌ ͲǤ͵ǡ ݇ ൌ ͷͲͲ.

Horizontal axis is False-Positive Rate (FPR), vertical axis is True-Positive

Rate (TPR). Orange line is the standard ROC curve for comparison.

45

Authorized licensed use limited to: ANNA UNIVERSITY. Downloaded on August 21,2024 at 03:59:46 UTC from IEEE Xplore. Restrictions apply.
 hybrid model approach, combining both linear regression and
CNN model to analyze patient data in multiple angles.

V. CONCLUSION
The results of the experiments have shown that the
combined model of linear regression and CNN outperforms
any of the models on itself. Based on the training and
evaluation of the model, we propose the new approach of
combining linear regression and CNN model as a novel
method to solving the problem of lung cancer survival
prediction with great potential. This model is cost-effective
Figure 8&9: Above is ROC (blue) of the CNN model. Horizontal axis is
and efficient with great versatility.
False-Positive Rate (FPR), vertical axis is True-Positive Rate (TPR). Orange It is also important to acknowledge this study’s limitations:
line is the standard ROC curve for comparison. Below is the confusion
matrix for the CNN model. 0 and 1 indicate dead and alive, respectively.
first, this study tested the approach with a rather limited
dataset. This means that statistically the results from model
training may not be able to accurately reflect the situation with
The two models are then combined and trained again, with greater datasets, while also potentially promoting the
the linear regression model responsible for analyzing clinical potentials of the proposed method at the same time. In addition,
data and CNN model for CT scan data (see Figs. 10 and 11). the approach is not yet developed to perform the best of its
abilities, with the limited sophistication of its structure and
various improvements that can be made. This provides
promising opportunities for further research work and builds
a basis for future developments.
This hybrid approach is an entirely new approach to lung
cancer survival prediction, and thus has great potentiality to
be developed. In the future, we will continue to modify the
model so that it can be even more effective in prediction and
adaptive in application.
To summarize, linear regression combined with CNN
modeling is a novel and advanced approach in the study of
lung cancer survival period prediction. Its accuracy provides
great potential to improve healthcare effectiveness, and we
believe that when implementation is achieved, these results
will benefit a great number of people.
ACKNOWLEDGMENTS
The first author, Xuandi Wang, is grateful for the
sponsorship and internship training by the Academy for
Advanced Research and Development (AARD). This project
is partially supported by the Scholarship of Future Scholars
of AARD, (http://www.ardacademy.org).
REFERENCES

[1] Dorwart, L. (2022, July 12). How much does lung cancer treatment
cost? Verywell Health. https://www.verywellhealth.com/lung-
cancer-treatment-cost-5217734#toc-medical-bill-terms
[2] Mayo Foundation for Medical Education and Research. (2024,
Figure 10&11: Above is ROC (blue) of the combined model. Horizontal axis February 7). What cancer survival rates can (and can’t) tell you.
is False-Positive Rate (FPR), vertical axis is True-Positive Rate (TPR). Mayo Clinic. https://www.mayoclinic.org/diseases-
Orange line is the standard ROC curve for comparison. Below is training conditions/cancer/in-depth/cancer/art-20044517
graph of the combined model in terms of Root Mean Squared Error (RMSE) [3] Altuhaifa, F. A., Win, K. T., & Su, G. (2023). Predicting lung cancer
survival based on clinical data using machine learning: A
The AUROC of the final model is 0.789, surpassing both review. Computers in biology and medicine, 165, 107338.
the individual result of the linear regression model and the https://doi.org/10.1016/j.compbiomed.2023.107338
CNN model. The RMSE training graph was also saw rapid [4] Cox, D. R. (1972). Regression Models and Life-Tables. Journal of
reduction to a near-0 value. This suggests the validity of the the Royal Statistical Society. Series B (Methodological), 34(2), 187–
220. http://www.jstor.org/stable/2985181

46

Authorized licensed use limited to: ANNA UNIVERSITY. Downloaded on August 21,2024 at 03:59:46 UTC from IEEE Xplore. Restrictions apply.
[5] Zhong, L., Yang, F., Sun, S. et al. Predicting lung cancer survival
prognosis based on the conditional survival bayesian network. BMC
Med Res Methodol 24, 16 (2024). https://doi.org/10.1186/s12874-
023-02043-y
[6] C. Haarburger, P. Weitz, O. Rippel and D. Merhof, "Image-Based
Survival Prediction for Lung Cancer Patients Using CNNS," 2019
IEEE 16th International Symposium on Biomedical Imaging (ISBI
2019), Venice, Italy, 2019, pp. 1197-1201, doi:
10.1109/ISBI.2019.8759499.
[7] Shreyesh Doppalapudi, Robin G. Qiu, Youakim Badr, Lung cancer
survival period prediction and understanding: Deep learning
approaches, International Journal of Medical Informatics, Volume
148, 2021, 104371, ISSN 1386-5056,
https://doi.org/10.1016/j.ijmedinf.2020.104371.
[8] Q. B. Baker, E. Khwaileh, M. Alharbi and Y. Jararweh, "Lung
Cancer Survival Time Prediction Using Machine Learning and Deep
Learning Techniques," 2023 Fourth International Conference on
Intelligent Data Science Technologies and Applications (IDSTA),
Kuwai, Kuwait, 2023, pp. 35-42, doi:
10.1109/IDSTA58916.2023.10317871.
[9] Sesen, M. B., Kadir, T., Alcantara, R. B., Fox, J., & Brady, M.
(2012). Survival prediction and treatment recommendation with
Bayesian techniques in lung cancer. AMIA ... Annual Symposium
proceedings. AMIA Symposium, 2012, 838–847.
[10] Jabbari, Fattaneh & Villaruz, Liza & Davis, Mike & Cooper,
Gregory. (2020). Lung Cancer Survival Prediction Using Instance-
Specific Bayesian Networks. 10.1007/978-3-030-59137-3_14.
[11] The Cancer Imaging Archive.
https://www.cancerimagingarchive.net/
[12] Aerts, H. J. W. L., Wee, L., Rios Velazquez, E., Leijenaar, R. T. H.,
Parmar, C., Grossmann, P., Carvalho, S., Bussink, J., Monshouwer,
R., Haibe-Kains, B., Rietveld, D., Hoebers, F., Rietbergen, M. M.,
Leemans, C. R., Dekker, A., Quackenbush, J., Gillies, R. J., Lambin,
P. (2014). Data From NSCLC-Radiomics (version 4) [Data set].
The Cancer Imaging
Archive. https://doi.org/10.7937/K9/TCIA.2015.PF0M9REI
[13] TNM classification of malignant tumours. UICC. (n.d.).
https://www.uicc.org/what-we-do/sharing-knowledge/tnm
[14] GfG. (2024, March 20). Linear regression in machine learning.
GeeksforGeeks. https://www.geeksforgeeks.org/ml-linear-
regression/
[15] Mithani, D. (2021, November 11). Overfitting and underfitting with
polynomial regression. Medium.
https://medium.com/@danish.mithani/overfitting-and-underfitting-
with-polynomial-regression-7a4ab2c8177a
[16] F. Chollet and Others, ‘Keras’, 2015. [Online]. Available:
https://keras.io.
[17] F. Pedregosa et al., ‘Scikit-learn: Machine Learning in
Python’, Journal of Machine Learning Research, vol. 12, pp. 2825–
2830, 2011.

47

Authorized licensed use limited to: ANNA UNIVERSITY. Downloaded on August 21,2024 at 03:59:46 UTC from IEEE Xplore. Restrictions apply.