1

Cancer Detection and

Segmentation in
Pathological Whole-Slide
Images
Project ID: 2853Nguyen
Name: Cam Nguyen
School: Bob Jones High School
Madison, AL, USA
 2

Research Question
• Liver cancer is the leading cause of cancer deaths worldwide.
Manual examination of histopathological slides to detect cancer
is time consuming and even experienced pathologists differ
opinions 27% of the time.
• Automation of detection of cancer would
• help reduce time and work burden for pathologists,
• provide a consistent and maybe even more accurate
diagnosis.
• In recent years, scientists have proposed using advances in
Artificial Intelligence (AI) such as deep learning algorithms to
segment and detect cancerous regions in histopathological
Whole Slide Images (WSI) [4, 6, 9]
• Most methods use advanced Semantic Segmentation Neural Nets
(SSN) such as FCN, Unet, LinkNet, FPN etc. to segment tumor
regions. However, there has been no systematic evaluation of
their performance.
• Each model’s performance also depends on several parameters
whose impact has been inadequately studied. An example of a Whole Slide Image (WSI) of a
cancerous liver tissue.
• In this work, we aim to provide a thorough evaluation of major Where is the tumor?
semantic segmentation models for detection and diagnosis of
liver cancer in histopathological WSI. We want to analyze the
dependence of the cancer segmentation accuracy on :
• The choice of a particular SSN model,
• The size of local image patches,
• The input image resolution,
• The tumor type (viable or whole).
 3

Approach

We have implemented in Python an image processing pipeline for cancer diagnosis.

The algorithm takes as input a WSI of a tissue sample scanned and digitized by a
WSI scanner. The WSI is first divided into a set of local image patches. A pre-trained
SSN is employed to predict a binary tumor mask for every patch. This mask is a
black and white image where white/black pixels indicate the presence/absence of
cancer at the location. These binary masks are then fused into a global tumor mask
that indicates the region of cancer in the input WSI.
 4

Generation of Patches
• A WSI is usually stored as an image pyramid allowing us to view it in different resolution
levels: 0, 1, 2 with level 0 as the highest resolution.
• While WSI have huge sizes (80k x 80k at level 0, 20k x 20k at level 1, 5k x 5k at level 2),
typical SSNs operate on images of much smaller sizes between 50-150 pixels and at a
single resolution. Therefore, WSI is divided into small local patches of the same size. The
patches may overlap.
• The choice of the patch size and the input resolution level may impact the accuracy of
the tumor segmentation algorithm.

Level 2
....

....
Level 1

Patch

Level 0
WSI

The pyramidal structure of a WSI Example of extraction of a local patch

 5

Semantic Segmentation Network (SSN)

Input image Segmentation mask
• SSN is a convolutional neural network (CNN) where
the input layer represents an image and the
output layer represents a segmentation mask.
• Most SSNs have an encoder-decoder architecture.
The encoder downscales the input image and
squeezes it into a smaller layer called bottleneck.
The decoder takes information in the bottleneck
encoder bottleneck decoder and upscales to a segmentation mask.
Convolutional Encoder Decoder Architecture • In this project, we evaluate the performance of the
six major SSN models:
Loss Error • Pyramid Attention Network (PAN) [7]
computation • Pyramid Scene Parsing Network (PSPNet) [11]
GroundTruth
Mask

• Unet [8]
Predicted Masks

• LinkNet [1]
A dataset Patch
Backpropagation
of WSIs Sampler • DeepLabV3 [2]
• Feature Pyramid Network (FPN) [5]
Image

Updated
• To train the networks, we used Pytorch and the
SSN Weights open-source Semantic Segmentation Models
package [10]. Each model was trained for 1000
backpropagation iterations.
The algorithm for SSN training
 Generation of Global Mask and Determination 6

of Cancer Status
• SSN produces for each pixel in each patch a number
from 0 to 1 that represents the probability of the
pixel being cancerous.
• The final segmentation mask is computed by taking
the average probability of each pixel over local
patches and rounding to 0 or 1.
• The final segmentation mask helps determine the
tumor burden, i.e. the amount of cancer present:
,%), $- ./,'0) !"#$%
!"#$% '"%()* =
,%), $- 1ℎ$0) !"#$%
• Note: We train two separate SSNs: one detects
viable tumors and one detects whole tumors.
An example of viable and whole tumor
• A viable tumor is the region containing the cell nest
and active cancerous parts of a tumor while a whole
tumor contains cell nests as well as dead tissue
surrounding the viable tumor and tumor capsules.
 7

Dataset and Performance Metrics

• To evaluate the cancer segmentation algorithms, we used the public dataset from the
PAIP 2019 grand challenge [https://paip2019.grand-challenge.org/]. The data was
collected by the Seoul National University Hospital. It provides 50 WSIs of liver tissue
that have groundtruth tumor masks (both viable and whole) marked by experts. Each
image contains one tumor lesion from one patient.
• We used 40 of those slides for training the SSN models. The remaining 10 were used for
performance evaluation.
• Performance metrics:
• The accuracy of the predicted tumor mask compared to the groundtruth is
computed using the Jaccard index, also known as the intersection over union (IOU)
[3]:

,%), $- $.)%0,4
3 4%)(/5!)(, 7%$"*(!%"!ℎ =
,%), $- "*/$*

• The accuracy of tumor burden estimation:

8559:;<=> =?@ABC@= = (1 − 4%)(/5!)( '"%()* − 7%$"*(!%"!ℎ '"%()* )
• We trained each of the 6 SSN models for various combinations of patch size (128x128,
256x256, 512x512), tumor type (viable, whole), and pyramid level (0, 1, 2) and
evaluated models’ performance in terms of the above metrics. In total, we trained 6 x 3
x 2 x 3 = 108 models in 4 months!
 8

Qualitative Results

These results were obtained with the PAN network at level 2 and patch size 512.
More example results are provided in the attached research paper
 9

Quantitative Results
Average Jaccard Index between predicted tumor mask and groundtruth for viable
tumors for different combinations of SSN model, patch size, and resolution levels

Average Jaccard Index between predicted tumor mask and groundtruth for whole Tumor Burden Estimation accuracy
tumors for different combinations of SSN model, patch size, and resolution levels
 10

Interpretation of Results
• Most of the tested SSNs were able to detect and segment viable tumor regions with a
relatively high accuracy. In the majority of the results, the predicted viable tumor masks
were mostly similar to the groundtruth except for scattered small size details.
• Our best average Jaccard Index value is 0.73 obtained with FPN at patch size 512 and
level 2. It is lower than the top scores reported at the PAIP challenge which are between
0.75-0.79. The method of [6], the only current published work PAIP 2019, combines the
result from multiple models through an ensemble approach. On the other hand, the
objective of our work is to investigate the behavior of individual models.
• As expected, the detection results of whole tumors are generally less accurate than that
of viable tumors due to the difficulty in detecting peripheral tissues which sometimes
may be visually similar to healthy tissue.
• DeepLabV3, FPN, and PAN outperformed the other three SSN models. While FPN
performed the best for segmentation of viable tumors, DeepLabV3 and PAN performed
well for both viable and whole tumors, and therefore got the best tumor burden
estimation accuracy. PAN appears to exhibit stable good performance across levels and
patch sizes.
• The patch size 512 exhibits the best performance, suggesting that a large patch better
captures global information which helps distinguish a tumor from other parts of the
liver.
• To our surprise, the pyramid levels 1 and 0 tend to have worse results than level 2.
Again, this suggests that the global information captured by level 2 could play a more
important role in detecting tumors than local details provided by levels 1 and 0. Another
possible reason could be that models at higher resolutions need to be trained for a
longer amount of time. We will further investigate this issue in future work.
 11

Conclusion
• We have implemented 6 major semantic segmentation models, trained and
quantitatively evaluated their performance on the PAIP dataset for various
parameter settings.
• Results show that SSN are able to detect and segment tumors in WSI with
relatively high accuracy. The accuracy of tumor burden estimates almost
reaches 90%.
• PAN and DeepLabV3 came out as the best choices among the tested
models
• A large patch size and low input resolution tend to lead to better
performance, although more experiments are needed to confirm this
observation.
• Whole tumors are more difficult to segment than viable tumors.
• Our work provide a better understanding of the behavior of deep learning
algorithms for tumor segmentation. It helps researchers quickly set up and fine
tune the algorithms for their work.

Acknowledgements:
I would like to thank Professor Yuankai Huo from the department of Electrical Engineering
and Computer Science at the Vanderbilt University for giving his valuable time and
feedback.
 12

References
1. A. Chaurasia and E. Culurciello. “LinkNet: Exploiting encoder representations for efficientsemantic
segmentation”. In: (2017), pp. 1–4.
2. L. Chen et al. “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous
Convolution, and Fully Connected CRFs”. In: IEEE Transactions on Pattern Analysis and Machine
Intelligence 40.4 (2018), pp. 834–848.
3. Chung NC et al. Jaccard/Tanimoto similarity test and estimation methods for biological presence-
absence data. BMC Bioinformatics. 2019 Dec 24;20(Suppl 15):644
4. Baris Gecer et al. “Detection and classification of cancer in whole slide breast histopathology
images using deep convolutional networks”. In: Pattern Recognition 84 (2018), pp. 345–356.
5. Jonathan Hui. Understanding Feature Pyramid Networks for object detection (FPN). 2020.
6. Mahendra Khened et al. A Generalized Deep Learning Framework for Whole-Slide Image
Segmentation and Analysis. 2020. arXiv:2001.00258 [eess.IV].
7. Hanchao Li et al. “Pyramid Attention Network for Semantic Segmentation”. In:
CoRRabs/1805.10180(2018). arXiv:1805.10180.
8. Olaf Ronneberger et al. “U-net: Convolutional networks for biomedical image segmentation”. In:
International Conference on Medical image computing and computer-assisted intervention.
Springer. 2015, pp. 234–241.
9. Mary Shapcott et al. “Deep Learning With Sampling in Colon Cancer Histology”. In: Frontiers in
Bioengineering and Biotechnology 7 (2019), p. 52.
10. Pavel Yakubovskiy. segmentation-models-pytorch. 2020.
url:https://pypi.org/project/segmentation-models-pytorch/.
11. Hengshuang Zhao et al. “Pyramid scene parsing network”. In: Proceedings of the IEEE conference
on computer vision and pattern recognition. 2017, pp. 2881–2890.