Twenty Years of Whole Slide Imaging - the Coming Phase Change
Download as PPTX, PDF1 like180 views
The document discusses the evolution and future of whole slide imaging in digital pathology, highlighting the impact of technological advancements on the classification and analysis of cancer. It emphasizes the integration of machine learning and deep learning techniques for precise cancer diagnoses and the importance of tumor spatial context and immune response in treatment outcomes. Various initiatives and research collaborations are mentioned, showcasing efforts to leverage imaging data for improved pathology assessments and patient prognosis.
Twenty Years of Whole Slide Imaging - the Coming Phase Change
- 1. Twenty Years of Whole Slide Imaging - the Coming Phase Change Joel Saltz MD, PhD Chair and Professor Department of Biomedical Informatics Professor Department of Pathology Stony Brook Medicine Pathology Visions, October 2, 2017
- 2. NO CONFLICTS TO DISCLOSE
- 5. The Virtual Microscope: Johns Hopkins 1997
- 6. Johns Hopkins School of Medicine Virtual Microscope
- 7. From 1998 Johns Hopkins Grand Rounds
- 8. The Virtual Microscope arose from University of Maryland College Park Active Data Repository Project
- 9. and National Science Foundation Grand Challenge Grant
- 10. Seven Years before Google EarthS
- 11. 1999 DARPA – HUBS Project Joel Saltz, Mike Becich and David Foran • Exploration of full slide digitized datasets • Classification of hematologic malignancies • Content Based Image Retrieval • Remotely steered microscope
- 12. Classification of Hematologic Malignancies – David Foran -- UMDNJ
- 13. Pathology Image Driven Decision Support • Improve reproducibility in traditional Pathology assessments (e.g. Gleason grade, NSCLC subtypes) • Precise scoring of well known criteria ( tumor infiltrating lymphocytes, mitoses and IHC staining) • Development of novel computational methods to employ Pathology image information to predict response to cancer treatment and outcomes.
- 14. Early Steps to Pathology Computer Aided Classification 2005-2010 Gurcan, Shamada, Kong, Saltz Hiro Shimada, Metin Gurcan, Jun Kong, Lee Cooper Joel Saltz BISTI/NIBIB Center for Grid Enabled Image Analysis - P20 EB000591, PI Saltz
- 15. Neuroblastoma Classification FH: favorable histology UH: unfavorable histology CANCER 2003; 98:2274-81 <5 yr Schwannian Development ≥50% Grossly visible Nodule(s) absent present Microscopic Neuroblastic foci absent present Ganglioneuroma (Schwannian stroma-dominant) Maturing subtype Mature subtype Ganglioneuroblastoma, Intermixed (Schwannian stroma-rich) FH FH Ganglioneuroblastoma, Nodular (composite, Schwannian stroma-rich/ stroma-dominant and stroma-poor) UH/FH* Variant forms* None to <50% Neuroblastoma (Schwannian stroma-poor) Poorly differentiated subtype Undifferentiated subtype Differentiating subtype Any age UH ≥200/5,000 cells Mitotic & karyorrhectic cells 100-200/5,000 cells <100/5,000 cells Any age ≥1.5 yr <1.5 yr UH UH FH ≥200/5,000 cells 100-200/5,000 cells <100/5,000 cells Any age UH ≥1.5 yr <1.5 yr ≥5 yr UH FH UH FH
- 16. Multi-Scale Machine Learning Based Shimada Classification System • Background Identification • Image Decomposition (Multi- resolution levels) • Image Segmentation (EMLDA) • Feature Construction (2nd order statistics, Tonal Features) • Feature Extraction (LDA) + Classification (Bayesian) • Multi-resolution Layer Controller (Confidence Region) No Yes Image Tile Initialization I = L Background? Label Create Image I(L) Segmentation Feature Construction Feature Extraction Classification Segmentation Feature Construction Feature Extraction Classifier Training Down-sampling Training Tiles Within Confidence Region ? I = I -1 I > 1? Yes Yes No No TRAINING TESTING
- 18. Deep Learning - Brain Tumor Classification – CVPR 2016
- 20. Digital Pathology as Precision Medicine • Statistical analyses and machine learning to link Radiology/Pathology features to “omics” and outcome biological phenomena • Image analysis and deep learning methods to extract features from images • Support queries against ensembles of features extracted from multiple datasets • Identify and segment trillions of objects – nuclei, glands, ducts, nodules, tumor niches • Analysis of integrated spatially mapped structural/”omic” information to gain insight into cancer mechanism and to choose best intervention
- 21. Quantitative Feature Analysis in Pathology: Emory In Silico Center for Brain Tumor Research (PI = Dan Brat, PD= Joel Saltz) 2009 - 2013
- 22. Using TCGA Data to Study Glioblastoma Diagnostic Improvement Molecular Classification Predictors of Progression
- 24. Oligodendroglioma Astrocytoma Nuclear Qualities Can we use image analysis of TCGA GBMs TO INFORM diagnostic criteria based on molecular or clinical endpoints? Application: Oligodendroglioma Component in GBM
- 25. Direct Study of Relationship Between Image Features vs Clinical Outcome, Response to Treatment, Molecular Information Cooper, Moreno Brat, Saltz, Kurc
- 26. Integrative Morphology/”omics” Quantitative Feature Analysis in Pathology: Emory In Silico Center for Brain Tumor Research (PI = Dan Brat, PD= Joel Saltz) NLM/NCI: Integrative Analysis/Digital Pathology R01LM011119, R01LM009239 (Dual PIs Joel Saltz, David Foran) Marcus Foundation Grant – Ari Kaufman, Joel Saltz
- 27. Associations
- 28. Integrative Analysis – Histology, “omics” to predict outcome • Kun Huang, Raghu Machiraju and collaborators • Research program to develop analytic methods to link Pathology features gene expression, methylation to outcome in many types of cancer
- 29. 1 Whole Slide Images 2 Representative Patches 3 Image Preprocessing 4 SuperPixel Segmentation 5 LBP Features Tissue Classification 6 Stromal tissue Epithelial tissue Cell Segmentation 7 Feature Extraction 8 Epithelial Features Stromal Features
- 30. Renal Papillary Cell Cancer 30
- 31. • Stony Brook, Institute for Systems Biology, MD Anderson, Emory group • TCGA Pan Cancer Immune Group – led by ISB researchers • Deep dive into linked molecular and image based characterization of cancer related immune response
- 32. Le Hou – Graduate Student Computer Science Vu Nguyen– Graduate Student Computer Science Anne Zhao – Pathology Informatics Biomedical Informatics, Pathology (now Surg Path Fellow SBM) Raj Gupta – Pathology Informatics Biomedical Informatics, Pathology Deep Learning and Lymphocytes: Stony Brook Digital Pathology Trainee Team The future of Digital Pathology
- 34. Importance of Immune System in Cancer Treatment and Prognosis • Tumor spatial context and cellular heterogeneity are important in cancer prognosis • Spatial TIL densities in different tumor regions have been shown to have high prognostic value – they may be superior to the standard TNM classification • Immune related assays used to determine Checkpoint Inhibitor immune therapy in several cancer types • Strong relationships with molecular measures of tumor immune response – results to soon appear in TCGA Pan Cancer Immune group publications • TIL maps being computed for SEER Pathology studies and will be routinely computed for data contributed to TCIA archive • Ongoing study to relate TIL patterns with immune gene expression groups and patient response
- 35. Imaging Based TIL Analysis Workflow Deep Learning Training, Validation and Prediction • Algorithm first trained on image patches • Several cooperating deep learning algorithms generate heat maps • Heat maps used to generate new predictions • Companion molecular statistical data analysis pipelines
- 40. Tools: Quantitative Imaging Pathology - QuIP Tool Set
- 41. Interactive Deep Learning Training Tool
- 44. Tools to Analyze Morphology and Spatially Mapped Molecular Data - U24 CA180924 • Specific Aim 1 Analysis pipelines for multi- scale, integrative image analysis. • Specific Aim 2: Database infrastructure to manage and query Pathomics features. • Specific Aim 3: HPC software that targets clusters, cloud computing, and leadership scale systems. • Specific Aim 4: Develop visualization middleware to relate Pathomics feature and image data and to integrate Pathomics image and “omic” data.
- 45. TCIA encourages and supports the cancer imaging open science community by hosting and managing Findable Accessible, Interoperable, and Reusable (FAIR) images and related data. http://www.cancerimagingarchive.net/ Clark, et al. J Digital Imag 26.6 (2013): 1045-1057. Cancer Imaging Archive – Integration of Pathology and Radiology for Community Clinical Studies
- 46. TCIA sustainment and scalability Platforms for quantitative imaging informatics in precision medicine Prior, Saltz, Sharma -- U24CA215109-01 • Identify quantitative imaging phenotypes across scale through the use of Radiomic/Pathomic analyses • Well-curated data for algorithm testing and validation. • Integrative Radiology/Pathology Image-Omics studies • Extend TCIA to support its rapidly growing user community and continue to promote research reproducibility and data reuse in cancer precision medical research.
- 47. Thanks!
- 48. ITCR Team Stony Brook University Joel Saltz Tahsin Kurc Yi Gao Allen Tannenbaum Erich Bremer Jonas Almeida Alina Jasniewski Fusheng Wang Tammy DiPrima Andrew White Le Hou Furqan Baig Mary Saltz Raj Gupta Emory University Ashish Sharma Adam Marcus Oak Ridge National Laboratory Scott Klasky Dave Pugmire Jeremy Logan Yale University Michael Krauthammer Harvard University Rick Cummings
- 49. Funding – Thanks! • This work was supported in part by U24CA180924, U24CA215109, NCIP/Leidos 14X138 and HHSN261200800001E from the NCI; R01LM011119-01 and R01LM009239 from the NLM • This research used resources provided by the National Science Foundation XSEDE Science Gateways program under grant TG-ASC130023 and the Keeneland Computing Facility at the Georgia Institute of Technology, which is supported by the NSF under Contract OCI- 0910735.