Deep Learning–Based Objective and Reproducible Osteosarcoma Chemotherapy Response Assessment and Outcome Prediction

Published:December 20, 2022DOI:
      Osteosarcoma is the most common primary bone cancer, whose standard treatment includes pre-operative chemotherapy followed by resection. Chemotherapy response is used for predicting prognosis and further management of patients. Necrosis is routinely assessed after chemotherapy from histology slides on resection specimens, where necrosis ratio is defined as the ratio of necrotic tumor/overall tumor. Patients with necrosis ratio ≥90% are known to have a better outcome. Manual microscopic review of necrosis ratio from multiple glass slides is semiquantitative and can have intraobserver and interobserver variability. In this study, an objective and reproducible deep learning–based approach is proposed to estimate necrosis ratio with outcome prediction from scanned hematoxylin and eosin whole slide images (WSIs). To conduct the study, 103 osteosarcoma cases with 3134 WSIs were collected. Deep Multi-Magnification Network was trained to segment multiple tissue subtypes, including viable tumor and necrotic tumor in pixel level and to calculate case-level necrosis ratio from multiple WSIs. Necrosis ratio estimated by the segmentation model highly correlates with necrosis ratio from pathology reports manually assessed by experts. Furthermore, patients were successfully stratified to predict overall survival with P = 2.4 × 10–6 and progression-free survival with P = 0.016. This study indicates deep learning can support pathologists as an objective tool to analyze osteosarcoma from histology for assessing treatment response and predicting patient outcome.

      Graphical abstract

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The American Journal of Pathology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ottaviani G.
        • Jaffe N.
        The epidemiology of osteosarcoma.
        Cancer Treat Res. 2009; 152: 3-13
        • Provisor A.J.
        • Ettinger L.J.
        • Nachman J.B.
        • Krailo M.D.
        • Makley J.T.
        • Yunis E.J.
        • Huvos A.G.
        • Betcher D.L.
        • Baum E.S.
        • Kisker C.T.
        • Miser J.S.
        Treatment of nonmetastatic osteosarcoma of the extremity with preoperative and postoperative chemotherapy: a report from the Children's Cancer Group.
        J Clin Oncol. 1997; 15: 76-84
        • Davis A.M.
        • Bell R.S.
        • Goodwin P.J.
        Prognostic factors in osteosarcoma: a critical review.
        J Clin Oncol. 1994; 12: 423-431
        • Glasser D.B.
        • Lane J.M.
        • Huvos A.G.
        • Marcove R.C.
        • Rosen G.
        Survival, prognosis, and therapeutic response in osteogenic sarcoma: the Memorial Hospital experience.
        Cancer. 1992; 69: 698-708
        • Huvos A.G.
        • Rosen G.
        • Marcove R.C.
        Primary osteogenic sarcoma: pathologic aspects in 20 patients after treatment with chemotherapy en bloc resection, and prosthetic bone replacement.
        Arch Pathol Lab Med. 1977; 101: 14-18
        • O'Kane G.M.
        • Cadoo K.A.
        • Walsh E.M.
        • Emerson R.
        • Dervan P.
        • O'Keane C.
        • Hurson B.
        • O'Toole G.
        • Dudeney S.
        • Kavanagh E.
        • Eustace S.
        • Carney D.N.
        Perioperative chemotherapy in the treatment of osteosarcoma: a 26-year single institution review.
        Clin Sarcoma Res. 2015; 5: 17
        • Raymond A.K.
        • Chawla S.P.
        • Carrasco C.H.
        • Ayala A.G.
        • Fanning C.V.
        • Grice B.
        • Armen T.
        • Plager C.
        • Papadopoulos N.E.
        • Edeiken J.
        Osteosarcoma chemotherapy effect: a prognostic factor.
        Semin Diagn Pathol. 1987; 4: 212-236
        • Rosen G.
        • Caparros B.
        • Huvos A.G.
        • Kosloff C.
        • Nirenberg A.
        • Cacavio A.
        • Marcove R.C.
        • Lane J.M.
        • Mehta B.
        • Urban C.
        Preoperative chemotherapy for osteogenic sarcoma: selection of postoperative adjuvant chemotherapy based on the response of the primary tumor to preoperative chemotherapy.
        Cancer. 1982; 49: 1221-1230
        • Kang J.-W.
        • Shin S.H.
        • Choi J.H.
        • Moon K.C.
        • Koh J.S.
        • kwon Jung C.
        • Park Y.-K.
        • Lee K.B.
        • Chung Y.-G.
        Inter-and intra-observer reliability in histologic evaluation of necrosis rate induced by neo-adjuvant chemotherapy for osteosarcoma.
        Int J Clin Exp Pathol. 2017; 10: 359-367
        • Srinidhi C.L.
        • Ciga O.
        • Martel A.L.
        Deep neural network models for computational histopathology: a survey.
        Med Image Anal. 2021; 67: 101813
        • Van der Laak J.
        • Litjens G.
        • Ciompi F.
        Deep learning in histopathology: the path to the clinic.
        Nat Med. 2021; 27: 775-784
        • Anisuzzaman D.
        • Barzekar H.
        • Tong L.
        • Luo J.
        • Yu Z.
        A deep learning study on osteosarcoma detection from histological images.
        Biomed Signal Process Control. 2021; 69: 102931
        • Arunachalam H.B.
        • Mishra R.
        • Daescu O.
        • Cederberg K.
        • Rakheja D.
        • Sengupta A.
        • Leonard D.
        • Hallac R.
        • Leavey P.
        Viable and necrotic tumor assessment from whole slide images of osteosarcoma using machine-learning and deep-learning models.
        PLoS One. 2019; 14: e0210706
        • Fu Y.
        • Xue P.
        • Ji H.
        • Cui W.
        • Dong E.
        Deep model with Siamese network for viable and necrotic tumor regions assessment in osteosarcoma.
        Med Phys. 2020; 47: 4895-4905
        • Mishra R.
        • Daescu O.
        • Leavey P.
        • Rakheja D.
        • Sengupta A.
        Histopathological diagnosis for viable and non-viable tumor prediction for osteosarcoma using convolutional neural network. International Symposium on Bioinformatics Research and Applications.
        Springer, 2017: 12-23
        • Mishra R.
        • Daescu O.
        • Leavey P.
        • Rakheja D.
        • Sengupta A.
        Convolutional neural network for histopathological analysis of osteosarcoma.
        J Comput Biol. 2018; 25: 313-325
        • Ho D.J.
        • Yarlagadda D.V.K.
        • D'Alfonso T.M.
        • Hanna M.G.
        • Grabenstetter A.
        • Ntiamoah P.
        • Brogi E.
        • Tan L.K.
        • Fuchs T.J.
        • Graphics
        Deep multi-magnification networks for multi-class breast cancer image segmentation.
        Comput Med Imaging Graph. 2021; 88: 101866
        • Ho D.J.
        • Agaram N.P.
        • Schüffler P.J.
        • Vanderbilt C.M.
        • Jean M.-H.
        • Hameed M.R.
        • Fuchs T.J.
        Deep interactive learning: an efficient labeling approach for deep learning-based osteosarcoma treatment response assessment. International Conference on Medical Image Computing and Computer-Assisted Intervention.
        Springer, 2020: 540-549
        • Otsu N.
        A threshold selection method from gray-level histograms.
        IEEE Trans Syst Man Cybern. 1979; 9: 62-66
        • Paszke A.
        • Gross S.
        • Massa F.
        • Lerer A.
        • Bradbury J.
        • Chanan G.
        • Killeen T.
        • Lin Z.
        • Gimelshein N.
        • Antiga L.
        Pytorch: an imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems.
        2019: 32
        • Schüffler P.J.
        • Geneslaw L.
        • Yarlagadda D.V.K.
        • Hanna M.G.
        • Samboy J.
        • Stamelos E.
        • Vanderbilt C.
        • Philip J.
        • Jean M.-H.
        • Corsale L.
        Integrated digital pathology at scale: a solution for clinical diagnostics and cancer research at a large academic medical center.
        J Am Med Inform Assoc. 2021; 28: 1874-1884
        • Wang C.
        • Durra H.Y.
        • Huang Y.
        • Manucha V.
        Interobserver reproducibility study of the histological patterns of primary lung adenocarcinoma with emphasis on a more complex glandular pattern distinct from the typical acinar pattern.
        Int J Surg Pathol. 2014; 22: 149-155
        • Warth A.
        • Stenzinger A.
        • von Brünneck A.-C.
        • Goeppert B.
        • Cortis J.
        • Petersen I.
        • Hoffmann H.
        • Schnabel P.A.
        • Weichert W.
        Interobserver variability in the application of the novel IASLC/ATS/ERS classification for pulmonary adenocarcinomas.
        Eur Respir J. 2012; 40: 1221-1227
        • Gomes D.S.
        • Porto S.S.
        • Balabram D.
        • Gobbi H.
        Inter-observer variability between general pathologists and a specialist in breast pathology in the diagnosis of lobular neoplasia, columnar cell lesions, atypical ductal hyperplasia and ductal carcinoma in situ of the breast.
        Diagn Pathol. 2014; 9: 1-9
        • Viray H.
        • Li K.
        • Long T.A.
        • Vasalos P.
        • Bridge J.A.
        • Jennings L.J.
        • Halling K.C.
        • Hameed M.
        • Rimm D.L.
        A prospective, multi-institutional diagnostic trial to determine pathologist accuracy in estimation of percentage of malignant cells.
        Arch Pathol Lab Med. 2013; 137: 1545-1549
        • Glasser D.B.
        Histologic Response to Pre-Operative Chemotherapy in Osteosarcoma: Appropriate Uses in Clinical Research.
        Columbia University, New York, NY1993
        • Li X.
        • Ashana A.O.
        • Moretti V.M.
        • Lackman R.D.
        The relation of tumour necrosis and survival in patients with osteosarcoma.
        Int Orthop. 2011; 35: 1847-1853
        • Graham S.
        • Vu Q.D.
        • Raza S.E.A.
        • Azam A.
        • Tsang Y.W.
        • Kwak J.T.
        • Rajpoot N.
        Hover-net: simultaneous segmentation and classification of nuclei in multi-tissue histology images.
        Med Image Anal. 2019; 58: 101563
        • Deshpande S.
        • Minhas F.
        • Graham S.
        • Rajpoot N.
        SAFRON: stitching across the Frontier Network for generating colorectal cancer histology images.
        Med Image Anal. 2022; 77: 102337
        • Janowczyk A.
        • Zuo R.
        • Gilmore H.
        • Feldman M.
        • Madabhushi A.
        HistoQC: an open-source quality control tool for digital pathology slides.
        JCO Clin Cancer Inform. 2019; 3: 1-7
        • Schömig-Markiefka B.
        • Pryalukhin A.
        • Hulla W.
        • Bychkov A.
        • Fukuoka J.
        • Madabhushi A.
        • Achter V.
        • Nieroda L.
        • Büttner R.
        • Quaas A.
        Quality control stress test for deep learning-based diagnostic model in digital pathology.
        Mod Pathol. 2021; 34: 2098-2108