Advertisement

A Deep Learning-Based Approach for Glomeruli Instance Segmentation from Multistained Renal Biopsy Pathologic Images

  • Lei Jiang
    Affiliations
    Electron Microscope Lab, Peking University People's Hospital, Beijing, China
    Search for articles by this author
  • Wenkai Chen
    Affiliations
    School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China
    Search for articles by this author
  • Bao Dong
    Affiliations
    Department of Nephrology, Peking University People's Hospital, Beijing, China
    Search for articles by this author
  • Ke Mei
    Affiliations
    School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China
    Search for articles by this author
  • Chuang Zhu
    Correspondence
    Address correspondence to Hongxia Shi, M.D., Electron Microscope Lab, Peking University People's Hospital, No. 11 Xizhimen S. St., Xicheng District, Beijing 100044, China; or Chuang Zhu, Ph.D., School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, No. 10, Xitucheng Rd., Haidian District, Beijing 100876, China.
    Affiliations
    School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China
    Search for articles by this author
  • Jun Liu
    Affiliations
    School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China
    Search for articles by this author
  • Meishun Cai
    Affiliations
    Department of Nephrology, Peking University People's Hospital, Beijing, China
    Search for articles by this author
  • Yu Yan
    Affiliations
    Department of Nephrology, Peking University People's Hospital, Beijing, China
    Search for articles by this author
  • Gongwei Wang
    Affiliations
    Department of Pathology, Peking University People's Hospital, Beijing, China
    Search for articles by this author
  • Li Zuo
    Affiliations
    Department of Nephrology, Peking University People's Hospital, Beijing, China
    Search for articles by this author
  • Hongxia Shi
    Correspondence
    Address correspondence to Hongxia Shi, M.D., Electron Microscope Lab, Peking University People's Hospital, No. 11 Xizhimen S. St., Xicheng District, Beijing 100044, China; or Chuang Zhu, Ph.D., School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, No. 10, Xitucheng Rd., Haidian District, Beijing 100876, China.
    Affiliations
    Electron Microscope Lab, Peking University People's Hospital, Beijing, China
    Search for articles by this author
      Glomeruli instance segmentation from pathologic images is a fundamental step in the automatic analysis of renal biopsies. Glomerular histologic manifestations vary widely among diseases and cases, and several special staining methods are necessary for pathologic diagnosis. A robust model is needed to segment and classify glomeruli with different staining methods and apply in cases with various glomerular pathologic changes. Herein, pathologic images from renal biopsy slides stained with three basic special staining methods were used to build the data sets. The snapshot group included 1970 glomeruli from 516 patients, and the whole-slide image group included 8665 glomeruli from 148 patients. Cascade Mask region-based convolutional neural net architecture was trained to detect, classify, and segment glomeruli into three categories: i) GN, structural normal; ii) global sclerosis; and iii) glomerular with other lesions. In the snapshot group, total glomeruli, GN, global sclerosis, and glomerular with other lesions achieved an F1 score of 0.914, 0.896, 0.681, and 0.756, respectively, which were comparable with those in the whole-slide image group (0.940, 0.839, 0.806, and 0.753, respectively). Among the three categories, GN achieved the best instance segmentation effect in both groups, as determined by average precision, average recall, F1 score, and Mask mean Intersection over Union. The present model segments and classifies multistained glomeruli with efficiency and robustness. It can be applied as the first step for more detailed glomerular histologic analysis.
      Histologic evaluation of glomeruli from renal biopsy tissues plays a key role in the diagnosis of various kidney diseases, and important in making treatment strategies and predicting prognosis.
      • Puelles V.G.
      • Bertram J.F.
      Counting glomeruli and podocytes: rationale and methodologies.
      ,
      • Lees G.E.
      • Cianciolo R.E.
      • Clubb F.J.
      Renal biopsy and pathologic evaluation of glomerular disease.
      Currently, histopathologic assessment of renal biopsies depends on manual judgment by pathologists. It is time-consuming and has unsatisfactory reproducibility, especially in situations involving grading or scoring the histologic features, in spite of efforts to improve the definitions, criteria, and classification methods.
      • Bajema I.M.
      • Wilhelmus S.
      • Alpers C.E.
      • Bruijn J.A.
      • Colvin R.B.
      • Cook H.T.
      • D'Agati V.D.
      • Ferrario F.
      • Haas M.
      • Jennette J.C.
      • Joh K.
      • Nast C.C.
      • Noel L.H.
      • Rijnink E.C.
      • Roberts I.S.D.
      • Seshan S.V.
      • Sethi S.
      • Fogo A.B.
      Revision of the International Society of Nephrology/Renal Pathology Society classification for lupus nephritis: clarification of definitions, and modified National Institutes of Health activity and chronicity indices.
      ,
      • Trimarchi H.
      • Barratt J.
      • Cattran D.C.
      • Cook H.T.
      • Coppo R.
      • Haas M.
      • Liu Z.H.
      • Roberts I.S.
      • Yuzawa Y.
      • Zhang H.
      • Feehally J.
      IgAN Classification Working Group of the International IgA Nephropathy Network and the Renal Pathology Society
      Oxford classification of IgA nephropathy 2016: an update from the IgA Nephropathy Classification Working Group.
      In recent years, deep-learning methods, especially convolutional neural nets (CNNs), have been applied often in medical image analysis and have achieved good results.
      • Gadermayr M.
      • Dombrowski A.-K.
      • Klinkhammer B.M.
      • Boor P.
      • Merhof D.
      CNN cascades for segmenting sparse objects in gigapixel whole slide images.
      • Esteva A.
      • Robicquet A.
      • Ramsundar B.
      • Kuleshov V.
      • DePristo M.
      • Chou K.
      • Cui C.
      • Corrado G.
      • Thrun S.
      • Dean J.
      A guide to deep learning in healthcare.
      • Wang S.
      • Yang D.M.
      • Rong R.
      • Zhan X.
      • Xiao G.
      Pathology image analysis using segmentation deep learning algorithms.
      • Komura D.
      • Ishikawa S.
      Machine learning approaches for pathologic diagnosis.
      Renal biopsy pathology, as an important branch of pathology, has unique complexity in image analysis and urgently needs this new technology to improve the efficiency and accuracy of diagnosis.
      Glomeruli extraction and recognition from pathologic images is a fundamental step in the automatic analysis of renal biopsies. Most of the recent work can generally be classified into two categories, object detection
      • Maree R.
      • Dallongeville S.
      • Olivo-Marin J.C.
      • Meas-Yedid V.
      An approach for detection of glomeruli in multisite digital pathology. IEEE International Symposium on Biomedical Imaging.
      • Temerinac-Ott M.
      • Forestier G.
      • Schmitz J.
      • Hermsen M.
      • Brasen J.H.
      • Feuerhake F.
      • Wemmert C.
      Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.
      • Simon O.
      • Yacoub R.
      • Jain S.
      • Tomaszewski J.E.
      • Sarder P.
      Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images.
      • Bukowy J.D.
      • Dayton A.
      • Cloutier D.
      • Manis A.D.
      • Staruschenko A.
      • Lombard J.H.
      • Solberg Woods L.C.
      • Beard D.A.
      • Cowley Jr., A.W.
      Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.
      and semantic segmentation
      • Kato T.
      • Relator R.
      • Ngouv H.
      • Hirohashi Y.
      • Takaki O.
      • Kakimoto T.
      • Okada K.
      • Segmental H.O.G.
      new descriptor for glomerulus detection in kidney microscopy image.
      • Sarder P.
      • Ginley B.
      • Tomaszewski J.E.
      Automated renal histopathology: digital extraction and quantification of renal pathology. SPIE Medical Imaging.
      • Sheehan S.
      • Mawe S.
      • Cianciolo R.E.
      • Korstanje R.
      • Mahoney J.M.
      Detection and classification of novel renal histologic phenotypes using deep neural networks.
      • Ginley B.
      • Lutnick B.
      • Jen K.-Y.
      • Fogo A.B.
      • Jain S.
      • Rosenberg A.
      • Walavalkar V.
      • Wilding G.
      • Tomaszewski J.E.
      • Yacoub R.
      • Rossi G.M.
      • Sarder P.
      Computational segmentation and classification of diabetic glomerulosclerosis.
      • Hermsen M.
      • de Bel T.
      • den Boer M.
      • Steenbergen E.J.
      • Kers J.
      • Florquin S.
      • Roelofs J.J.T.H.
      • Stegall M.D.
      • Alexander M.P.
      • Smith B.H.
      • Smeets B.
      • Hilbrands L.B.
      • van der Laak J.A.W.M.
      Deep learning–based histopathologic assessment of kidney tissue.
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      • Zeng C.
      • Nan Y.
      • Xu F.
      • Lei Q.
      • Li F.
      • Chen T.
      • Liang S.
      • Hou X.
      • Lv B.
      • Liang D.
      • Luo W.
      • Lv C.
      • Li X.
      • Xie G.
      • Liu Z.
      Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.
      (Table 1). For object detection, Temerinac-Ott et al
      • Temerinac-Ott M.
      • Forestier G.
      • Schmitz J.
      • Hermsen M.
      • Brasen J.H.
      • Feuerhake F.
      • Wemmert C.
      Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.
      used CNN to get classification results from differently stained sections. They used it to improve glomeruli detection on one staining and achieved 10% to 20% higher F1 scores than those with the HOG detector. Bukowy et al
      • Bukowy J.D.
      • Dayton A.
      • Cloutier D.
      • Manis A.D.
      • Staruschenko A.
      • Lombard J.H.
      • Solberg Woods L.C.
      • Beard D.A.
      • Cowley Jr., A.W.
      Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.
      used region-based convolutional neural net (R-CNN) and CNN for final classification as glomerulus or background objects, and achieved an average precision and recall of 96.94% and 96.79%, respectively. With the development of deep-learning methods, DeepLab-V2,
      • Hermsen M.
      • de Bel T.
      • den Boer M.
      • Steenbergen E.J.
      • Kers J.
      • Florquin S.
      • Roelofs J.J.T.H.
      • Stegall M.D.
      • Alexander M.P.
      • Smith B.H.
      • Smeets B.
      • Hilbrands L.B.
      • van der Laak J.A.W.M.
      Deep learning–based histopathologic assessment of kidney tissue.
      SegNet,
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      and U-Net
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      ,
      • Zeng C.
      • Nan Y.
      • Xu F.
      • Lei Q.
      • Li F.
      • Chen T.
      • Liang S.
      • Hou X.
      • Lv B.
      • Liang D.
      • Luo W.
      • Lv C.
      • Li X.
      • Xie G.
      • Liu Z.
      Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.
      have been used to deal with segmentation tasks in semantic segmentation. Object detection tasks can detect and classify glomeruli, but the boundary of glomeruli is unknown. Although semantic segmentation can determine the boundary, the glomeruli number cannot be calculated and different glomeruli cannot be distinguished in the same image. Therefore, to overcome the above limitations, the current method attempts to achieve the goal of instance segmentation. The algorithm of the Cascade Mask R-CNN
      • Cai Z.
      • Vasconcelos N.
      • Cascade R.-C.N.N.
      High Quality Object Detection and Instance Segmentation.
      model was improved to better fit the data set so that sufficient image information can be acquired for further research.
      Table 1Previous Research on Glomeruli Detection and Segmentation
      SourceData/slideApproachPerformanceSource of renal tissue imagesStaining method
      Detection
       Maree et al
      • Maree R.
      • Dallongeville S.
      • Olivo-Marin J.C.
      • Meas-Yedid V.
      An approach for detection of glomeruli in multisite digital pathology. IEEE International Symposium on Biomedical Imaging.
      Total: 200Ellipsoidal shape + decision tree87% F1Human (renal biopsies)Masson trichrome
       Temerinac-Ott et al
      • Temerinac-Ott M.
      • Forestier G.
      • Schmitz J.
      • Hermsen M.
      • Brasen J.H.
      • Feuerhake F.
      • Wemmert C.
      Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.
      Train: 16

      Test: 4
      Mutual information + CNN66.38%–81.75% F11. Human (nephrectomy specimens)Jones H&E, PAS, Sirius Red, and immunostained for CD10
      2. Human (needle biopsies from transplanted kidneys)Jones H&E, PAS, and Sirius Red
       Simon et al
      • Simon O.
      • Yacoub R.
      • Jain S.
      • Tomaszewski J.E.
      • Sarder P.
      Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images.
      Train: 25

      Test: 9
      Multiradial LBP + SVM90.4% Precision,

      76.7% recall
      1. A standard streptozocin mouse modelH&E
      2. RatsH&E, PAS, Jones silver, Gömöri trichrome, and Congo Red
      3. Human (DN patients with chronic kidney disease stage II and stage III; control: normal kidneys)PAS
       Bukowy et al
      • Bukowy J.D.
      • Dayton A.
      • Cloutier D.
      • Manis A.D.
      • Staruschenko A.
      • Lombard J.H.
      • Solberg Woods L.C.
      • Beard D.A.
      • Cowley Jr., A.W.
      Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.
      Train: 72

      Test: 13
      Faster R-CNN96.9% Precision,

      96.8% recall
      Rats of various genetic backgroundsMasson trichrome
      Segmentation
       Kato et al
      • Kato T.
      • Relator R.
      • Ngouv H.
      • Hirohashi Y.
      • Takaki O.
      • Kakimoto T.
      • Okada K.
      • Segmental H.O.G.
      new descriptor for glomerulus detection in kidney microscopy image.
      Train:
      Not mentioned.


      Test: 20
      S-HOG + SVM86.6% F1SD and SDT ratsImmunostained for desmin
       Sarder et al
      • Sarder P.
      • Ginley B.
      • Tomaszewski J.E.
      Automated renal histopathology: digital extraction and quantification of renal pathology. SPIE Medical Imaging.
      Train:
      Not mentioned.


      Test: 15
      Gabor filters87.8% AccuracyNormal healthy untreated rats and micePAS and H&E
       Sheehan et al
      • Sheehan S.
      • Mawe S.
      • Cianciolo R.E.
      • Korstanje R.
      • Mahoney J.M.
      Detection and classification of novel renal histologic phenotypes using deep neural networks.
      Total: 90AlexNet + SVM92% Recall,

      90% specificity
      Mice with different genotypesPAS
       Ginley et al
      • Ginley B.
      • Lutnick B.
      • Jen K.-Y.
      • Fogo A.B.
      • Jain S.
      • Rosenberg A.
      • Walavalkar V.
      • Wilding G.
      • Tomaszewski J.E.
      • Yacoub R.
      • Rossi G.M.
      • Sarder P.
      Computational segmentation and classification of diabetic glomerulosclerosis.
      Train: 41

      Test: 13
      DeepLab version 293% AccuracyHuman (patients with DN)PAS
       Hermsen et al
      • Hermsen M.
      • de Bel T.
      • den Boer M.
      • Steenbergen E.J.
      • Kers J.
      • Florquin S.
      • Roelofs J.J.T.H.
      • Stegall M.D.
      • Alexander M.P.
      • Smith B.H.
      • Smeets B.
      • Hilbrands L.B.
      • van der Laak J.A.W.M.
      Deep learning–based histopathologic assessment of kidney tissue.
      Train: 37

      Test: 3
      5 U-Nets ensemble79% Dice1. Human (transplant biopsies)PAS
      2. Human (nephrectomy specimens)
       Bueno et al
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      Train: 38

      Test: 9
      SegNet-VGG1999.8% Precision,

      99.2% F1
      Human (AIDPATH kidney database)PAS
       Zeng et al
      • Zeng C.
      • Nan Y.
      • Xu F.
      • Lei Q.
      • Li F.
      • Chen T.
      • Liang S.
      • Hou X.
      • Lv B.
      • Liang D.
      • Luo W.
      • Lv C.
      • Li X.
      • Xie G.
      • Liu Z.
      Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.
      Test: 22

      Train: 360
      U-Net-SSIM + marked watershed93% Specificity,

      93.1% precision,

      94.0% F1,

      94.9% recall,

      90.1% Dice
      Human (patients with IgA nephropathy)PAS
      AIDPATH, Academia and Industry Collaboration for Digital Pathology; CNN, convolutional neural net; Dice, dice coefficient; DN, diabetes nephropathy; F1, F1 score; H&E, hematoxylin-eosin; LBP, local binary patterns; PAS, periodic acid–Schiff; R-CNN, region-based CNN; SD, sprague-dawley; SDT, spontaneously diabetic torii; SVM, support vector machine; S-HOG, segmental histogram of oriented gradient.
      Not mentioned.
      Glomerular lesions can involve any part of a glomerulus, including endothelial cells, mesangial cells, podocytes, and mesangial matrix.
      • Sethi S.
      • Haas M.
      • Markowitz G.S.
      • D'Agati V.D.
      • Rennke H.G.
      • Jennette J.C.
      • Bajema I.M.
      • Alpers C.E.
      • Chang A.
      • Cornell L.D.
      • Cosio F.G.
      • Fogo A.B.
      • Glassock R.J.
      • Hariharan S.
      • Kambham N.
      • Lager D.J.
      • Leung N.
      • Mengel M.
      • Nath K.A.
      • Roberts I.S.
      • Rovin B.H.
      • Seshan S.V.
      • Smith R.J.H.
      • Walker P.D.
      • Winearls C.G.
      • Appel G.B.
      • Alexander M.P.
      • Cattran D.C.
      • Casado C.A.
      • Cook H.T.
      • De Vriese A.S.
      • Radhakrishnan J.
      • Racusen L.C.
      • Ronco P.
      • Fervenza F.C.
      Mayo Clinic/Renal Pathology Society consensus report on pathologic classification, diagnosis, and reporting of GN.
      In addition, the severity of lesions varies widely. Hence, the appearance of different glomeruli varies greatly. An ideal model should be able to identify and segment all types of glomeruli. Most of the existing models analyzing human kidney tissues for glomeruli detection and segmentation are based on a single disease, such as diabetic nephropathy,
      • Ginley B.
      • Lutnick B.
      • Jen K.-Y.
      • Fogo A.B.
      • Jain S.
      • Rosenberg A.
      • Walavalkar V.
      • Wilding G.
      • Tomaszewski J.E.
      • Yacoub R.
      • Rossi G.M.
      • Sarder P.
      Computational segmentation and classification of diabetic glomerulosclerosis.
      transplant biopsies,
      • Hermsen M.
      • de Bel T.
      • den Boer M.
      • Steenbergen E.J.
      • Kers J.
      • Florquin S.
      • Roelofs J.J.T.H.
      • Stegall M.D.
      • Alexander M.P.
      • Smith B.H.
      • Smeets B.
      • Hilbrands L.B.
      • van der Laak J.A.W.M.
      Deep learning–based histopathologic assessment of kidney tissue.
      and IgA nephropathy,
      • Zeng C.
      • Nan Y.
      • Xu F.
      • Lei Q.
      • Li F.
      • Chen T.
      • Liang S.
      • Hou X.
      • Lv B.
      • Liang D.
      • Luo W.
      • Lv C.
      • Li X.
      • Xie G.
      • Liu Z.
      Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.
      except in case of Bueno et al,
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      who used 47 whole-slide images (WSIs) from AIDPATH kidney database.
      In addition to hematoxylin-eosin staining, for light microscopy, a variety of histochemical stains, including periodic acid–Schiff (PAS), periodic acid–silver methenamine (PASM), and Masson trichrome stains are used to evaluate renal tissues. Each staining method has its unique value in elucidating specific histologic features and is essential for renal pathologic analysis. Both the variabilities of glomerular histologic manifestations and of staining methods increase the complexity of glomeruli detection and segmentation in pathologic images. Among the recent research analyzing human kidney tissues for glomeruli detection and segmentation, all results were obtained from PAS-stained sections.
      • Ginley B.
      • Lutnick B.
      • Jen K.-Y.
      • Fogo A.B.
      • Jain S.
      • Rosenberg A.
      • Walavalkar V.
      • Wilding G.
      • Tomaszewski J.E.
      • Yacoub R.
      • Rossi G.M.
      • Sarder P.
      Computational segmentation and classification of diabetic glomerulosclerosis.
      • Hermsen M.
      • de Bel T.
      • den Boer M.
      • Steenbergen E.J.
      • Kers J.
      • Florquin S.
      • Roelofs J.J.T.H.
      • Stegall M.D.
      • Alexander M.P.
      • Smith B.H.
      • Smeets B.
      • Hilbrands L.B.
      • van der Laak J.A.W.M.
      Deep learning–based histopathologic assessment of kidney tissue.
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      • Zeng C.
      • Nan Y.
      • Xu F.
      • Lei Q.
      • Li F.
      • Chen T.
      • Liang S.
      • Hou X.
      • Lv B.
      • Liang D.
      • Luo W.
      • Lv C.
      • Li X.
      • Xie G.
      • Liu Z.
      Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.
      and there is no study that can segment the glomeruli from the images with multiple staining methods using a single model.
      In this research, an algorithmic model with the following features was constructed: i) it can recognize and segment glomeruli with various histologic appearances from different diseases, ii) it has good robustness so that it can be applied to different special stained slides, and iii) it can make a preliminary classification of the glomeruli. For the above purposes, we present a deep-learning–based approach for the object extraction and three-classification of glomeruli.

      Materials and Methods

      Tissue Samples

      Kidney biopsy samples from patients diagnosed with various kidney diseases from January 2018 to June 2019 in Department of Nephrology of Peking University People's Hospital (Beijing, China) were collected. All biopsy samples were processed according to standard techniques for light microscopy, immunofluorescence, and electron microscopy. For light microscopy, formalin-fixed, paraffin-embedded tissues were cut into sections (2 μm thick) and stained with one of the following techniques: hematoxylin-eosin, PAS, PASM, or Masson. The images from sections stained with PAS, PASM, and Masson were selected.

      Imaging

      The digital images used in this study were obtained from the two sources listed below.

      Snapshot Images from Image Library

      A total of 2 to 10 digital images were routinely captured in typical lesion areas from each case using Leica DFC550 digital camera (Singapore) attached to a Leica DM4000 B LED optical microscope (Weizla, Germany) and saved in the image library of our department. The average size of each image was approximately 1360 × 1024 × 3 pixels, corresponding to a field of 2.176 × 1.632 mm2, which resulted in a length scale of 0.85 mm/pixel.
      Images from patients diagnosed with various kidney diseases from January 2018 to April 2019, with high technical staining quality and containing at least one glomerular area, were selected.

      Whole-Slide Images

      Slides were scanned at 40× using Precie 500B scanner (UNIC Technologies Inc., Beijing, China) for WSIs, with a 40× objective and numerical aperture = 0.8; the resolution was 0.12 μm/pixel for all acquired images.
      Images from patients diagnosed with various kidney diseases from May 2019 to June 2019, with high technical staining quality, were selected.

      Glomerular Data Set Generation

      Snapshot Images from Image Library

      In total, 1123 images from 516 patients, covering >30 pathologic types of kidney diseases, were collected (Tables 2 and 3). The diagnoses of the patients selected in this group are shown in Table 4. The total number of glomeruli was 1970 (PASM, 779; PAS, 492; and Masson, 699) (Table 5). The amplification multiples were 10× (65 glomeruli), 20× (1482 glomeruli), and 40× (423 glomeruli).
      Table 2The Number and Distribution of Images and Patients in Two Groups of Data Set
      Data setSnapshotsWSIs
      Training set1018/465 (2.19)296/118 (2.51)
      Test set105/51 (2.06)52/30 (1.73)
      Total1123/516 (2.18)348/148 (2.35)
      Data are provided as number of images/number of patients (images/patient).
      WSI, whole-slide image.
      Table 3The Number and Distribution of WSIs and Snapshot Images With Each Stain
      VariablePASMPASMassonAll
      Snapshots
       Training set3822793541018
       Test set392244105
      WSIs
       Training set1079396296
       Test set17171852
      Masson, Masson trichrome stain; PAS, periodic acid–Schiff; PASM, periodic acid–silver methenamine; WSI, whole-slide image.
      Table 4Diagnoses of the Patients in the Snapshots Group
      DiagnosisPatients, NPatients, %
      IgA nephropathy14528.10
      Atypical membranous nephropathy12123.45
      Diabetic nephropathy479.11
      Lupus nephritis265.04
      Membranous nephropathy224.26
      Focal and segmental glomerulosclerosis183.49
      Henoch-Schonlein purpura nephritis163.10
      Amyloidosis132.52
      ANCA-associated small-vessel glomerulonephritis122.33
      Minimal change disease91.74
      Hepatitis B–associated glomerulonephritis71.36
      IgA nephropathy + diabetic nephropathy
      Compound diseases.
      71.36
      Subacute tubulointerstitial nephritis81.55
      Crescentic glomerulonephritis61.16
      Obesity-related glomerulopathy40.78
      Mild mesangial proliferative glomerulopathy40.78
      Ischemic renal injury40.78
      Acute tubular injury40.78
      Acute tubular necrosis40.78
      Malignant hypertensive renal injury30.58
      Focal proliferative glomerulonephritis30.58
      Atypical membranous nephropathy + diabetic nephropathy
      Compound diseases.
      30.58
      IgA nephropathy + malignant hypertensive renal injury
      Compound diseases.
      30.58
      IgA nephropathy + membranous nephropathy
      Compound diseases.
      30.58
      Membranoproliferative glomerulonephritis30.58
      Interstitial infiltration of leukemia cells20.39
      Cast nephropathy20.39
      Minimal change disease + ischemic renal injury
      Compound diseases.
      20.39
      Endocapillary proliferative glomerulonephritis30.58
      Amyloidosis + membranous nephropathy
      Compound diseases.
      10.19
      Light chain proximal tubulopathy10.19
      Renocortical necrosis10.19
      Lipoprotein glomerulopathy10.19
      Chronic tubulointersitial nephritis10.19
      Fabry disease10.19
      IgA nephropathy + ANCA-associated small-vessel glomerulonephritis
      Compound diseases.
      10.19
      IgA nephropathy + minimal change disease
      Compound diseases.
      10.19
      Lupus nephritis + thrombotic microangiopathy
      Compound diseases.
      10.19
      Membranous nephropathy + acute tubular injury
      Compound diseases.
      10.19
      Thin basement membrane nephropathy10.19
      Thrombotic microangiopathy10.19
      Total516100.00
      ANCA, anti-neutrophil cytoplasmic antibody.
      Compound diseases.
      Table 5The Number and Distribution of Glomeruli in Two Groups of Data Set
      VariableGNGLGSAll
      Snapshots
       Original training set8657212171803
       Training set after data enhancement51904326130210,818
       Test set816521167
      WSIs
       Original training set3848225810877193
       Training set after data enhancement23,08812,548651643,152
       Test set8415131181472
      GL, glomerulus with abnormal structure; GN, glomerulus with basically normal structure; GS, global sclerosis; WSI, whole-slide image.

      Whole-Slide Images

      A total of 348 WSIs were obtained from 148 patients with >20 pathologic types of kidney diseases (Tables 2 and 3). The total number of glomeruli was 8665 (PASM, 3248; PAS, 2525; and Masson, 2892) (Table 5). The diagnoses of the patients selected in this group are shown in Table 6.
      Table 6Diagnoses of the Patients in the WSI Group
      DiagnosisPatients, NPatients, %
      IgA nephropathy7550.68
      Atypical membranous nephropathy2214.86
      Lupus nephritis74.73
      Diabetic nephropathy64.05
      Hepatitis B–associated glomerulonephritis53.38
      Minimal change disease53.38
      Mild mesangial proliferative glomerulopathy42.70
      Focal and segmental glomerulosclerosis42.70
      Membranous nephropathy42.70
      IgA nephropathy + diabetic nephropathy
      Compound diseases.
      32.03
      Endocapillary proliferative glomerulonephritis21.35
      Thrombotic microangiopathy21.35
      Henoch-Schonlein purpura nephritis21.35
      Membranoproliferative glomerulonephritis10.68
      Amyloidosis10.68
      ANCA-associated small-vessel glomerulonephritis10.68
      Focal and segmental glomerulosclerosis + acute tubular injury
      Compound diseases.
      10.68
      IgA nephropathy + membranous nephropathy
      Compound diseases.
      10.68
      IgA nephropathy + obesity-related glomerulopathy10.68
      Subacute tubulointerstitial nephritis10.68
      Total148100.00
      ANCA, anti-neutrophil cytoplasmic antibody; WSI, whole-slide image.
      Compound diseases.
      Because of the high resolution of WSIs and the limited computing resources, the entire WSIs cannot be fed into the model as input. So, the original WSIs were cropped into 2048 × 2048 pixel patches. In the cropping process, some glomeruli might be divided into two patches (incomplete cut). To avoid waste of data resources and to better use the data for getting a better model, the stride of the cutting window was three-fourths of the window's length. These images were then processed by script and checked manually to ensure that all irrelevant areas had been filtered out. The small patches were sent to the model, and the results were mapped back to WSIs, according to the cropped coordinates.

      Labeling Glomeruli

      Each image was labeled into one of the three categories:
      • i)
        GN: The glomeruli with completely open capillary tufts were considered structural normal (GN). This category included normal glomeruli, glomeruli with mild lesions (such as mild mesangial hypercellularity or mild mesangial matrix expansion), and glomeruli with thickened glomerular basement membrane without other lesions.
      • ii)
        GS: The glomeruli with complete obliteration of the entire glomerular tuft and with loss of their normal structure were considered as global sclerosis (GS).
      • iii)
        GL: The glomeruli with lesions other than GS and leading to loss of any part of their structure were considered as glomerular with other lesions (GL). This category included segmental sclerosis, moderate to severe mesangial hypercellularity or expansion, crescents, and apparent endothelial proliferation.
      Each lesion was defined according to the multicenter Nephrotic Syndrome Study Network digital pathology scoring system.
      • Barisoni L.
      • Troost J.P.
      • Nast C.
      • Bagnasco S.
      • Avila-Casado C.
      • Hodgin J.
      • Palmer M.
      • Rosenberg A.
      • Gasim A.
      • Liensziewski C.
      • Merlino L.
      • Chien H.P.
      • Chang A.
      • Meehan S.M.
      • Gaut J.
      • Song P.
      • Holzman L.
      • Gibson D.
      • Kretzler M.
      • Gillespie B.W.
      • Hewitt S.M.
      Reproducibility of the NEPTUNE descriptor-based scoring system on whole-slide images and histologic and ultrastructural digital images.
      Moderate to severe mesangial hypercellularity was defined as six or more mesangial cells/mesangial area.
      • Roberts I.S.
      • Cook H.T.
      • Troyanov S.
      • Alpers C.E.
      • Amore A.
      • Barratt J.
      • Berthoux F.
      • Bonsib S.
      • Bruijn J.A.
      • Cattran D.C.
      • Coppo R.
      • D'Agati V.
      • D'Amico G.
      • Emancipator S.
      • Emma F.
      • Feehally J.
      • Ferrario F.
      • Fervenza F.C.
      • Florquin S.
      • Fogo A.
      • Geddes C.C.
      • Groene H.J.
      • Haas M.
      • Herzenberg A.M.
      • Hill P.A.
      • Hogg R.J.
      • Hsu S.I.
      • Jennette J.C.
      • Joh K.
      • Julian B.A.
      • Kawamura T.
      • Lai F.M.
      • Li L.S.
      • et al.
      Working Group of the International Ig A Nephropathy Network and the Renal Pathology Society
      The Oxford classification of IgA nephropathy: pathology definitions, correlations, and reproducibility.
      Each image was independently reviewed and labeled by two nephropathologists (L.J. and G.W.). If there was disagreement, a third nephropathologist (expert) (B.D.) reviewed the image and discussed the labeling with the other two pathologists, to reach the final agreement.

      Data Set and Data Augmentation

      Snapshot Images from Image Library

      The images were randomly split into two parts: 1803 glomeruli in the training set and 167 glomeruli in the test set. For the training set, data augmentation was used to increase the amount of training data and diversify the data distribution, so as to improve the robustness of the model. The specific transformation included random rotate 90 degrees, horizontal flip, vertical flip, transpose, and random crop. Thus, the final training set included a total of 10,818 glomeruli (Table 5).
      Table 7Detection and Segmentation Performance Metrics of Snapshot Images
      VariableAPARF1 scoreMask mIoU
      PASM
       GN1.0000.9140.9550.833
       GL0.8800.8800.8800.812
       GS0.8000.6670.7270.692
       Detection of total glomeruli
      Regardless of the classification.
      0.9830.9240.9530.946
      PAS
       GN1.0001.0001.0000.943
       GL0.7500.6000.6670.692
       GS0.6150.8890.7270.692
       Detection of total glomeruli
      Regardless of the classification.
      0.8430.9000.8710.886
      Masson
       GN0.7690.8570.8110.707
       GL0.6900.6670.6780.647
       GS0.4440.8000.5710.459
       Detection of total glomeruli
      Regardless of the classification.
      0.8570.9430.8980.907
      All
       GN0.8900.9010.8960.790
       GL0.7740.7380.7560.708
       GS0.5930.8000.6810.638
       Detection of total glomeruli
      Regardless of the classification.
      0.9010.9280.9140.914
      AP, average precision; AR, average recall; GL, glomerulus with abnormal structure; GN, glomerulus with basically normal structure; GS, global sclerosis; Mask mIoU, mean intersection/union ratio of the segmentation results; Masson, Masson trichrome stain; PAS, periodic acid–Schiff; PASM, periodic acid–silver methenamine.
      Regardless of the classification.

      Whole-Slide Images

      Similar to the snapshot images, cropped WSIs were randomly split into two parts: 7193 in the training set and 1472 in the test set. After data augmentation, the final training set included a total of 43,152 glomeruli (Table 5).

      Model Training for Glomerular Instance Segmentation

      The architectures of the model are shown in Figure 1. Cascade Mask R-CNN architecture, based on CNNs, was trained by using the glomeruli training set. The adopted Cascade Mask R-CNN, which is built by extending Cascade R-CNN
      • Cai Z.
      • Vasconcelos N.
      • Cascade R.-C.N.N.
      High Quality Object Detection and Instance Segmentation.
      by adding a branch for predicting an object mask in every stage, can detect, classify, and segment three types of glomeruli. ResNet-50 was used as the backbone of the model.
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition. IEEE Conference on Computer Vision & Pattern Recognition.
      B0 is the region proposal network,
      • Ren S.
      • He K.
      • Girshick R.
      • Sun J.
      • Faster R.-C.N.N.
      Towards real-time object detection with region proposal networks.
      and other component details in the architectures are the same as in the Mask R-CNN.
      • He K.
      • Gkioxari G.
      • Dollár P.
      • Girshick R.
      Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV).
      Among them, the backbone consists of many convolution blocks that have the ability to extract image features; convolution blocks usually share some basic components, such as convolution layer and pooling layer. The convolutional layer is the fundamental component that is composed of various learnable kernels (filters). A pooling layer (usually applied after a convolutional layer) is used to compute feature maps condensed in a smaller representation with the goal of achieving some invariance. Region proposal network comprises two sibling fully connected layers—a box-regression layer and a box-classification layer follow convolution blocks. It can propose regions for use in subsequent model modules.
      Figure thumbnail gr1
      Figure 1Architectures of the Cascade Mask region-based convolutional neural net. Bounding boxes of the glomeruli represent the detection results, which come after bounding box branch (B) 3, and the segmentation results come after segmentation branch (S). B0, proposal in architecture; C, classification branch; Conv, backbone convolutions; GL, glomerulus with abnormal structure; GN, glomerulus with normal structure; GS, global sclerosis; H, network head; Pool, region-wise feature extraction.
      During the training procedure, part of the structure was modified to fit the data set and the hyperparameter was continuously changed after analyzing the results to get the model with the best performance. Specifically, the final classification layer was replaced from the network and retrained with the data set using the three output labels (GN, GL, and GS).
      To adapt the model to different stainings, images with three different stains were combined to guide the model learning in the training stage, and the images were normalized to alleviate the domain differences between differently stained images. In the training process, some regular terms were added into the loss function, so that the model could avoid overfitting and had stronger robustness. Before being sent to the network for training, the images with different digital amplification were normalized to be under the same data distribution. The training process and the testing process used the same approach of normalization, so the model could be adapted to different resolutions.
      The optimizer of choice for training was Stochastic Gradient Descent, with the learning rate of 0.005, the momentum of 0.9, weight decay of 0.0001, and epoch of 12. The learning rate dropped on epoch 8 and 11. Smooth L1 was used as the loss function for the result of boundary box detection, and cross entropy was used as the loss function for classification and segmentation. The implementation is based on MMDetection,
      • Chen K.
      • Wang J.
      • Pang J.
      • Cao Y.
      • Xiong Y.
      • Li X.
      • Sun S.
      • Feng W.
      • Liu Z.
      • Xu J.
      MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.
      and code is available (https://github.com/bupt-ai-cz/glomeruli-instance-segmentation, last accessed April 1, 2021).

      Performance Metrics

      True positives (TPs) were defined as intersection over union (IoU) value >0.75, and the results with IoU value <0.75 were all judged to have failed detection.
      Four metrics were used to evaluate the proposed method: average precision (AP) as the ratio of correctly predicting glomeruli and the sum of predicted TP and false-positive (FP) observations (whereby high precision is regarded to low FP rate); average recall (AR) as the ratio of correctly predicting glomeruli and the sum of predicted TP and false-negative (FN) observations (whereby high recall is regarded to low FN rate); and F1 score as the weighted average of precision and recall (whereby high F1 score is regarded to high precision and recall rates). Mask mean Intersection over Union (mIoU) was used as the mean IoU to evaluate the segmentation results; the areas of intersection and union were calculated by the segmentation part from ground truth (GT) and model prediction results (P). AP, AR, and F1 score are detection metrics, and Mask mIoU is a segmentation metric.
      The precision, recall, F1 score, and Mask IoU can be calculated as follows:
      Precision=TPTP+FP
      (1)


      Recall=TPTP+FN
      (2)


      F1score=2∗Precision∗RecallPrecision+Recall
      (3)


      MaskIoU=PGTPGT
      (4)


      Ethical Considerations

      This study was approved by the ethics committee of Peking University People's Hospital (2019PHB176-01).

      Results

      Glomerular Instance Segmentation Using Snapshot Images

      Examples of the CNN pipeline applied to snap images are shown in Figure 2. Among the three categories, glomeruli with normal structure achieved the best instance segmentation effect (0.890, 0.901, 0.896, and 0.790 for AP, AR, F1 score, and Mask mIoU, respectively), followed by glomeruli with abnormal structure (0.774, 0.738, 0.756, and 0.708 for AP, AR, F1 score, and Mask mIoU, respectively). Global sclerosis got the lowest scores (0.593, 0.800, 0.681, and 0.638 for AP, AR, F1 score, and Mask mIoU, respectively). AP, AR, F1 score, and Mask mIoU for total glomeruli ignoring the classification were also calculated, which were 0.901, 0.928, 0.914, and 0.914, respectively. The stain-wise performance was calculated, and the details are shown in Table 7. The orders of F1 scores were as follows: GN (PAS > PASM > Masson), GL (PASM > Masson > PAS), GS (PASM = PAS > Masson), and total (PASM > Masson > PAS).
      Figure thumbnail gr2
      Figure 2Segmentation and classification results of glomeruli in snapshot images. A, C, and E: Annotation by pathologists. B, D, and F: Artificial intelligence segmentation results. AF: The top (A and B), middle (C and D), and bottom (E and F) panels were periodic acid–silver methenamine, periodic acid–Schiff, and Masson trichrome staining, respectively. GL, glomerulus with other abnormal structure; GN, glomerulus with normal structure; GS, global sclerosis.

      Glomerular Instance Segmentation Using WSIs

      To test the model's result, the WSI was first cropped into small patches, and then sent to the model, which finally mapped the results back to the WSI, according to the cropped coordinates. Mapping relation is shown in Figure 3.
      Figure thumbnail gr3
      Figure 3Example of glomerulus instance segmentation in whole-slide images. GL, glomerulus with other abnormal structure; GN, glomerulus with normal structure.
      Examples of the CNN pipeline applied to WSIs are shown in Figure 4. Among the three categories, glomerulus with basically normal structure achieved the best instance segmentation effect (0.802, 0.880, 0.839, and 0.702 for AP, AR, F1 score, and Mask mIoU, respectively), followed by global sclerosis (0.757, 0.862, 0.806, and 0.623 for AP, AR, F1 score, and Mask mIoU, respectively). The glomeruli with abnormal structure got the lowest scores (0.866, 0.666, 0.753, and 0.602 for AP, AR, F1 score, and Mask mIoU, respectively). AP, AR, F1 score, and Mask mIoU for total glomeruli ignoring the classification were also calculated, which were 0.957, 0.924, 0.940, and 0.873, respectively. The stain-wise performance was calculated, and the details are shown in Table 8. The orders of F1 scores were as follows: GN (PASM > PAS > Masson), GL (PASM > Masson > PAS), GS (Masson > PASM > PAS), and total (PASM > PAS > Masson).
      Figure thumbnail gr4
      Figure 4Segmentation and classification results of glomeruli in whole-slide images. A, C, and E: Annotation by pathologists. B, D, and F: Artificial intelligence segmentation results. AF: The top (A and B), middle (C and D), and bottom (E and F) panels were periodic acid–silver methenamine, periodic acid–Schiff, and Masson trichrome staining, respectively. GL, glomerulus with other abnormal structure; GN, glomerulus with normal structure; GS, global sclerosis.
      Table 8Detection and Segmentation Performance Metrics of WSIs
      VariableAPARF1 scoreMask mIoU
      PASM
       GN0.8470.9080.8760.742
       GL0.8500.7650.8050.660
       GS0.7610.8360.7970.621
       Detection of total glomeruli
      Regardless of the classification.
      0.9470.9610.9540.881
      PAS
       GN0.8420.8630.8520.697
       GL0.8230.6140.7030.544
       GS0.7330.8460.7860.646
       Detection of total glomeruli
      Regardless of the classification.
      0.9620.9030.9320.874
      Masson
       GN0.6830.8520.7580.616
       GL0.9140.6120.7330.630
       GS0.7670.9170.8530.616
       Detection of total glomeruli
      Regardless of the classification.
      0.9660.8940.9290.856
      All
       GN0.8020.8800.8390.702
       GL0.8660.6660.7530.602
       GS0.7570.8620.8060.623
       Detection of total glomeruli
      Regardless of the classification.
      0.9570.9240.9400.873
      AP, average precision; AR, average recall; GL, glomerulus with abnormal structure; GN, glomerulus with basically normal structure; GS, global sclerosis; Mask mIoU, mean intersection/union ratio of the segmentation results; Masson, Masson trichrome stain; PAS, periodic acid–Schiff; PASM, periodic acid–silver methenamine; WSI, whole-slide image.
      Regardless of the classification.

      Discussion

      For computer-aided automatic histologic assessment of a renal biopsy, the fundamental step is to extract the glomeruli with variable histologic features, as a human pathologist usually does in routine diagnosis works. To achieve this goal, the data sets built in this study included >30 types of kidney diseases, covering most of the common kidney diseases, including combined situations, to develop a model that can detect both normal glomeruli and glomeruli with variable lesions with varying degrees of severity. So, the model presented in this study was designed to be consistent with the pathologist's daily workflow. This will lay a foundation for more detailed assessments of glomerular lesions in later research.
      PASM, Masson, and PAS staining methods have been routinely used to show main structures and lesions of kidney tissues. To better meet the needs of future work, slides with all those three staining methods were contained in the data set to build a unified model with good robustness. Images with different staining methods mean different statistical distributions of matrix values. On the whole, the result of PASM staining was the best, followed by PAS and Masson. For each classification, the performance orders were a little bit different. This might due to the different proportion of each stain. But because of the power of the algorithm, as long as enough data resources are obtained to train the model, it can perform well in images with each staining method. It will be simple and effective in use if a single model is applied with no need to consider the staining method. This study also confirmed the ability and adaptability of a deep-learning algorithm in the pathologic image analysis area.
      There have been several studies to classify normal glomeruli and globally sclerosed glomeruli.
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      ,
      • Marsh J.N.
      • Matlock M.K.
      • Kudose S.
      • Liu T.C.
      • Stappenbeck T.S.
      • Gaut J.P.
      • Swamidass S.J.
      Deep learning global glomerulosclerosis in transplant kidney frozen sections.
      ,
      • Kannan S.
      • Morgan L.A.
      • Liang B.
      • Cheung M.G.
      • Lin C.Q.
      • Mun D.
      • Nader R.G.
      • Belghasem M.E.
      • Henderson J.M.
      • Francis J.M.
      • Chitalia V.C.
      • Kolachalama V.B.
      Segmentation of glomeruli within trichrome images using deep learning.
      In the present study, the glomeruli were classified into three categories, adding the GL tag to label the glomeruli with some lesions other than global sclerosis. Those lesions were selected because they are correlated with various glomerular diseases, and it is necessary to recognize them during pathologic analysis. Mild mesangial lesions, such as mild mesangial hypercellularity or mild mesangial matrix expansion, usually do not cause an obvious change of glomerular structure, so in the current step, they were labeled as GN and grouped with the normal glomeruli. In future research, it will be necessary to make more accurate annotations and design a more specific algorithm model to detect and segment each lesion.
      Semantic segmentation networks have already been used in the classification and segmentation
      • Gadermayr M.
      • Dombrowski A.-K.
      • Klinkhammer B.M.
      • Boor P.
      • Merhof D.
      CNN cascades for segmenting sparse objects in gigapixel whole slide images.
      ,
      • Bueno G.
      • Fernandez-Carrobles M.M.
      • Gonzalez-Lopez L.
      • Deniz O.
      Glomerulosclerosis identification in whole slide images using semantic segmentation.
      ,
      • Marsh J.N.
      • Matlock M.K.
      • Kudose S.
      • Liu T.C.
      • Stappenbeck T.S.
      • Gaut J.P.
      • Swamidass S.J.
      Deep learning global glomerulosclerosis in transplant kidney frozen sections.
      • Kannan S.
      • Morgan L.A.
      • Liang B.
      • Cheung M.G.
      • Lin C.Q.
      • Mun D.
      • Nader R.G.
      • Belghasem M.E.
      • Henderson J.M.
      • Francis J.M.
      • Chitalia V.C.
      • Kolachalama V.B.
      Segmentation of glomeruli within trichrome images using deep learning.
      • Barros G.O.
      • Navarro B.
      • Duarte A.
      • Dos-Santos W.L.C.
      PathoSpotter-K: a computational tool for the automatic identification of glomerular lesions in histological images of kidneys.
      of glomeruli. But semantic segmentation is to classify all pixels in an image, so different glomeruli cannot be segmented separately. In clinical practice, counting the number of glomeruli is the first step of the pathologic analysis. Therefore, instance segmentation of glomeruli is necessary to meet the clinical needs. In this study, a Cascade Mask R-CNN
      • Cai Z.
      • Vasconcelos N.
      • Cascade R.-C.N.N.
      High Quality Object Detection and Instance Segmentation.
      ,
      • He K.
      • Zhang X.
      • Ren S.
      • Sun J.
      Deep residual learning for image recognition. IEEE Conference on Computer Vision & Pattern Recognition.
      • Ren S.
      • He K.
      • Girshick R.
      • Sun J.
      • Faster R.-C.N.N.
      Towards real-time object detection with region proposal networks.
      • He K.
      • Gkioxari G.
      • Dollár P.
      • Girshick R.
      Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV).
      algorithm model was trained for the instance segmentation of human glomeruli in routinely stained sections, using snapshot images and WSIs, respectively. The algorithm step of instance segmentation contained object detection and segmentation. The purpose of the object detection step is to locate the coordinates of glomeruli and classify their categories. IoU was used to describe the intersection with the real target border, and the IoU threshold was used as a reference to divide the positive samples and negative samples in training. Using Cascade Mask R-CNN, by means of cascade structure, different thresholds can be set so the contradiction of setting threshold value in a single module detection network can be avoided, and the accuracy of target detection can be greatly improved. For about segmentation step, the model segmented each detected glomerulus at the pixel level.
      In the model building process, the strategy of the nonmaximum suppression (NMS) algorithm, which from R-CNN
      • Girshick R.
      • Donahue J.
      • Darrell T.
      • Malik J.
      Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
      was changed to suppress the redundant detection boxes. NMS algorithm calculates the IoU between each detection box. If some IoU values are too high, the algorithm will only save one detection box, which has the highest score. The general NMS algorithm only suppresses the results in the same class. The NMS algorithm was changed to suppress the results in all classes to prevent the generation of some incorrect results with the wrong classification. The improvement of NMS algorithm is effective for the data set.
      Most of the failed detections were incomplete glomeruli located at the image boundary, as shown in Figure 5. Therefore, appropriate stride should be set during cutting patches to ensure that all glomeruli can be complete in at least one patch. In this way, when the results are mapped back to WSI, the detection results of glomeruli on the whole WSI will be better.
      Figure thumbnail gr5
      Figure 5A failed detection case. Green bounding boxed area denotes successful detection, and red bounding boxed area denotes failed detection.
      For some images of glomerular clusters, the segmentation results of the model output may overlap. To solve this problem, only the results with high confidence scores were reserved in the overlapping area.
      For the classification of glomerular lesions, GN got the highest accuracy. The order of the performance metrics was just consistent with that of the sample size of each classification. To improve the accuracies of classifying GL and GS, more glomeruli with these tags need to be selected into the data set. Another reason might be that some glomeruli in GL and GS classification were seriously damaged and hard to be correctly detected from inflammatory interstitial tissue, such as a glomerular filled with crescent or a small sclerosis glomerulus without Bowman capsule. This also can be resolved by increasing the number of glomeruli with difficultly identified structures.
      Although the process of biopsy digitization has been increasingly performed in diagnostic centers, there are still many smaller centers that do not have the digital scanner, and it is just these smaller centers that need more computer assistance to improve the accuracy of diagnosis. So, the images for training the model were from two sources: snapshot images from the image library and scanned WSIs. The snapshot images were taken from typical lesion areas of each case, so the proportion of pathologic glomeruli was relatively higher than that of glomeruli from WSIs. Although the total glomerular number was limited (only 1487 glomeruli), they belonged to 516 patients and included >30 pathologic types of kidney diseases. All of those were conducive to the construction of an algorithm model for the recognition of glomeruli with varied lesions. The WSIs were from 148 patients, less than those of snapshot images, and they also contained fewer pathologic types of kidney diseases, whereas the total number of glomeruli was nearly four times that of the former. The performance metrics, which reflected overall accuracy of recognition, were comparable in two groups, especially when detecting total glomeruli, regardless of classification; AP, AR, and F1 score were >0.9. This might be explained that large sample size made up for the lack of diversity of lesions in the WSI group, to some extent. To build an ideal data set, both sample size and lesion diversity should be considered in future studies.
      This study also has some limitations. First, as has been mentioned above, the number of glomeruli with seriously damaged structures should be increased to improve segmentation accuracy. Second, the abnormal glomeruli detected in this model only included lesions that obviously changed the glomerular structures, whereas the thickness of the basement membrane was not considered. It needs a more refined model to detect such lesions. Third, in terms of classification, the result of basically normal glomeruli is the best, and the other two categories need to be judged and modified by physicians on the basis of the automatic classification of the model. We are working on an application with an interactive window to help physicians adjust the classification as they apply the model, and the results can be used to further refine the model.
      In conclusion, this study presents a model that can segment both normal glomeruli and glomeruli with various lesions, suitable for three kinds of special staining methods at the same time, and can detect and classify glomeruli into three types. In clinical practice, the present model can help physicians to calculate the number of glomeruli, quickly locate glomeruli in diagnosis, and lay the foundation for further analysis of the internal structures of glomeruli. Therefore, the current study can be regarded as the first step in the whole automation system. Images can be put into this part to get the bounding box and classification results, then cropped into smaller patches by bounding box value and sent to the next parts to analyze more detailed histologic information, such as mesangial hypercellularity and crescents.

      Acknowledgments

      We thank Xin Li and Chunying Shao for excellent technical assistance.

      Author Contributions

      L.J., C.Z., and J.L. designed the research. L.J., W.C., and K.M. analyzed and interpreted the data. L.J. and W.C. drafted the article. B.D., J.L., and G.W. provided intellectual content of critical importance to the work. C.Z., H.S., M.C., Y.Y., and L.Z. revised the article. C.Z. and H.S. approved the version to be published.
      H.S. and C.Z. are the guarantors of this work and, as such, had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

      References

        • Puelles V.G.
        • Bertram J.F.
        Counting glomeruli and podocytes: rationale and methodologies.
        Curr Opin Nephrol Hypertens. 2015; 24: 224-230
        • Lees G.E.
        • Cianciolo R.E.
        • Clubb F.J.
        Renal biopsy and pathologic evaluation of glomerular disease.
        Top Companion Anim Med. 2011; 26: 143-153
        • Bajema I.M.
        • Wilhelmus S.
        • Alpers C.E.
        • Bruijn J.A.
        • Colvin R.B.
        • Cook H.T.
        • D'Agati V.D.
        • Ferrario F.
        • Haas M.
        • Jennette J.C.
        • Joh K.
        • Nast C.C.
        • Noel L.H.
        • Rijnink E.C.
        • Roberts I.S.D.
        • Seshan S.V.
        • Sethi S.
        • Fogo A.B.
        Revision of the International Society of Nephrology/Renal Pathology Society classification for lupus nephritis: clarification of definitions, and modified National Institutes of Health activity and chronicity indices.
        Kidney Int. 2018; 93: 789-796
        • Trimarchi H.
        • Barratt J.
        • Cattran D.C.
        • Cook H.T.
        • Coppo R.
        • Haas M.
        • Liu Z.H.
        • Roberts I.S.
        • Yuzawa Y.
        • Zhang H.
        • Feehally J.
        • IgAN Classification Working Group of the International IgA Nephropathy Network and the Renal Pathology Society
        Oxford classification of IgA nephropathy 2016: an update from the IgA Nephropathy Classification Working Group.
        Kidney Int. 2017; 91: 1014-1021
        • Gadermayr M.
        • Dombrowski A.-K.
        • Klinkhammer B.M.
        • Boor P.
        • Merhof D.
        CNN cascades for segmenting sparse objects in gigapixel whole slide images.
        Comput Med Imaging Graphics. 2019; 71: 40-48
        • Esteva A.
        • Robicquet A.
        • Ramsundar B.
        • Kuleshov V.
        • DePristo M.
        • Chou K.
        • Cui C.
        • Corrado G.
        • Thrun S.
        • Dean J.
        A guide to deep learning in healthcare.
        Nat Med. 2019; 25: 24-29
        • Wang S.
        • Yang D.M.
        • Rong R.
        • Zhan X.
        • Xiao G.
        Pathology image analysis using segmentation deep learning algorithms.
        Am J Pathol. 2019; 189: 1686-1698
        • Komura D.
        • Ishikawa S.
        Machine learning approaches for pathologic diagnosis.
        Virchows Arch. 2019; 475: 131-138
        • Maree R.
        • Dallongeville S.
        • Olivo-Marin J.C.
        • Meas-Yedid V.
        An approach for detection of glomeruli in multisite digital pathology. IEEE International Symposium on Biomedical Imaging.
        2016: 1033-1036 (IEEE)
        • Temerinac-Ott M.
        • Forestier G.
        • Schmitz J.
        • Hermsen M.
        • Brasen J.H.
        • Feuerhake F.
        • Wemmert C.
        Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities. 10th International Symposium on Image and Signal Processing and Analysis.
        2017: 19-24 (Ljubljana, Slovenia)
        • Simon O.
        • Yacoub R.
        • Jain S.
        • Tomaszewski J.E.
        • Sarder P.
        Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images.
        Sci Rep. 2018; 8: 2032
        • Bukowy J.D.
        • Dayton A.
        • Cloutier D.
        • Manis A.D.
        • Staruschenko A.
        • Lombard J.H.
        • Solberg Woods L.C.
        • Beard D.A.
        • Cowley Jr., A.W.
        Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections.
        J Am Soc Nephrol. 2018; 29: 2081-2088
        • Kato T.
        • Relator R.
        • Ngouv H.
        • Hirohashi Y.
        • Takaki O.
        • Kakimoto T.
        • Okada K.
        • Segmental H.O.G.
        new descriptor for glomerulus detection in kidney microscopy image.
        BMC Bioinformatics. 2015; 16: 1-16
        • Sarder P.
        • Ginley B.
        • Tomaszewski J.E.
        Automated renal histopathology: digital extraction and quantification of renal pathology. SPIE Medical Imaging.
        2016 (San Diego, CA)
        • Sheehan S.
        • Mawe S.
        • Cianciolo R.E.
        • Korstanje R.
        • Mahoney J.M.
        Detection and classification of novel renal histologic phenotypes using deep neural networks.
        Am J Pathol. 2019; 189: 1786-1796
        • Ginley B.
        • Lutnick B.
        • Jen K.-Y.
        • Fogo A.B.
        • Jain S.
        • Rosenberg A.
        • Walavalkar V.
        • Wilding G.
        • Tomaszewski J.E.
        • Yacoub R.
        • Rossi G.M.
        • Sarder P.
        Computational segmentation and classification of diabetic glomerulosclerosis.
        J Am Soc Nephrol. 2019; 30: 1953-1967
        • Hermsen M.
        • de Bel T.
        • den Boer M.
        • Steenbergen E.J.
        • Kers J.
        • Florquin S.
        • Roelofs J.J.T.H.
        • Stegall M.D.
        • Alexander M.P.
        • Smith B.H.
        • Smeets B.
        • Hilbrands L.B.
        • van der Laak J.A.W.M.
        Deep learning–based histopathologic assessment of kidney tissue.
        J Am Soc Nephrol. 2019; 30: 1968-1979
        • Bueno G.
        • Fernandez-Carrobles M.M.
        • Gonzalez-Lopez L.
        • Deniz O.
        Glomerulosclerosis identification in whole slide images using semantic segmentation.
        Comput Methods Programs Biomed. 2020; 184: 105273
        • Zeng C.
        • Nan Y.
        • Xu F.
        • Lei Q.
        • Li F.
        • Chen T.
        • Liang S.
        • Hou X.
        • Lv B.
        • Liang D.
        • Luo W.
        • Lv C.
        • Li X.
        • Xie G.
        • Liu Z.
        Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning.
        J Pathol. 2020; 252: 53-64
        • Cai Z.
        • Vasconcelos N.
        • Cascade R.-C.N.N.
        High Quality Object Detection and Instance Segmentation.
        High Quality Object Detection and Instance Segmentation. 2021; 43: 1483-1498
        • Sethi S.
        • Haas M.
        • Markowitz G.S.
        • D'Agati V.D.
        • Rennke H.G.
        • Jennette J.C.
        • Bajema I.M.
        • Alpers C.E.
        • Chang A.
        • Cornell L.D.
        • Cosio F.G.
        • Fogo A.B.
        • Glassock R.J.
        • Hariharan S.
        • Kambham N.
        • Lager D.J.
        • Leung N.
        • Mengel M.
        • Nath K.A.
        • Roberts I.S.
        • Rovin B.H.
        • Seshan S.V.
        • Smith R.J.H.
        • Walker P.D.
        • Winearls C.G.
        • Appel G.B.
        • Alexander M.P.
        • Cattran D.C.
        • Casado C.A.
        • Cook H.T.
        • De Vriese A.S.
        • Radhakrishnan J.
        • Racusen L.C.
        • Ronco P.
        • Fervenza F.C.
        Mayo Clinic/Renal Pathology Society consensus report on pathologic classification, diagnosis, and reporting of GN.
        J Am Soc Nephrol. 2016; 27: 1278-1287
        • Barisoni L.
        • Troost J.P.
        • Nast C.
        • Bagnasco S.
        • Avila-Casado C.
        • Hodgin J.
        • Palmer M.
        • Rosenberg A.
        • Gasim A.
        • Liensziewski C.
        • Merlino L.
        • Chien H.P.
        • Chang A.
        • Meehan S.M.
        • Gaut J.
        • Song P.
        • Holzman L.
        • Gibson D.
        • Kretzler M.
        • Gillespie B.W.
        • Hewitt S.M.
        Reproducibility of the NEPTUNE descriptor-based scoring system on whole-slide images and histologic and ultrastructural digital images.
        Mod Pathol. 2016; 29: 671-684
        • Roberts I.S.
        • Cook H.T.
        • Troyanov S.
        • Alpers C.E.
        • Amore A.
        • Barratt J.
        • Berthoux F.
        • Bonsib S.
        • Bruijn J.A.
        • Cattran D.C.
        • Coppo R.
        • D'Agati V.
        • D'Amico G.
        • Emancipator S.
        • Emma F.
        • Feehally J.
        • Ferrario F.
        • Fervenza F.C.
        • Florquin S.
        • Fogo A.
        • Geddes C.C.
        • Groene H.J.
        • Haas M.
        • Herzenberg A.M.
        • Hill P.A.
        • Hogg R.J.
        • Hsu S.I.
        • Jennette J.C.
        • Joh K.
        • Julian B.A.
        • Kawamura T.
        • Lai F.M.
        • Li L.S.
        • et al.
        • Working Group of the International Ig A Nephropathy Network and the Renal Pathology Society
        The Oxford classification of IgA nephropathy: pathology definitions, correlations, and reproducibility.
        Kidney Int. 2009; 76: 546-556
        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Deep residual learning for image recognition. IEEE Conference on Computer Vision & Pattern Recognition.
        2016: 770-778 (IEEE Computer Society)
        • Ren S.
        • He K.
        • Girshick R.
        • Sun J.
        • Faster R.-C.N.N.
        Towards real-time object detection with region proposal networks.
        IEEE Trans Pattern Anal Mach Intell. 2016; 39: 1137-1149
        • He K.
        • Gkioxari G.
        • Dollár P.
        • Girshick R.
        Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV).
        IEEE, Venice, Italy2017
        • Chen K.
        • Wang J.
        • Pang J.
        • Cao Y.
        • Xiong Y.
        • Li X.
        • Sun S.
        • Feng W.
        • Liu Z.
        • Xu J.
        MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.
        2019 (1906.07155v1)
        • Marsh J.N.
        • Matlock M.K.
        • Kudose S.
        • Liu T.C.
        • Stappenbeck T.S.
        • Gaut J.P.
        • Swamidass S.J.
        Deep learning global glomerulosclerosis in transplant kidney frozen sections.
        IEEE Trans Med Imaging. 2018; 37: 2718-2728
        • Kannan S.
        • Morgan L.A.
        • Liang B.
        • Cheung M.G.
        • Lin C.Q.
        • Mun D.
        • Nader R.G.
        • Belghasem M.E.
        • Henderson J.M.
        • Francis J.M.
        • Chitalia V.C.
        • Kolachalama V.B.
        Segmentation of glomeruli within trichrome images using deep learning.
        Kidney Int Rep. 2019; 4: 955-962
        • Barros G.O.
        • Navarro B.
        • Duarte A.
        • Dos-Santos W.L.C.
        PathoSpotter-K: a computational tool for the automatic identification of glomerular lesions in histological images of kidneys.
        Sci Rep. 2017; 7: 46769
        • Girshick R.
        • Donahue J.
        • Darrell T.
        • Malik J.
        Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
        IEEE, Columbus, OH2014