Advertisement

Three-Dimensional Vessel Segmentation in Whole-Tissue and Whole-Block Imaging Using a Deep Neural Network

Proof-of-Concept Study
Open ArchivePublished:December 23, 2020DOI:https://doi.org/10.1016/j.ajpath.2020.12.008
      In the field of pathology, micro–computed tomography (micro-CT) has become an attractive imaging modality because it enables full analysis of the three-dimensional characteristics of a tissue sample or organ in a noninvasive manner. However, because of the complexity of the three-dimensional information, understanding would be improved by development of analytical methods and software such as those implemented for clinical CT. As the accurate identification of tissue components is critical for this purpose, we have developed a deep neural network (DNN) to analyze whole-tissue images (WTIs) and whole-block images (WBIs) of neoplastic cancer tissue using micro-CT. The aim of this study was to segment vessels from WTIs and WBIs in a volumetric segmentation method using DNN. To accelerate the segmentation process while retaining accuracy, a convolutional block in DNN was improved by introducing a residual inception block. Three colorectal tissue samples were collected and one WTI and 70 WBIs were acquired by a micro-CT scanner. The implemented segmentation method was then tested on the WTI and WBIs. As a proof-of-concept study, our method successfully segmented the vessels on all WTI and WBIs of the colorectal tissue sample. In addition, despite the large size of the images for analysis, all segmentation processes were completed in 10 minutes.
      Whole-slide imaging has gained importance as an imaging modality in pathology because it enables analog to digital conversion for integration with image analysis and/or artificial intelligence in future diagnosis. However, digitization in the field of pathology lags behind other medical imaging modalities because analog observation with microscope remains popular among many pathologists. More than two decades of research have revealed important roles of whole-slide image (WSI), among which three-dimensional (3D) imaging is a symbolic achievement.
      • Casero R.
      • Siedleck U.
      • Jones E.S.
      • Gruscheski L.
      • Gibb M.
      • Schneider J.E.
      • Kohl P.
      • Grau V.
      Transformation diffusion reconstruction of three-dimensional histology volumes from two-dimensional image stacks.
      • Gaffling S.
      • Daum V.
      • Hornegger J.
      Landmark-constrained 3-D histological imaging: a morphology-preserving approach.
      • Schwier M.
      • Böhler T.
      • Hahn H.K.
      • Dahmen U.
      • Dirsch O.
      Registration of histological whole slide images guided by vessel structures.
      • Cifor A.
      • Bai L.
      • Pitiot A.
      Smoothness-guided 3-D reconstruction of 2-D histological images.
      • Xu Y.
      • Pickering G.
      • Nong Z.
      • Glibson E.
      • Arpino J.M.
      • Yin H.
      • Ward A.D.
      A method for 3D histopathology reconstruction supporting mouse microvasculature analysis.
      • Roberts N.
      • Magee D.
      • Song Y.
      • Brabazon K.
      • Shires M.
      • Crellin D.
      • Orsi N.M.
      • Quirke R.
      • Quirke P.
      • Treano D.
      Toward routine use of 3D histopathology as a research tool.
      However, whole tissues from the patient cannot be examined because in accordance with institutional regulations, 50% of the tumor volume must remain untouched after the study. In addition, WSI-based 3D imaging is time-consuming, labor intensive, and costly. Therefore, to realize a clinical application for pathology, a noninvasive and simple 3D imaging technique is required.
      In recent years, micro–computed tomography (micro-CT) has become an attractive imaging modality because it enables full analysis of the 3D characteristics of a tissue sample or organ in a noninvasive manner.
      • Teplov A.
      • Tabata K.
      • Fu X.
      • Uraoka N.
      • Roehr H.A.
      • Ntiamoah P.
      • Humm J.L.
      • Sirintrapun S.J.
      • Murray M.P.
      • Shia J.
      • Travis W.D.
      • Brogi E.
      • Hameed M.
      • Yagi Y.
      Development of standard operating procedure (SOP) of micro-computed tomography (micro-CT) in pathology.
      • Ketsamenis O.L.
      • Olding M.
      • Wamer J.A.
      • Chatelet D.S.
      • Jones M.G.
      • Sgalla G.
      • Smit B.
      • Larkin O.J.
      • Haing I.
      • Richeldi L.
      • Sinclair I.
      • Lackie P.M.
      • Schnider P.
      X-ray micro-computed tomography for nondestructive three-dimensional (3D) X-ray histology.
      • Troschel F.M.
      • Gottumkkala R.V.
      • DiCorpo D.
      • Mario J.
      • Ott H.C.
      • Wright C.D.
      • Muniappan A.
      • Lanuti M.
      • Yang K.
      • JO Shepard
      • Nardi V.
      • Michaelson J.S.
      • Hariri L.P.
      • Fintelmann F.J.
      Feasibility of perioperative micro-computed tomography of human lung cancer specimens.
      • Albers J.
      • Markus M.A.
      • Alves F.
      • Dullin C.
      X-ray based virtual histology allows guided sectioning of heavy ion stained murine lungs for histological analysis.
      • Tanabe N.
      • Vasilescu D.M.
      • Kirby M.
      • Coxson H.O.
      • Verleden S.E.
      • Vanaudenaerde B.M.
      • Kinose D.
      • Nakano Y.
      • Paré P.D.
      • Hogg J.C.
      Analysis of airway pathology in COPD using a combination of computed tomography, micro-computed tomography and histology.
      • Mai C.
      • Verleden S.E.
      • McDonough J.E.
      • Willems S.
      • Wever W.D.
      • Coolen J.
      • Dubbeldam A.
      • Raemdonck D.E.V.
      • Verbeken E.K.
      • verleden G.M.
      • Hogg J.C.
      • Vanaudenaerde B.M.
      • Wuyts W.A.
      • Verschakelen J.A.
      Thin-section CT features of idiopathic pulmonary fibrosis correlated with micro-CT and histologic analysis.
      The imaging principle is the same as that of conventional clinical CT. Although the field of view is much smaller, microscale images of high spatial resolution can be obtained. The best spatial resolution of micro-CT image is inferior to that of WSI with ×40 magnification; however, micro-CT images have potential as a substitute for low-magnification WSI because they enable observation of the morphologic structure of the entire tissue noninvasively. Teplov et al
      • Teplov A.
      • Tabata K.
      • Fu X.
      • Uraoka N.
      • Roehr H.A.
      • Ntiamoah P.
      • Humm J.L.
      • Sirintrapun S.J.
      • Murray M.P.
      • Shia J.
      • Travis W.D.
      • Brogi E.
      • Hameed M.
      • Yagi Y.
      Development of standard operating procedure (SOP) of micro-computed tomography (micro-CT) in pathology.
      examined fresh samples, fixed samples, and formalin-fixed, paraffin-embedded (FFPE) block samples of various organs. They developed operational procedures and effective imaging parameters for fresh, fixed, and FFPE block samples; in addition, they found that the X-ray dose from micro-CT scanning does not affect protein expression. In their study of interstitial pneumonia, Ketsamenis et al
      • Ketsamenis O.L.
      • Olding M.
      • Wamer J.A.
      • Chatelet D.S.
      • Jones M.G.
      • Sgalla G.
      • Smit B.
      • Larkin O.J.
      • Haing I.
      • Richeldi L.
      • Sinclair I.
      • Lackie P.M.
      • Schnider P.
      X-ray micro-computed tomography for nondestructive three-dimensional (3D) X-ray histology.
      demonstrated that the small airways, blood vessels, and alveoli of the lungs could be clearly visualized on micro-CT. Albers et al
      • Albers J.
      • Markus M.A.
      • Alves F.
      • Dullin C.
      X-ray based virtual histology allows guided sectioning of heavy ion stained murine lungs for histological analysis.
      scanned murine lung samples with free-propagation phase-contrast micro-CT and reported that micro-CT has the potential to improve workflow aspects of histologic analysis, such as determination of the cutting position. Xu et al
      • Xu B.
      • Teplov A.
      • Ibrahim K.
      • Inoue T.
      • Stueben B.
      • Kitabi N.
      • Hameed M.
      • Yagi Y.
      • Ghosein R.
      Detection and assessment of capsular invasion, vascular invasion and lymph node metastasis volume in thyroid carcinoma using microCT scanning of paraffin tissue blocks (3D whole block imaging): a proof of concept.
      assessed thyroid carcinoma using micro-CT images of paraffin tissue blocks. According to their report, capsular invasion and vascular invasion could be detected and volume of nodal metastasis could be assessed by micro-CT images, even if the images were not present on serially sectioned hematoxylin and eosin (H&E)–stained slides. In addition, they mentioned that the volume of the metastatic foci measured by micro-CT image differed by 13.125-fold compared with that measured by WSI of H&E slide. The reported spatial resolution of a micro-CT image of an FFPE block sample is approximately 5 to 10 μm,
      • Teplov A.
      • Tabata K.
      • Fu X.
      • Uraoka N.
      • Roehr H.A.
      • Ntiamoah P.
      • Humm J.L.
      • Sirintrapun S.J.
      • Murray M.P.
      • Shia J.
      • Travis W.D.
      • Brogi E.
      • Hameed M.
      • Yagi Y.
      Development of standard operating procedure (SOP) of micro-computed tomography (micro-CT) in pathology.
      ,
      • Ketsamenis O.L.
      • Olding M.
      • Wamer J.A.
      • Chatelet D.S.
      • Jones M.G.
      • Sgalla G.
      • Smit B.
      • Larkin O.J.
      • Haing I.
      • Richeldi L.
      • Sinclair I.
      • Lackie P.M.
      • Schnider P.
      X-ray micro-computed tomography for nondestructive three-dimensional (3D) X-ray histology.
      which is comparable to that of WSI (up to ×10.0 magnification) used for evaluation of morphologic structure. To three-dimensionally visualize the FFPE block sample with 4-mm–thick tissue by WSI, approximately 1000 slides must be generated; however, micro-CT can achieve it without sectioning and tissue exhaustion. The findings of these previous studies suggest that, in terms of clinical diagnosis, micro-CT is a useful tool that can provide 3D structural tissue information that enriches the histopathologic diagnosis.
      However, because it is difficult to interpret the vast amount of 3D information, there is a need for analytical methods, such as those implemented in the software used with clinical CT. The most highly demanded technique is the accurate identification of each tissue component. Tissue identification has great potential for examining the extension of each component feature and the relationship between these components. For example, vessel extension or tumor growth can be analyzed after tissue identification, thus providing the pathologist with additional diagnostic information that enables a more accurate morphologic assessment. Several previous studies have reported the development of tissue identification techniques. Wollatz et al
      • Wollatz L.
      • Johnston S.J.
      • Lackie P.M.
      • Cox S.J.
      3D histopathology—a lung tissue segmentation workflow for microfocus X-ray-computed tomography scans.
      built a semi-automatic segmentation workflow with open-source software for segmenting airways and blood vessels from lung tissue samples. Although their system could run on a standard personal computer, it needed an improvement in performance. Zhao et al
      • Zhao F.
      • Gao P.
      • Hu H.
      • He X.
      • Hou Y.
      • He X.
      Efficient kidney segmentation in micro-CT based on multi-atlas registration and random forests.
      developed a kidney segmentation method based on multi-atlas registration and random forests, and applied it to murine micro-CT images. Although their method shows great potential for segmentation purposes, it will not be easily applied to clinical histopathologic diagnosis as it was developed specifically for analysis of murine kidney.
      While there is high demand for the development of various image analyses, this manuscript focuses on vessel segmentation because there is strong interest in the 3D structural relationships between vessels and tumors in many clinical fields. Sometimes a contrast between vessel and other tissues in the CT is low, depending on the object. In such case, angiography, which uses a contrast agent, has been generally applied to enhance the vessel region.
      • Lee J.
      • Beighley P.
      • Ritman E.
      • Smith N.
      Automatic segmentation of 3D micro-CT coronary vascular images.
      Classic methods, such as Hessian matrix–based method,
      • Lee J.
      • Beighley P.
      • Ritman E.
      • Smith N.
      Automatic segmentation of 3D micro-CT coronary vascular images.
      make it difficult to segment the vessel from low-contrast micro-CT images. Because one of our targets is FFPE block, generated according to the standard clinical procedure, use of the additional angiogram step is not assumed. Therefore, to achieve highly accurate segmentation on the micro-CT image without the contrast agent, we constructed a deep neural network (DNN), which has become a widely used technique in recent years. As a proof-of-concept study, we focused on the diagnosis of colorectal cancer and performed vessel segmentation on three micro-CT images acquired from one colorectal tissue sample. The following sections will explain the image acquisition procedure, outline the network structure for vessel segmentation, and report the experimental results.

      Materials and Methods

      Materials

      Figure 1 outlines the procedure developed for application of micro-CT to clinical histopathologic diagnosis. A resected fresh or fixed tissue sample is examined by micro-CT scanner in low-resolution mode. The spatial resolution of the obtained micro-CT image is ≥50 μm. The micro-CT image of those samples is called a whole-tissue image (WTI) in this study. FFPE block samples are prepared, and these are also examined by micro-CT scanner. The micro-CT image of the FFPE block sample is termed a whole-block image (WBI). After micro-CT examinations, stained slide samples are prepared and digitized by whole-slide scanners. All acquired micro-CT images and whole-slide images are stored in the image server linked with each scanner, and synchronized with a storage server via a database server. The database server manages the synchronized images on the storage server and associates the study identifier and the corresponding images. The pathologist searches and views the arbitrary images through the database server. If image analysis is requested from a client personal computer, the database server submits the tasks to a high-performance computing server and provides the analysis results to the client personal computer.
      Figure thumbnail gr1
      Figure 1Schematic illustration of our system and workflow. All acquired images are stored into the storage server, and the deep neural network developed in this study is implemented and run on a high-performance computing server. DB, database; FFPE, formalin fixed, paraffin embedded; ID, identifier; Micro-CT, micro–computed tomography; PC, personal computer.
      Figure 2 shows the data flow from tissue sample attainment to performance evaluation in this study. Three resected colorectal tissue samples were used in the experiment. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study. Of these, two fixed tissue samples were divided into several sections along their long axis, and each dissected colorectal tissue sample was divided into two block samples. Each block sample underwent tissue processing and was then embedded into a cassette. Because the block samples were larger than the standard block cassette (28 mm × 40 mm) used for colorectal tissue samples, according to the standard procedure of our institute, we used whole-mount block cassettes of 75 mm × 52 mm. This procedure allowed us to reduce the number of samples to be scanned and to extend the field of view on micro-CT. Forty FFPE blocks were then immobilized with a carbon fiber paddle and individually examined by micro-CT scanner, resulting in 40 WBIs. After micro-CT image acquisition, the FFPE blocks were thin sliced. All slices were deparaffinized, stained using hematoxylin and eosin, and then scanned by a whole-slide scanner. The resulting images were termed whole-mount slide images (WMIs). Because following the annotation process was a highly time-consuming task, 10 pairs of WBIs and WMIs were selected to generate a data set for network training. As the original WBIs were too large and the vessels were sparsely located, 25 arbitrary-sized volumes containing vessels were manually extracted from WBIs. The average size of the extracted volumes was 200 × 200 × 80 voxels. Vessels with 2 to 15 voxel diameters were annotated as the ground truth (GT) on each extracted volume by two individuals, with reference to the corresponding WMI. Annotation was complete when there was agreement between the annotators.
      Figure thumbnail gr2
      Figure 2Data flow from sample attainment to performance evaluation. Black arrows indicate flow of network training; green arrows, flow of visual assessment using whole-tissue image (WTI); blue arrows, flow of visual assessment and performance evaluation using two whole-block images (WBIs). FFPE, formalin fixed, paraffin embedded; WMI, whole-mount slide image.
      One remaining fixed tissue sample was immobilized in the polystyrene foam container and examined by micro-CT scanner after formalin fixation, resulting in one WTI. A total of 30 FFPE blocks were obtained from the fixed tissue sample and examined by micro-CT scanner. For cost and labor reasons, two WBIs (B1 and B2) were selected and used to evaluate the performance of our method. These two WBIs were selected because the tissue blocks were unadjacent, and had plenty of vessels versus those in the other WBIs. After micro-CT examination, 20 consecutive H&E slices were obtained from the FFPE blocks corresponding to B1 and B2, and their WMIs were acquired. GT for the performance evaluation was obtained by manual annotation on WMIs, and corresponding slices of WBIs to WMIs were also manually selected. Annotators were the same individuals who generated the GTs for the training data set. As it is extremely time-consuming to annotate all vessels on a WMI, in each case we focused on an area of high-vessel density, and evaluated performance on the area.
      The micro-CT imaging system comprises a custom-built micro-CT scanner (Nikon Metrology NV, Hertfordshire, UK) and CT Pro image reconstruction software version 2.0 (Nikon Metrology NV). The following scan parameters were adjusted for WTI and WBI, as determined previously
      • Teplov A.
      • Tabata K.
      • Fu X.
      • Uraoka N.
      • Roehr H.A.
      • Ntiamoah P.
      • Humm J.L.
      • Sirintrapun S.J.
      • Murray M.P.
      • Shia J.
      • Travis W.D.
      • Brogi E.
      • Hameed M.
      • Yagi Y.
      Development of standard operating procedure (SOP) of micro-computed tomography (micro-CT) in pathology.
      : beam energy, beam current, exposure time, number of projections, frames per projection, and the distance from the X-ray source to the sample (Table 1). For fixed tissues, scan duration was set to 15 minutes, and the best combination of scan parameters was investigated. After every micro-CT image examination, we confirmed successful image reconstruction by visual assessment using VGSTUDIO MAX image analysis software version 1.2.1 (Volume Graphics, Heidelberg, Germany). The whole-mount slides were scanned by NanoZoomer S60 (Hamamatsu Photonics K.K., Hamamatsu, Japan) in ×20 magnification mode. In this mode, pixel size was 0.46 × 0.46 μm2. For each WBI case, we selected the four WMIs that had the greatest similarity with the WBI slices. The same annotators in the GT generation visually evaluated the similarity between WMI and WBI slices by focusing on the position and shape of the vessels. The WMI positions were manually corrected by referring to the corresponding WBI slice. The image format of the WMIs was converted from NDPI to BMP on an image viewer (NDP.view2; Hamamatsu Photonics K.K.). Three pairs of corresponding coordinates were selected manually from a slice of WBI and WMI, and transformation parameters based on affine transformation with shift, rotation, and flip operations
      • Chang S.
      • Cheng F.
      • Hsu W.
      • Wu G.
      Fast algorithm for point pattern matching: invariant to translations, rotations and scale changes.
      were calculated. WMIs were transformed by calculated transformation parameters using ImageJ software version 1.52a (NIH, Bethesda, MD; https://imagej.nih.gov/ij).
      Table 1Acquisition Parameters of Micro-CT
      ObjectBeam energy, kVBeam current, μAExposure time, millisecondsProjections, nFrames per projectionSource-to-object distance, mm
      Fixed tissue9022225018012325
      FFPE block70100325048244120–175
      FFPE, formalin fixed, paraffin embedded; Micro-CT, micro–computed tomography.

      Vessel Segmentation Method

      In the present study, we implemented a vessel segmentation function using DNN. A U-Net
      • Ronneberger O.
      • Fischer P.
      • Brox T.
      U-Net convolutional networks for biomedical image segmentation.
      –like network was created (Figure 3). Inspired by V-Net
      • Milletari F.
      • Navab N.
      • Ahmadi S.A.
      V-Net fully convolutional neural networks for volumetric medical image segmentation.
      and 3D U-Net,
      • Çiçek Ö.
      • Abdulkadir A.
      • Lienkamp S.S.
      • Brox T.
      • Ronneberger O.
      3D U-Net: learning dense volumetric segmentation from sparse annotation.
      volumetric semantic segmentation was constructed. These previously reported DNNs used the same sized convolution kernels at each convolutional block. Many convolutional blocks must be stacked and repeated for segmenting thick vessels. Particularly in the case of 3D DNN, this greatly increases the number of adjustable parameters and leads to low processing efficiency and a long training time. Accordingly, our convolutional block design aimed to extend the convolution area and increase the variation of the convolution processes while limiting the number of parameters.
      Figure thumbnail gr3
      Figure 3Schematic illustration of network architecture. After multichannelization, convolutional blocks and down convolution are recurrently conducted. After bottom convolutional block, up convolution, skip connection, and convolutional block are recurrently conducted until image sizes become same as those of input volumes. Ch, channel; ReLU, rectified linear unit.
      First, input patch images were multichannelized by convolution layer with a 1 × 1 × 1 kernel and C channels. The convolutional block and down convolution were then conducted recurrently. The convolutional block, four convolution paths, consisting of small convolution layers, were built (Figure 4) . The input feature volumes were divided into four groups along a channel dimension, and then input into each convolution path. This adopted structure was inspired by residual inception block
      • Szegedy C.
      • Loffe S.
      • Vanhoucke V.
      • Alemi A.
      Inception-v4, inception-ResNet and the impact of residual connections on learning.
      to extend the effective area and increase the convolution styles. If the input feature volumes had 4c (= C) channels, standard 3 × 3 × 3 and 5 × 5 × 5 convolutions adjusted 112c and 504c parameters, respectively. However, the number of adjustable convolutional parameters in our convolutional block became 138c. Although the effective area of the convolution block was a little sparse, it was maximally 9 × 9 × 9 voxels, and the number of parameters was similar to the standard 3 × 3 × 3 convolution. The concatenated four kinds of convolution results were added with the input feature volumes. Batch normalization
      • Ioffe S.
      • Szegedy C.
      Batch normalization: accelerating deep network training by reducing internal covariate shift.
      and leaky-rectified linear unit
      • Maas A.L.
      • Hannun A.Y.
      • Ng A.Y.
      Rectifier nonlinearities improve neural network acoustic models.
      activation were conducted after the addition process. Batch normalization was defined as follows:
      xi¯=xiμBσB2+ε,
      (1)


      Figure thumbnail gr4
      Figure 4Convolutional block structure. Four kinds of convolution processes are first conducted on divided input volumes along channel (ch) dimension. The concatenated four kinds of convolution results are added with the input feature volumes. Conv3D, three-dimensional convolution; Norm, normalization; ReLU, rectified linear unit.
      where xi and xi¯ represent layer inputs and standardized layer inputs, respectively; and μB, σB, and ε represent the mean and SD of the batch and a small constant value, such as 1.0 × 10−3, respectively. The layer output is obtained with scale and shift operations as yi=γxi¯+β. The γ and β represent trainable scale and bias parameters, respectively. Leaky-rectified linear unit is a rectified linear unit base activation function, defined as follows:
      f(x)={x ifx>0,0.3x otherwise
      (2)


      The slope parameter was originally proposed as 0.01 but we used 0.3, the default value of Keras API (Google LLC, Mountain View, CA), to implement the DNN. The down convolution provided twofold channels of downsized feature volumes by applying the 2 × 2 × 2 kernel and stride 2. After bottom convolutional block, up convolution consisted of up sampling with linear interpolation, and convolution was conducted with a 1 × 1 × 1 kernel. The number of channels was halved through the up-convolution process. Up-convolved results were concatenated with feature volumes provided at the same depth, after which convolutional block was executed again. After repeating up convolution and convolutional block, the feature volumes were dechannelized into one channel and activated with a sigmoid function. The activation result was used as a predicted value. The loss between the predicted value and the GT was calculated by binary cross entropy, as described by the following equation:
      H=1Ni=0N{yi·log[p(yiˆ)]+(1yi)·log[1p(yiˆ)]}
      (3)


      where y, yˆ represent the GT having 0 or 1, respectively; and predicted value p(·) represents the probability of the vessel for all N voxels. Herein, batch normalization and leaky-rectified linear unit layers were executed after every convolution process in multichannel, down-convolution, up-convolution, and dechannel layers.
      In general, although the inception module reduces the number of parameters, DNN still uses large amounts of graphics processing unit memory to analyze the 3D structure. Therefore, the patch image size must be small, which means that large-caliber vessels cannot be segmented by our DNN. To solve this problem, a multiresolution strategy was adapted in the prediction step. A Gaussian blur filter with σ = 1 was applied to the original image. An image pyramid was then generated from the original image, and patch images to be input to the network were extracted from the image at each resolution level. The network outputs at low resolution levels were up sampled to the original resolution with linear interpolation. The final segmentation result was obtained by applying the voxel-wise maximum operation with all up-sampled results.

      Experiments

      In machine learning field, intensity normalization or standardization is generally applied for all data before network input. However, because the range of intensity strongly varied with each WBI, the conventional method+, using mean intensity and SD, was unsuitable for this study. Therefore, several vessel intensities were measured manually on each WBI by using ImageJ software, and a mean intensity of the vessel was calculated. All intensities of each WBI were then divided by the mean intensity. After standardization, the vessel intensities on WBI approached 1. Small patch images with 64 × 64 × 32 voxels were randomly extracted from the standardized volumes for training. Contrast control, rotation, and flip processes were conducted as data augmentation during the extraction process. Contrast was adjusted simply, according to the following equation:
      Vout=aVin+b
      (4)


      where Vin and Vout represent input and output intensities, respectively. The a and b represent adjustive parameters that were determined randomly for each patch image. The ranges of a and b were from 0.95 to 1.05 and from −0.05 to 0.05, respectively, and the distribution of random numbers was uniform. Rotation of 0, 90, 180, and 270 degrees was conducted along the z axis only. Flip was conducted for each axis. Execution of flip for each axis was determined randomly.
      The implementation environment was as follows: Intel Xeon E5-2698 version 4 2.2 GHz with 20 cores, 80 GB RAM, and 4-Tesla V100 with 5120 cores and 16 GB RAM (Nvidia, Santa Clara, CA). The proposed network was implemented by Tensorflow 1.14.0 (Google LLC) and Keras 2.2.5 (Google LLC) in a Python 3.6 (Python Software Foundation, Beaverton, OR) environment. The number of channels in a multichannel layer was empirically set to 20. The numbers of network depth and convolutional blocks at each time were 4 and 4, respectively. Adam optimizer was used for network training, and the initial learning rate was 1.0 × 10−5. The batch size was set to 48. A total of 100,000 epochs were conducted with a train-on-batch mode, and batch was updated on every epoch. Therefore, total training time was approximately 4 days.
      Performance evaluation was conducted with true positive rate (TPR) and positive predictive value (PPV), defined as follows:
      TPR=TP/(TP+FN),
      (5)


      PPV=TP/(TP+FP),
      (6)


      where TP, FN, and FP represent the numbers of true positives, false negatives, and false positives, respectively. Both TPR and PPV can take a value from 0 to 1, where a higher value represents good results. Tissue deformation and breakage that occurred during H&E slide preparation made it difficult to obtain WMI and WBI slices that corresponded exactly, and prevented pixel-wise evaluation. Therefore, we evaluated the area-wise TPR and PPV. Figure 5 shows schematic illustration of judgements of TP, FP, and FN. If a part of a vessel corresponding to GT was segmented, it was counted as a TP. Differences of shape and position were out of consideration in this judgement because of tissue deformation and breakage. FNs and FPs were also counted, using the same judgment criterion. These judgements were also conducted by the same annotators described above. In addition, performance for thick and thin vessels was measured separately. Vessels with the shortest diameters, within 3 voxels in the micro-CT image, were categorized as thin vessels.
      Figure thumbnail gr5
      Figure 5Judgement criteria of true positive (TP), false positive, and false negative. If corresponding vessel area was activated, it was counted as a true positive. Differences of shape and position were out of consideration because of tissue deformation and breakage.

      Results

      Image size and voxel size of WTI were 1300 × 1200 × 1400 voxels and 63 × 63 × 63 μm3/voxel, respectively, and those of WBIs were approximately 1000 × 2000 × 200 voxels and 25 × 25 × 25 μm3/voxel, respectively. The trained network was evaluated using the WTI and WBIs of B1 and B2. Because the original WTI was huge and noisy, each axis was down sampled to half size before the evaluation. Figures 6 and 7 show 3D images of WTI and WBIs, respectively, with superimposition of the segmented vessels. An image analysis software (Dragonfly version 3.0; Object Research Systems, Montreal, QC, Canada) was used to generate the superimposed images and visualize all images. The entire structure of thick vessels could be observed on the WTI, and the shape of the vessels appeared smooth. Because WTI was down sampled, some thin vessels were partially segmented. In contrast, as the spatial resolution of WBI was five times higher than that of down-sampled WTI, many thin vessels could be segmented in both WBIs. Even complicated vessel structures could be visualized clearly (Figure 7). The constructed DNN could complete the segmentation process in about 10 minutes for all images, whereas it takes ≥8 hours to manually annotate vessel regions on a patch image with 200 × 200 × 120 voxels.
      Figure thumbnail gr6
      Figure 6Three-dimensional views of whole-tissue image. Left panel: Micro–computed tomography (micro-CT) image. Right panel: Superimposed image, showing the segmentation result (red) superimposed on the micro-CT image (gray).
      Figure thumbnail gr7
      Figure 7Three-dimensional views of two whole-block images (WBIs). Left column: Micro–computed tomography (micro-CT) images. Middle column: Superimposed images of the segmentation result (red) on the contrast-enhanced micro-CT image (gray). Right column: Segmentation results. Top row: WBI of case B1. Bottom row: WBI of case B2.
      Table 2 summarizes segmentation performance. The corresponding WBIs, segmentation results, WMIs, and GTs used for performance evaluation are shown in Figures 8 and 9. Total TPR and PPV were 0.735 and 0.889, respectively, which might appear insufficient; however, the performance for thick vessels fared better versus those of thin vessels. TPR and PPV for thick vessels were 0.878 and 0.941, respectively, whereas TPR and PPV for thin vessels were 0.518 and 0.778, respectively. Most vessels were segmented, although some thin vessels could not be segmented (Figures 8 and 9). Although there were many vessels of different thicknesses, the constructed network was able to segment them. Nerves were also segmented because of the slight difference between nerves and thin vessels. It would be difficult to accurately differentiate thin vessels and nerves, even in manual segmentation. However, there were also unsegmented vessels, as indicated. In the WBI of B1, the unsegmented vessel had a zigzag shape rather than being smooth and tubular. In addition, there were some black artifacts on the vessel. Because these vessels were not included in the training data set, these strange-shaped vessels with artifacts could not be segmented. In the WBI of B2, the intensities of the unsegmented vessels were lower than those of the other vessels, and their outlines were unclear in comparison. Updating the training data set is undoubtedly a high-priority task; however, the micro-CT image acquisition and reconstruction methods must also be improved to reduce artifacts and enhance the contrast between tissues.
      Table 2Segmentation Performance
      VariableThick vesselThin vesselTotal
      True positive rate0.878 (144/164)0.519 (56/108)0.735 (200/272)
      Positive predictive value0.941 (144/153)0.778 (56/72)0.889 (200/225)
      Data are given as proportion (n/total n).
      Figure thumbnail gr8
      Figure 8Examples of whole-block images of B1 at various depths. Left column: Micro–computed tomography images. Middle column: Results of segmentation. Middle right column: Whole-mount slide images. Right column: Ground truth. Green arrowheads indicate oversegmentation; orange arrowheads, unsegmented regions. Scale bars = 1 mm.
      Figure thumbnail gr9
      Figure 9Examples of whole-block images of B2 at various depths. Left column: Micro–computed tomography images. Middle left column: Results of segmentation. Middle right column: Whole-mount slide images. Right column: Ground truth. Green arrowheads indicate oversegmentation; orange arrowheads, unsegmented regions. Scale bars = 2 mm.

      Discussion

      One of the limitations of our method was the intensity standardization. Although the micro-CT scanner can three-dimensionally visualize many kinds of objects, such as living things, minerals, and so on, the micro-CT scanner used in this study was still an α version for the biomedical field. Therefore, the intensities of micro-CT image of biomedical materials were unstable, which was the main reason for manually conducting the intensity standardization. Although an automatic calibration method should be introduced in the future, the mean intensity of the vessels for each WTI and WBI were manually determined and all the intensities corrected. In addition, because all intensities were simply divided by the mean intensity, spatial-dependent variation should be considered by introducing nonlinear and local intensity transformation. One of candidate methods for automatic calibration will be an unsupervised feature analysis with convolutional autoencoder.
      • Chen M.
      • Shi X.
      • Zhang Y.
      • Wu D.
      • Guizani M.
      Deep feature learning for medical image analysis with convolutional autoencoder neural network.
      As a preprocessing, by using such method, all small patch images will be roughly categorized into several tissue patterns. Several candidate regions of vessel will be then detected, and intensities of entire WBI will be standardized by using analysis result of intensity distributions and positions of detected candidate regions.
      Second, the network training and performance evaluation was conducted using only small parts of WBIs and area-based metrics. For network training, even for the 25 arbitrary-sized volumes extracted from 10 WBIs, three-dimensional annotation to generate the ground truth was an extremely laborious task, taking at least 8 hours for each volume. To expand the training data set, the number of annotators must be increased and annotation procedure should also be reconstructed efficiently. For the performance evaluation, consecutive WMIs were acquired to confirm vessel regions exactly, because some tissues and vessel are almost indistinguishable on the micro-CT image from visual assessment. Since making many WMIs cost a lot of money, in addition to labor; two WBIs were selected and used for evaluation in this study. In addition, if the pixel-wise accuracy is measured in the entire WBI, hundreds of WMIs and a highly accurate image registration method will be required to find the corresponding WBI slice to WMI and correct the deformation of the tissue sample. These processes would require a long computation time and costly slide preparation.
      As well as vessel segmentation, the purposes of this project include segmentation of tumor lesions, measurement of tumor invasion depth, and identification of metastatic lymph nodes. Although these necessary functions will be implemented, the total time required for analysis could sharply increase, depending on the number of analyses and their complexity. Therefore, it will be necessary to accelerate the analyses. We have empirically determined network parameters, such as depth of the network, the number of convolutional blocks, and the convolution sizes; however, it has not verified that the network structure is sufficiently compact. Network distillation methods have recently been proposed to rebuild compact networks while retaining the prediction accuracy.
      • Hinton G.
      • Vinyals O.
      • Dean J.
      Distilling the knowledge in a neural network.
      • Jimmy L.B.
      • Caruana R.
      Do deep nets really need to be deep?.
      • Zhang Y.
      • Xiang T.
      • Hospedales T.M.
      • Lu H.
      Deep mutual learning.
      To improve the processing efficiency, it will be necessary to apply such distillation methods when all functions are to be integrated.
      Micro-CT image may not completely displace WSI or microscopic observation with the glass slide. However, combining WSI and micro-CT image has great potential to improve the histopathologic diagnosis because 3D morphologic structure can be also evaluated easily without additional resection and tissue exhaustion. Xu et al
      • Xu B.
      • Teplov A.
      • Ibrahim K.
      • Inoue T.
      • Stueben B.
      • Kitabi N.
      • Hameed M.
      • Yagi Y.
      • Ghosein R.
      Detection and assessment of capsular invasion, vascular invasion and lymph node metastasis volume in thyroid carcinoma using microCT scanning of paraffin tissue blocks (3D whole block imaging): a proof of concept.
      mentioned that micro-CT image can detect additional foci of capsular invasion of thyroid carcinoma missed by the initial H&E slide, and volume of nodal metastasis measured on micro-CT image was most accurate compared with the conventional two-dimensional measurement on WSI. In addition, they suggested that among other things, micro-CT has the potential to increase the detection rate of capsular invasion and better define criteria for capsular invasion and vascular invasion. On the other hand, as the initial study, we implemented the vessel segmentation function for micro-CT image by using deep neural network. Although segmentation performance will have to be improved, the resultant images were able to present the entire structure of vessels. The constructed network can extend to segment not only vessels, but also other tissues. If segmentation and detection methods for micro-CT image is enriched, automation of assessment, such as mentioned by Xu et al,
      • Xu B.
      • Teplov A.
      • Ibrahim K.
      • Inoue T.
      • Stueben B.
      • Kitabi N.
      • Hameed M.
      • Yagi Y.
      • Ghosein R.
      Detection and assessment of capsular invasion, vascular invasion and lymph node metastasis volume in thyroid carcinoma using microCT scanning of paraffin tissue blocks (3D whole block imaging): a proof of concept.
      can be achieved.

      Conclusions

      We have developed a deep neural network to analyze WTIs and WBIs of human tissue, using micro-CT. As an initial step, this network was utilized successfully to segment vessels in WTI and WBI samples of colorectal cancer tissue. Despite the large size of the images for analysis (eg, 1000 × 2000 × 200 voxels), the segmentation task was completed in 10 minutes. As we used only one sample of colorectal tissue for the performance evaluation in this proof-of-concept study, the next step is to extend the study to multiple specimens from neoplasms of various organs, including colorectum. At the same time, segmentation performance, particularly for thin vessels, must be improved. To achieve this, securing an annotator to generate the ground truths for both network training and evaluation will be an immediate task. In the future, we plan to expand the functions of neural network to measure and segment structures, such as lymph nodes and focal tumor lesions, with the expectation that these analyses may elucidate patterns of invasion that could inform diagnostic and therapeutic decisions.

      Acknowledgment

      We thank Nikon Metrology NV (Leuven, Belgium) loaning equipment and supporting the micro-CT system.

      References

        • Casero R.
        • Siedleck U.
        • Jones E.S.
        • Gruscheski L.
        • Gibb M.
        • Schneider J.E.
        • Kohl P.
        • Grau V.
        Transformation diffusion reconstruction of three-dimensional histology volumes from two-dimensional image stacks.
        Med Image Anal. 2017; 38: 184-204
        • Gaffling S.
        • Daum V.
        • Hornegger J.
        Landmark-constrained 3-D histological imaging: a morphology-preserving approach.
        in: Vision, Modeling, and Visualization. Geneva, Switzerland, Eurographics Association2011: 309-316
        • Schwier M.
        • Böhler T.
        • Hahn H.K.
        • Dahmen U.
        • Dirsch O.
        Registration of histological whole slide images guided by vessel structures.
        J Pathol Inform. 2013; 4 Suppl: S10
        • Cifor A.
        • Bai L.
        • Pitiot A.
        Smoothness-guided 3-D reconstruction of 2-D histological images.
        NeuroImage. 2011; 56: 197-211
        • Xu Y.
        • Pickering G.
        • Nong Z.
        • Glibson E.
        • Arpino J.M.
        • Yin H.
        • Ward A.D.
        A method for 3D histopathology reconstruction supporting mouse microvasculature analysis.
        PLoS One. 2015; 10: e0126817
        • Roberts N.
        • Magee D.
        • Song Y.
        • Brabazon K.
        • Shires M.
        • Crellin D.
        • Orsi N.M.
        • Quirke R.
        • Quirke P.
        • Treano D.
        Toward routine use of 3D histopathology as a research tool.
        Am J Pathol. 2012; 180: 1835-1842
        • Teplov A.
        • Tabata K.
        • Fu X.
        • Uraoka N.
        • Roehr H.A.
        • Ntiamoah P.
        • Humm J.L.
        • Sirintrapun S.J.
        • Murray M.P.
        • Shia J.
        • Travis W.D.
        • Brogi E.
        • Hameed M.
        • Yagi Y.
        Development of standard operating procedure (SOP) of micro-computed tomography (micro-CT) in pathology.
        Diagn Pathol. 2019; 5https://doi.org/10.17629/www.diagnosticpathology.eu-2019-5:273
        • Ketsamenis O.L.
        • Olding M.
        • Wamer J.A.
        • Chatelet D.S.
        • Jones M.G.
        • Sgalla G.
        • Smit B.
        • Larkin O.J.
        • Haing I.
        • Richeldi L.
        • Sinclair I.
        • Lackie P.M.
        • Schnider P.
        X-ray micro-computed tomography for nondestructive three-dimensional (3D) X-ray histology.
        Am J Pathol. 2019; 189: 1608-1620
        • Troschel F.M.
        • Gottumkkala R.V.
        • DiCorpo D.
        • Mario J.
        • Ott H.C.
        • Wright C.D.
        • Muniappan A.
        • Lanuti M.
        • Yang K.
        • JO Shepard
        • Nardi V.
        • Michaelson J.S.
        • Hariri L.P.
        • Fintelmann F.J.
        Feasibility of perioperative micro-computed tomography of human lung cancer specimens.
        Arch Pathol Lab Med. 2019; 143: 319-325
        • Albers J.
        • Markus M.A.
        • Alves F.
        • Dullin C.
        X-ray based virtual histology allows guided sectioning of heavy ion stained murine lungs for histological analysis.
        Sci Rep. 2018; 8: 7712
        • Tanabe N.
        • Vasilescu D.M.
        • Kirby M.
        • Coxson H.O.
        • Verleden S.E.
        • Vanaudenaerde B.M.
        • Kinose D.
        • Nakano Y.
        • Paré P.D.
        • Hogg J.C.
        Analysis of airway pathology in COPD using a combination of computed tomography, micro-computed tomography and histology.
        Eur Respir J. 2018; 51: 1701245
        • Mai C.
        • Verleden S.E.
        • McDonough J.E.
        • Willems S.
        • Wever W.D.
        • Coolen J.
        • Dubbeldam A.
        • Raemdonck D.E.V.
        • Verbeken E.K.
        • verleden G.M.
        • Hogg J.C.
        • Vanaudenaerde B.M.
        • Wuyts W.A.
        • Verschakelen J.A.
        Thin-section CT features of idiopathic pulmonary fibrosis correlated with micro-CT and histologic analysis.
        Radiology. 2017; 283: 252-263
        • Xu B.
        • Teplov A.
        • Ibrahim K.
        • Inoue T.
        • Stueben B.
        • Kitabi N.
        • Hameed M.
        • Yagi Y.
        • Ghosein R.
        Detection and assessment of capsular invasion, vascular invasion and lymph node metastasis volume in thyroid carcinoma using microCT scanning of paraffin tissue blocks (3D whole block imaging): a proof of concept.
        Mod Pathol. 2020; 33: 2449-2457
        • Wollatz L.
        • Johnston S.J.
        • Lackie P.M.
        • Cox S.J.
        3D histopathology—a lung tissue segmentation workflow for microfocus X-ray-computed tomography scans.
        J Digit Imaging. 2017; 30: 772-781
        • Zhao F.
        • Gao P.
        • Hu H.
        • He X.
        • Hou Y.
        • He X.
        Efficient kidney segmentation in micro-CT based on multi-atlas registration and random forests.
        IEEE Access. 2018; 6: 43712-43723
        • Lee J.
        • Beighley P.
        • Ritman E.
        • Smith N.
        Automatic segmentation of 3D micro-CT coronary vascular images.
        Med Image Anal. 2007; 11: 630-647
        • Chang S.
        • Cheng F.
        • Hsu W.
        • Wu G.
        Fast algorithm for point pattern matching: invariant to translations, rotations and scale changes.
        Pattern Recognit. 1997; 30: 311-320
        • Ronneberger O.
        • Fischer P.
        • Brox T.
        U-Net convolutional networks for biomedical image segmentation.
        2015 (arXiv:1505.04597v1)
        • Milletari F.
        • Navab N.
        • Ahmadi S.A.
        V-Net fully convolutional neural networks for volumetric medical image segmentation.
        2016 (arXiv:1606.04797v1)
        • Çiçek Ö.
        • Abdulkadir A.
        • Lienkamp S.S.
        • Brox T.
        • Ronneberger O.
        3D U-Net: learning dense volumetric segmentation from sparse annotation.
        2016 (arXiv:1606.06650v1)
        • Szegedy C.
        • Loffe S.
        • Vanhoucke V.
        • Alemi A.
        Inception-v4, inception-ResNet and the impact of residual connections on learning.
        2016 (arXiv:1602.07261v2)
        • Ioffe S.
        • Szegedy C.
        Batch normalization: accelerating deep network training by reducing internal covariate shift.
        2015 (arXiv:1502.03167v3)
        • Maas A.L.
        • Hannun A.Y.
        • Ng A.Y.
        Rectifier nonlinearities improve neural network acoustic models.
        Proc ICML. 2013; 30
        • Chen M.
        • Shi X.
        • Zhang Y.
        • Wu D.
        • Guizani M.
        Deep feature learning for medical image analysis with convolutional autoencoder neural network.
        IEEE Trans on Big Data. 2017; https://doi.org/10.1109/TBDATA.2017.2717439
        • Hinton G.
        • Vinyals O.
        • Dean J.
        Distilling the knowledge in a neural network.
        2015 (arXiv:1503.02531v1)
        • Jimmy L.B.
        • Caruana R.
        Do deep nets really need to be deep?.
        2014 (arXiv:1312.6184v7)
        • Zhang Y.
        • Xiang T.
        • Hospedales T.M.
        • Lu H.
        Deep mutual learning.
        2017 (arXiv:1706.00384v1)