Densely connected U-net and auto-encoder based multi-task learning framework for nuclear segmentation in histopathological images

Abstract

Histopathological image analysis provides quantitative profiles of diseased tissues and cells that enable professionals to identify cell alterations and find the actual cause of a patient's illness. Detection and segmentation of cell nuclei are some of the essential steps in histopathological image analysis to diagnose and prognose various medical conditions like cancer. However, low-contrast of histopathological images and lack of sufficient supervised datasets affect the segmentation performance adversely. A number of deep learning networks have been developed for nuclei segmentation, but these networks are complex with large numbers of parameters which require huge computational capacity and longer training and testing time. U-Net is one of the extensively used networks for medical image segmentation. In this thesis, we choose the U-Net as baseline and adapt it in three ways. First, we modify the decoder of the U-Net by using residual-inception connections and dilated convolutions for feature reconstruction and eliminate skip connections which reduces the parameter size. In the second approach, we change the encoder of the U-Net by replacing the regular convolution layers with densely connected convolutional blocks. In the dense convolution block, each convolution layer has all the feature maps from its preceding layers which ensure maximum information flow causing a 10% reduction in validation loss compared to the U-Net. After that, we place auxiliary segmentation blocks prior to pooling layers in the encoder of the densely connected U-Net in order to minimize the loss incurred for downsampling operation. These blocks ensure maximum feature exploitation, provide additional supervision signals and increase the gradient signal. We achieve 77.19% dice coefficient and 62.86% Jaccard index with validation loss 0.088 for this model. It is demonstrated that each of the U-Net variants outperform the U-Net with smaller parameter space in the TNBC dataset for dice coefficient and Jaccard index. We evaluate the performance of a preprocessing technique termed as contrast limited adaptive histogram equalization (CLAHE) to mitigate the adverse effect of low contrast. It is found that the U-Net performs better on the preprocessed dataset compared to the raw dataset. Finally, we propose a shallow auto-encoder based segmentation network with only 35105 numbers of parameters which was trained in a multi-task learning framework by optimizing four loss functions with the help of a pre-trained auto-encoder. This small segmentation network performs segmentation with 69.63% dice coefficient and 53.41% Jaccard indexwhichare comparable to that of U-Net, but there is still a lot of room for improvement in this network.