Automatic cell nuclei segmentation in histopathology images using boundary preserving guided attention based deep neural network

Abstract

Precise cell nucleus segmentation is very critical in many biologically related analy- ses and disease diagnoses. Two of the major challenges in this task are the precise delineation of the small-shaped nucleus and the characterization of the edge region. In this thesis, deep neural network based nucleus segmentation methods are proposed where a unique idea of generating various types of boundary aware guiding signal is introduced to guide the spatial information of the segmentation architectures through attention mechanism.

In the first method, the attention module of the segmentation network is guided by a separate boundary extractor shallow encoder-decoder network which minimizes a sep- arate objective function from a synthetically generated nucleus edge mask. The edge aware information found from different decoder stages of this shallow network acts as the guiding signal. Although the network offers comparatively better segmentation per- formance with respect to the other network, the complexity of the network increases because of the utilization of separate shallow network.

Considering the inspiring characteristics of various transformation techniques, a con- tourlet driven attention network, namely ConDANet, is developed which utilizes con- tourlettransformedsignalastheguidingsignal. Thecontourlettransformbasedcontrol- ling signal generation scheme exploits the advantage of the multi-scale time frequency localization and provides a high degree of directionality. Additionally, the wavelet pool- ing strategy is incorporated to the network which preserves the textural content of the nucleus.

Furthermore, a boundary aware wavelet guided network (BAWGNet) is proposed which utilizes the wavelet transform based guiding signal generation along with three separate loss function for optimization purpose. A boundary aware unit is designed that captures thenuclei’s boundaryinformationbyemploying auniqueboundaryaware lossfunction, ensuring accurate prediction of the nuclei pixels in the edge region.

The proposed method is employed for analyzing three publicly available histopathology datasets to manifest its effectiveness. Using the proposed framework, significant perfor- mance improvement is found over the other state-of-the-art techniques while evaluating on these datasets.