Detection and segmentation in medical imaging: types of deep learning models

In the field of medical image analysis, object detection and segmentation is an important discipline. This article introduces the types of models commonly used for this purpose.

Image classification is a global classification of the pixels of an image. That is to say that objects are identified from all the pixels of the image, but are not located. Object detection and segmentation are, on the other hand, an individual classification of the image’s pixels, where the objects present in the image are located. This distinction is crucial when choosing the architecture of the machine learning model, since the detection and segmentation tasks require the generation of Regions Of Interest (ROI) containing the objects. In medical imaging, these objects can correspond to anatomical structures (e.g. organs) or anomalies (e.g. pulmonary nodules).

Object detection and segmentation models are mainly based on 4 types of approaches:

1. the proposal of pixel regions
2. the single-shot with a global classification/regression of the image
3. latent representation
4. visual saliency maps, the latter is less commonly used, but is well suited for modeling visual attention [1].

The candidate region proposal approach

This approach is based on the use of Region Proposal Networks (RPN). This type of model was proposed by [2] to meet the need for individual classification of pixels in the image that involves the detection of objects.
From a convolutional neural network (CNN), a region proposal algorithm is added between the image feature extraction and image classification layers. The one proposed by [2] is composed of 3 modules:

1. application of a selective search algorithm to the image to obtain a set of feature maps representing the candidate image regions for object detection;
2. deformation and cropping of the proposed image regions, to obtain the input data for a CNN that computes the probability that a candidate region contains an object, and generates the anchor box of those whose probability is close to 1;
3. regression by a Support Vector Machine (SVM) of the anchor boxes estimated by the CNN against those representing the target values (i.e. the object classes). The anchor boxes are thus classified according to the type of object they contain.

Selective Search algorithm

The first module of the region proposal algorithm analyzes the image using a selective search algorithm [3]. It allows the extraction of a set of region proposals that may contain an object. By using selective search, the method of detection by region proposals is similar to that of visual saliency maps (developed later in the article). 
The selective search algorithm considers the image as a hierarchical graph. It is iteratively segmented by a superpixel algorithm [4] using 5 similarity criteria. The number of iterations is conditioned by n, the number of expected region proposals defined a priori (in [2], n = 2000).

The similarity criteria used are:

1. color,
2. texture,
3. size,
4. shape,
5. a global similarity indicator which is a linear combination of the 4 previous measures. [5]

The segmented regions and their anchor box obtained by the selective search are then proposed to the second module of the region proposal algorithm.

Image pyramid method 

Instead of selective search, the region proposal algorithm can also use the image pyramid method. With the latter, the image is convolved by filters at different scale levels (i.e. pyramid layers) to detect objects of different sizes. However, the image pyramid method is costly in processing time, and its performance is sensitive to variations in the scale of the pyramid layers as well as to the size of the applied filters.

There are several types of pyramid architectures:

1. featurized image pyramid, where an image pyramid is used to build a pyramid of feature maps. The feature maps are computed independently on each scale level of the image, and allow the computation of object detection predictions for each of them;
2. single feature map, where a single scale feature map, rather than an image pyramid, is used to compute object detection predictions at the highest level of abstraction of the convolutional neural network;
3. pyramidal feature hierarchy, where a single-scale feature map is used to compute object detection predictions at each level of abstraction of the convolutional neural network;
4. feature pyramid network, where object detection predictions are computed independently at each level of abstraction of the feature pyramid.

“Single-Shot” Detection approach

Unlike RPN models, single-shot models define object detection as a global regression problem of the image pixels. In order to reduce the processing time, they do away with the region proposal module. They directly analyze the image with a CNN to extract the bounding boxes of the detected objects, and classify them. They are mainly used for real-time object detection.
For this, a set of bounding boxes is predefined, either manually or by a clustering algorithm from the training dataset. During the image analysis using the CNN, the feature maps are gridded. For each cell in the grid, the probability of containing an object is calculated. Each cell center whose probability is close to 1 is considered as the potential center of a predefined bounding box. When regressing the predefined bounding boxes with the target values, only the bounding boxes with a good similarity measure (e.g. Jaccard index) are selected as final values.

Recent work in medical imaging detection uses both RPN and global (i.e., single-shot) classification/regression architectures, and combinations of architectures are also employed [7][8][9][10][11].

Latent representation approach

Latent representation models can be used for the segmentation of regions of interest. A latent variable is deduced from the observation of other variables. Also, the latent representation of the image allows the model to learn the discriminative features of the regions of interest. In medical image analysis, segmentation tasks are common [13], hence their mention in this article. These tasks are different from detection tasks. Indeed, they lead to the generation of binary masks that require a higher accuracy than with bounding boxes. Auto-Encoder architectures use latent representations, one of the best known in the medical imaging domain is U-Net [14]. This model architecture was designed to address the challenges of biomedical image segmentation. We will come back to this in a future article.

Salience map approach

Visual saliency maps methods are machine learning models that initially do not use deep neural networks as their architecture. These models integrate two joint methods. The first one, "bottom-up", is the study of the local characteristics of the image to detect the object present in the majority. This one appears and automatically captures the attention because of its salience (for example, a red tomato among green apples). The RARE algorithm is a good example of an exclusively bottom-up architecture.

The second one, "top-down", is the addition of a priori information (e.g. a binary segmentation mask) as input to the algorithm, allowing the exclusion of regions of the image that should not be considered as salient [15]. [16] proposes a comparison of salient models applied to medical imaging. The best ranked models have different detection approaches: Bayesian approach, decision theory approach, graphical approach, pattern recognition approach. In the field of automatic medical image analysis, this type of model is favored for anomaly detection. Currently, in medical image segmentation, latent representation models are preferred to salient models. However, visual saliency maps are still preferred a priori and a posteriori in the analysis of medical images by deep learning models. In the first case, they add additional information during supervised learning [15]. In the second case, they provide an explanation of the model's decision [17], whose process is still seen as a black box by radiologists.