Data augmentation for medical image analysis in deep learning

Deep learning in general, but particularly in medical imaging, requires a large amount of training data in order to obtain good performance and avoid overfitting. To meet these challenges, increasing the quantity of training data is a common solution. It allows a small dataset to be rebalanced or enriched  for any reason (time-consuming manual annotations, lack of accessible data...).

The augmentation techniques used must make sense with respect to the type of analysis desired and therefore positively influence the performance of the model during the learning phase: by applying a large number of augmentations, the performance will not necessarily be better. There are several types of transformations for medical images, acquired by computed tomography (CT) or magnetic resonance (MRI). These can be performed on the whole dataset or applied according to certain probabilities of occurrence.

They can be classified into different categories according to the type of transformation applied. 

Geometric transformations

1. Rotation

This transformation, as its name indicates, consists of rotating the original image according to a desired angle. In medical image analysis, it is common to augment a dataset with random rotations at different angles ranging from 10° to 175° [1] or from -15° to +15° as well as multiples of 45° [2].

2. Flips

The image flips are performed along an axis of symmetry. For medical image enhancement, they can be performed vertically as well as horizontally, because images can be acquired in supine or prone position, and contain anatomical variations (e.g. situs inversus). Therefore, an organ, whatever its location in the body, will always be the same organ. Contrary to the case of natural images where, depending on the type of object represented, vertical reflections (e.g. a cat) are of no real interest, because the reflected image is considered unrealistic. [1]

3. Shift

This transformation can be performed along the x and/or y axis randomly [3][4]. The transformed image keeps the same orientation as the original image and is moved in the applied direction.

4. Crop and zoom

Both methods have the same visual result, some parts of the image are cropped to keep only a part of it.
When cropping, the image dimensions will be reduced (e.g. cropping a 512x512 image to a 250x250 image) and the aspect ratio is not necessarily maintained.
When zooming, the crop retains the same aspect ratio as the original image. The image is resized identically to the original image. To do this, the pixels created to recover the original dimensions have a value interpolated from a function defined before applying the zoom to the original image (e.g. the average brightness of the neighboring pixels). As a result, a loss of quality can be observed. [5][6]
This type of transformation is usually applied randomly to the dataset. [7]

 

5. Crop with padding

In the same way as in the previous section, a crop is performed but in order to restore/retain the dimensions of the original image, a fill is performed (with a pixel value of 0) around the cropped image.

6. Shear

Horizontal shearing (along the x-axis) and/or vertical shearing (along the y-axis) is an affine transformation that consists of shifting in opposite directions the top and bottom of the image (horizontal shearing) or the right and left of the image (vertical shearing). Unlike the previous methods, the image is distorted [8]. Shearings performed on medical images have for example parameter values ranging from 0.1 to 0.35 [1] or with an x/y ratio of up to 15% [9].

Miscellaneous transformations

1. Adding noise

Noise is a random variation in brightness within the image. It degrades the quality of the original image and can be of several types: white Gaussian noise, Poisson noise, impulse noise (salt and pepper). [10]

2. Gaussian filter

This convolution filter has a kernel of Gaussian shape. It scans the entire original image to denoise or blur it, by applying a Gaussian function to the pixels of the image. This one realizes a weighted average of the convolved pixels by granting a more important weight to the pixels near the center of the kernel (contrary to an average filter where each pixel present in the kernel has the same weight). [11]

3. Contrast

The contrast of an image is increased when the darker pixels of this image are darkened and the lighter pixels are lightened: a contrasted image will therefore contain a greater quantity of black and white (see histogram of the example below). You can see the increase in contrast on the histogram of the image, because the gap between the brightest pixels and darkest pixels of the image is greater: the histogram is more spread out, as can be seen when we go from the histogram of the image (b) to the histogram of the image (a). The contrast of an image corresponds to this spread. [12] Thus when the contrast continues to be increased, the number of white and black pixels contained in the image also increases, as can be seen by comparing the histograms of images (a) and (c). The contrast variations performed for data augmentation from medical images can range from 0.8 to 1.2, for example, and are often applied randomly to the dataset. [2]

4. Brightness

In the HSV (Hue, Saturation, Value) perceptual space, grayscale images have a zero value of saturation and no specific hue. These images are value dependent, i.e. the lower the value of the latter, the darker the image. In order to increase the size of the data set in medical imaging, brightness variations belonging to the interval [-0.1; 0.1] can be applied randomly. [2]

5. Sharpening

This transformation increases the contrast at the edges of the image. Light and dark voxels located in the same area will be lightened and darkened after this transformation. The application is performed with a certain radius around the edge pixel, it corresponds to the number of pixels surrounding this target pixel and the larger the radius, the greater the accentuation of the edges. The same is true for the noise present in the image if it isn’t attenuated beforehand [13]. By focusing on the boundaries between regions within the image and not on the overall contrast, the image appears sharper.

Context of application

Medical image analysis by deep learning covers a multitude of pathologies and anatomical structures (lungs, kidneys, liver, brain...). This important diversity can also be seen in the imaging modalities on which research in this field is carried out, where one finds 3D imaging (magnetic resonance imaging, tomodensitometry, positron emission tomography...), 2D slices (ultrasound...) or 2D projections (mammography, standard radiography...). 
The set of augmentation methods presented is equally well used for medical image classification, detection or segmentation tasks. However, we note that some image transformation methods are more popular than others, such as rotations, reflections, shifts and noise addition. This popularity can be explained by their influence on the speed of convergence during training and on the performance of the model. Better accuracy is achieved with the application of each of the following augmentations: rotations, shears, Gaussian filters and reflections [1]. If one or more of these augmentation techniques are applied appropriately on the training set, then the model performance will improve over that obtained from a dataset without augmentation. These transformations can be applied to the whole training data set in a homogeneous way, or have a random occurrence frequency, and/or applied to different parameter values (e.g. rotations that can vary from -15° to 15°).

What We Keep?

There are a multitude of data transformation and augmentation methods for medical image analysis in deep learning, a large part of them has been presented previously. But it's not enough to apply all of them mechanically to generate a better performing model, it will depend on the context of the image and the problem to solve. To our knowledge, some transformations have proven to be more popular than others. This is partly due to their positive influence on model performance. In recent years, a new method of data augmentation has become more and more popular through the use of generative adversarial networks (GAN). This type of algorithm is based on the unsupervised learning method. GAN is of particular interest in the field of medical imaging where it can be used as a data augmentation in deep learning by generating new data similar to the training data and more realistic. The generation of these new data is possible thanks to the competition of two neural networks: one (generator) generating plausible images and the other (discriminator) having as an objective to distinguish the generated images from the real images. In this way, a large number of new data and their respective annotations can be generated, which saves a lot of time during data preparation.  [2] [4]

When comparing data augmentation techniques for medical image analysis with those for natural image analysis, there is a strong similarity in the choice of methods used. Geometric transformations (rotations, horizontal reflections, cropping, shifting), the addition of random noise, gamma correction and the use of GANs are also found in this field of analysis [14] [15] [16]. Contrary to medical images, it’s possible to apply additional transformations influencing the color such as the modification of the saturation or the hue of the natural image [17]. Regardless of the type of image analyzed, data augmentation is applied in order to improve the performance of the model and avoid overfitting.