Deep learning algorithms for the detection of objects (abnormality, anatomical structure) in medical imaging require datasets annotated with precision to achieve acceptable results for medical use. This annotation is very time-consuming and requires the expertise of radiologists. To facilitate this step, and to minimize the annotation time, the user will need an annotation tool which:
- takes into account DICOM files, to simplify the processing of data from the care facility’s PACS (Picture Archiving and Communication System),
- provides all the necessary tools for manual detection by the radiologist, of the region of interest (ROI), and
- allows the user to simply export these annotations in a format compatible with the data processing pipeline of a deep learning algorithm.
Our current medical image analysis project uses deep learning object detection algorithms. After defining the annotation formats required for this type of task (cf. article Medical image annotations for deep learning), we were able to begin the data preparation phase. During the review, we searched for a free DICOM image annotation software to improve our dataset with the corresponding ROI annotations. For our project, we have selected a few tools proposed by [1] [2], presented below. Others were excluded (ePad [3], XANAT [4], GeoS, MammoApplet, etc.), for one or more of the following reasons:
- operated via virtual machines, or directly integrated with medical image collection and storage tools,
- does not support DICOM or NIfTI medical data formats,
- is no longer free,
- not accessible/maintained.
Which free sofware to segment medical images?
DICOMworks:
We have already used this software [5] for anonymizing DICOM files. It offers, in addition, annotation tools such as tags and measures: dot, angle, arrow, circle, free form, text, distance measurement, gap measurement.
Annotations can be copied from one image to another in the series.
Pros: it also offers an anonymization tool, which avoids the need to use several tools to carry out the preparation of the data of an image analysis project, and numerous export formats (JPEG, PNG, DICOM, TIFF, BMP, etc) are available.
Cons: does not allow the user to create binary masks, few manual or semi-automatic segmentation tools.
ITK-Snap:
This software is an open-source collaborative project between the University of Pennsylvania and Utah. It allows segmentation of 3D medical images [6].
The set up is easy. Once the software is launched, the user has access to the startup guide, previously loaded images and previously saved workspaces. He can also directly load a new image/DICOM file, or a new workspace.
Once the series of images opened, the user interface consists of two main areas: the annotation tools management panel and the viewing area in the center.
The side panel is the user’s toolbox from which he can:
- select the desired annotation tool and configure it,
- create/modify the labels of the annotations performed, and
- manipulate the 3D rendering.
The viewing area is divided into 4 views: one for each axis (axial, sagittal, coronal), and one for the 3D rendering of the active segmentation.
The manipulation, annotation and segmentation tools available are:
- crosshair mode, which enables the user to position the 3D cursor in the 3 orthogonal planes,
- zoom,
- marking in the form of lines, text, distance and angle measurements,
- paintbrush or polygon tools for manual segmentation, and
- semi-automatic segmentation by actively selecting the object’s contours with the snake tool.
Once the segmentation is completed, the user can obtain volume and statistical information related to the segmented object.
The user can also change windowing values, access the series’ metadata and segmented object information, using the Layer Inspector.
ITK-snap has an interpolation tool to label an object in the 3 orthogonal planes.v
Finally, the user can export and save the segmentation in several formats, including the most common in image processing: NIfTI, NRRD, VTK (the latter is used to obtain a meshing of the surface).
Pros: the software is very complete and easy to use. The tools are designed specifically for medical data formats, making it specifically suitable for segmenting medical images.
Cons: none, according to us. However, it does not support bounding box annotations.
Some recent scientific papers using ITK-Snap for CT image segmentation [7] [8] [9] [10].
RIL-contour:
This software was developed by the Radiological Computing Laboratory of Mayo Clinic, Rochester, United States [11]. It is installed using the command line with the Anaconda IDE. To launch it, the user must also use the command line. Both the installation and the launch are easily done with minimal experience of the shell. Once opened, RIL-contour consists of three elements: the navigation panel and loading datasets, the segmentation/image manipulation toolbar, and the viewing area.
The first thing to do is to create a project, defining the directory containing the dataset and the directory where the segmentations will be saved.
The data is automatically converted to NIfTI before being loaded into this new project and can be displayed in the viewing area. The user can change the windowing of the series in order to facilitate the identification of anatomical structures.
Prior to segmentation, ROI labels must be defined.
During the segmentation using the brush, a bounding box is defined around the area being treated. Once completed, the user can lock the ROIs. This option is very convenient as it avoids colouring outside the lines when segmenting neighbouring ROIs (see screenshot below).
The user can visualize the result of the segmentation in 3D.
RIL-contour provides ROI statistics by volume or slice.
The software also offers the use of machine learning models for segmentation or classification. The user can load its own models and then apply them to the dataset. We were unable to test this functionality because we did not have segmentation models available.
Annotations can be exported in NIfTI format.
Pros: easy to install, video tutorials to get started, user friendly segmentation tools, the user can load segmentation algorithms for automatic processing, use of the main medical data formats (DICOM, NIfTI).
Cons: the tool is very extensive, the user needs to take the time to go through the menus to unlock certain features or functionalities. Absence of markup tools (i.e. line, free form, etc...).
Some recent scientific articles using RIL-contour for segmentation [12] [13] [14].
Sefexa:
Sefexa is a segmentation software for 2D images [15]. It allows the processing of the following formats: PNG, GIF, TIFF, JPEG, BMP. Unfortunately, it does not support DICOM or NIfTI formats. Nevertheless, we will briefly introduce it, as the data sets for medical image analysis projects are sometimes saved in more "classic" image formats.
Sefexa is simple to install and user friendly. It allows the user to do:
- manual segmentation with free plotting, and
- semi-automatic segmentation
- either by thresholding pixel intensities, or
- by using the graph-cut approach, identifying with various brush strokes the different objects in the image (see screenshot below).
Pros: user friendly.
Cons: does not support DICOM or NIfTI formats. It only offers 2D segmentation.
Some recent scientific articles using Sefexa for segmentation [16] [17].
3D Slicer:
3D Slicer is an open source software dedicated to medical image processing as well as three-dimensional visualization [18].
After installation, when the software is started, the user can import DICOM files. The user can also find, in the drop-down menu located in the toolbar at the top, the different modules offered by 3D Slicer, including the "Segment Editor" which allows the user to perform segmentations.
Once the images have been opened, the user can go to the "Segment Editor" module and access the segmentation tools located in the side panel.
In this side panel, the user can:
- create new segments,
- access all the segmentation tools,
- rename and assign colors to each segment, and
- display the 3D visualization.
The viewing area, located in the central part of the screen, allows the user to visualize together different slice views, axial, coronal and sagittal, as well as the 3D visualization. In this area, one can use the different mouse shortcuts to zoom in, move the slices and navigate between the slices. Also, by activating the modification of the windowing, the user can adapt it as he wishes with the mouse.
The tools available in the "Segment Editor" module are divided into 3 categories:
1. Tools for manual segmentation as
a. the brush,
b. the drawing tool,
c. the eraser,
2. Tools for semi-automatic segmentation as
a. level (intensity) tracking,
b. grow from seeds,
c. thresholding,
d. scissors,
e. islands,
3. Complementary tools for segmentation as
a. fill between slices (interpolation),
b. filters,
c. logical operators.
Once the segmentations have been performed, the user can access different statistical data from the "Quantification" module, followed by "Segment Statistics". These data are displayed in table form, and contain volume information as well as other statistics related to the segmented anatomical structure.
The export of the segmented data is done in NRRD format, and the table with volume and statistical information is saved in TXT, TSV and CSV formats.
Finally, it may be interesting to mention that from the "Markups" module the user can also use bounding boxes (by positioning a region of interest noted ROI) to perform annotation. This data can be exported in JSON format.
Pros: Very complete software with a large number of tools (manual and semi-automatic) allowing the segmentation of medical images (DICOM format). The presence of additional tools for segmentation can facilitate this step. Data export is done in NRRD format, thus compatible with data analysis projects.
Cons: Software not solely dedicated to segmentation and may therefore require some time to adapt.
Comparative table of free annotation software
We evaluated the software presented using the following criteria:
- supported image formats suitable for medical imaging, in particular with DICOM or NIfTI formats,
- plenitude of the annotation tools,
- ease of installation and use, and
- formats available for exporting annotations, notably compatible with a deep learning pipeline.
Rating: 0 criterion not met, 5 criterion met, NA criterion not applicable (e.g. impossible to evaluate the tool according to this criterion)
| Software | Criterion 1 | Criterion 2 | Criterion 3 | Criterion 4 |
| DICOM works | 4 Only DICOM, no NIfTI format | 2 Offers markup tools only | 5 | 5 |
| ITK-Snap | 5 | 4 Exclusively dedicated to the segmentation task | 5 | 5 |
| RIL-contour | 5 | 5 | 4 Anaconda IDE needed and the user needs to know how to use the “ command prompt” | 4 Only exports to NIfTI format |
| Sefexa | 1 Only GIF, TIFF, PNG, JPG, no medical image formats such as DICOM/NIfTI | 3 Only 2D segmentation tools available | 5 | 2 Only exports to PNG format |
| 3D Slicer | 5 | 5 | 4 Requires time to get the hang of the software | 5 Exports to NRRD forma |
To conclude, what is the best software for annotating medical images?
To train deep learning detection algorithms in medical imaging, the most common way is to include, with the CT image, additional information to restrict the search space, such as bounding boxes, binary masks, anchor points. To do this, the data preparation phase must include an image annotation step. Among the softwares identified and tested for our project, two have been selected: ITK-snap and 3D slicer (see comparison for more details). These software have been designed for the annotation of medical images using the most common formats in this field: DICOM, NIfTI. They both present the markup and semantic segmentation tools commonly used in preparing data for AI; namely the region, border and threshold search tools. For more details about these tools, read the article "Medical image annotations for deep learning".
[1] Aryanto KY., Oudkerk M., van Ooijen PM. Free DICOM de-identification tools in clinical research: functioning and safety of patient privacy. Eur Radiol. 2015;25(12):3685-3695. doi:10.1007/s00330-015-3794-0
[2] Chehab K., et al. Study of Healthcare Professionals’ Interaction in the Patient Records Based on Annotations, Conference Proceedings, The Impact of Digital Technologies on Public Health in Developed and Developing Countries, 2020, Springer International Publishing, Cham, 978-3-030-51517-1, 316-328, https://doi.org/10.1007/978-3-030-51517-1_28
[3] Rubin DL., Ugur AM., Altindag C., Alkim E. ePAD: An Image Annotation and Analysis Platform for Quantitative Imaging. Tomography 2019;5(1):170–183. https://epad.stanford.edu/
[4] https://www.xnat.org/ https://bitbucket.org/icrimaginginformatics/ohif-viewer-xnat-plugin/src/master/
[5] Puech PA, Boussel L, Belfkih S, Lemaitre L, Douek P, Beuscart R. DicomWorks: software for reviewing DICOM studies and promoting low-cost teleradiology. J Digit Imaging. 2007;20(2):122-130. https://doi.org/10.1007/s10278-007-9018-7
[6] Yushkevich PA. et al. User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage. 2006 Jul 1; 31(3):1116-28.
[7] Veeraraghavan H., Friedman C.F., DeLair D.F. et al. Machine learning-based prediction of microsatellite instability and high tumor mutation burden from contrast-enhanced computed tomography in endometrial cancers. Sci Rep 10, 17769 (2020). https://doi.org/10.1038/s41598-020-72475-9
[8] Löffler MT. et al. A Vertebral Segmentation Dataset with Fracture Grading
Radiology: Artificial Intelligence 2020 2:4
https://doi.org/10.1148/ryai.2020190138
[9] Zhang Y., et al. A Deep Learning Framework for Pancreas Segmentation with Multi-atlas Registration and 3D Level-set, Medical Image Analysis, 2020, 101884, ISSN 1361-8415, https://doi.org/10.1016/j.media.2020.101884
[10] Xia Y. et al. 3D semi-supervised learning with uncertainty-aware multi-view co-training, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 3646-3655
[11] Philbrick KA., Weston AD., Akkus Z. et al. RIL-Contour: a Medical Imaging Dataset Annotation Tool for and with Deep Learning. J Digit Imaging. 2019 Aug;32(4):571-581. https://doi.org/10.1007/s10278-019-00232-0 PMID: 31089974; PMCID: PMC6646456.
[12] Weston, A. D., Korfiatis, P., Philbrick, et al. Complete abdomen and pelvis segmentation using U‐net variant architecture. Medical physics. (2020).
[13] Cai JC., Akkus Z., Philbrick KA. et al. Fully Automated Segmentation of Head CT Neuroanatomy Using Deep Learning. Radiology: Artificial Intelligence, 2(5), e190183. (2020).
[14] Buciuc M., Conte GM., Scharf E. Endocannabinoid response in acute ischemic stroke: elevated 2-arachidonoylglycerol. medRxiv. (2020).
[15] http://www.fexovi.com/sefexa.html
[16] Bhowmik MK., Nath N., Datta A., Ghosh AK. Shape feature based automatic abnormality detection of cervico-vaginal pap smears. In 2nd International Conference on Biomedical Imaging, Signal Processing (ICBSP 2017), published by Journal of Image and Graphics (JOIG), New Jersey, USA. (2017, December).
[17] Edalati-rad A., Mosleh M. Improving Brain Tumor Diagnosis Using MRI Segmentation Based on Collaboration of Beta Mixture Model and Learning Automata. Arabian Journal for Science and Engineering, 44(4), 2945-2957. (2019).
[18] 3D Slicer. https://www.slicer.org/
