The hidden value of uncertainty in medical segmentation

Abstract

The pipeline of most segmentation algorithms begins and ends with a curated dataset. In medical imaging, these datasets will be the result of significant effort from clinicians, with each contour representing the result of a sequence of informed decisions. With so much effort and experience involved in the creation of these datasets, we must consider what information about annotation uncertainty is lost over the course of the pipeline. Annotation uncertainty is an inherent property of any dataset containing expert interpretations, and is especially important to quantify in a safety-critical domain such as medical imaging.

In this Perspective, we examine the consequences of losing information concerning annotation uncertainty, not only for dataset representation, but also for the reliability of model evaluation. We describe this as annotation-induced evaluation uncertainty: variation in reported model performance arising from the particular annotation used for evaluation, when multiple distinct expert annotations may be plausible. By contrast, considerable research attention has been given to model uncertainty over the last decade, including the development of probabilistic representations for predicted segmentations that allow nuanced spatial information to be included in uncertainty quantification. Comparable representations for annotation uncertainty should complement these developments, enabling evaluation to account for uncertainty not only in the prediction, but also in the annotation. This would allow more reliable and informative evaluations of medical segmentation models.