Segmentation foundation models matching inter-rater variability on unseen anatomies, pathologies and scanners

• Specialist models remain strong in 3D. Isensee et al. ("nnU-Net Revisited", arXiv:2404.09556) found that CNN U-Nets in the nnU-Net framework, scaled, outperform transformer and Mamba architectures, and that many claimed superiority results do not survive rigorous validation.
• Promptable foundation models. MedSAM (Nature Communications 2024) trained on about 1.57 million image-mask pairs across 10 modalities and was evaluated on 86 internal and 60 external tasks, reporting better accuracy and robustness than modality-specific specialists. Later systems (MedSAM2, BiomedParse, nnInteractive, SAM-Med3D) extend to 3D and text prompts.
• Mixed head-to-head evidence. A comparison in MedicoSAM (arXiv:2501.11734) found SAM-derived models competitive with or slightly better than nnU-Net in 2D while nnU-Net was best in 3D, with SAM-based models failing on two datasets. A lung-CT tumour benchmark (arXiv:2505.01239) reported that foundation models, especially MedSAM 2, outperform conventional models for tumour delineation. A brain MRI study (arXiv:2506.20786) found SAM and SAM 2 reach high Dice only with accurate box prompts and concluded nnU-Net remains dominant because accurate prompts are impractical. All of these are preprints or secondary reports in the sources reviewed.
• Evaluation gaps. Metrics Reloaded (Nature Methods 2024) shows that metric choice changes rankings; inter- and intra-observer variability complicates the use of human annotations as a reference (a 2025 Journal of Imaging paper proposes blinded validation designs).
• Not shown. No model has been shown non-inferior to inter-rater variability on a broad pre-registered benchmark of unseen tasks with a validated failure detector.

A reliable general segmenter would remove the annotation bottleneck for radiotherapy contouring, surgical planning, longitudinal tumour tracking and imaging biomarkers, and make quantitative imaging usable at sites without machine-learning teams. Failure detection is as important as accuracy for safe clinical deployment.

A refereed, pre-registered, multi-site evaluation (organised by a challenge platform such as Grand Challenge or MICCAI) meeting criteria (i)-(iii) with released data, code and annotator-level labels, and a prospective study showing criterion (iv) in at least one clinical workflow.