Dear Editor,
We read with great interest the study by Kim et al.1 comparing four commercially available artificial intelligence (AI) solutions for acute intracranial hemorrhage detection on non-contrast computed tomography (CT). Their head-to-head, real-world evaluation is particularly relevant because hospitals increasingly need comparative evidence to guide procurement and work-flow integration.
The authors appropriately evaluated each solution as deployed using the product-reported binary classification to calculate sensitivity, specificity, precision, and F1 score. This answers an important practical question: How does each product perform in its current configuration? A complementary question becomes important, however, when these results are used for procurement: How much of the observed difference reflects discriminatory capability, and how much reflects the product-specific operating point?
This distinction matters because changing the decision threshold of the same AI model can materially alter the sensitivity–specificity trade-off. Developers can select different operating points to favor either sensitivity or specificity.2 In the present study, solutions A–C provided continuous probability scores, whereas solution D provided only binary classifications.1 Accordingly, higher specificity, precision, or F1 score at native binary outputs should be interpreted as performance of the deployed product configuration rather than as threshold-independent superiority of the underlying algorithm.
Both perspectives are clinically useful. Native-output performance estimates what an institution may experience after deployment. For products providing continuous scores, a second, threshold-standardized analysis could compare specificity and false-positive burden at a prespecified matched sensitivity, or sensitivity at matched specificity. Operating-point matching has been used in large diagnostic AI comparisons to improve clinical interpretability.3 Reporting false-positive notifications per 1,000 CT examinations at a clinically acceptable sensitivity could further translate statistical differences into expected work-flow burden.
Because solution D did not provide a continuous score, threshold-standardized comparison could not be performed for that product from the reported outputs. Whether its operating point can be locally configured is therefore a separate product characteristic that may also matter for procurement.
Kim et al.1 provide a valuable framework for evidence-based AI selection. Future head-to-head evaluations could strengthen this framework by reporting both native-product and threshold-standardized performance, thereby separating two clinically distinct questions: How does the product perform as deployed, and how does it compare when institutions require the same diagnostic safety target?
