ABSTRACT

PURPOSE

To evaluate the real-world multimetric performance of four commercially available computed tomography (CT)-based artificial intelligence (AI) solutions for acute intracranial hemorrhage (AIH).

METHODS

Patients who underwent non-contrast brain CT for suspected AIH in our emergency room between February and March 2024 were screened. After applying the inclusion and exclusion criteria, 436 CT scans were included in the final analysis. Three neuroradiologists established the ground truth for AIH and hemorrhage volume. For detection performance, the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and Brier score were calculated based on the available probability score, whereas sensitivity, specificity, precision, and F1 score were calculated based on binary classification. Bland–Altman analysis was performed to assess volumetric agreement for AIH between each algorithm’s calculations and the neuroradiologists’ measurements.

RESULTS

A total of 436 patients (mean age, 62 years ± 20; male patients, 209) were enrolled. The AUROC (0.96 to 0.99) and sensitivity (0.85 to 0.92) were high across all solutions, with no statistically significant differences in pairwise comparisons (P > 0.05). However, solution B demonstrated the highest AUPRC [0.98, 95% confidence interval (CI): 0.94, 1.00] and the lowest Brier score [0.02 (95% CI: 0.02, 0.03)]. In binary performance, both solutions B and D exhibited significantly higher specificity (1.00 and 0.99), precision (0.90 to 0.98), and F1 score (0.87 to 0.94) than the other solutions (P < 0.05).  For volumetric agreement of AIH, solution D showed the lowest mean difference [−0.87 mm³ (95% CI: −1.47, −0.27)] and the narrowest limits of agreement (−13.4 to 11.6) relative to the neuroradiologists’ measurements.

CONCLUSION

In a real-world emergency setting, all four commercially available CT-based AI solutions for AIH demonstrated uniformly excellent performance; however, meaningful differences emerged in confirmatory performance and volumetric agreement. These distinct, algorithm-specific trade-offs provide practical guidance for selecting and integrating appropriate AI solutions to improve AIH diagnosis and management workflows.

CLINICAL SIGNIFICANCE

The algorithm-specific performance trade-offs identified in this study suggest that no single AI solution is universally optimal; solutions with superior confirmatory performance may reduce unnecessary notifications in high-volume emergency settings, whereas those with more consistent volumetric agreement may better support treatment planning and longitudinal monitoring. A structured, multimetric evaluation aligned with institutional priorities is essential for evidence-based AI procurement in acute stroke imaging.

The rapid expansion of commercially available artificial intelligence (AI) applications in radiology has led to the widespread clinical availability of stroke-focused algorithms, particularly for the detection of acute intracranial hemorrhage (AIH) on non-contrast computed tomography (CT).^1-3 As an increasing number of AI solutions receive regulatory approval and reimbursement pathways expand,⁴ hospitals operating under constrained resources are faced with a growing range of options and must evaluate and select the most appropriate tools for integration into acute stroke workflows.^5-8

AIH represents a time-sensitive neurologic emergency.⁹ Rapid detection is essential for initiating appropriate management strategies, including blood pressure control, anticoagulation reversal, neurosurgical consultation, and intensive monitoring.^10-12 Beyond binary detection, hemorrhage volume is a well-established prognostic marker that plays a central role in risk stratification, treatment decision-making, and longitudinal assessment. Accordingly, AI tools in this domain are positioned not only as triage systems but also as quantitative decision-support instruments.^4-6,13,14

Despite the proliferation of commercial AI solutions, independent head-to-head comparisons under real-world clinical conditions remain critically lacking.¹⁵ Procurement decisions are frequently informed by vendor-reported performance metrics or single-metric validation studies, which do not fully reflect clinical priorities such as discriminative ability, confirmatory performance, calibration quality, and quantitative agreement. Moreover, different algorithms may be optimized for distinct tasks, raising the possibility that they are not functionally interchangeable.^16-19

Therefore, we performed an external, real-world, multimetric evaluation of four CT-based AI solutions for AIH that were approved and designated as innovative medical devices by the Korean Ministry of Food and Drug Safety. By assessing both discrimination performance and volumetric agreement, including subtype-specific analyses, this study aims to provide practical, evidence-based guidance for the selection and clinical integration of AI tools in acute stroke imaging.

Methods

This retrospective study was performed in accordance with the principles of the Declaration of Helsinki and approved by Eunpyeong St. Mary’s Hospital’s Institutional Review Board (IRB; protocol number: PC25RISI0154, approval date: 14 August 2025). The requirement for informed consent was waived by the IRB because of the retrospective nature of the study.

Sample eligibility

A total of 615 non-contrast brain CT scans of adult patients with suspected AIH, including intraparenchymal hemorrhage (IPH), subarachnoid hemorrhage (SAH), subdural hemorrhage (SDH), epidural hemorrhage, and intraventricular hemorrhage (IVH), obtained between February and March 2024 in Eunpyeong St. Mary’s Hospital emergency room were eligible for inclusion. During this period, eligible scans were included using the following criteria: (1) the first CT scan was performed during the patient’s clinical course, (2) the cranial structure was preserved, and (3) the CT image quality was acceptable for radiologist interpretation, which was defined as the absence of significant motion or metal artifacts that would preclude diagnostic evaluation, consistent with prior validation studies that excluded non-diagnostic-quality images to ensure a fair and standardized comparison across the evaluated AI solutions.^{8, 20-22} The exclusion criteria were as follows: (1) mixed-stage intracranial hemorrhage, considering the intended use of the enrolled AI solutions; (2) patients with intracranial surgical materials (e.g., clips, coils, or drainage catheters); and (3) intracranial hemorrhages caused by other co-pathologic factors (e.g., tumor bleeding).^{8, 20-22} All eligible CT scans were reviewed by a board-certified neuroradiologist (J.K., with 12 years of relevant experience) according to the inclusion and exclusion criteria. After review, 63 follow-up scans, 15 scans of patients who had undergone craniectomy, and 21 scans with significant motion artifacts were excluded. Among the potentially eligible 516 scans, 7 scans presenting mixed-stage intracranial hemorrhage, 71 scans with significant metal artifacts, and 2 scans presenting tumor hemorrhage were excluded. Finally, 436 non-contrast brain CT scans were included in this study.

Computed tomography scanning protocol

CT scans were performed using one of two CT machines at the study institution. Machine A was a 128-slice single-source CT scanner (SOMATOM Edge, Siemens Healthineers, Erlangen, Germany) with a tube potential range of 70–140 kVp and a tube current range of 20–800 mA, and machine B was a dual-source CT scanner (SOMATOM Force, Siemens Healthineers) with a tube potential range of 70–150 kVp and a tube current range of 20–1,300 mA. The acquisition parameters were as follows: slice thickness, 4 mm without a gap; rotation time, 1.0 s; pitch, 1; automatic tube voltage modulation (CARE kV, Siemens Healthineers) using a reference kV of 120; automatic tube current selection (CAREDose 4D, Siemens Healthineers) using a reference mAs of 250; and collimation, 128 × 0.6 for machine A and 192 × 0.6 for machine B.

Commercially available artificial intelligence software

In this study, four commercially available CT-based AI solutions for detecting AIH (solution A, JLK-ICH, version 4.1.0.2, JLK Inc.; solution B, HyperInsight–ICH, version 2.1.3, Purple AI Inc.; solution C, Heuron-ICH, version 1.0.5.13, Heuron Inc.; and solution D, AVIEW NeuroCAD, version 1.1, Coreline Soft Inc.) were used based on predefined criteria to reflect both regulatory status and real-world clinical applicability. Specifically, the included solutions (1) were designated as innovative medical devices by the Korean Ministry of Food and Drug Safety or had equivalent regulatory approval, (2) were practically accessible and deployable within our institutional environment at the time of the study, and (3) had supporting evidence from clinical validation studies.^6-8 All four solutions were designated as innovative medical devices by the Korean Ministry of Food and Drug Safety, and solutions A, B, and C were additionally approved by the U.S. Food and Drug Administration.^22-24

All AI solutions were deployed on-premises within Eunpyeong St. Mary’s Hospital’s secure internal network environment. CT images were processed locally without transfer to external servers or vendor platforms. The AI analyses were performed retrospectively using anonymized datasets, and each solution was applied under standardized conditions to ensure a fair comparison.

Solutions A, B, and C provided binary classifications with probability scores and calculated volumes for AIH. However, solution D provided only binary classification without probability scores. Detailed information on the algorithms used in each AI solution is provided in the Supplementary Material. None of the CT images included in this study were used for the training, validation, or internal testing of any of the evaluated AI solutions.

Ground truth and volume measurement for acute intracranial hemorrhage

To establish the ground truth for AIH, three board-certified neuroradiologists (J.M., S.W.O., and H.Y.L., with 12, 18, and 20 years of experience in brain imaging, respectively) independently reviewed the identical set of 436 non-contrast brain CT scans. The three neuroradiologists diagnosed AIH based solely on CT findings and were blinded to the patients’ clinical information and the results of the AI solutions. In cases of disagreement, the ground truth was determined by consensus with reference to other available imaging studies. AIH volume in each case was also independently measured by the three neuroradiologists using manual free drawing in ITK-SNAP (version 4.0.1, http://www.itksnap.org). For objective evaluation of AIH, the Hounsfield unit (HU) cut-off for AIH was > 30 HU.^{25, 26} Therefore, a segmentation map for AIH with a threshold of 30–150 HU was provided to each neuroradiologist before initiating manual drawing. Each neuroradiologist edited the provided AIH segmentation map using the free-draw function. The reference standard for AIH volume was defined as the mean of the three independent measurements.

Statistical analysis

Considering the exploratory and retrospective head-to-head comparison of four AI algorithms in this study, the sample size was determined by the number of eligible non-contrast head CT examinations available during the study period, which was substantially greater than the preliminary sample size calculation based on a significance level of 0.05, a statistical power of 0.8, a specificity of 0.90 from a previous meta-analysis, a specificity of 0.98 from previous validation studies, and a dropout rate of 10%, yielding a minimum required sample size of 202 cases.^5-8,27

To evaluate detection performance, a two-step analysis was conducted. First, an analysis based on probability scores was performed among solutions A, B, and C because solution D did not provide probability scores. Second, an analysis based on binary classification was performed among solutions A, B, C, and D based on their reported binary results. For the first analysis, the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were calculated. In addition, the Brier score was calculated to compare the overall quality of probability predictions across algorithms under identical conditions. For the second analysis, binary classification metrics, including sensitivity, specificity, precision, negative predictive value, false-positive ratio, false-negative ratio, and F1 score, were calculated. These performance comparisons were also conducted between the traumatic and non-traumatic AIH cohorts in a sub-analysis. In addition, a sub-analysis of detection-failure cases by solution was conducted. AIH volumes were compared between true-positive and false-negative cases. In addition, considering existing studies,^28-30 the false-negative rate was compared between the small (< 10 mm³) and large (> 10 mm³) AIH groups.

To evaluate the solutions’ AIH volume calculation, Bland–Altman analysis was performed to assess volumetric agreement between each solution’s calculation and the neuroradiologists’ measurement. Inter-reader agreement of volume measurement among the three neuroradiologists was evaluated using the intraclass correlation coefficient (ICC). The ICCs were classified as follows: < 0.2, very poor; ≥ 0.2 and < 0.4, poor; ≥ 0.4 and < 0.6, moderate; ≥ 0.6 and < 0.8, good; and ≥ 0.8, excellent.

The baseline characteristics, including age, sex, AIH incidence, Glasgow Coma Scale scores, and modified Rankin Scale scores, were compared between patients with and without AIH, and AIH volume was compared between true-positive and false-negative cases using independent t-tests or Mann–Whitney U tests after the Shapiro–Wilk normality test. Diagnostic performance and volumetric agreement metrics among solutions were compared using patient-level paired bootstrap resampling (1,000 iterations). For each bootstrap sample, performance metrics were computed for each algorithm, and pairwise differences were obtained. Two-sided P values were calculated from the empirical distribution of bootstrap differences, and 95% confidence intervals (CIs) were derived using the percentile method. This bootstrap-based comparison was applied to both probability-based metrics and binary classification metrics. Stratified assessments by AIH subtype were additionally performed as sub-analyses.

All statistical analyses were conducted using MedCalc (version 23.2.1, MedCalc Software Ltd) and Python (version 3.14, Python Software Foundation) using scikit-learn. Multiple comparisons were controlled using the Benjamini–Hochberg procedure. Statistical significance was set at P < 0.05.

Results

Patient characteristics

A total of 436 patients, each with one initial CT scan, were included in the final analysis. A flow chart of patient enrollment is presented in Figure 1. The mean age of the total patient cohort was 62 ± 20 years [standard deviation (SD)], the proportion of male participants was 48%, and the proportion of AIH cases was 12% (n = 52; traumatic, 27; non-traumatic, 25). There were significant differences in median Glasgow Coma Scale scores and modified Rankin Scale scores between the AIH-positive and AIH-negative groups (P < 0.001). The mean volume of AIH cases was 33.15 mm³ ± 48.24 (SD). The results are summarized in Table 1. The detailed numbers of AIH subtypes and volume information are presented in Supplementary Table 1 and Supplementary Figure 1.

Detection performance based on probability score

Solution B showed the highest AUROC [0.99 (95% CI: 0.97, 1.00)]; however, there were no significant differences in pairwise comparisons among solutions A, B, and C (P > 0.05). In contrast, the highest AUPRC [0.97 (95% CI: 0.94, 1.00)] and the lowest Brier score [0.02 (95% CI: 0.01, 0.03)] of solution B were significantly different from those of the other solutions (P < 0.01).

 In sub-analyses according to AIH type, all solutions achieved AUROCs greater than 0.95 across all AIH subtypes. However, solution B consistently showed the highest AUPRCs (0.95 to 1.00) and the lowest Brier scores (0.01 to 0.02). Solution A showed the lowest AUPRCs (0.56 to 0.63), whereas solution C showed the highest Brier scores (0.15 to 0.16) across AIH types. These results are summarized in Table 2, Figure 2, and Supplementary Figure 2. In sub-analyses by etiology, there were no significant differences between traumatic and non-traumatic AIH across solutions
(P > 0.05) (Supplementary Table 2).

Detection performance based on binary classification

For sensitivity, solution C showed the highest value [0.92 (95% CI: 0.84, 0.98)]; however, there was no significant difference in pairwise comparisons among the solutions (P > 0.05). Solution B showed the highest specificity [1.00 (95% CI: 0.99, 1.00)], precision [0.98 (95% CI: 0.93, 1.00)], and F1 score [0.94 (95% CI: 0.89, 0.98)]; however, these metrics were not significantly different from those of solution D. In sub-analyses according to AIH type, solutions A and B showed the highest sensitivity (0.97) for IPH. In addition, perfect sensitivity (1.00) was achieved by solution C for SAH, by solutions B and C for SDH, and by all four solutions for IVH. In contrast, solutions B and D consistently demonstrated the highest specificity and precision, resulting in the highest F1 scores among the four solutions across AIH types. These results are presented in Table 3, Supplementary Table 3, and Supplementary Figure 3. In sub-analyses by etiology, solutions C and D showed the highest sensitivity in traumatic AIH [0.93 (95% CI: 0.81, 1.00)] and non-traumatic AIH [0.96 (95% CI: 0.87, 1.00)], respectively. For all other metrics, solution B showed the highest specificity (1.00), precision (0.96), and F1 scores (0.92–0.94) in both traumatic and non-traumatic AIH. However, there were no significant differences between traumatic and non-traumatic AIH across all solutions
(P > 0.05) (Supplementary Table 2).

False-negative analysis according to acute intracranial hemorrhage volume

Across all four solutions, the median AIH volumes of false-negative cases (range, 0.30 to 0.73 mm³) were significantly smaller than those of true-positive cases (range, 15.50 to 16.74 mm³) (P < 0.01). Volume-stratified analysis demonstrated that the false-negative rate was significantly higher for small AIHs (0.17 to 0.35) than for large AIHs (0.00) across all solutions (P < 0.01). These findings are summarized in Supplementary Tables 4 and 5.

Volumetric agreement between each solution and ground truth

The ICC of the measured AIH volume among the three neuroradiologists showed excellent inter-reader agreement [1.00 (95% CI: 0.99, 1.00)]. Among the four solutions, solution D showed the lowest mean difference [−0.87 (95% CI: −1.47, −0.27)] and the narrowest limits of agreement (−13.4 to 11.6) for overall AIH. In sub-analyses, solution D showed the lowest mean differences in IPH [−6.49 (95% CI: −12.89, −0.09)], SAH [−5.74 (95% CI: −8.83, −2.65)], SDH [−8.66 (95% CI: −17.24, −0.07)], and IVH [−6.62 (95% CI: −15.21, 1.96)] across all solutions. These results are presented in Table 4 and Figure 3. Examples of the AIH heatmaps generated by each solution are illustrated in Figure 4.

Discussion

In this real-world comparative study of four commercially available CT-based AI solutions for AIH, we observed distinct, task-specific performance profiles across algorithms. All four solutions demonstrated uniformly high discriminative performance; however, solutions B and D exhibited superior precision-oriented performance among all evaluated solutions. This pattern was consistent regardless of AIH subtype, and no significant performance differences were observed based on hemorrhage etiology. Regarding missed detections, false negatives occurred predominantly in small-volume hemorrhages, particularly SAH. In terms of volumetric agreement, solution D demonstrated the most stable and accurate performance across subtypes.

The clinical relevance of between-algorithm differences is well reflected in specific metrics. In this study, all solutions showed high sensitivity and AUROC, metrics emphasizing case discrimination. Among them, solutions B and D demonstrated consistently strong performance in false-positive control, as reflected by precision-oriented metrics. These results suggest consistent performance and a robust risk-scoring framework with a favorable operating point. Therefore, interpretation of the results requires caution because, in class-imbalanced settings such as this real-world cohort, precision-oriented confirmatory metrics may better reflect clinically meaningful differences.¹⁵ In particular, differences in sensitivity or false-negative rate represent variations in the probability of a missed diagnosis at the point of care,^10-12 whereas differences in specificity translate directly into differential false-positive notification burdens in emergency workflows.⁵ Therefore, differences in multimetric performance should be considered when evaluating clinical utility, and the optimal algorithm choice may depend on institutional priorities.^{15, 31} Notably, despite the distinct pathophysiological mechanisms, underlying comorbidities, and prognostic implications associated with traumatic vs. non-traumatic AIH,³² no significant performance differences were observed between the two etiologic subgroups across all four solutions, suggesting that etiology did not influence algorithm performance in this cohort.

False-negative analysis according to AIH volume revealed that missed cases occurred predominantly in small-volume AIH. For small AIH, which is generally associated with more favorable clinical outcomes,^28-30 the false-negative rate was consistently higher across all solutions than for large AIH, for which perfect sensitivity was observed. In particular, SAH accounted for the majority of missed cases, whereas IVH was reliably detected across all solutions. These results highlight volume and subtype as key determinants of AI failure in AIH detection and suggest that future model development should specifically address and overcome weaknesses such as small-volume SAH.

The volumetric analysis demonstrated distinct performance characteristics across the evaluated solutions. Solutions A and B showed intermediate agreement, indicating reasonable accuracy with some variability. In contrast, solution C exhibited greater variability in volume estimation. Among the solutions, solution D exhibited minimal systematic bias and the most consistent volumetric estimation. Given the excellent inter-reader agreement of the reference standard, measurement error in the ground truth is unlikely to account for the observed differences in volumetric estimation across solutions, suggesting that these differences reflect algorithmic rather than annotator variability. From a clinical standpoint, volumetric accuracy represents an important surrogate for prognostic assessment and longitudinal monitoring in intracranial hemorrhage.^{9, 11, 28, 30, 33} Therefore, accurate and reproducible volume estimation is important for evaluating interval changes on follow-up. Although solution D did not demonstrate the highest detection performance, its superior volumetric agreement suggests a distinct strength in quantitative assessment rather than case-level triage.^33-36 The consistent performance of solution D across multiple hemorrhage subtypes further supports its potential utility.

In this study, the differences in performance across solutions are likely attributable to a combination of training dataset characteristics and architectural design choices.^6-8 Although the evaluated solutions obtained regulatory approval based on multi-institutional, multi-racial validation datasets demonstrating robust generalizability, differences in the scale, composition, and case distribution of their respective training data may contribute to the algorithm-specific performance profiles observed in this evaluation.^{5, 7, 8, 22-24} Collectively, these findings suggest that commercially available AI tools for AIH are not interchangeable. Rather than being universally superior, the algorithms appear to emphasize different performance dimensions.^10-12,37,38

Both accurate detection and reliable volume estimation are important; however, their relative priorities can vary across stages of care. In resource-limited settings with reduced staffing or expertise, AI may function as a second-opinion tool to reduce missed diagnoses, emphasizing detection performance. In high-volume emergency settings, solutions with high sensitivity and rapid notification may serve as reliable red-flag systems to prioritize urgent cases. In contrast, in well-resourced tertiary centers, where detection accuracy is less limiting, AI-driven volume quantification may reduce repetitive manual measurement tasks and support longitudinal monitoring.^9-11 This perspective suggests that no single solution is universally optimal and that algorithm selection should be guided by the clinical context and institutional workflow rather than by aggregate performance metrics alone. By characterizing algorithm-specific performance profiles across these dimensions, our multimetric framework provides actionable guidance for matching AI solutions to the specific demands of each stage of care rather than relying on a single aggregate metric for procurement decisions.

To the best of our knowledge, this is the first external, head-to-head, multimetric comparison of four concurrently regulatory-approved commercial AI solutions under real-world emergency conditions. This study directly addresses the critical gap identified by recent systematic reviews regarding the scarcity of independent head-to-head comparisons in this field.^{15, 18, 39} Unlike vendor-reported benchmarks or single-algorithm validations,^22-24 our comprehensive multimetric framework provides the granularity needed to move beyond the binary question of whether AI can detect hemorrhage toward the more clinically pressing question of which algorithm best serves a given institutional context. Although the observed performance differences likely reflect dataset characteristics such as case mix, subtype distribution, and prevalence rather than intrinsic algorithmic superiority,^22-24,40,41 these differences carry direct practical consequences. Therefore, amid the increasing number of commercially available AI solutions, we propose that AI procurement in acute stroke imaging should be guided not by a single performance metric but by a structured, multimetric evaluation aligned with institutional priorities—a framework this study is designed to support.

Our study had several limitations. First, this was a retrospective, single-center study utilizing CT scanners from a single vendor, which may limit generalizability to other CT vendors. However, considering the consistent cross-vendor performance of the solutions,^5-8,22-24 vendor-specific performance is unlikely to fully explain the algorithm-specific differences observed in this study. In addition, the evaluated AI solutions were limited to those available at Eunpyeong St. Mary’s Hospital and may not represent the full spectrum of commercially available algorithms, potentially limiting generalizability. Second, the selection of AI solutions and the case enrollment criteria may limit generalizability. Although geographic-specific bias is substantially mitigated by the characteristics of prior training and validation datasets,^5-8,22-24 these factors remain potentially vulnerable to selection bias. Third, regarding subgroup analyses, individual cases could contain multiple hemorrhage subtypes with relatively small per-subgroup case counts, possibly resulting in non-independent analyses. Thus, even though previous validations have shown consistent results, subtype-level analyses should be interpreted as hypothesis-generating rather than confirmatory. Fourth, each solution’s AIH volumetric agreement was evaluated; however, topographic information was not evaluated.⁴² Finally, we did not evaluate the clinical impact of the AI solutions, which is the most important consideration when selecting a solution. For example, other important factors (e.g., processing speed, software usability, or workflow compatibility) may affect patient outcomes.^{43, 44} Therefore, further prospective, multicenter, and comparative clinical evaluations of these AI solutions should be performed in the future.

In conclusion, commercially available CT-based AI tools for AIH demonstrated distinct, task-specific performance profiles in a real-world emergency setting. Although all solutions exhibited excellent discriminative performance, some solutions demonstrated superior confirmatory performance, and certain solutions showed more consistent AIH volumetric agreement with expert measurements. Therefore, these results provide practical guidance for selecting and integrating appropriate AI algorithms to improve AIH diagnosis and management workflows.