ABSTRACT

PURPOSE

Unstructured, free-text dictation (FT), the current standard in breast magnetic resonance imaging (MRI) reporting, is considered time-consuming and prone to error. The purpose of this study is to assess the usability and performance of a novel, software-based guided reporting (GR) strategy in breast MRI.

METHODS

Eighty examinations previously evaluated for a clinical indication (e.g., mass and focus/non-mass enhancement) with FT were reevaluated by three specialized radiologists using GR. Each radiologist had a different number of cases (R1, n = 24; R2, n = 20; R3, n = 36). Usability was assessed by subjective feedback, and quality was assessed by comparing the completeness of automatically generated GR reports with that of their FT counterparts. Errors in GR were categorized and analyzed for debugging with a final software version. Combined reading and reporting times and learning curves were analyzed.

RESULTS

Usability was rated high by all readers. No non-sense, omission/commission, or translational errors were detected with the GR method. Spelling and grammar errors were observed in 3/80 patient reports (3.8%) with GR (exclusively in the discussion section) and in 36/80 patient reports (45%) with FT. Between FT and GR, 41 patient reports revealed no content differences, 33 revealed minor differences, and 6 revealed major differences that resulted in changes in treatment. The errors in all patient reports with major content differences were categorized as content omission errors caused by improper software operation (n = 2) or by missing content in software v. 0.8 displayable with v. 1.7 (n = 4). The mean combined reading and reporting time was 576 s (standard deviation: 327 s; min: 155 s; max: 1,517 s). The mean times for each reader were 485, 557, and 754 s, and the respective learning curves evaluated by regression models revealed statistically significant slopes (P = 0.002; P = 0.0002; P < 0.0001). Overall times were shorter compared with external references that used FT. The mean combined reading and reporting time of MRI examinations using FT was 1,043 s and decreased by 44.8% with GR.

CONCLUSION

GR allows for complete reporting with minimized error rates and reduced combined reading and reporting times. The streamlining of the process (evidenced by lower reading times) for the readers in this study proves that GR can be learned quickly. Reducing reporting errors leads to fewer therapeutic faults and lawsuits against radiologists. It is known that delays in radiology reporting hinder early treatment and lead to poorer patient outcomes.

CLINICAL SIGNIFICANCE

While the number of scans and images per examination is continuously rising, staff shortages create a bottleneck in radiology departments. The IT-based GR method can be a major boon, improving radiologist efficiency, report quality, and the quality of simultaneously generated data.

Main points

• The software-based guided reporting (GR) strategy is a novel technology for structured reporting in medicine.

• GR allows for complete radiology reporting with minimized error rates and reduced reading times.

• The shortening of reporting times is key for successful implementation of GR into the clinical workflow.

• Differences in content between GR and unstructured, free-text dictation are not caused by limitations of the software itself but by insufficient, user-dependent operation of the software. Thus, the introduction of this disruptive technology requires intensive training and adjustment to the process.

• The great potential of structured datasets will open doors for the future of radiology with respect to big data analysis, automatic, real-time International Statistical Classification of Diseases and Related Health Problems 10 coding, and the efficient integration of artificial intelligence.

Radiology reporting has not changed significantly since the beginning of the 20^th century. By general consensus, radiologists should communicate unambiguously with referring physicians according to the six C’s of communication: clarity, correctness, concision, completeness, consistency, and a high level of confidence.¹ Few linguistic guidelines exist for the structure of radiological findings.^{2, 3} Most radiology reports are crafted by free-text dictation (FT), whereby individual styles of speech may not meet the expectations of a referring physician.^{4, 5} In addition, a high percentage of reports contain errors,⁶ with the amount of radiology errors recently assessed as 3%–5%.⁷ Errors can occur before, during, or after the reporting period, may cause direct, indirect, permanent, or temporary harm to patients,⁸ and can be classified into 12 subgroups.⁹ Contrary to the low incidence of “poor communication-related error” reported by Kim and Mansfield⁹ 2014, Brady⁷ concludes that poorly written or incoherent reports are a significant source of potential harm to patients.

Additionally, the time it takes radiologists to generate reports^{10, 11}creates a bottleneck. Waiting for imaging and its accompanying report affects length of hospital stay and the quality of patient care. In a Canadian evaluation, each additional hour spent waiting for magnetic resonance imaging (MRI) increased acute length of hospital stay by 1.2 h.¹² Modern MRI scanners produce increasingly detailed images in ever shorter times, often making image acquisition faster than analysis and reporting.

Modern radiology reporting systems are expected to reduce error rates, increase comprehensibility, and shorten report generation times. Templates and structured reporting (SR) methods do not meet all these requirements and have been applied in clinical routine only to a limited extent.^13-15 Guided reporting (GR) is a new strategy for radiology reporting. To the best of our knowledge, this study is the first to investigate and evaluate the performance of GR in breast MRI. Our purpose is to overcome the known disadvantages of SR, which include greater time investment and effort, inconsistencies in interpretation, and poor user experiences (UX).¹⁶

Methods

Study design

The Ethics Committee of Carl von Ossietzky University in Oldenburg, Germany approved the protocol for this retrospective study (project no: 2023-217, date: 20.11.2023). This is an experimental retrospective study conducted to assess the usability, report quality, and reading time of GR software (RadioReport®) in breast MRI (Figure 1). The study used a pre-market version (v. 0.8) of the MR mammography module to validate the usability of the software and identify further areas for improvement needed for the finalization of a market version. Eighty MRI examinations covering the full spectrum of clinical breast MRI findings [mass (n = 57), focus or non-focus enhancement (NME) (n = 8), and exclusion of a mass (n = 15)] were selected from the department of radiology at University Hospital Oldenburg (Oldenburg, Germany). Broad consent for retrospective evaluation is routinely obtained at the study hospital. These cases had been reported more than 8 weeks prior using FT and were reevaluated by the same radiologist.

The GR evaluation in this study was completed by three radiologists specialized in breast diagnostics with 16, 13, and 11 years of experience in breast MRI. The three radiologists completed a 90-min online training session on how to use the software. Application support was available by phone. Readers were instructed to begin the image reading and reporting processes simultaneously. Reader 1 had 24 cases, Reader 2 had 20 cases, and Reader 3 had 36 cases. The number of orthographic errors in the FT and GR reports was evaluated.

Guided reporting software

The pre-market v. 0.8 of the GR software was used as a stand-alone web tool. The software automatically generates radiological reports based on decoded complex radiological decision trees. A comprehensive query system (Figure 2) guides the user through the reporting process, and a dedicated module for breast MRI is available. The program includes mandatory fields, predefined paths for information input, and plausibility checks, thus guaranteeing completeness in reporting. The radiologist is guided through 10 folders (patient/indication, anamnesis, technique, anatomy, background parenchymal enhancement, findings, lymph nodes/organs, bones, summary, and report). A key difference between GR and SR is the former’s anatomy-based approach: every body region and modality is covered in a single module [e.g., thorax computed tomography (CT), abdomen MRI, breast MRI]. Each module covers all relevant pathologies for its specific anatomical region (e.g., tumors, inflammation, post-surgery pathologies), and a combination of different pathologies can be easily reported (rather than having to combine different templates for a single body region).¹⁷ Instead of having to offer and continuously update hundreds of templates, almost all MRI, CT, and breast imaging indications can be reported using 23 modules.

Each folder (e.g., “axillary lymphadenopathy” in the lymph nodes/organs folder; Figure 2b) opens with a mouse click and displays further items on the next level of depth (e.g., right/left). Each sub-item consists of several subordinated levels (e.g., level I/II/III). Detailed descriptions can be added (e.g., max. short-axis diameter in mm). Moreover, the software features information files. By selecting different items and sub-items, the software concomitantly generates semantic sentences from predefined and approved text phrases stored in its database.

Additional information can be entered manually in the Discussion section (using speech recognition or keyboard entry) to enable a hybrid approach,¹⁸ wherein GR with fully machine-readable data and individual information entries are combined to enhance flexibility and acceptance. The commercial and European conformity (CE)-certified version (v. 1.7) of the software includes an add-on of bullet points in the Impression section (i.e., to highlight the exclusion of suspected pathologies) (Figure 3).

With a final click, a fully automated text structured in standard compliant sections (e.g., patient data, indication, anamnesis, technique, findings, impression, and recommendations; Figures 3, 4) is generated. Diagnoses are automatically coded according to the 10^th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10).¹⁹

For the second phase of the study, v. 1.7 of the software was used. Here, technical and medical feedback was implemented, and the complexity of the breast MRI module was increased from 33,756 lines of code in v. 0.8 to 57,213 in v. 1.7 (front- and backend). Five hundred-thirteen logic lines were used in v. 1.7.

Evaluation

Usability and guided reporting quality

Usability was evaluated based on feedback from every reader along three parameters: intuitive operation, the practicability of simultaneous image analysis and reporting, and user confidence in the automatically generated report compared with FT (assessed on a 4-point scale: very good, good, medium, and poor). Readers were asked if they experienced a change in confidence while operating the software and if they would specifically recommend GR for certain subgroups.

Report quality was evaluated by comparing the completeness of the automatically generated GR product with its FT counterpart, as well as noting any differences between the two strategies. This evaluation was performed by consensus by two senior radiologists who organized the reports into three scores. A report received a score of A if there were no differences in content between FT and GR, a score of B if there were only minor differences in content that did not result in differences in treatment, and a score of C if there were major differences in content that resulted in differences in treatment. Entries in the Discussion section were not included in this evaluation.

Content omission errors (COEs) in GR, v. 0.8, for patient reports receiving a score of B or C were further evaluated and categorized as follows: category 1 errors were caused by insufficient operation of the software; category 2 errors were caused by missing content in the v. 0.8 software displayable with v. 1.7; and category 3 errors were caused by missing content in the v. 0.8 software that was not displayable with v. 1.7. Errors in reporting were further categorized according to a modified classification system proposed by Hawkins et al.⁶ as follows: (a) non-sense; (b) spelling/grammar; (c) omission/commission; and (d) translation (Table 1).

Reporting process and time

Reading time (s) was defined as the overall combined image reading time and report creation time, evaluated based on the implemented tracking option. Individual use of information files was tracked automatically. Overall reading times were compared between different readers as well as different lesions. The learning curve for reading time over the study period was evaluated for every reader using a multiple regression analysis. The reporting process was additionally analyzed on a segmental (folder) level and compared on an interindividual basis. FT was not evaluated in this portion of the study; however, benchmark data from large radiological information system (RIS) evaluations (n = 170,901 reports, including 5,622 MRI reports) were used as reference.¹⁰ These benchmark data were extracted from two large teaching hospitals with 23 full-time radiologists in New Zealand. Outliers with exceptionally long reporting times (>60 min for MRI) were culled, removing 9.5% of the total. Automatically generated GR ICD-10 codes¹⁹ were compared with manual FT coding by a senior radiologist (A.H.).

Statistical analysis

Differences in reading duration by number of cases per reader were assessed using a trend test. A regression analysis was used to assess trends, first by reader and then using a mixed model for repeated measures (MMRM). In the MMRM, the data of all three readers were used within the same model, with the factor, “reader”, used to indicate repeated measures. For sensitivity analysis in case of abnormality, a non-parametric Jonckheere–Terpstra trend test was used. Analyses were performed using SAS v. 9.4 software (SAS Institute Inc., Cary, NC, USA).

Results

Usability was rated high by all readers. Intuitive operation was rated very good by two readers and good by one reader. The coworking of image analysis and reporting was rated very good by all readers, and confidence in the automatically generated report compared with its FT counterpart was rated good. All readers reported increasing confidence during the study.

With respect to content differences between GR and FT, 41 patient reports (51.3%) received an A score (no differences in content), 33 patient reports (41.3%) received a B score (minor differences in content not resulting in differences in treatment), and 6 patient reports (7.5%) received a C score (major differences in content resulting in differences in treatment) (Table 2).

Of the 39 patient reports scoring B or C, 26 (66.7%) involved category 1 errors (COEs in v. 0.8 caused by insufficient operation of the software), and 13 (33.3%) involved category 2 errors (COEs caused by missing content in v. 0.8 of the software but displayable with v. 1.7). No category 3 errors (COEs caused by missing content in the v. 0.8 software and not displayable with v. 1.7) were identified.

No non-sense, omission/commission, or translation errors were observed in the GR reports. Spelling/grammar errors were observed in 3/80 (3.8%) GR patient reports (exclusively in the free-text discussion section), compared with 36/80 (45%) in FT patient reports.

Three patients were excluded from the overall reading time analysis due to software issues (system instability resolved by remote maintenance). The mean reading time was 9 min 36 s (standard deviation: 5 min 27 s; min: 2 min 35 s; max: 25 min 17 s). The mean reading time per reader was 12 min 34 s (Reader 1), 9 min 17 s (Reader 2), and 8 min 5 s (Reader 3). The mean reading time for reports on patients without lesions was 11 min 16 s, 9 min 31 s for reports on patients with mass(es), and 7 min 31 s for reports on patients with focus/NME. The overall reading time was significantly shorter than FT radiology report reference values. The mean reading time for MRI examinations in general was 17 min 23 s (assessed from 1,629 h for 5,622 examinations).¹⁰ Thus, compared with MRI examinations in general, reading time was shortened by 44.8%. More specifically, for breast MRI, a study-ascribable time (SAT) of 35 min was reported.¹¹ In contrast to the reading time analyzed in,¹⁰ the SAT includes additional steps to reading and reporting, such as interpretation and clarification of request, prescription of the examination protocol, and communication with the referrer.¹¹

The information fields in the software were unsurprisingly limited to two uses for one reader, as the users were highly specialized in breast imaging.

A learning curve with significant streamlining of the reading process (assessed by time to completion) during the training course was observed for all readers (Figure 5). Regression models show reading time learning curve slopes of –0.59 (P = 0.002) for Reader 1, –0.52 (P = 0.0002) for Reader 2, and –0.31 (P < 0.0001) for Reader 3, with an average slope of –0.36 (P < 0.0001). As a certain deviance from normality was found in the data, a sensitivity analysis was performed using a non-parametric Jonckheere–Terpstra trend test. This test supported the results of the parametric regression analyses (Reader 1: P = 0.0157; Reader 2: P = 0.0015; Reader 3: P < 0.0001). A detailed analysis of the steps in the reporting process revealed differences between the readers (Figure 6). Reader 2 invested the greatest time allotment (38.4%) (Figure 6b) to folders 1 and 2 (patient/indication and anamnesis) of the GR process (compared with 33.3% and 27.9%, for Readers 1 and 3, respectively) (Figure 6a, c), suggesting that Reader 2 analyzed the image dataset in detail before beginning the reporting process. The relative time allotment for folder 6 (findings, which is the crux of a MR mammography report) was longer for Reader 3 (28.8%) compared with Readers 1 and 2 (21.9% and 21.0%, respectively). For folder 10 (report), the relative time allotment was longer for Reader 2 (17.6%) compared with Readers 1 and 3 (9.0% and 9.3%, respectively). This folder is used to check the text of the automatically generated report and the free-text entry of the discussion section, meaning that Readers 1 and 3 spent objectively less time in the acceptance or modification of the report text generated by the software than Reader 2.

The ICD-10 coding of the automatically generated GR reports was identical to manual FT coding and included the following codes: C22.9 (malignant neoplasm of the liver, not specified as primary or secondary; n = 1), C50.11 (malignant neoplasm of the central portion of the breast, n = 2), C50.21 (malignant neoplasm of the upper, inner quadrant; n = 2), C50.31 (malignant neoplasm of the lower, inner quadrant; n = 1), C50.41 (malignant neoplasm of the upper, outer quadrant; n = 7), C50.51 (malignant neoplasm of the lower, outer quadrant; n = 4), C50.81 (malignant neoplasm of overlapping sites of the breast; n = 34), C50.9 (malignant neoplasm of unspecified site of the breast; n = 1), D13.4 (benign neoplasm of the liver; n = 1), D24 (benign neoplasm of the breast; n = 1), N60.1 (diffuse cystic mastopathy; n = 3), N62 (hypertrophy of the breast; n = 2), N63 (unspecified lump in the breast; n = 15), R59.0 (enlarged lymph nodes; n = 17), T85.4 (mechanical complication of breast prosthesis and implant; n = 4), Z80.3 (family history of malignant neoplasm of the breast; n = 7), and Z85.3 (personal history of malignant neoplasm of the breast; n = 20). In total, 122 ICD-10 codes were generated.

Discussion

Differences between guided and structured reporting and the clinical need for guided reporting technology

GR is a modern, information technology (IT)-based solution developed to improve the workflow and quality of radiology reports. A standalone version of the software was used in this study, as opposed to integration into existing IT that occurs in clinical routine (e.g., picture archiving and communication systems and RIS. The benefits of SR have been reported in several publications.^{20, 21} Traditional narrative reporting is associated with high variability and is prone to error. According to Hawkins et al.⁶, 41% of radiology reports contain errors, and 33% contain non-grammatical errors that can lead to quality issues in patient care (e.g., errors in electronic patient records) and poor-quality routine data for research. SR allows for a significant improvement in the quality of written reports. Text is generated automatically based on information input and is edited in a concise and standardized language. SR may be easy to implement into clinical workflow for uncomplicated cases; however, patients with complex pathologies involving different diseases (e.g., breast carcinoma combined with a benign bone tumor in the humerus) with a wide range of findings require greater flexibility. Thus, percentages of unstructured free-text reporting in clinical settings will remain comparatively high.^{20, 21}

GR is a further development of SR. Major differences include reporting by anatomical region, rather than by pathology, with a manageable number of 23 modules covering the whole spectrum of clinical MRI and CT examinations (plus X-ray mammography and breast ultrasound); the mapping of a complex cognitive decision tree developed by imaging specialists with a predefined point-by-point approach guiding radiologists through the reporting process, and the conscious exclusion of free text in the findings section and in the core components of the Impression section. The software provides error-free report texts: non-grammatical omission errors are reduced by the implementation of mandatory fields in the most relevant parts of the decision tree; plausibility checks reduce the risk of missense errors due to human failure; an intuitive and clear UX design focuses on risk reduction of commission errors; and intrinsic contradictions within the imaging report (e.g., between the findings and the impression sections) are prevented by waiving free-text entries. Digumarthy et al.²² reported that 33 of 47876 radiology reports (876; 0.0007%) contained side discrepancies between the findings and impression sections. These discrepancies were revealed to be uncommon but quantifiable. Most discrepancies occurred in complex radiology reports involving multiple bilateral lesions with numerous citations of lateralization used to describe lesion distribution and location.²²

Usability and performance of guided reporting: learning how to use the technology

Our starting hypothesis that the usability of GR software is intuitive and enables radiologists to produce high-quality breast MRI reports was confirmed by the high ratings of all readers. Of the 80 reports, 51.3% revealed no content differences between FT and GR, 41.3 % contained minor differences in content between FT and GR not resulting in differences in treatment, and 7.5% contained major differences in content resulting in differences in treatment. All those reports containing major differences were COEs caused either by insufficient operation of the software or by an omission in the v. 0.8 software prototype displayable with the commercially available v. 1.7. Analysis revealed the need for extended training in 2/6 cases with the lowest score C (major differences in content, resulting in differences in treatment) and for additions to the decision tree in 4/6 cases. The importance of intensive training in the novel technology was clear from our study results. In conclusion, GR reports are complete, error-free, and without contradictions between the Findings and Impression sections. The error rate of 45% (mostly spelling and grammatical) in our evaluation of free-text narrative reporting by dictation can be lowered with the use of GR. Attached images and localizers within the GR system increase comprehensibility for referring physicians. It had been shown previously that reports with embedded images save time, increase physician confidence in treatment decisions, and may influence patient management.^{23, 24}

Increased reporting speed

The major attraction for radiologists in the use of GR in their daily routines is the reduction in reporting time. Staff shortages are a current bane in radiology; the number of scans and images per examination are rapidly rising, while the number of specialist radiologists remains more or less stagnant. In the U.S., MRI and CT rates increased by 124% and 110%, respectively, between 2000 and 2016.²⁵ In contrast, the total number of radiologists increased by only 39% between 1995 and 2011.²⁶ Our study was able to demonstrate a shortening of the combined reading and reporting time with GR compared with reference values from a narrative reporting process. The mean reading time for general MRI examinations was 17 min 39 s¹⁰ and 35 min for breast MRI specifically.¹¹ Radiologist reporting times are a key component of radiology department workload assessment. Although of pivotal importance for management decisions, reliable measurements remain challenging.²⁷ Different methods have been described. Site-specific median reporting times (MRTs) are considered more precise than the SATs²⁷ introduced by Pitman et al.¹¹ in 2018. As the MRT described by Cowan et al.^{10, 27} was very similar to our site-specific method, it was selected as a comparator. Reading time was decreased by 44.8% (general MRI). The learning curve for all readers promises great time and cost savings potential.

The necessity of leaving prosaic radiology reporting

Numerous studies have found that structured, disease-focused dictation templates improve the comprehensiveness and appropriateness of radiologist reports.^{18, 20, 21, 28, 29} Structure in reporting can help reduce errors (particularly typographical errors), train residents in the evaluation of complex examinations, and assist residents in recalling critical findings.²⁹ Structure in reporting is recommended in many MRI indications.^{30, 31}

In accordance, the following recommendation was published by the European Society of Radiology in 2018: “moving from conventional prose reports to SR is endorsed as a positive development, and must be an international effort, with international design and adoption of SR templates that can be translated and adapted in local environments as needed. Industry involvement is key to success, based on international data standards and guidelines”.³² In addition, IT-based reporting tools, and especially novel GR technology, are the basis for additional export formats of structured data (i.e., direct ICD-10 coding), as used in our study. Patient-friendly reporting, created with a single click, is essential to providing patient-centered and value-based care in the radiology of tomorrow.³³

The limitations of our study include the setting of FT as the gold standard, preventing intra-individual comparison between FT and GR regarding reporting time, as well as its relatively small sample size and single-institution, retrospective design. Future studies with double reading and reporting in a randomized order using the commercial, CE-certified version of GR software are needed to validate the reduced combined reading and reporting time and to quantify reduced radiology turnaround times. An international multicenter design with a wider range of reader experiences would provide further insight into workflow improvements in different regional and education settings. An evaluation of all readers reading all to assess intra- and inter-reader reliability and variability was not performed and should be conducted in future studies to confirm the robustness of the advantages of GR technology. In addition, testing for missense errors in GR reports, and how they compare with those in FT reports, would need direct comparative analysis, with the images as a gold standard. Missense errors as defined by Hawkins et al.⁶ (including their subclassifications into E1: translation errors, E2: errors of omission, and E3: human error, any of which could change the meaning of a phrase/sentence) could not be detected in our evaluation because FT was used as the gold standard.

In conclusion, GR allows for complete radiology reporting with a minimized error rate and a reduced combined reading and reporting time. Introducing this disruptive reporting strategy into clinical workflow requires management adjustments and intensive training, as has been emphasized by reports showing minor differences between GR and FT that are not caused by limitations of the software itself but by user-dependent operation. Shortening of reporting time is key for successful implementation into the clinical workflow. The great potential of structured datasets will open the doors for the future of radiology with respect to big data analysis, automatic ICD coding, and efficient artificial intelligence development. SR, and GR even more so, has the potential to facilitate developments in machine learning for radiological applications.³⁴