Reproducibility of rCBV in glioblastomas using T2*-weighted perfusion MRI: an evaluation of sampling, normalization, and experience
PDF
Cite
Share
Request
Neuroradiology - Original Article
P: 124-134
March 2024

Reproducibility of rCBV in glioblastomas using T2*-weighted perfusion MRI: an evaluation of sampling, normalization, and experience

Diagn Interv Radiol 2024;30(2):124-134
1. University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
No information available.
No information available
Received Date: 27.07.2023
Accepted Date: 17.09.2023
Publish Date: 06.03.2024
PDF
Cite
Share
Request

ABSTRACT

CONCLUSION

The inter-reader reproducibility of rCBV measurement was poor to moderate regardless of whether it was calculated by neuroradiologists, general radiologists, or residents, which may indicate the need for automated methods. Choosing five ROIs and using the centrum semiovale as a reference area may increase reliability for all users.

RESULTS

For individuals, the highest inter-reader reproducibility was observed between neuroradiologists [ICC: 0.527; 95% confidence interval (CI): 0.21–0.74] and between residents (ICC: 0.513; 95% CI: 0.20–0.73). There was poor reproducibility in the analyses of individuals with different levels of experience (ICC range: 0.296–0.335) and in reader-wise and group-wise pooling (ICC range: 0.296–0.335 and 0.397–0.427, respectively). However, an increase in ICC values was observed when five ROIs were used. In an analysis of all strategies, the ICC for the centrum semiovale was significantly higher than that for contralateral white matter (P < 0.001).

METHODS

In this institutional review board-approved single-center study, 30 patients with glioblastoma were retrospectively evaluated with DSC–MRI at a 3.0 Tesla scanner. Three groups of reviewers, including neuroradiologists, general radiologists, and radiology residents, calculated the rCBV based on the number of regions of interest (ROIs) and reference areas. For statistical analysis of feature reproducibility, the intraclass correlation coefficient (ICC) and Bland–Altman plots were used. Analyses were made among individuals, reader groups, reader-group pooling, and a population that contained all of them.

PURPOSE

The reproducibility of relative cerebral blood volume (rCBV) measurements among readers with different levels of experience is a concern. This study aimed to investigate the inter-reader reproducibility of rCBV measurement of glioblastomas using the hotspot method in dynamic susceptibility contrast perfusion magnetic resonance imaging (DSC–MRI) with various strategies.

Main points

• Reproducibility of relative cerebral blood volume (rCBV) measurement among readers with different experiences is a concern.

• The inter-reader reproducibility of rCBV measurement was poor to moderate when using the hotspot method, regardless of whether it was calculated by neuroradiologists, general radiologists, or radiology residents.

• Sampling five regions of interest and selecting the centrum semiovale for normalization improved reproducibility.

In neuroradiology practice, dynamic susceptibility contrast perfusion magnetic resonance imaging (DSC–MRI) is used extensively as an advanced method for the diagnosis, grading, and post-treatment follow-up of glioblastomas.1,2 The DSC–MRI technique depends on a susceptibility-caused signal loss on T2*-weighted images resulting from a bolus passage of gadolinium-based contrast media. Cerebral blood volume (CBV) is the most commonly used parameter of DSC–MRI and it defines the area under the concentration–time curve.3 CBV is basically an absolute value however, it has some assumptions/conditional requirements. Therefore, to obtain a relative quantification, it is usually rated to a reference point, such as contralateral white matter, the centrum semiovale, or arterial input function.4 The relative CBV (rCBV) is the most robust and commonly used DSC–MRI parameter for the radiological characterization of glioblastomas.3,5 It has been proven that the rCBV value is highly correlated with tumor grade, vascularity, and prognosis.2,3 Moreover, rCBV has been shown to be useful in distinguishing tumor progression from its mimickers, such as pseudoprogression and radiation necrosis.3,6,7

To calculate rCBV, radiologists draw regions of interest (ROIs) in the most hyperperfused area of a tumor and a normal-appearing reference area.8 In ROI analysis, determining the hyperperfused area on CBV map images and selecting the normal-appearing reference area are two important operator-dependent subjective issues. Despite its operator dependency and inter-observer variability, this is the most widely used method in clinical practice.9,10,11,12

Radiologists with different experience levels can potentially assess DSC–MRI. However, the assessment of DSC–MRI parameters by radiologists with different levels of experience, such as radiology residents, general radiologists, and neuroradiologists, may lead to inconsistent evaluations in the diagnosis and treatment processes. It is crucial to improve a reliable and standard analysis method for rCBV measurement to eliminate incompatibility between different users. It has been recommended that DSC–MRI measurements should be reviewed by two experienced radiologists and an adjudicator should be consulted in the event of disagreement.13 However, this recommendation is time-consuming and not always applicable; therefore, it is impractical for clinical practice. In clinical practice, the assessment of DSC–MRI by a single radiology resident (when preparing reports), a general radiologist, or an experienced neuroradiologist is not uncommon. Residents and general radiologists do not have the opportunity to consult an experienced neuroradiologist in all DSC–MRI examinations.

To our knowledge, there is limited literature on the reproducibility aspects of rCBV in terms of sampling, normalization location, and reader experience. Therefore, we sought to fill this knowledge gap in the literature by assessing the inter-reader reproducibility of rCBV measurements in glioblastoma cases in pre-surgical settings, calculated either by radiology residents, inexperienced general radiologists, or experienced neuroradiologists. Second purpose was to investigate the effect of the number of ROI and the selection of reference areas on inter-reader reproducibility.

Methods

Ethics

For this retrospective study, institutional review board approval was acquired from the Local Medical Ethics Committee of Basaksehir Cam and Sakura City Hospital (no: 2023-18; decision date: 11/01/2023). The requirement for written informed consent was waived by the ethics committee due to the study’s retrospective design. The study adhered to the principles of the Helsinki Declaration.

Study population

In this single-center retrospective study, 30 patients with newly diagnosed glioblastoma were consecutively included between July 2021 and January 2023. The reason for including 30 cases was to meet the minimum requirement for a reliability analysis.14 This study was conducted in a tertiary academic hospital.

Eligibility criteria

The inclusion criteria were as follows: i, a definitive histopathologic diagnosis of glioblastoma according to the World Health Organization’s 2021 classification;15 ii, ≥18 years old at the time of DSC–MRI.

The following criteria determined exclusion: i, history of cranial surgical resection or biopsy, or radiotherapy, chemotherapy, corticosteroid, or anti-angiogenic treatment before imaging; ii, MRI scans with severe artifacts that impeded the diagnostic evaluation.

Magnetic resonance imaging acquisition protocol and processing

All MRI scans were performed at a 3.0 Tesla system (Ingenia, Philips Healthcare) using 32-channel phased-array head coils in the supine position. The routine DSC–MRI examination protocol included axial spin–echo T1-weighted imaging [repetition time (TR)/echo time (TE): 600/10 ms; field of view (FOV): 230 mm; slice thickness: 4 mm; matrix: 208 × 165; number of excitations (NEX): 1], a 3D axial fluid-attenuated inversion recovery sequence (TR/TE: 4,800/340 ms; inversion time: 1,650 ms; FOV: 230 mm; slice thickness: 4 mm; matrix: 272 × 243; NEX: 1), subsequent DSC–MRI data, and finally, 3D postcontrast T1-weighted imaging (TR/TE: 600/10 ms; FOV: 230 mm; slice thickness: 4 mm; matrix: 208 × 165; NEX: 1).

DSC–MRI was obtained on an axial plan with a gradient-echo echoplanar imaging technique using the following parameters: TR/TE: 1,500/30 ms; FOV: 237 × 237 mm; matrix: 128 × 128; section thickness: 3 mm; flip angle: 60°; voxel size: 2.33 × 2.39 × 4.00 mm. An intravenous bolus injection of gadolinium-based contrast agent was given at a dose of 0.1 mmol/kg and a speed of 5 mL/s, followed by a 20-mL saline flush. Before the dynamic phase, a saturation pre-bolus of contrast agent was administered as a preload to reduce contaminating T1 effects from contrast agent leakage. The DSC–MRI protocol of this study was in line with consensus recommendations.16

Dynamic susceptibility contrast perfusion magnetic resonance imaging processing and observer setting

The DSC–MRI datasets were processed using the IntelliSpace Portal (Philips). Maximum rCBV values were calculated independently by three groups of readers (six readers in total). Individual readers were denoted as R1, R2, R3, R4, R5, and R6.

Readers R1 and R2 were neuroradiologists with 5 years of experience in neuroimaging in research settings. Readers R3 and R4 were general radiologists with 4 years of experience in general radiology who were working in all divisions of radiology. Readers R5 and R6 were third-year radiology residents who had completed neuroradiology rotations at the beginning of this study. All readers were blinded to the clinical information and histopathological results of patients. All data were anonymized.

Inter-reader reproducibility was assessed using various strategies as follows: i, between individuals with the same experience levels (R1 vs. R2, R3 vs. R4, and R5 vs. R6); ii, between individuals with different experience levels (R1 vs. R3, R3 vs. R5, and R1 vs. R5). Since two readers in each group had similar experience levels and to avoid complicating the analysis, only one reader was randomly selected from each group and compared with the other readers; iii, between reader groups with different experience levels (neuroradiologists vs. general radiologists, neuroradiologists vs. residents, and general radiologists vs. residents); iv, inter-reader reproducibility of group-wise (neuroradiologists vs. general radiologists vs. residents) and reader-wise (R1 vs. R2 vs. R3 vs. R4 vs. R5 vs. R6) pooling.

Region of interest analysis

The reviewers were encouraged to place five different ROIs within the tumor that visually appeared as mostly hyperperfused on colored relative CBV map images (hotspot method). The CBV value of the first ROIs (CBV1), the highest CBV value among the first three ROIs (CBV3), and the highest CBV value among five ROIs (CBV5) were recorded. Only the highest CBV value among the three and five ROIs was used (not their means). Then, the reviewers were instructed to place an ROI of the same size on the contralateral normal-appearing white matter in the same axial section as the tumor’s ROI and the contralateral normal-appearing centrum semiovale, which are known to be the most reliable reference areas.13 The CBV values obtained from CBV1, CBV3, and CBV5 were rated to these reference areas to obtain a normalized rCBV. Statistical analyses were performed separately for each reference area.

All circular ROIs were drawn manually by the readers on CBV map images and ranged between 40 and 60 mm2. In the ROI analyses, care was taken to avoid hemorrhagic, necrotic, or cystic regions, normal grey matter, and intralesional non-tumor large vessels that might affect the values. A multi-ROI analysis of a glioblastoma case is represented in Figure 1.

Figure 1

Statistical analysis

The statistical analysis was performed using R 4.3 (main packages: “MKinfer” and “ggpubr” for reliability analyses and inferential statistics; tool: JASP for descriptive statistics only)17 and Python 3.7 (main package: pingouin for reliability analyses) environments.18 To assess feature reliability, the mean and 95% CI values of the intraclass correlation coefficient (ICC) were calculated. The ICC was based on the type of ICC (2,1) according to Shrout and Fleiss’s convention.19 The interpretation scale for the ICC was as follows: <0.5: poor; ≤0.5 to <0.75: moderate; ≤0.75 to <0.9: good; and ≥90: excellent.14 In addition to the ICC analysis, non-parametric Bland–Altman analyses were performed to evaluate the differences in measurements and the limits of agreement, relying on median and 2.5–97.5th percentiles, respectively. The Shapiro–Wilk test was used to determine the normality of continuous variables. Depending on the group distributions, a paired non-parametric test, the Wilcoxon signed-rank test, was used to assess pairwise statistical differences in continuous variables. Statistical results were considered significant if P <0.050. In the case of multiple comparisons, the results were considered significant if the adjusted P values were <0.050 after multiplicity correction using the Bonferroni method.

Results

Patient characteristics

In total, 30 consecutive glioblastoma cases were enrolled, 17 were male, and 13 were female. The mean age of the patients (standard deviation) was 61.1 (9.7) years (range: 38–78 years).

According to the number of ROIs and reference areas, the median rCBV values ranged between 13.7 and 20.1 for neuroradiologists, 18.1 and 22.1 for general radiologists, and 10.1 and 12.8 for residents. The median and the interquartile range (IQR) of the rCBV values of all readers, which were calculated by using the hotspot method, are presented in Supplementary Table 1.

Table 1

Inter-reader reproducibility for individuals with similar experience levels

The inter-reader reproducibility of the rCBV measurements was poor to moderate (ICC range: 0.288–0.527). The highest inter-reader reproducibility of the rCBV measurements was obtained between neuroradiologists (R1 vs. R2) using one ROI and normalization with white matter (ICC: 0.527; 95% CI: 0.21–0.74) and between residents (R5 vs. R6) using one ROI and normalization with the centrum semiovale (ICC: 0.513; 95% CI: 0.20–0.73). The ICC value of general radiologists (R3 vs. R4) increased from 0.312 to 0.370, with a higher number of ROIs (from one to five) using the centrum semiovale as a reference area. The ICC value of each analysis is presented in detail in Table 1.

Table 1

Inter-reader reproducibility for individuals with different experience levels

In all analyses, the inter-reader reproducibility of the rCBV measurements was poor (ICC range: 0.296–0.335). However, an increase in ICC values was observed when five ROIs were used instead of one or three ROIs, even if contralateral white matter or the centrum semiovale is used for normalization (Supplementary Table 2).

Table 2

Inter-reader reproducibility for reader groups with different experience levels

In the analysis of reader groups, the inter-reader reproducibility of rCBV measurements was moderate (ICC range: 0.566–0.640) for neuroradiologists vs. general radiologists. On the other hand, inter-reader reproducibility was poor for all the other group-based analyses. The ICC values ranged between 0.350 and 0.422 for neuroradiologists vs. residents and between 0.254 and 0.334 for general radiologists vs. residents (Table 2).

Table 2

Inter-reader reproducibility for group-wise and reader-wise pooling

In both the group-wise (all groups pooled) and reader-wise (all readers pooled) pooling analyses, inter-reader reproducibility was poor. The ICC ranged between 0.397 and 0.427 and between 0.296 and 0.335 for group-wise pooling and reader-wise pooling, respectively. Similar to the results of individuals with different experience levels, an increase in ICC values was observed when five ROIs were used. All the ICC values are presented in Table 3.

Table 3

Analysis of the overall reader population of perfusion magnetic resonance imaging

An additional analysis, that included all the readers, groups, and pooled analyses, was performed (n = 66). The ICC values of all the analyses performed are summarized in Figure 2.

Figure 2

While the inter-reader reproducibility of the general radiologists (R3 vs. R4) was poor (Table 1), six of the top 10 most reproducible analyses involved neuroradiologists vs. general radiologists. Among all the analyses, the top two in terms of inter-reader reproducibility was for neuroradiologists vs. general radiologists using the centrum semiovale, with an ICC value of 0.640 (95% CI: 0.32–0.82) and 0.583 (95% CI: 0.29–0.78) for one ROI and five ROIs, respectively. The top 10 most reproducible results among all the analyses are presented in Table 4.

Table 1
Table 4

In this analysis, the median of the ICC value was 0.349 (IQR: 0.116) for the centrum semiovale and 0.305 (IQR: 0.107) for white matter. The inter-reader agreement was higher, with a statistically significant difference for the centrum semiovale (P < 0.001). Comparisons were performed according to the number of ROIs. The median of the ICC value was 0.335 (IQR: 0.117) for five ROIs, 0.321 (IQR: 0.128) for one ROI, and 0.316 (IQR: 0.112) for three ROIs. There was a significant difference between three and five ROIs, including all reference areas (P < 0.001). In addition, a significant difference was observed between three and five ROIs when using one of the two reference areas (P < 0.010). There was no significant difference in the use of one or three ROIs (P > 0.050). All results are summarized in Figure 3.

Figure 3

Figures 4 and 5 show Bland–Altman plots of the readers based on centrum semiovale normalization with the same and different experience levels, respectively. Supplementary Figures 1 and 2 show Bland–Altman plots based on white matter normalization. In all the Bland–Altman analyses, the vast majority of ROI measurements were within the upper and lower agreement limits. In the analysis based on white matter normalization, most of ROI measurements by neuroradiologists were quite close to the zero line (Supplementary Figure 1). The same condition was observed for residents using centrum semiovale normalization (Figure 4). In general, ROI measurements were far from the zero line for general radiologists (Figure 4 and Supplementary Figure 1).

Figure 4
Figure 5
Figure 1
Figure 2

Discussion

In the present study, we investigated the inter-reader reproducibility of rCBV measurements in patients with untreated glioblastomas. Generally, a moderate inter-reader agreement was observed when analyses were made between individuals with the similar experience level. In contrast, there was poor inter-reader reproducibility when the analyses were made between different experience levels. According to all the analyses performed, the inter-reader agreement of rCBV measurements when using the centrum semiovale as a reference area was significantly higher than when using contralateral white matter. In addition, the ICC values for the placement of five ROIs were significantly higher than with one or three ROIs.

In clinical radiology practice, T2*-weighted perfusion MRI is often used for gliomas. The radiology community pays relatively little attention to the reproducibility of the derived parameters, despite their importance. In this study, we focused on preoperative glioblastoma cases to make the findings more evident, i.e., to assess the reliability of obviously high perfusion values. In clinical practice, reproducible perfusion parameters are essential for the consistent target area selection of gliomas in stereotactic biopsies and for establishing consistent baseline perfusion parameter values for use in post-treatment follow-up scans. Additionally, the reproducibility of these parameters is necessary for research consistency to ensure that the results of different studies are comparable, which may increase the validity of the conclusions drawn from pooled data and meta-analyses.

In previously published reliability studies, only normal-appearing contralateral white matter20,21 or the contralateral centrum semiovale13 has been generally selected as a reference area. In these studies, three observers were selected to assess DSC–MRI, including only neuroradiologists13,21 or neuroradiologists and a resident.20,22 We noticed that there was sparse radiological research literature on inter-reader reproducibility analyses of rCBV measurements for a population that includes neuroradiologists, general radiologists, and residents in the same study. According to our experience and knowledge, in clinical practice, general radiologists and residents (when preparing reports to present to neuroradiologists) may have to evaluate DSC–MRI, although this is not as common as with neuroradiologists. Therefore, inter-reader reproducibility of general radiologists and residents within themselves and between other reviewers is a matter of concern.

Definitive interpretations of DSC-MRI in patients with glioblastoma should be conducted and interpreted by specialized neuroradiologists. The most valuable advantage and originality of our paper is that it included two different groups in addition to neuroradiologists in the same study. For all analyses, normalization with the centrum semiovale acquired better inter-reader reproducibility than with white matter. According to our observation, the reason for this difference could be explained by the fact that the centrum semiovale is a large homogenous area that is mostly visible in more than two axial slices, and it suffers less from partial volume artifacts compared with contralateral white matter. In a recently published retrospective study, Roques et al.20 evaluated the inter-observer reproducibility of rCBV measurements in 27 cases of untreated glioblastoma. In that study, three observers (two neuroradiologists and a radiology resident) calculated the maximum rCBV values independently using the hotspot method, similar to our study. However, differently, they used only contralateral white matter as a reference to normalize the CBV value. Their inter-observer reproducibility for maximum rCBV value measurements was fair [ICC: 0.46 (0.22–0.67)], but their inter-observer reproducibility was found to be poor to fair (ICC range: 0.30–0.47) when a resident was added to the analysis. Our research confirms the variability of the hotspot method, similar to the results of Roques et al.’s20 study. However, their study did not include general radiologists and did not assess inter-observer reliability among radiology residents or general radiologists, which are the main differences of our study. Another advantage of our research is the use of leakage correction with a gadolinium preload to avoid the underestimation of CBV values. Furthermore, our study assessed the effect of selecting two different reference areas on inter-observer reliability.

The present study has the following limitations: a small sample size, a retrospective nature, and involvement of only a single institution. In this research, we only studied cases with untreated glioblastomas, in which DSC–MRI perfusion is anticipated to be increased. This study was conducted only on the initial DSC–MRI examinations of untreated glioblastomas and did not include the evaluation of post-treatment perfusion MRI features. Including only the baseline DSC-MRIs in our study may have contributed to reliability.

In conclusion, there is poor to moderate inter-reader reproducibility of rCBV measurements using the hotspot method, regardless of whether they are calculated by neuroradiologists, general radiologists, or radiology residents. This may indicate the need for automated methods. Selecting five ROIs and using the centrum semiovale as a reference area for normalization may increase the inter-reader reproducibility of measurements.

References

1
Shiroishi MS, Castellazzi G, Boxerman JL, et al. Principles of T2*-weighted dynamic susceptibility contrast MRI technique in brain tumor imaging. J Magn Reson Imaging. 2015;41(2):296-313.
2
Thust SC, Heiland S, Falini A, et al. Glioma imaging in Europe: a survey of 220 centres and recommendations for best clinical practice. Eur Radiol. 2018;28(8):3306-3317.
3
Essig M, Shiroishi MS, Nguyen TB, et al. Perfusion MRI: the five most frequently asked technical questions. AJR Am J Roentgenol. 2013;200(1):24-34.
4
Saini J, Gupta RK, Kumar M, et al. Comparative evaluation of cerebral gliomas using rCBV measurements during sequential acquisition of T1-perfusion and T2*-perfusion MRI. PLoS One. 2019;14(4):e0215400.
5
Cha S, Knopp EA, Johnson G, Wetzel SG, Litt AW, Zagzag D. Intracranial mass lesions: dynamic contrast-enhanced susceptibility-weighted echo-planar perfusion MR imaging. Radiology. 2002;223(1):11-29.
6
Law M, Yang S, Wang H, et al. Glioma grading: sensitivity, specificity, and predictive values of perfusion MR imaging and proton MR spectroscopic imaging compared with conventional MR imaging. AJNR Am J Neuroradiol. 2003;24(10):1989-1998.
7
Boxerman JL, Ellingson BM, Jeyapalan S, et al. Longitudinal DSC-MRI for distinguishing tumor recurrence from pseudoprogression in patients with a high-grade glioma. Am J Clin Oncol. 2017;40(3):228-234.
8
Sadeghi N, D’Haene N, Decaestecker C, et al. Apparent diffusion coefficient and cerebral blood volume in brain gliomas: relation to tumor cell density and tumor microvessel density based on stereotactic biopsies. AJNR Am J Neuroradiol. 2008;29(3):476-482.
9
Rani N, Singh B, Kumar N, et al. Differentiation of recurrent/residual glioma from radiation necrosis using semi quantitative 99mTc MDM (bis-methionine-DTPA) brain SPECT/CT and dynamic susceptibility contrast-enhanced MR perfusion: a comparative study. Clin Nucl Med. 2018;43(3):74-81.
10
Santarosa C, Castellano A, Conte GM, et al. Dynamic contrast-enhanced and dynamic susceptibility contrast perfusion MR imaging for glioma grading: preliminary comparison of vessel compartment and permeability parameters using hotspot and histogram analysis. Eur J Radiol. 2016;85(6):1147-1156.
11
Dijkstra H, Sijens PE, van der Hoorn A, van Laar PJ. Inter-observer reproducibility of quantitative dynamic susceptibility contrast and diffusion MRI parameters in histogram analysis of gliomas. Acta Radiol. 2020;61(1):76-84.
12
Oei MTH, Meijer FJA, Mordang JJ, et al. Observer variability of reference tissue selection for relativecerebral blood volume measurements in glioma patients. Eur Radiol. 2018;28(9):3902-3911.
13
Smits M, Bendszus M, Collette S, et al. Repeatability and reproducibility of relative cerebral blood volume measurement of recurrent glioma in a multicentre trial setting. Eur J Cancer. 2019;114:89-96.
14
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163.
15
Louis DN, Perry A, Wesseling P, et al. The 2021 WHO Classification of Tumors of the Central Nervous System: a summary. Neuro Oncol. 2021;23(8):1231-1251.
16
Boxerman JL, Quarles CC, Hu LS, et al. Consensus recommendations for a dynamic susceptibility contrast MRI protocol for use in high-grade gliomas. Neuro Oncol. 2020;22(9):1262-1275.
17
R Core Team. R: A Language and Environment for Statistical Computing. 2022.
18
Van Rossum G, Drake FL. Python 3 Reference Manual. 2009.
19
Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-428.
20
Roques M, Raveneau M, Adam G, et al. Reproducibility of volume analysis of dynamic susceptibility contrast perfusion-weighted imaging in untreated glioblastomas. Neuroradiology. 2022;64(9):1763-1771.
21
Jung SC, Choi SH, Yeom JA, et al. Cerebral blood volume analysis in glioblastomas using dynamic susceptibility contrast-enhanced perfusion MRI: a comparison of manual and semiautomatic segmentation methods. PLoS One. 2013;8(8):e69323.
22
Kouwenberg V, van Santwijk L, Meijer FJA, Henssen D. Reliability of dynamic susceptibility contrast perfusion metrics in pre- and post-treatment glioma. Cancer Imaging. 2022;22(1):28.