RSNA-ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge 2021

BraTS2021

The Brain Tumor Segmentation (BraTS) challenge celebrates its 10th anniversary, and this year is jointly organized by the Radiological Society of North America (RSNA), the American Society of Neuroradiology (ASNR), and the Medical Image Computing and Computer Assisted Interventions (MICCAI) society.

The RSNA-ASNR-MICCAI BraTS 2021 challenge utilizes multi-institutional pre-operative baseline multi-parametric magnetic resonance imaging (mpMRI) scans, and focuses on the evaluation of state-of-the-art methods for (Task 1) the segmentation of intrinsically heterogeneous brain glioblastoma sub-regions in mpMRI scans. Furthemore, this BraTS 2021 challenge also focuses on the evaluation of (Task 2) classification methods to predict the MGMT promoter methylation status.

Participants are free to choose whether they want to focus only on one or both tasks.

 

(All deadlines are for 23:59 Eastern Time)

1 July Registration opens (Task 1: Segmentation)
7 July

Task 1 Training Phase starts: (Release of training data + associated ground truth).

15 July Task 2 Launch (Registration & Training phase starts)  
30 July Validation phase (Release of validation data, with hidden ground truth).
13 Aug Submission of short papers, reporting method & preliminary results.
27 Aug Contacting methods top-ranked in validation phase, to prepare slides for oral presentation at MICCAI.
4 Oct. Challenge at MICCAI. Presentation of top-ranked validation phase methods.
20-30 Oct Final ranking phase on unseen testing data.
29 Nov Challenge conclusion at RSNA 2021. Announcement of top 3 ranked teams & distribution of awards.
12 Dec Extended Camera-Ready LNCS paper submission deadline.
   
  (All deadlines are for 23:59 Eastern Time)

Glioma, and particularly glioblastoma, and diffuse astrocytic glioma with molecular features of glioblastoma (WHO Grade 4 astrocytoma), are the most common and aggressive malignant primary tumor of the central nervous system in adults, with extreme intrinsic heterogeneity in appearance, shape, and histology. Glioblastoma patients have very poor prognosis, and the current standard of care comprises surgery, followed by radiotherapy and chemotherapy. MGMT (O[6]-methylguanine-DNA methyltransferase) is a DNA repair enzyme that the methylation of its promoter in newly diagnosed glioblastoma has been identified as a favorable prognostic factor and a predictor of chemotherapy response. Thus determination of MGMT promoter methylation status in newly diagnosed glioblastoma can influence treatment decision making.

The International Brain Tumor Segmentation (BraTS) Challenges —which have been running since 2012— assess state-of-the-art machine learning methods used for brain tumor image analysis in mpMRI scans.

Participants are free to choose whether they want to focus only on one or both tasks.

Task 1: Brain Tumor Segmentation in mpMRI scans.

The participants are called to address this task by using the provided clinically-acquired training data to develop their method and produce segmentation labels of the different glioma sub-regions. The sub-regions considered for evaluation are the "enhancing tumor" (ET), the "tumor core" (TC), and the "whole tumor" (WT) [see figure below]. The ET is described by areas that show hyper-intensity in T1Gd when compared to T1, but also when compared to “healthy” white matter in T1Gd. The TC describes the bulk of the tumor, which is what is typically resected. The TC entails the ET, as well as the necrotic (NCR) parts of the tumor. The appearance of NCR is typically hypo-intense in T1-Gd when compared to T1. The WT describes the complete extent of the disease, as it entails the TC and the peritumoral edematous/invaded tissue (ED), which is typically depicted by hyper-intense signal in FLAIR.

The provided segmentation labels have values of 1 for NCR, 2 for ED, 4 for ET, and 0 for everything else.
 

The participants are called to upload their method in a containerized way for evaluation. More details can be found at the Synapse platform, which is the official performance evaluation and ranking platform for the tumor sub-region segmentation task.

Evaluation Approach

Consistent with the configuration of previous BraTS challenges, we intend to use the "Dice score", and the "Hausdorff distance (95%)". Expanding upon this evaluation scheme, we will also provide the metrics of "Sensitivity" and "Specificity", allowing to determine potential over- or under-segmentations of the tumor sub-regions by participating methods.

 

Task 2: Radiogenomics: Prediction of the MGMT promoter methylation status in mpMRI scans.

The participants are called to use the provided mpMRI data to extract imaging/radiomic features that they consider appropriate, and analyze them through machine learning algorithms, in an attempt to predict the MGMT promoter methylation status. The participants do not need to be limited to volumetric parameters, but can also consider intensity, morphologic, histogram-based, and textural features, as well as spatial information, deep learning features, and glioma diffusion properties extracted from glioma growth models.

Note that participants will be evaluated for the predicted MGMT status of the subjects indicated in the accompanying spreadsheet.

The participants are called to upload their method in a containerized way for evaluation. More details will be provided soon at Kaggle, which is the official performance evaluation and ranking platform for the radiogenomic classification task.

Evaluation Approach

The participating teams will be evaluated and ranked based on the area under the receiver operating characteristics curve for the classification of the MGMT status as methylated and unmethylated. Predictions of the participating teams will also be assessed based on accuracy (i.e., the number of correctly classified patients) with respect to this grouping.

To register for participation and get access to the BraTS 2021 data, you can follow the instructions given at the "Registration/Data Request" section below.

Ample multi-institutional routine clinically-acquired multi-parametric MRI (mpMRI) scans of glioma, with pathologically confirmed diagnosis and available MGMT promoter methylation status (for the glioblastoma cases with such associated data), are used as the training, validation, and testing data for this year’s BraTS challenge.

Specifically, the datasets used in this year's challenge have been updated, since BraTS'20, with many more routine clinically-acquired mpMRI scans. Ground truth annotations of the tumor sub-regions are created and approved by expert neuroradiologists for every subject included in the training, validation, and testing datasets to quantitatively evaluate the predicted tumor segmentations of Task 1, whereas the quantitative evaluation of Task 2 is performed according to clinical data.

Validation data will be released on July, to allow participants obtain preliminary results in unseen data and also report it in their submitted short LNCS papers (due on August 13), in addition to their cross-validated results on the training data. The ground truth of the validation data will not be provided to the participants, but multiple submissions to the online evaluation platform will be allowed.  The participating teams top-ranked in the validation phase will be invited by August 27, to prepare their slides for a short oral presentation of their method during the BraTS challenge at MICCAI.

Finally, all participants will be evaluated and ranked on the same unseen testing data, which will not be made available to the participants, but the participants are required to upload their method in the Synapse evaluation platform (due on 20-30 October).

The final top-ranked participating teams will be announced during the BraTS challenge at RSNA.

Imaging Data Description

All BraTS mpMRI scans are available as NIfTI files (.nii.gz) for Task 1 (Segmentation) and as DICOM (.dcm) files for Task 2 (Classification). These mpMRI scans describe a) native (T1) and b) post-contrast T1-weighted (T1Gd), c) T2-weighted (T2), and d) T2 Fluid Attenuated Inversion Recovery (T2-FLAIR) volumes, and were acquired with different clinical protocols and various scanners from multiple institutions, mentioned as data contributors here. We intend to release all corresponding de-identified DICOM (.dcm) and NIFTI (.nii.gz) files for both tasks after the conclusion of the challenge.

All the imaging datasets have been segmented manually, by one to four raters, following the same annotation protocol, and their annotations were approved by experienced board-certified neuro-radiologists. Annotations comprise the GD-enhancing tumor (ET — label 4), the peritumoral edematous/invaded tissue (ED — label 2), and the necrotic tumor core (NCR — label 1), as described both in the BraTS 2012-2013 TMI paper and in the latest BraTS summarizing paper. The ground truth data were created after their pre-processing, i.e., co-registered to the same anatomical template, interpolated to the same resolution (1 mm^3) and skull-stripped.

Comparison with Previous BraTS datasets

The BraTS 2021 data of 2,000 cases (8,000 mpMRI scans) represent a superset of the BraTS 2020 data of 660 cases (2640 mpMRI scans). The BraTS 2020-2017 data, differs significantly from the data provided during the previous BraTS challenges (i.e., 2016 and backwards). The only data that have been previously used and are utilized again (during BraTS'17-'20) are the images and annotations of BraTS'12-'13, which have been manually annotated by clinical experts in the past. The data used during BraTS'14-'16 (from TCIA) have been discarded, as they described a mixture of pre- and post-operative scans and their ground truth labels have been annotated by the fusion of segmentation results from algorithms that ranked highly during BraTS'12 and '13. For BraTS'17, expert neuroradiologists have radiologically assessed the complete original TCIA glioma collections (TCGA-GBM, n=262 and TCGA-LGG, n=199) and categorized each scan as pre- or post-operative. Subsequently, all the pre-operative TCIA scans (135 GBM and 108 LGG) were annotated by experts for the various glioma sub-regions and included in this year's BraTS datasets. The exact procedures for these cases can be found in this manuscript.

This year we also provide the naming convention and direct filename mapping between the data of BraTS'21-'17, and the subjects used from the data collections of TCGA-GBM, TCGA-LGG, IvyGAP, and CPTAC-GBM, available through The Cancer Imaging Archive (TCIA) to further facilitate research beyond the directly BraTS related tasks.

MGMT Promoter Methylation Data Description

The MGMT promoter methylation status data, defined as a binary label (0:unmethylated, 1:methylated), are included in a comma-separated value (.csv) file with correspondences to the pseudo-identifiers of the imaging data.

Use of Data Beyond BraTS

Participants are allowed to use additional public but NOT private data (from their own institutions) for extending the provided BraTS data, for the training of the algorithm chosen to be ranked. Similarly, using models that were pretrained on private datasets is NOT allowed. This is due to our intentions to provide a fair comparison among the participating methods. However, participants are allowed to use additional public and/or private data (from their own institutions), only if for scientific publication purposes and they explicitly mention this in their submitted manuscripts and also report results using only the BraTS'21 data to discuss potential result differences.

Data Usage Agreement / Citations

You are free to use and/or refer to the BraTS datasets in your own research, provided that you always cite the following three manuscripts:

[1] U.Baid, et al., "The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification", arXiv:2107.02314, 2021.

[2] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, et al. "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)", IEEE Transactions on Medical Imaging 34(10), 1993-2024 (2015) DOI: 10.1109/TMI.2014.2377694

[3] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J.S. Kirby, et al., "Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features", Nature Scientific Data, 4:170117 (2017) DOI: 10.1038/sdata.2017.117

In addition, if there are no restrictions imposed from the journal/conference you submit your paper about citing "Data Citations", please be specific and also cite the following:

[4] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, et al., "Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-GBM collection", The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.KLXWJJ1Q

[5] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, et al., "Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-LGG collection", The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.GJQ7R0EF

 

Note: Use of the BraTS datasets for creating and submitting benchmark results for publication on MLPerf.org is considered non-commercial use. It is further acceptable to republish results published on MLPerf.org, as well as to create unverified benchmark results consistent with the MLPerf.org rules in other locations. Please note that you should always adhere to the BraTS data usage guidelines and cite appropriately the aforementioned publications, as well as to the terms of use required by MLPerf.org.

 

Feel free to send any communication related to the BraTS challenge to brats@cbica.upenn.edu

Training Data availability. Register for this year's challenge, to get access to the DICOM and NIFTI, skull-stripped, and annotated training data.

Validation Data availability. An independent set of validation scans will be made available to the participants in July, with the intention to allow them assess the generalizability of their methods in unseen data, via the official evaluation platforms. At this point a public leaderboard will also be made available.

Short Paper submission deadline (August 13). Participants will have to evaluate their methods on the training and validation datasets, and submit their short paper (8-10 LNCS pages — together with the "LNCS Consent to Publish" form), describing their segmentation method and results to the BrainLes CMT submission system, and make sure you choose BRATS as the "Track". Please ensure that you include the appropriate citations, mentioned at the bottom of the "Data" section. This unified scheme should allow for appropriate preliminary comparisons and the creation of the MICCAI BrainLes conference proceedings. Participants are allowed to submit longer papers to the MICCAI 2021 BrainLes Workshop, by choosing "BrainLes" as the "Track". BraTS papers will be part of the BrainLes workshop proceedings distributed by Springer LNCS. All paper submissions should use the LNCS template, available both in LaTeX and in MS Word format, directly from Springer (link here).

Oral Presentations at MICCAI. The top-ranked participants of the validation phase that have also submitted a short paper, will be contacted by August 27 to prepare slides for orally presenting their method during the BraTS session in MICCAI.

Final ranking phase (October 20-30). The BraTS 2021 test scans will not be made available to the participating teams. Participants will need to submit their method in a containerized form (more details will follow up a the respective evaluation platforms) to be evaluated on the hidden testing data.

Announcement of Final Results (Nov 29). Ranking and final results of the challenge will be reported during the BraTS'21 challenge, which will run as part of the Annual Scientific Meeting of the RSNA 2021 .

Post-conference LNCS paper (Dec 12). All participated methods will be invited to extend their papers to 11-14 pages for inclusion to the Springer LNCS proceedings of the BrainLes Workshop.

Joint post-conference journal paper. All participating teams will be involved to the joint manuscript summarizing the results of RSNA-ASNR-MICCAI BraTS 2021 Challenge, that will be submitted to a high-impact journal in the field.

 

Feel free to send any communication related to the BraTS challenge to brats@cbica.upenn.edu.

Challenge data may be used for all purposes, provided that the challenge is appropriately referenced using the citations given at the bottom of this page.

To request the training and the validation data of the BraTS 2021 challenge, please follow the registration steps below. Please note that the i) training data includes ground truth annotations, ii) validation data does not include annotations, and iii) testing data are not available to the public or to the challenge participants.

 

 

  • TO REGISTER FOR TASK 1, create an account at the Synapse platform, and fill out this Google form, to get access to the related training data. Note that the Synapse platform is the official performance evaluation and ranking platform for the tumor sub-region segmentation task.

 

  • TO REGISTER FOR TASK 1, create an account at the Kaggle platform, to get access to the related training data. Note that the Kaggle platform is the official performance evaluation and ranking platform for the radiogenomic classification task.

 

 

You are free to use and/or refer to the BraTS datasets in your own research, provided that you always cite the following three manuscripts:

[1] U.Baid, et al., "The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification", arXiv:2107.02314, 2021.

[2] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, et al. "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)", IEEE Transactions on Medical Imaging 34(10), 1993-2024 (2015) DOI: 10.1109/TMI.2014.2377694

[3] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J.S. Kirby, et al., "Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features", Nature Scientific Data, 4:170117 (2017) DOI: 10.1038/sdata.2017.117

In addition, if there are no restrictions imposed from the journal/conference you submit your paper about citing "Data Citations", please be specific and also cite the following:

[4] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, et al., "Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-GBM collection", The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.KLXWJJ1Q

[5] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, et al., "Segmentation Labels and Radiomic Features for the Pre-operative Scans of the TCGA-LGG collection", The Cancer Imaging Archive, 2017. DOI: 10.7937/K9/TCIA.2017.GJQ7R0EF

 

Feel free to send any communication related to the BraTS challenge to brats@cbica.upenn.edu

BraTS 2021 builds upon its 9 previous successful instances:

Feel free to send any communication related to the BraTS challenge to brats@cbica.upenn.edu

Organizing Committee

(in alphabetical order)

  • Spyridon (Spyros) Bakas, Ph.D.,   —   [Lead Organizer]    Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA

  • Ujjwal Baid, Ph.D.,    Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA

  • Christopher Carr,    Radiological Society of North America (RSNA), Oak Brook, IL, USA

  • Evan Calabrese, M.D., Ph.D.,    Department of Radiology & Biomedical Imaging, University of California San Francisco, CA,
    USA

  • Errol Colak, M.D.,    Unity Health Toronto, University of Toronto, Toronto, ON

  • Keyvan Farahani, Ph.D.,    Center for Biomedical Informatics and Information Technology (CBIIT), National Cancer Institute (NCI),
    National Institutes of Health (NIH), Bethesda, MD, USA 

  • Adam Flanders, M.D.,    Thomas Jefferson University Hospital, Philadelphia, PA, USA

  • Felipe Kitamura, M.D., Ph.D.,    Diagnósticos da América SA (Dasa) and Universidade Federal de São Paulo (UNIFESP), São Paulo, Brazil

  • Bjoern Menze, Ph.D.,    University of Zurich, Switzerland

  • Luciano M. Prevedello, M.D., M.P.H.,    The Ohio State University Wexner Medical Center, Columbus, OH, USA

  • Jeffrey D. Rudie, M.D., Ph.D.,    Department of Radiology & Biomedical Imaging, University of California San Francisco, CA,
    USA 

  • Russell Taki Shinohara, Ph.D.,    University of Pennsylvania, Philadelphia, PA, USA

 

Data Contributors

(in order of decreasing data contributions)

Clinical Evaluation and Annotation Approval

  • Satyam Ghodasara, M.D., Ph.D.,    Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA

  • Suyash Mohan, MD,    Division of Neuroradiology, Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA

  • Michel Bilello, MD, Ph.D.,    Division of Neuroradiology, Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA

 

Annotation Volunteers

*The complete list of volunteer annotators from the American Society of Neuroradiology (ASNR) will be updated here soon.

 

Awards Sponsor

Feel free to send any communication related to the BraTS challenge to brats@cbica.upenn.edu