In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients\u2014manually annotated by up to four raters\u2014and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%\u201385%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.<\/p>\n","protected":false},"excerpt":{"rendered":"
In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients\u2014manually annotated by up to four raters\u2014and […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"msr-content-type":[3],"msr-research-highlight":[],"research-area":[13556,13562],"msr-publication-type":[193715],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[246694,251923,265980,246691,265977,248920,248398,246682,256360],"msr-conference":[],"msr-journal":[265983],"msr-impact-theme":[261673],"msr-pillar":[],"class_list":["post-167562","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-locale-en_us","msr-field-of-study-artificial-intelligence","msr-field-of-study-benchmark-computing","msr-field-of-study-brain-tumor","msr-field-of-study-computer-science","msr-field-of-study-cut","msr-field-of-study-data-mining","msr-field-of-study-image-segmentation","msr-field-of-study-pattern-recognition","msr-field-of-study-set-abstract-data-type"],"msr_publishername":"","msr_edition":"","msr_affiliation":"","msr_published_date":"2015-9-30","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"IEEE Transactions on Medical Imaging","msr_volume":"34","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"10","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"http:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=6975210","msr_doi":"","msr_publication_uploader":[{"type":"doi","viewUrl":"false","id":"false","title":"10.1109\/TMI.2014.2377694","label_id":"243106","label":0},{"type":"url","viewUrl":"false","id":"false","title":"https:\/\/digitalcommons.odu.edu\/cgi\/viewcontent.cgi?article=1146&context=ece_fac_pubs","label_id":"243132","label":0},{"type":"url","viewUrl":"false","id":"false","title":"https:\/\/hal.inria.fr\/hal-00935640\/document","label_id":"243132","label":0}],"msr_related_uploader":"","msr_attachments":[{"id":0,"url":"http:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=6975210"}],"msr-author-ordering":[{"type":"guest","value":"bjoern-h-menze","user_id":873573,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=bjoern-h-menze"},{"type":"guest","value":"andras-jakab","user_id":873576,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=andras-jakab"},{"type":"guest","value":"stefan-bauer","user_id":873579,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=stefan-bauer"},{"type":"guest","value":"jayashree-kalpathy-cramer","user_id":873582,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=jayashree-kalpathy-cramer"},{"type":"guest","value":"keyvan-farahani","user_id":873585,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=keyvan-farahani"},{"type":"guest","value":"justin-kirby","user_id":873588,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=justin-kirby"},{"type":"guest","value":"yuliya-burren","user_id":873591,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=yuliya-burren"},{"type":"guest","value":"nicole-porz","user_id":873594,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nicole-porz"},{"type":"guest","value":"johannes-slotboom","user_id":873597,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=johannes-slotboom"},{"type":"guest","value":"roland-wiest","user_id":873600,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=roland-wiest"},{"type":"guest","value":"levente-lanczi","user_id":873603,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=levente-lanczi"},{"type":"guest","value":"elizabeth-gerstner","user_id":873606,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=elizabeth-gerstner"},{"type":"guest","value":"marc-andre-weber","user_id":873609,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=marc-andre-weber"},{"type":"guest","value":"tal-arbel","user_id":873612,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=tal-arbel"},{"type":"guest","value":"brian-b-avants","user_id":873615,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=brian-b-avants"},{"type":"guest","value":"nicholas-ayache","user_id":871143,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nicholas-ayache"},{"type":"guest","value":"patricia-buendia","user_id":873618,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=patricia-buendia"},{"type":"guest","value":"d-louis-collins","user_id":873621,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=d-louis-collins"},{"type":"guest","value":"nicolas-cordier","user_id":873624,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nicolas-cordier"},{"type":"guest","value":"jason-j-corso","user_id":873627,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=jason-j-corso"},{"type":"user_nicename","value":"Antonio Criminisi","user_id":41790,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Antonio Criminisi"},{"type":"guest","value":"tilak-das","user_id":873630,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=tilak-das"},{"type":"guest","value":"herve-delingette","user_id":871140,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=herve-delingette"},{"type":"guest","value":"cagatay-demiralp","user_id":873633,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=cagatay-demiralp"},{"type":"guest","value":"christopher-r-durst","user_id":873636,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=christopher-r-durst"},{"type":"guest","value":"michel-dojat","user_id":873639,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=michel-dojat"},{"type":"guest","value":"senan-doyle","user_id":873642,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=senan-doyle"},{"type":"guest","value":"joana-festa","user_id":873645,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=joana-festa"},{"type":"guest","value":"florence-forbes","user_id":873648,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=florence-forbes"},{"type":"guest","value":"ezequiel-geremia","user_id":873651,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=ezequiel-geremia"},{"type":"guest","value":"ben-glocker-2","user_id":869628,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=ben-glocker-2"},{"type":"guest","value":"polina-golland","user_id":873654,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=polina-golland"},{"type":"guest","value":"xiaotao-guo","user_id":873657,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=xiaotao-guo"},{"type":"guest","value":"andac-hamamci","user_id":873660,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=andac-hamamci"},{"type":"guest","value":"khan-m-iftekharuddin","user_id":873663,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=khan-m-iftekharuddin"},{"type":"guest","value":"raj-jena","user_id":873666,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=raj-jena"},{"type":"guest","value":"nigel-m-john","user_id":873669,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nigel-m-john"},{"type":"guest","value":"ender-konukoglu","user_id":873672,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=ender-konukoglu"},{"type":"guest","value":"danial-lashkari","user_id":873675,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=danial-lashkari"},{"type":"guest","value":"jose-antonio-mariz","user_id":873678,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=jose-antonio-mariz"},{"type":"guest","value":"raphael-meier","user_id":873681,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=raphael-meier"},{"type":"guest","value":"sergio-pereira","user_id":873684,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=sergio-pereira"},{"type":"guest","value":"doina-precup","user_id":873687,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=doina-precup"},{"type":"guest","value":"stephen-j-price","user_id":873690,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=stephen-j-price"},{"type":"guest","value":"tammy-riklin-raviv","user_id":873693,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=tammy-riklin-raviv"},{"type":"guest","value":"syed-m-s-reza","user_id":873696,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=syed-m-s-reza"},{"type":"guest","value":"michael-ryan","user_id":873699,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=michael-ryan"},{"type":"guest","value":"duygu-sarikaya","user_id":873702,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=duygu-sarikaya"},{"type":"guest","value":"lawrence-schwartz","user_id":873705,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=lawrence-schwartz"},{"type":"guest","value":"hoo-chang-shin","user_id":873708,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=hoo-chang-shin"},{"type":"user_nicename","value":"Jamie Shotton","user_id":32162,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jamie Shotton"},{"type":"guest","value":"carlos-a-silva","user_id":873711,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=carlos-a-silva"},{"type":"guest","value":"nuno-sousa","user_id":873714,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nuno-sousa"},{"type":"guest","value":"nagesh-k-subbanna","user_id":873717,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nagesh-k-subbanna"},{"type":"guest","value":"gabor-szekely","user_id":873720,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=gabor-szekely"},{"type":"guest","value":"thomas-j-taylor","user_id":873723,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=thomas-j-taylor"},{"type":"guest","value":"owen-m-thomas","user_id":873726,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=owen-m-thomas"},{"type":"guest","value":"nicholas-j-tustison","user_id":873729,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=nicholas-j-tustison"},{"type":"guest","value":"gozde-unal","user_id":873732,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=gozde-unal"},{"type":"guest","value":"flor-vasseur","user_id":873735,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=flor-vasseur"},{"type":"guest","value":"max-wintermark","user_id":873738,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=max-wintermark"},{"type":"guest","value":"dong-hye-ye","user_id":873741,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=dong-hye-ye"},{"type":"guest","value":"liang-zhao","user_id":873744,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=liang-zhao"},{"type":"guest","value":"binsheng-zhao","user_id":873747,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=binsheng-zhao"},{"type":"guest","value":"d-zikic","user_id":871362,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=d-zikic"},{"type":"guest","value":"marcel-prastawa","user_id":873750,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=marcel-prastawa"},{"type":"guest","value":"mauricio-reyes","user_id":873753,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=mauricio-reyes"},{"type":"guest","value":"koen-van-leemput","user_id":873756,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=koen-van-leemput"}],"msr_impact_theme":["Health"],"msr_research_lab":[],"msr_event":[],"msr_group":[],"msr_project":[169659],"publication":[],"video":[],"download":[],"msr_publication_type":"article","related_content":{"projects":[{"ID":169659,"post_title":"Project InnerEye - Democratizing Medical Imaging AI","post_name":"medical-image-analysis","post_type":"msr-project","post_date":"2008-10-07 05:22:18","post_modified":"2023-07-28 05:51:32","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/medical-image-analysis\/","post_excerpt":"InnerEye is a research project that uses state of the art\u00a0machine learning\u00a0technology to build innovative tools for the automatic, quantitative analysis of three-dimensional medical images.","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/169659"}]}}]},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/167562"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":5,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/167562\/revisions"}],"predecessor-version":[{"id":873762,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/167562\/revisions\/873762"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=167562"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=167562"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=167562"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=167562"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=167562"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=167562"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=167562"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=167562"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=167562"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=167562"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=167562"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=167562"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=167562"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=167562"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=167562"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=167562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}