{"id":487466,"date":"2019-01-14T09:43:36","date_gmt":"2019-01-14T17:43:36","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=487466"},"modified":"2023-03-29T19:31:38","modified_gmt":"2023-03-30T02:31:38","slug":"figureqa-dataset","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/figureqa-dataset\/","title":{"rendered":"FigureQA Dataset"},"content":{"rendered":"<p>Answering questions about a given image is a difficult task, requiring both an understanding of the image and the accompanying query. Microsoft research Montreal&#8217;s <strong>FigureQA<\/strong> dataset introduces a new visual reasoning task for research, specific to graphical plots and figures. The task comes with an additional twist: all of the questions are relational, requiring the comparison of several or all elements of the underlying plot.<\/p>\n<p>Images are comprised on five types of figures commonly found in analytical documents. Fifteen question types were selected for the dataset concerning quantitative attributes in relational <strong>global<\/strong> and <strong>one-vs-one<\/strong> contexts. These include properties like minimum and maximum, greater and less than, medians, curve roughness, and area under the curve (AUC). All questions in the training and validation sets have either a yes or no answer.<\/p>\n<p>For more details concerning the task, dataset, and our experiments, please read our paper: <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/1710.07300\">FigureQA: An Annotated Figure Dataset for Visual Reasoning<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n<h3 style=\"padding-top: 25px;\">Click on a figure below to enlarge it and see some of its questions, answers, and bounding boxes.<\/h3>\n<h3><\/h3>\n<ul id='gallery-1' class='gallery galleryid-487466 gallery-columns-4 gallery-size-grid-third stripped ms-row fixed-small'><li class='s-col-6-24 xs-margin-bottom-sp1 s-margin-bottom-sp2'><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-one.png\" data-mfp-src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-one.png\" data-caption=\"Vertical Graph Bar\" class=\"gallery-item\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-one-480x280.png\" alt=\"Vertical Graph Bar\" class=\"db full-width\" \/><\/a>\n\t\t\t\t<dd class='wp-caption-text gallery-caption' id='gallery-1-487493'>\n\t\t\t\tVertical Graph Bar\n\t\t\t\t<\/dd><\/li><li class='s-col-6-24 xs-margin-bottom-sp1 s-margin-bottom-sp2'><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-two.png\" data-mfp-src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-two.png\" data-caption=\"Horizontal Graph Bar\" class=\"gallery-item\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-two-480x280.png\" alt=\"Horizontal Graph Bar\" class=\"db full-width\" \/><\/a>\n\t\t\t\t<dd class='wp-caption-text gallery-caption' id='gallery-1-487490'>\n\t\t\t\tHorizontal Graph Bar\n\t\t\t\t<\/dd><\/li><li class='s-col-6-24 xs-margin-bottom-sp1 s-margin-bottom-sp2'><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-three.png\" data-mfp-src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-three.png\" data-caption=\"Line Graph\" class=\"gallery-item\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-three-480x280.png\" alt=\"Line Graph\" class=\"db full-width\" \/><\/a>\n\t\t\t\t<dd class='wp-caption-text gallery-caption' id='gallery-1-487487'>\n\t\t\t\tLine Graph\n\t\t\t\t<\/dd><\/li><li class='s-col-6-24 xs-margin-bottom-sp1 s-margin-bottom-sp2'><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-four.png\" data-mfp-src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-four.png\" data-caption=\"Dot Line Graph\" class=\"gallery-item\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-four-428x280.png\" alt=\"Dot Line Graph\" class=\"db full-width\" \/><\/a>\n\t\t\t\t<dd class='wp-caption-text gallery-caption' id='gallery-1-487484'>\n\t\t\t\tDot Line Graph\n\t\t\t\t<\/dd><\/li><br style=\"clear: both\" \/><li class='s-col-6-24 xs-margin-bottom-sp1 s-margin-bottom-sp2'><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-five.png\" data-mfp-src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-five.png\" data-caption=\"Pie Chart\" class=\"gallery-item\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/05\/figure-graph-five-480x280.png\" alt=\"Pie Chart\" class=\"db full-width\" \/><\/a>\n\t\t\t\t<dd class='wp-caption-text gallery-caption' id='gallery-1-487481'>\n\t\t\t\tPie Chart\n\t\t\t\t<\/dd><\/li>\n\t\t\t<br style='clear: both' \/>\n\t\t<\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>Answering questions about a given image is a difficult task, requiring both an understanding of the image and the accompanying query. Microsoft research Montreal&#8217;s FigureQA dataset introduces a new visual reasoning task for research, specific to graphical plots and figures. The task comes with an additional twist: all of the questions are relational, requiring the [&hellip;]<\/p>\n","protected":false},"featured_media":487478,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-487466","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[],"related-downloads":[471993],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[{"id":0,"name":"Highlights","content":"<h3>Highlights<\/h3>\r\n[row]\r\n[column class=\"l-col-6-24\"]\r\n<div style=\"text-align: center;vertical-align: middle;background-color: #104586;min-width: 150px;min-height: 200px;padding: 15px\">\r\n<div style=\"color: white;font-size: 30px;padding-bottom: 10px\">100,000<\/div>\r\n&nbsp;\r\n<div style=\"color: white;font-size: 15px\">Figure images in the training set<\/div>\r\n<\/div>\r\n[\/column]\r\n\r\n[column class=\"l-col-6-24\"]\r\n<div style=\"text-align: center;vertical-align: middle;background-color: #104586;min-width: 150px;min-height: 200px;padding: 15px\">\r\n<div style=\"color: white;font-size: 30px;padding-bottom: 10px\">1,327,368<\/div>\r\n&nbsp;\r\n<div style=\"color: white;font-size: 15px\">Question-answer pairs in the training set<\/div>\r\n<\/div>\r\n[\/column]\r\n\r\n[column class=\"l-col-6-24\"]\r\n<div style=\"text-align: center;vertical-align: middle;background-color: #104586;min-width: 150px;min-height: 200px;padding: 15px\">\r\n<div style=\"color: white;font-size: 30px;padding-bottom: 10px\">100<\/div>\r\n&nbsp;\r\n<div style=\"color: white;font-size: 15px\">Unique colors and possible names for figure plot elements<\/div>\r\n<\/div>\r\n[\/column]\r\n\r\n[column class=\"l-col-6-24\"]\r\n<div style=\"text-align: center;vertical-align: middle;background-color: #104586;min-width: 150px;min-height: 200px;padding: 15px\">\r\n<div style=\"color: white;font-size: 30px;padding-bottom: 10px\">15<\/div>\r\n&nbsp;\r\n<div style=\"color: white;font-size: 15px\">Question types for quantitative attributes<\/div>\r\n<\/div>\r\n[\/column]\r\n\r\n[\/row]\r\n<h3 style=\"padding-top: 25px\">Details<\/h3>\r\n<table style=\"border-spacing: 0px;border-collapse: separate;border: 1px solid #d0d0d0;height: 225px\" width=\"100%\" cellspacing=\"0\" cellpadding=\"15\">\r\n<thead>\r\n<tr style=\"background-color: #104586\">\r\n<td style=\"padding: 15px;border: inherit\" valign=\"middle\" width=\"20%\"><span style=\"color: #ffffff\">Dataset Split<\/span><\/td>\r\n<td style=\"padding: 15px;border: inherit\" valign=\"middle\" width=\"20%\"><span style=\"color: #ffffff\"># Images<\/span><\/td>\r\n<td style=\"padding: 15px;border: inherit\" valign=\"middle\" width=\"20%\"><span style=\"color: #ffffff\"># Questions<\/span><\/td>\r\n<td style=\"padding: 15px;border: inherit\" valign=\"middle\" width=\"20%\"><span style=\"color: #ffffff\">Has Answers &amp; Annotations?<\/span><\/td>\r\n<td style=\"padding: 15px;border: inherit\" valign=\"middle\" width=\"20%\"><span style=\"color: #ffffff\">Color Scheme<\/span><\/td>\r\n<\/tr>\r\n<\/thead>\r\n<tbody>\r\n<tr>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Train<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">100,000<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">1,327,368<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Yes<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Scheme 1<\/td>\r\n<\/tr>\r\n<tr style=\"background-color: #f2f2f2\">\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Validation 1<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">20,000<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">265,106<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Yes<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Scheme 1<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Validation 2<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">20,000<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">265,798<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Yes<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Scheme 2<\/td>\r\n<\/tr>\r\n<tr style=\"background-color: #f2f2f2\">\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Test 1<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">20,000<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">265,024<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">No<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Scheme 1<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Test 2<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">20,000<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">265,402<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">No<\/td>\r\n<td style=\"vertical-align: middle;padding: 15px;border: inherit\">Scheme 2<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n&nbsp;\r\n<h3 style=\"padding-top: 25px\">Unique Features<\/h3>\r\nAdditionally, the following features make FigureQA a distinct visual question-answering (VQA) and reasoning dataset:\r\n<ul>\r\n \t<li>It is entirely synthetically generated. Any number of samples can be generated in a configurable and extensible manner.<\/li>\r\n \t<li>Each figure image is accompanied by the source data used to create it. This data can be used as input features or a learning target, and can be used to formulate questions and answers.<\/li>\r\n \t<li>Rich bounding box annotations for all plot elements are extracted automatically and included with each generated figure image.<\/li>\r\n<\/ul>\r\n<h3 style=\"padding-top: 25px\">Figure Color Schemes<\/h3>\r\nTo color and identify plot elements, 100 colors where selected from the X11 named color set. Colors were selected to have a large color distance from white, the background color, with some modifications to the names to enhance readability.\r\n\r\nIn order to evaluate models on unseen color combinations, we provide validation and test sets with two color schemes consisting of alternating disjoint color sets. Each figure is colored with one set according to the training color scheme, then the other color set in the test set using the test color scheme. This ensures that all colors are learned during training, and is consistent with the one used in the CLEVR dataset.\r\n\r\nFor example:\r\n\r\n[row]\r\n[column class=\"m-col-12-24\"]\r\n<h4>Scheme 1<\/h4>\r\n<ul>\r\n \t<li>Vertical bar graphs, line charts, and pie charts are colored using 50 unique colors in <strong>set A<\/strong>, including <span style=\"color: #ff0000\"><strong>crimson<\/strong><\/span>, <span style=\"color: #339966\"><strong>seafoam<\/strong><\/span>, and <span style=\"color: #3366ff\"><strong>royal blue<\/strong><\/span>.<\/li>\r\n \t<li>Horizontal bar graphs and dot line charts are colored using 50 unique colors in <strong>set B<\/strong>, including <span style=\"color: #f08080\"><strong>light coral<\/strong><\/span>, <span style=\"color: #a0522d\"><strong>sienna<\/strong><\/span>, and <span style=\"color: #333399\"><strong>web purple<\/strong><\/span>.<\/li>\r\n<\/ul>\r\n[\/column]\r\n[column class=\"m-col-12-24\"]\r\n<h4>Scheme 2<\/h4>\r\n<ul>\r\n \t<li>Vertical bar graphs, line charts, and pie charts are colored using 50 unique colors in <strong>set B<\/strong>, including <span style=\"color: #f08080\"><strong>light coral<\/strong><\/span>, <span style=\"color: #a0522d\"><strong>sienna<\/strong><\/span>, and <span style=\"color: #333399\"><strong>web purple<\/strong><\/span>.<\/li>\r\n \t<li>Horizontal bar graphs and dot line charts are colored using 50 unique colors in <strong>set A<\/strong>, including <span style=\"color: #ff0000\"><strong>crimson<\/strong><\/span>, <span style=\"color: #339966\"><strong>seafoam<\/strong><\/span>, and <span style=\"color: #3366ff\"><strong>royal blue<\/strong><\/span>.<\/li>\r\n<\/ul>\r\n[\/column][\/row]"},{"id":1,"name":"Download","content":"The dataset is split into five separate packages which differ by train\/validation\/test split and the colour-to-figure assignment scheme that is used. A sample of one thousand images from the training set is also included for those curious or eager to work with the data. Each split of the dataset is differentiated by the color scheme that was used, namely \"1\" or \"2\". The entire dataset size is <strong>3.53 GB compressed<\/strong> and <strong>5.78 GB uncompressed<\/strong>.\r\n<ul class=\"stripped no-margin-bottom no-margin-last ms-row\">\r\n \t<li class=\"s-col-2-4 m-col-1-4 l-col-4-4 margin-bottom-sp1\"><a class=\"button-solid x-hidden-focus\" href=\"https:\/\/www.microsoft.com\/en-hk\/download\/details.aspx?id=100635\" target=\"_blank\" rel=\"noopener\">Agree &amp; Download<\/a><\/li>\r\n<\/ul>"},{"id":2,"name":"Evaluation","content":"If you would like to evaluate your results on the FigureQA test sets, please compile your predicted answers in the format documented below and email them to the authors.\r\n\r\nPlease send your results to Samira and Adam A. at <a href=\"mailto:figureqa@microsoft.com\">figureqa@microsoft.com<\/a>.\r\n\r\nWe provide the same statistics on each of the <strong>test1<\/strong> and <strong>test2<\/strong> sets as those in the FigureQA paper, namely:\r\n<ul>\r\n \t<li>Overall accuracy.<\/li>\r\n \t<li>Accuracy by question type.<\/li>\r\n \t<li>Accuracy by figure type.<\/li>\r\n<\/ul>\r\n<h3 style=\"padding-top: 25px\">Predictions Format<\/h3>\r\nResults must be provided in a CSV format, having the first line as a header and each row like so:\r\n<p align=\"center\"><span style=\"background-color: #e6e6e6;font-weight: bold;text-align: center\">question_index,image_index,question_id,question_string,answer<\/span><\/p>\r\nHere <span style=\"background-color: #e6e6e6;font-weight: bold\">question_index<\/span> refers to the sample's index in the <span style=\"background-color: #e6e6e6;font-weight: bold\">qa_pairs<\/span> array within <strong>qa_pairs.json<\/strong>. All other fields are the same as in <strong>qa_pairs.json<\/strong>, documented here.\r\n\r\nHere are example results files for <a href=\"http:\/\/datasets.maluuba.com\/static\/datasets-website\/template_test1.csv\">test1<\/a> and <a href=\"http:\/\/datasets.maluuba.com\/static\/datasets-website\/template_test2.csv\">test2<\/a>. Simply replace the final column field <span style=\"background-color: #e6e6e6;font-weight: bold\">&lt;\u00a00\/1\u00a0&gt;<\/span> with your prediction as a <span style=\"background-color: #e6e6e6;font-weight: bold\">\u00a00\u00a0<\/span> or <span style=\"background-color: #e6e6e6;font-weight: bold\">\u00a01\u00a0<\/span> in each row.\r\n\r\nTo evaluate your results faster, please ensure that your results file is <strong>UTF-8 encoded<\/strong> and can be read using the <a href=\"https:\/\/pandas.pydata.org\/pandas-docs\/stable\/generated\/pandas.read_csv.html\" target=\"_blank\" rel=\"noopener\"><strong>read_csv<\/strong><\/a> function from the Python Pandas library."}],"slides":[],"related-researchers":[{"type":"guest","display_name":"Samira Ebrahimi Kahou","user_id":621291,"people_section":"Section name 1","alias":""},{"type":"guest","display_name":"Vincent Michalski","user_id":479133,"people_section":"Section name 1","alias":""},{"type":"user_nicename","display_name":"Adam Atkinson","user_id":37095,"people_section":"Section name 1","alias":"adatkins"},{"type":"guest","display_name":"Akos Kadar","user_id":487469,"people_section":"Section name 1","alias":""},{"type":"guest","display_name":"Yoshua Bengio","user_id":487475,"people_section":"Section name 1","alias":""},{"type":"user_nicename","display_name":"Mahmoud Adada","user_id":37176,"people_section":"Section name 1","alias":"maadada"},{"type":"user_nicename","display_name":"Rahul Mehrotra","user_id":37203,"people_section":"Section name 1","alias":"ramehrot"}],"msr_research_lab":[437514],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/487466"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":15,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/487466\/revisions"}],"predecessor-version":[{"id":931842,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/487466\/revisions\/931842"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/487478"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=487466"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=487466"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=487466"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=487466"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=487466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}