{"id":755974,"date":"2021-06-24T10:54:10","date_gmt":"2021-06-24T17:54:10","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=755974"},"modified":"2021-06-24T10:54:12","modified_gmt":"2021-06-24T17:54:12","slug":"microsoft-lreasoner-leads-the-reclor-challenge-on-logical-reasoning","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/microsoft-lreasoner-leads-the-reclor-challenge-on-logical-reasoning\/","title":{"rendered":"Microsoft LReasoner leads the ReClor challenge on logical reasoning"},"content":{"rendered":"\n<figure class=\"wp-block-image alignwide size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1024x576.jpg\" alt=\"\" class=\"wp-image-755998\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-16x9.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1920x1080.jpg 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>For many years AI researchers have sought to build upon traditional machine learning, which trains technology to process facts and learn from them, and develop machine reasoning, in which programs apply logic to data and solve problems \u2013 comparable to the way humans think. For a system to analyze multiple sets of logical arguments, it requires both critical thinking and combinatorial reasoning abilities.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/reclor-a-reading-comprehension-dataset-requiring-logical-reasoning\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning\" data-bi-aN=\"margin-callout\" data-bi-cN=\"ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning\">\n\t\t\t\tReClor: A Reading Comprehension Dataset Requiring Logical Reasoning&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>One of the current benchmarks for evaluating a system\u2019s logical reasoning ability is <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/reclor-a-reading-comprehension-dataset-requiring-logical-reasoning\/\">ReClor, a Reading Comprehension Dataset Requiring Logical Reasoning<\/a>. ReClor is a dataset built from logical reasoning problems used in standardized admission tests, including the Law School Admission Test (LSAT) and Graduate Management Admission Test (GMAT). <\/p>\n\n\n\n<p>Today, we are excited to announce that Microsoft\u2019s LReasoner system is the top-rated performer on the official ReCLor leaderboard. LReasoner also significantly exceeded human performance, as measured by the average accuracy of 10 college students who each answered 10 randomly selected test questions and reported in the ReClor paper. <\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div class=\"annotations \" data-bi-aN=\"citation\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 \">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Article <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/microsoft-research-asia\/articles\/microsoft-lreasoner-leads-the-reclor-challenge-on-logical-reasoning\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Microsoft LReasoner leads the ReClor challenge on logical reasoning\" data-bi-aN=\"citation\" data-bi-cN=\"Microsoft LReasoner leads the ReClor challenge on logical reasoning\">\n\t\t\t\tMicrosoft LReasoner leads the ReClor challenge on logical reasoning&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div class=\"annotations \" data-bi-aN=\"citation\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 \">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/logic-driven-context-extension-and-data-augmentation-for-logical-reasoning-of-text\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text\" data-bi-aN=\"citation\" data-bi-cN=\"Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text\">\n\t\t\t\tLogic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eval.ai\/web\/challenges\/challenge-page\/503\/leaderboard\/1347\"><img loading=\"lazy\" decoding=\"async\" width=\"692\" height=\"424\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-1_RECLOR.png\" alt=\"Figure 1: A screenshot from the ReClor leaderboard shows LReasoner at the top, with an overall score of 76.1, a Test-Easy score of 87.05, and a Test-Hard score of 67.5. At number two on the leaderboard is RainaCUED (Electra) with scores of 67.1, 80.91, and 56.25. At number three on the leaderboard is zhiweihu (ALBERT) with scores of 62.6, 73.64, and 53.93.\" class=\"wp-image-755980\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-1_RECLOR.png 692w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-1_RECLOR-300x184.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-1_RECLOR-16x10.png 16w\" sizes=\"(max-width: 692px) 100vw, 692px\" \/><span class=\"sr-only\"> (opens in new tab)<\/span><\/a><figcaption>Figure1\uff1aLReasoner achieves the state-of-the-art performance on the official <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/eval.ai\/web\/challenges\/challenge-page\/503\/leaderboard\/1347\">ReClor leaderboard<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/figcaption><\/figure><\/div>\n\n\n\n<p>The LSAT is a standardized exam established in 1947 that has become an essential benchmark in the law school admissions process. An example of the LSAT\u2019s challenging logical reasoning questions is shown in Figure 2. To answer such a question, a system needs to understand the logical symbols like \u201chave keyboarding skills\u201d, make complex inferences to extend the existing logical expressions according to logical rules and then match the logical expressions with the answer options.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"470\" height=\"598\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-2.png\" alt=\"Figure 2: An example of a logical reasoning test question: \nContext: If you have no keyboarding skills at all, you will not be able to use a computer. And if you are not able to use a computer you will not be able to write your essays using a computer program.  \nQuestion: If the statements above are true, which of the following must be true? \nOptions: \nA.\tIf you are not able to write your essays using a word processing program, you have no keyboarding skills\nB.\tIf you're able to write your essays using a word processing program, you have at least some keyboarding skills\nC.\tIf you are not able to write your essays using a word processing program, you are not able to use a computer\nD.\tIf you have some keyboarding skills, you will be able to write your essays using a word processing program\nOption B is listed in green, indicating that it is the most plausible answer.\n\" class=\"wp-image-755983\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-2.png 470w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-2-236x300.png 236w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure-2-9x12.png 9w\" sizes=\"(max-width: 470px) 100vw, 470px\" \/><figcaption>Figure 2: example of a logical reasoning test question<\/figcaption><\/figure><\/div>\n\n\n\n<p>To address this logical reasoning challenge in a realistic scenario, the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/group\/natural-language-processing\/\">Natural Language Computing Group<\/a> of Microsoft Research Asia proposed the LReasoner system, which helps the model find the answer to a problem by recognizing logical symbols and logical expressions in the text.<\/p>\n\n\n\n<p>LReasoner improves the reasoning ability of the previous pre-trained language models by using two novel techniques (illustrated in Figure 2): (1) logic-driven context extension framework, which aims to first identify logic symbols from the context and then infer extended logical expressions through logical equivalence law, and (2) logic-driven data augmentation algorithm, which applies contrastive learning to find logically different context to help the model better capture the logical information, especially the logical negation and conditional relationship.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"366\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure3_reclor-1024x366.png\" alt=\"Three panel diagram showing logic-driven context extension framework. More details in the paper. \" class=\"wp-image-755986\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure3_reclor-1024x366.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure3_reclor-300x107.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure3_reclor-768x275.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure3_reclor-16x6.png 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/Figure3_reclor.png 1269w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 3: Logic-driven context extension framework<\/figcaption><\/figure><\/div>\n\n\n\n<p>In surpassing human performance as demonstrated in Figure 1, the LReasoner has taken an essential step towards deeper logical reasoning by AI. The LReasoner system is also one of the first attempts by researchers to apply machine reasoning to real scenarios. In the future, the Natural Language Computing Group of Microsoft Research Asia will continue to explore new tasks and new methods in the field of machine reasoning and promote the research of knowledgeable and interpretable artificial intelligence.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>For many years AI researchers have sought to build upon traditional machine learning, which trains technology to process facts and learn from them, and develop machine reasoning, in which programs apply logic to data and solve problems \u2013 comparable to the way humans think. For a system to analyze multiple sets of logical arguments, it [&hellip;]<\/p>\n","protected":false},"author":38838,"featured_media":755998,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13545],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-755974","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144735],"related-projects":[],"related-events":[],"related-researchers":[{"type":"guest","value":"natural-language-computing-group","user_id":"756685","display_name":"Natural Language Computing Group  ","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/group\/natural-language-computing\/\" aria-label=\"Visit the profile page for Natural Language Computing Group  \">Natural Language Computing Group  <\/a>","is_active":true,"last_first":"Natural Language Computing Group ","people_section":0,"alias":"natural-language-computing-group"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-960x540.jpg\" class=\"img-object-cover\" alt=\"Graphic with the word ReClor in the center along side an icon of a lightbulb. Binary code makes up the backdrop of this graphic.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-16x9.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/06\/1400x788_Reclor_no_logo_still_with_code-1920x1080.jpg 1920w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/group\/natural-language-computing\/\" title=\"Go to researcher profile for Natural Language Computing Group  \" aria-label=\"Go to researcher profile for Natural Language Computing Group  \" data-bi-type=\"byline author\" data-bi-cN=\"Natural Language Computing Group  \">Natural Language Computing Group  <\/a>","formattedDate":"June 24, 2021","formattedExcerpt":"For many years AI researchers have sought to build upon traditional machine learning, which trains technology to process facts and learn from them, and develop machine reasoning, in which programs apply logic to data and solve problems \u2013 comparable to the way humans think. For&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/755974"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=755974"}],"version-history":[{"count":13,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/755974\/revisions"}],"predecessor-version":[{"id":756751,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/755974\/revisions\/756751"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/755998"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=755974"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=755974"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=755974"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=755974"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=755974"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=755974"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=755974"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=755974"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=755974"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=755974"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=755974"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}