{"id":743044,"date":"2021-06-22T19:44:57","date_gmt":"2021-06-23T02:44:57","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=743044"},"modified":"2021-06-22T19:44:57","modified_gmt":"2021-06-23T02:44:57","slug":"microsoft-lreasoner-leads-the-reclor-challenge-on-logical-reasoning","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/microsoft-lreasoner-leads-the-reclor-challenge-on-logical-reasoning\/","title":{"rendered":"Microsoft LReasoner leads the ReClor challenge on logical reasoning"},"content":{"rendered":"
Recently, the industry has witnessed the growth of highly advanced and powerful AI language models. When the industry marvels at its variety of skills, like drawing, writing, and game-playing, it also worries for its IQ. For example, if you try to ask an advanced language model the following question:<\/p>\n
Question: How many eyes does the sun have?<\/em> The reason for this type of mistake is that when the language model was asked, it did not infer the relationship between the sun and the eyes. If you look for the reason from technical aspect, there is a possible explanation that most of the current natural processing technologies use the \u201cpre-training + fine-tuning\u201d paradigm. This paradigm achieves superior performance on tasks that require shallow semantic matching and understanding of text. However, whether the pre-trained language model really has reasoning ability and whether it can cope with tasks that require complex reasoning ability is still a problem needs to be solved in current research.<\/p>\n In order to solve the logical reasoning problem of the machine, the Natural Language Computing Group of Microsoft Research Asia proposed the LReasoner system, which assists the model to find the answer to the problem by recognizing logical symbols and expressions expressed in the text.<\/p>\n When the researchers test the LReasoner system on the ReCLor dataset, which focuses on the logical reasoning part of Law School Admission Test (LSAT), the system achieved the current SOTA (state-of-the-art performance) in the official evaluation leaderboard of the dataset. As a result, it significantly outperforms human performance (Note: human performance refers to the average accuracy of 10 college students given in the ReClor paper) reported in the ReClor paper (Table 1).<\/p>\n
\nModel: Sun has one eye.<\/em>
\nCorrect answer from humans: The sun is a star, and it has no eyes.<\/em><\/p>\n