{"id":1010406,"date":"2024-03-05T06:00:00","date_gmt":"2024-03-05T14:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1010406"},"modified":"2024-03-06T08:41:49","modified_gmt":"2024-03-06T16:41:49","slug":"orca-math-demonstrating-the-potential-of-slms-with-model-specialization","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/orca-math-demonstrating-the-potential-of-slms-with-model-specialization\/","title":{"rendered":"Orca-Math: Demonstrating the potential of SLMs with model specialization"},"content":{"rendered":"\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1.png\" alt=\"abstract wave lines on a gradient background\" class=\"wp-image-1010433\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1.png 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-240x135.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-1280x720.png 1280w\" sizes=\"(max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n\n<p>Our work on <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4\/\" target=\"_blank\" rel=\"noreferrer noopener\">Orca<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/orca-2-teaching-small-language-models-how-to-reason\/\" target=\"_blank\" rel=\"noreferrer noopener\">Orca 2<\/a> demonstrated the power of improved training signals and methods to enhance the reasoning abilities of smaller language models, getting closer to the levels found in much larger language models. Orca-Math is another step in this direction, where we explore the capabilities of small language models (SLMs) when specialized in a certain area, in this case solving grade school math problems, which has long been recognized as a complex task for SLMs.<\/p>\n\n\n\n<p>Orca-Math is a 7 billion parameters model created by fine-tuning the Mistral 7B model. Orca-Math achieves 86.81% on GSM8k pass@1, exceeding the performance of much bigger models including general models (e.g. LLAMA-2-70, Gemini Pro and GPT-3.5) and math-specific models (e.g. MetaMath-70B and WizardMa8th-70B). Note that the base model (Mistral-7B) achieves 37.83% on GSM8K.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1277\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-scaled.jpg\" alt=\"Alt Text: Bar graph comparing GSM8K score of different models with an upward trend in quality. The models are LLAMA-2-70, GPT-3.5, Gemini Pro,  WizardMath-70B, MetaMath-70B and Orca-Math-7B. The graph shows that the Orca-Math-7B model outperforms other bigger models on GSM8K.\" class=\"wp-image-1011258\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-scaled.jpg 2560w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-300x150.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-1024x511.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-768x383.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-1536x766.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-2048x1022.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/03\/orca-math-240x120.jpg 240w\" sizes=\"(max-width: 2560px) 100vw, 2560px\" \/><figcaption class=\"wp-element-caption\">&nbsp;Bar graph comparing GSM8K score of different models with an upward trend in quality. The models are LLAMA-2-70, GPT-3.5, Gemini Pro,&nbsp; WizardMath-70B, MetaMath-70B and Orca-Math-7B. The graph shows that the Orca-Math-7B model outperforms other bigger models on GSM8K.<\/figcaption><\/figure>\n\n\n\n<p>The state-of-the-art (SOTA) performance of the Orca-Math model can be attributed to two key insights:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Training on high-quality synthetic data with 200,000 math problems, created using multi-agents (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/autogen\/\" target=\"_blank\" rel=\"noreferrer noopener\">AutoGen<\/a>). This is smaller than other math datasets, which could have millions of problems. The smaller model and smaller dataset mean faster and cheaper training.<\/li>\n\n\n\n<li>In addition to traditional supervised fine-tuning, the model was trained using an iterative learning process, where the model is allowed to practice solving problems and continues to improve based on feedback from a teacher.<\/li>\n<\/ul>\n\n\n\n<p>Our findings show that smaller models are valuable in specialized settings, where they can match the performance of much larger models while also highlighting the potential of continual learning and using feedback to improve language models. We are making the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/huggingface.co\/datasets\/microsoft\/orca-math-word-problems-200k\">dataset<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> publicly available, along with a <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2402.14830\">report<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> describing the training procedure to encourage research on the improvement and specialization of smaller language models.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"teaching-slms-math\">Teaching SLMs math<\/h2>\n\n\n\n<p>Solving mathematical word problems has long been recognized as a complex task for SLMs. Models that achieve over 80% accuracy on the GSM8K benchmark (GSM8K, which stands for Grade School Math 8K, is a dataset of 8,500 high-quality grade school mathematical word problems that require multi-step reasoning) typically exceed 30 billion parameters.<\/p>\n\n\n\n<p>To reach higher levels of performance with smaller models, researchers often train SLMs to generate code, or use calculators to help avoid calculation errors. Additionally, they employ a technique called ensembling, in which the model is called up to 100 times, with each call reattempting to solve the problem.&nbsp;Ensembling provides a substantial boost in accuracy but at a significant increase in compute cost increase, due to multiple calls to the model.&nbsp;<\/p>\n\n\n\n<p>This research aims to explore how far we can push the native ability of smaller language models when they are specialized to solve math problems, without the use of external tools, verifiers or ensembling. More specifically, we focus on two directions:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"agentinstruct\">AgentInstruct<\/h3>\n\n\n\n<p>Previous work on synthetic data creation often uses frontier models to generate similar problems based on a seed problem. Providing paraphrases of the seed with different numbers and attributes can be useful for creating training data for the smaller model. We propose employing multi-agent flows, using AutoGen, to create new problems and solutions, which can not only create more demonstrations of the problem but also increase the diversity and range of difficulty of the problems.&nbsp;<\/p>\n\n\n\n<p>To generate more challenging problems, we create a setup with a team of agents working collaboratively to create a dataset geared toward a predefined objective. For example, we can use two agents, namely <em>Suggester<\/em> and <em>Editor<\/em>. The Suggester examines a problem and proposes several methods for increasing its complexity, while the Editor takes the original word problem and the Suggester&#8217;s recommendations to generate an updated, more challenging problem. This iterative process can occur over multiple rounds, with each round further increasing the complexity of the previously generated problem. A third agent can then <em>verify<\/em> that the problem is solvable and create the solution.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"iterative-learning\">Iterative learning<\/h3>\n\n\n\n<p>Using&nbsp;high-quality training data that may&nbsp;elicit richer learning signals (e.g. explanations) has been shown to significantly improve&nbsp;SLM\u2019s abilities in acquiring skills that had only emerged before at much larger scale.<\/p>\n\n\n\n<p>This paradigm fits under a teacher-student approach where the large model (the teacher) is&nbsp;<br>creating demonstrations for the SLM (the student) to learn from. In this work, we extend the teacher-student paradigm to iterative learning settings as follows:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Teaching by demonstration<\/strong>: In this stage, we train the SLM by using AgentInstruct to demonstrate problems and their solutions.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Practice and feedback: <\/strong>We let the SLM practice solving problems on its own. For every problem, we allow the SLM to create multiple solutions. We then use the teacher model to provide feedback on the SLM solutions. If the SLM is unable to correctly solve the problem, even after multiple attempts, we use a solution provided by the teacher.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Iterative improvement: <\/strong>We use the teacher feedback to create preference data showing the SLM both good and bad solutions to the same problem, and then retrain the SLM.<\/li>\n<\/ul>\n\n\n\n<p>The practice, feedback, and iterative improvement steps can be repeated multiple times.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h2>\n\n\n\n<p>Our findings show that smaller models are valuable in specialized settings where they can match the performance of much larger models but with a limited scope. By training Orca-Math on a small dataset of 200,000 math problems, we have achieved performance levels that rival or surpass those of much larger models.<\/p>\n\n\n\n<p>The relatively small size of the dataset also shows the potential of using multi-agent flows to simulate the process of data and feedback generation. The small dataset size has implications for the cost of training and highlights that training data with richer learning signals can improve the efficiency of the learning process. Our findings also highlight the potential of continual learning and the improvement of language models, where the model iteratively improves as it receives more feedback from a person or another model.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Microsoft\u2019s Orca-Math, a specialized small language model, outperforms much larger models in solving math problems that require multi-step reasoning and shows the potential of using feedback to improve language models. Learn more.<\/p>\n","protected":false},"author":37583,"featured_media":1010433,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13545],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1010406","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[992148],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[983295,973047],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Arindam Mitra","user_id":42978,"display_name":"Arindam Mitra","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/armitra\/\" aria-label=\"Visit the profile page for Arindam Mitra\">Arindam Mitra<\/a>","is_active":false,"last_first":"Mitra, Arindam","people_section":0,"alias":"armitra"},{"type":"user_nicename","value":"Hamed Khanpour","user_id":38055,"display_name":"Hamed Khanpour","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/hakhanpo\/\" aria-label=\"Visit the profile page for Hamed Khanpour\">Hamed Khanpour<\/a>","is_active":false,"last_first":"Khanpour, Hamed","people_section":0,"alias":"hakhanpo"},{"type":"user_nicename","value":"Corby Rosset","user_id":41997,"display_name":"Corby Rosset","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/corbyrosset\/\" aria-label=\"Visit the profile page for Corby Rosset\">Corby Rosset<\/a>","is_active":false,"last_first":"Rosset, Corby","people_section":0,"alias":"corbyrosset"},{"type":"user_nicename","value":"Ahmed Awadallah","user_id":31979,"display_name":"Ahmed Awadallah","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/hassanam\/\" aria-label=\"Visit the profile page for Ahmed Awadallah\">Ahmed Awadallah<\/a>","is_active":false,"last_first":"Awadallah, Ahmed","people_section":0,"alias":"hassanam"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-960x540.png\" class=\"img-object-cover\" alt=\"abstract wave lines on a gradient background\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-960x540.png 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-1024x576.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-768x432.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-240x135.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-640x360.png 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1-1280x720.png 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/02\/ORCA-BlogHeroFeature-1400x788-1.png 1400w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/armitra\/\" title=\"Go to researcher profile for Arindam Mitra\" aria-label=\"Go to researcher profile for Arindam Mitra\" data-bi-type=\"byline author\" data-bi-cN=\"Arindam Mitra\">Arindam Mitra<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/hakhanpo\/\" title=\"Go to researcher profile for Hamed Khanpour\" aria-label=\"Go to researcher profile for Hamed Khanpour\" data-bi-type=\"byline author\" data-bi-cN=\"Hamed Khanpour\">Hamed Khanpour<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/corbyrosset\/\" title=\"Go to researcher profile for Corby Rosset\" aria-label=\"Go to researcher profile for Corby Rosset\" data-bi-type=\"byline author\" data-bi-cN=\"Corby Rosset\">Corby Rosset<\/a>, and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/hassanam\/\" title=\"Go to researcher profile for Ahmed Awadallah\" aria-label=\"Go to researcher profile for Ahmed Awadallah\" data-bi-type=\"byline author\" data-bi-cN=\"Ahmed Awadallah\">Ahmed Awadallah<\/a>","formattedDate":"March 5, 2024","formattedExcerpt":"Microsoft\u2019s Orca-Math, a specialized small language model, outperforms much larger models in solving math problems that require multi-step reasoning and shows the potential of using feedback to improve language models. Learn more.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1010406"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/37583"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1010406"}],"version-history":[{"count":38,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1010406\/revisions"}],"predecessor-version":[{"id":1012320,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1010406\/revisions\/1012320"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1010433"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1010406"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1010406"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1010406"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1010406"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1010406"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1010406"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1010406"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1010406"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1010406"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1010406"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1010406"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}