{"id":830284,"date":"2022-03-31T10:00:00","date_gmt":"2022-03-31T17:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=830284"},"modified":"2022-08-17T09:03:48","modified_gmt":"2022-08-17T16:03:48","slug":"jigsaw-fixes-bugs-in-machine-written-software","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/jigsaw-fixes-bugs-in-machine-written-software\/","title":{"rendered":"Jigsaw fixes bugs in machine-written software"},"content":{"rendered":"\n<figure class=\"wp-block-image alignwide size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_animation_hero_no_logo.gif\" alt=\"A flowchart showing inputs pre-processed before being fed into large language models including GPT-3, Codex, and others. The post-process output is returned to the end-user for verification. If they find the output incorrect, it is edited by them, and the learning is fed back into the pre-process and post-process mechanisms to improve them further. \" class=\"wp-image-830503\"\/><\/figure>\n\n\n\n<p>Large pre-trained language models such as GPT-3, Codex, and others can be tuned to generate code from natural language specifications of programmer intent. Such automated models have the potential to improve productivity for every programmer in the world. But since the models can struggle to understand program semantics, the quality of the resulting code can\u2019t be guaranteed.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/jigsaw-large-language-models-meet-program-synthesis\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Jigsaw: Large Language Models meet Program Synthesis\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Jigsaw: Large Language Models meet Program Synthesis\">\n\t\t\t\tJigsaw: Large Language Models meet Program Synthesis&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>In our research paper, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/jigsaw-large-language-models-meet-program-synthesis\/\">Jigsaw: Large Language Models meet Program Synthesis<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, which has been accepted at the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/conf.researchr.org\/details\/icse-2022\/icse-2022-papers\/178\/Jigsaw-Large-Language-Models-meet-Program-Synthesis\" target=\"_blank\" rel=\"noreferrer noopener\">International Conference on Software Engineering<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> (ICSE 2022), we introduce a new tool that can improve the performance of these large language models. Jigsaw deploys post-processing techniques that understand the programs\u2019 syntax and semantics and then leverages user feedback to improve future performance. Jigsaw is designed to synthesize code for Python Pandas API using multi-modal inputs.<\/p>\n\n\n\n<p>Our experience suggests that as these large language models evolve for synthesizing code from intent, Jigsaw can play an important role in improving the accuracy of the systems.<\/p>\n\n\n\n<h2 id=\"the-promise-and-perils-of-machine-written-software\">The promise, and perils, of machine-written software<\/h2>\n\n\n\n<p>Large language models like OpenAI\u2019s Codex are redefining the landscape of programming. A software developer, while solving a programming task, can provide a description in English for an intended code fragment and Codex can synthesize the intended code in languages like Python or JavaScript. However, the synthesized code might be incorrect and might even fail to compile or run. Codex users are responsible for vetting the code before using it. With <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/project-jigsaw\/\">Project Jigsaw<\/a>, we aim to automate some of this vetting to boost the productivity of developers who are using large language models like Codex for code synthesis.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Project<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/project-jigsaw\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Project Jigsaw\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Project Jigsaw\">\n\t\t\t\tProject Jigsaw&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>Suppose Codex provides a code fragment to a software developer. The developer might then undertake a basic vetting by checking whether the code compiles. If it doesn\u2019t compile, then the developer might be able to use the error messages of the compiler to repair it. Once the code eventually does compile, a typical developer will test it on an input to check whether the code is producing the intended output or not. Again, the code might fail (raise an exception or produce incorrect output) and the developer would need to repair it further. <em>We show that this process can be completely automated.<\/em> Jigsaw takes as input an English description of the intended code, as well as an I\/O example. In this way, it pairs an input with the associated output, and provides the quality assurance that the output Python code will compile and generate the intended output on the provided input.<\/p>\n\n\n\n<p>In our <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/conf.researchr.org\/home\/icse-2022\" target=\"_blank\" rel=\"noreferrer noopener\">ICSE 2022<\/a> paper, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/jigsaw-large-language-models-meet-program-synthesis\/\">Jigsaw: Large Language Models meet Program Synthesis<\/a>, we evaluate this approach on Python Pandas. Pandas is a widely used API in data science, with hundreds of functions for manipulating dataframes, or tables with rows and columns. Instead of asking a developer to memorize the usage of all these functions, an arguably better approach is to use Jigsaw. With Jigsaw, the user provides a description of the intended transformation in English, an input dataframe, and the corresponding output dataframe, and then lets Jigsaw synthesize the intended code. For example, suppose a developer wants to remove the prefix \u201cName: \u201d from the column \u201ccountry\u201d in the table below. Using Pandas, this can be solved performing the following operation:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><span align=\"center\" style=\"font-size:16.0pt\">df&#91;<span style=\"color: #32CD32\">'c'<\/span>] = df&#91;<span style=\"color: #32CD32\">'c'<\/span>].str.replace(<span style=\"color: #32CD32\">'Name: '<\/span>, <span style=\"color: #32CD32\">''<\/span>)<\/span><\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Two tables with two columns labeled \u201ccountry\u201d and \u201cval\u201d, identifying three countries. In the first table, the rows are labeled: Name: India, Name: USA, and UK. In the second table, the word \u201cName\u201d is removed, so the rows read: India, USA, UK. The \u201cval\u201d column remains the same in both the tables. \" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw_table.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"322\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw_table.jpg\" alt=\"Two tables with two columns labeled \u201ccountry\u201d and \u201cval\u201d, identifying three countries. In the first table, the rows are labeled: Name: India, Name: USA, and UK. In the second table, the word \u201cName\u201d is removed, so the rows read: India, USA, UK. The \u201cval\u201d column remains the same in both the tables. \" class=\"wp-image-830296\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw_table.jpg 945w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw_table-300x102.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw_table-768x262.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw_table-240x82.jpg 240w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/a><figcaption><strong>Figure 1:<\/strong> Input dataframe and output dataframe. Jigsaw removes the superfluous word \u201cName: \u201d from the column labelled \u201ccountry\u201d.<\/figcaption><\/figure>\n\n\n\n<p>A developer who is new to Pandas will need to figure out the functions and their arguments to put together this code fragment or post the query and example to a forum like StackOverflow and wait for a good Samaritan to respond. In addition, they might have to tweak the response, at times considerably, based on the context. In contrast, it is much more convenient to provide the English query with an input-output table (or dataframe).<\/p>\n\n\n\n<h2 id=\"how-jigsaw-works\">How Jigsaw works<\/h2>\n\n\n\n<p>Jigsaw takes the English query and pre-processes it with appropriate context to build an input that can be fed to a large language model. The model is treated as a black box and Jigsaw has been evaluated both with GPT-3 and Codex. The advantage of this design is that it enables plug-and-play with the latest and greatest available models. Once the model generates an output code, Jigsaw checks whether it satisfies the I\/O example. If so, then Jigsaw is done! The model output is already correct. In our experiments, we found this happened about 30% of the time. If the code fails, then the repair process starts in a post-processing phase.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"A flowchart showing inputs pre-processed before being fed into large language models including GPT-3, Codex, and others. The post-process output is returned to the end-user for verification. If they find the output incorrect, it is edited by them, and the learning is fed back into the pre-process and post-process mechanisms to improve them further. \" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw-process-flow.png\"><img loading=\"lazy\" decoding=\"async\" width=\"624\" height=\"243\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw-process-flow.png\" alt=\"A flowchart showing inputs pre-processed before being fed into large language models including GPT-3, Codex, and others. The post-process output is returned to the end-user for verification. If they find the output incorrect, it is edited by them, and the learning is fed back into the pre-process and post-process mechanisms to improve them further. \" class=\"wp-image-830299\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw-process-flow.png 624w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw-process-flow-300x117.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/Jigsaw-process-flow-240x93.png 240w\" sizes=\"(max-width: 624px) 100vw, 624px\" \/><\/a><figcaption><strong>Figure 2:<\/strong> Inputs are pre-processed before being fed into large language models including GPT-3, Codex, and others. The post-process output is returned to the end-user for verification and editing, if necessary. The learnings are fed back into the pre-process and post-process mechanisms to improve them further.<\/figcaption><\/figure>\n\n\n\n<p>During post-processing, Jigsaw applies three kinds of transformations to repair the code. Each of these transformations is motivated by the failure modes that we have observed in GPT-3 and Codex. Surprisingly, both GPT-3 and Codex fail in similar ways and hence Jigsaw\u2019s post-processing to address these failure modes is useful for both.<\/p>\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"1085526\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Microsoft research podcast<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/whats-your-story-lex-story\/\" aria-label=\"What\u2019s Your Story: Lex Story\" data-bi-cN=\"What\u2019s Your Story: Lex Story\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/08\/Lex-Story_WYS_Hero_Feature_1400x788.jpg\" alt=\"photo of Lex Story for the What's Your Story episode of the Microsoft Research podcast\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">What\u2019s Your Story: Lex Story<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p class=\"large\">Model maker and fabricator Lex Story helps bring research to life through prototyping. He discusses his take on failure; the encouragement and advice that has supported his pursuit of art and science; and the sabbatical that might inspire his next career move.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button\">\n\t\t\t\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/whats-your-story-lex-story\/\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" aria-label=\"Listen now\" data-bi-cN=\"What\u2019s Your Story: Lex Story\" target=\"_blank\">\n\t\t\t\t\t\t\tListen now\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t\n\n\n<h2 id=\"variable-transformations\">Variable transformations<\/h2>\n\n\n\n<p>We have observed that Codex can produce output that uses incorrect variable names. For example, most publicly available code uses names like df1, df2, etc. for dataframes. So, the Codex output also uses these names. Now, if the developer uses g1, g2, etc. as dataframe names, the Codex output is probably going to use df1, df2, etc. and fail. Other times Codex confuses variable names provided to it. For instance, it produces <code>df2.merge(df1)instead of df1.merge(df2)<\/code>. To fix these kinds of errors, Jigsaw replaces names in Codex generated code with all possible names in the scope until it finds a program that satisfies the I\/O example. We find this simple transformation to be quite useful in many cases.<\/p>\n\n\n\n<h2 id=\"argument-transformations\">Argument transformations<\/h2>\n\n\n\n<p>Sometimes Codex generated code calls the expected API functions but with some of the arguments incorrect. For example:<\/p>\n\n\n\n<p>a.) Query \u2013 Drop all the rows that are duplicated in column <code>\u2018inputB\u2019<\/code><\/p>\n\n\n\n<pre class=\"wp-block-code has-large-font-size\"><code><span style=\"font-size:16.0pt\">dfout = dfin.drop_duplicates(subset=&#91;'inputB']) <span style=\"color: red\"># Model<\/span><\/span>\n<span style=\"font-size:16.0pt\">dfout = dfin.drop_duplicates(subset=&#91;'inputB'],<mark>keep=False<\/mark>) <span style=\"color: #32CD32\"># Correct<\/span><\/span><\/code><\/pre>\n\n\n\n<p>b.) Query \u2013 Replace Canada with <code>CAN<\/code> in column country of df<\/p>\n\n\n\n<pre class=\"wp-block-code has-medium-font-size\"><code><span style=\"font-size:16.0pt\">df = df.replace({'Canada':'CAN'}) <span style=\"color: red\"># Model<\/span><\/span>\n<span style=\"font-size:16.0pt\">df = df.replace(<mark>{'country':{'Canada':'CAN'}<\/mark>) <span style=\"color: #32CD32\"># Correct<\/span><\/span><\/code><\/pre>\n\n\n\n<p>To fix such errors, Jigsaw systematically enumerates over all possible arguments, using the function and argument sequences generated by Codex as a starting point, until it finds a program that satisfies the I\/O example.<\/p>\n\n\n\n<h2 id=\"ast-to-ast-transformations\">AST-to-AST transformations<\/h2>\n\n\n\n<p>An AST (abstract-syntax-tree) is a representation of code in the form of a tree. Since models like Codex work at a syntactic level, they might produce code which is syntactically very close to the intended program, but some characters might be incorrect. For example:<\/p>\n\n\n\n<p>a.) Query \u2013 Select rows of dfin where value in bar is <code><38<\/code> or <code>>60<\/code><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><span style=\"font-size:16.0pt\">dfout = dfin&#91;dfin&#91;'bar']<38|dfin&#91;'bar']>60] <span style=\"color: red\"># Model<\/span><\/span>\n<span style=\"font-size:16.0pt\">dfout = dfin&#91;<mark>(<\/mark>dfin&#91;'bar']<38<mark>)<\/mark>|<mark>(<\/mark>dfin&#91;'bar']>60<mark>)<\/mark>] <span style=\"color: #32CD32\"># Correct<\/span><\/span><\/code><\/pre>\n\n\n\n<p><em>Mistake &#8211; missing parentheses change precedence and cause exception<\/em><\/p>\n\n\n\n<p>b.) Query \u2013 Count the number of duplicated rows in df<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><span style=\"font-size:16.0pt\">out = df.duplicated() <span style=\"color: red\"># Model<\/span><\/span>\n<span style=\"font-size:16.0pt\">out = df.duplicated()<mark>.sum()<\/mark> <span style=\"color: #32CD32\"># Correct<\/span><\/span><\/code><\/pre>\n\n\n\n<p><em>Mistake \u2013 missing required summation to get the count<\/em><\/p>\n\n\n\n<p>To fix this failure mode, Jigsaw provides AST-to-AST transformations that are learned over time. The user would need to fix the code themselves \u2014 then the Jigsaw UI will capture the edit, generalize the edit to a more widely applicable transformation, and learn this transformation. With usage, the number of transformations increases, and Jigsaw becomes more and more effective.<\/p>\n\n\n\n<h2 id=\"evaluation\">Evaluation<\/h2>\n\n\n\n<p>We evaluated Codex and Jigsaw (with Codex) on various datasets and measured accuracy, which is the percentage of tasks in the dataset where the system produces the intended result. Codex gives an accuracy of about 30% out-of-the-box, which is what is expected from <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2107.03374\" target=\"_blank\" rel=\"noreferrer noopener\">OpenAI\u2019s paper<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> as well. Jigsaw improves the accuracy to >60% and, through user feedback, the accuracy improves to >80%.<\/p>\n\n\n\n<h2 id=\"the-road-ahead\">The road ahead<\/h2>\n\n\n\n<p>We have released the datasets that we used to evaluate Jigsaw in the public domain. Each dataset includes multiple tasks, where each task has an English query and an I\/O example. Solving a task requires generating a Pandas code that maps the input dataframe provided to the corresponding output dataframe. We hope that this dataset will help evaluate and compare other systems. Although there are datasets where the tasks have only English queries or only I\/O examples, the Jigsaw datasets are the first to contain both English queries and the associated I\/O examples.<\/p>\n\n\n\n<p>As these language models continue to evolve and become more powerful, we believe that Jigsaw will still be required for providing the guardrails and making these models viable in real-world scenarios. This is just addressing the tip of the iceberg for research problems in this area and many questions remain to be answered:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Can these language models be trained to learn semantics associated with code?<\/li><li>Can better preprocessing and postprocessing steps be integrated into Jigsaw? For example, we are looking at static analysis techniques to improve the post-processing.<\/li><li>Are I\/O examples effective for other APIs apart from Python Pandas? How do we tackle scenarios where I\/O examples are not available? How do we adapt Jigsaw for languages like JavaScript and general code in Python?<\/li><li>The developer overhead of providing an example over just providing a natural language query needs further evaluation and investigation.<\/li><\/ol>\n\n\n\n<p>These are some of the interesting directions we are pursuing. As we refine and improve Jigsaw, we believe it can play an important role in improving programmer productivity through automation. We continue to work on generalizing our experience with the Python Pandas API to work across other APIs and other languages.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-content-justification-center is-content-justification-center is-layout-flex wp-container-core-buttons-is-layout-1 wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a data-bi-type=\"button\" class=\"wp-block-button__link\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/jigsaw-large-language-models-meet-program-synthesis\/\" target=\"_blank\" rel=\"noreferrer noopener\">Read the paper<\/a><\/div>\n\n\n\n<div class=\"wp-block-button is-style-fill-github\"><a data-bi-type=\"button\" class=\"wp-block-button__link\" href=\"https:\/\/github.com\/microsoft\/JigsawDataset\" target=\"_blank\" rel=\"noreferrer noopener\">Download the dataset<\/a><\/div>\n<\/div>\n\n\n\n<div style=\"height:10px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p><strong>Other contributors:<\/strong><\/p>\n\n\n\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/naman-ntc.github.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Naman Jain<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, Research fellow at Microsoft Research India Lab<\/p>\n\n\n\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/skandavaidyanath.github.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Skanda Vaidyanath<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, Intern at Microsoft Research India Lab, currently pursuing master&#8217;s degree at Stanford<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Large pre-trained language models such as GPT-3, Codex, and others can be tuned to generate code from natural language specifications of programmer intent. Such automated models have the potential to improve productivity for every programmer in the world. But since the models can struggle to understand program semantics, the quality of the resulting code can\u2019t [&hellip;]<\/p>\n","protected":false},"author":37583,"featured_media":830509,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13560],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[244008],"msr-podcast-series":[],"class_list":["post-830284","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-programming-languages-software-engineering","msr-locale-en_us","msr-post-option-blog-homepage-featured","msr-promo-type-blog-post"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199562],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[809728],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Rahul Sharma","user_id":36308,"display_name":"Rahul Sharma","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/rahsha\/\" aria-label=\"Visit the profile page for Rahul Sharma\">Rahul Sharma<\/a>","is_active":false,"last_first":"Sharma, Rahul","people_section":0,"alias":"rahsha"},{"type":"user_nicename","value":"Suresh Parthasarathy","user_id":33762,"display_name":"Suresh Parthasarathy","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/supartha\/\" aria-label=\"Visit the profile page for Suresh Parthasarathy\">Suresh Parthasarathy<\/a>","is_active":false,"last_first":"Parthasarathy, Suresh","people_section":0,"alias":"supartha"},{"type":"user_nicename","value":"Sriram Rajamani","user_id":33711,"display_name":"Sriram Rajamani","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/sriram\/\" aria-label=\"Visit the profile page for Sriram Rajamani\">Sriram Rajamani<\/a>","is_active":false,"last_first":"Rajamani, Sriram","people_section":0,"alias":"sriram"},{"type":"user_nicename","value":"Nagarajan Natarajan","user_id":37311,"display_name":"Nagarajan Natarajan","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/nagarajn\/\" aria-label=\"Visit the profile page for Nagarajan Natarajan\">Nagarajan Natarajan<\/a>","is_active":false,"last_first":"Natarajan, Nagarajan","people_section":0,"alias":"nagarajn"},{"type":"user_nicename","value":"Arun Iyer","user_id":36299,"display_name":"Arun Iyer","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ariy\/\" aria-label=\"Visit the profile page for Arun Iyer\">Arun Iyer<\/a>","is_active":false,"last_first":"Iyer, Arun","people_section":0,"alias":"ariy"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-960x540.jpg\" class=\"img-object-cover\" alt=\"A flowchart showing inputs pre-processed before being fed into large language models including GPT-3, Codex, and others. The post-process output is returned to the end-user for verification. If they find the output incorrect, it is edited by them, and the learning is fed back into the pre-process and post-process mechanisms to improve them further.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-1024x577.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/03\/1400x788_Jigsaw_still-1920x1080.jpg 1920w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"March 31, 2022","formattedExcerpt":"Large pre-trained language models such as GPT-3, Codex, and others can be tuned to generate code from natural language specifications of programmer intent. Such automated models have the potential to improve productivity for every programmer in the world. But since the models can struggle to&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/830284"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/37583"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=830284"}],"version-history":[{"count":47,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/830284\/revisions"}],"predecessor-version":[{"id":870561,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/830284\/revisions\/870561"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/830509"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=830284"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=830284"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=830284"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=830284"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=830284"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=830284"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=830284"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=830284"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=830284"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=830284"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=830284"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}