{"id":1056792,"date":"2024-07-12T00:35:05","date_gmt":"2024-07-12T07:35:05","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-research-item&#038;p=1056792"},"modified":"2024-09-05T07:18:23","modified_gmt":"2024-09-05T14:18:23","slug":"lets-fix-this-together-conversational-debugging-with-github-copilot","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/lets-fix-this-together-conversational-debugging-with-github-copilot\/","title":{"rendered":"Let\u2019s Fix this Together: Conversational Debugging with GitHub Copilot"},"content":{"rendered":"<p><span dir=\"ltr\" role=\"presentation\">Despite advancements in IDE tooling, code understanding<\/span><span dir=\"ltr\" role=\"presentation\">, generation, and automated repair, debugging continues <\/span><span dir=\"ltr\" role=\"presentation\">to present significant challenges. Existing debugging strategies <\/span><span dir=\"ltr\" role=\"presentation\">available to developers in literature are often too mechanical and <\/span><span dir=\"ltr\" role=\"presentation\">rigid for day-to-day issues. Recent advances in Large Language <\/span><span dir=\"ltr\" role=\"presentation\">Models (LLMs) promise practical solutions that allow for more <\/span><span dir=\"ltr\" role=\"presentation\">free-form debugging strategies. While LLMs offer satisfactory <\/span><span dir=\"ltr\" role=\"presentation\">assistance<\/span> <span dir=\"ltr\" role=\"presentation\">in<\/span> <span dir=\"ltr\" role=\"presentation\">some<\/span> <span dir=\"ltr\" role=\"presentation\">cases,<\/span> <span dir=\"ltr\" role=\"presentation\">they<\/span> <span dir=\"ltr\" role=\"presentation\">often<\/span> <span dir=\"ltr\" role=\"presentation\">leap<\/span> <span dir=\"ltr\" role=\"presentation\">to<\/span> <span dir=\"ltr\" role=\"presentation\">action<\/span> <span dir=\"ltr\" role=\"presentation\">without <\/span><span dir=\"ltr\" role=\"presentation\">sufficient context, making implicit assumptions and providing <\/span><span dir=\"ltr\" role=\"presentation\">inaccurate responses. Moreover, the dialogue between developers <\/span><span dir=\"ltr\" role=\"presentation\">and LLMs predominantly takes the form of question-answer <\/span><span dir=\"ltr\" role=\"presentation\">pairs, placing the burden of formulating the correct questions <\/span><span dir=\"ltr\" role=\"presentation\">and sustaining multi-turn conversations on the developer.<br \/>\n<\/span><br role=\"presentation\" \/><span dir=\"ltr\" role=\"presentation\">We introduce R<\/span><span dir=\"ltr\" role=\"presentation\">OBIN<\/span><span dir=\"ltr\" role=\"presentation\">, a novel multi-agent conversational AI-<\/span><span dir=\"ltr\" role=\"presentation\">assistant within GitHub Copilot Chat, specifically designed for <\/span><span dir=\"ltr\" role=\"presentation\">debugging. R<\/span><span dir=\"ltr\" role=\"presentation\">OBIN<\/span> <span dir=\"ltr\" role=\"presentation\">moves beyond the question-answer pairs by <\/span><span dir=\"ltr\" role=\"presentation\">introducing the<\/span> <span dir=\"ltr\" role=\"presentation\">investigate & respond<\/span> <span dir=\"ltr\" role=\"presentation\">pattern, that focuses on <\/span><span dir=\"ltr\" role=\"presentation\">using information gathered automatically from the IDE or gath<\/span><span dir=\"ltr\" role=\"presentation\">ered interactively from the developer before responding. R<\/span><span dir=\"ltr\" role=\"presentation\">OBIN <\/span><span dir=\"ltr\" role=\"presentation\">incorporates<\/span> <span dir=\"ltr\" role=\"presentation\">a<\/span> <span dir=\"ltr\" role=\"presentation\">general<\/span> <span dir=\"ltr\" role=\"presentation\">debugging<\/span> <span dir=\"ltr\" role=\"presentation\">strategy<\/span> <span dir=\"ltr\" role=\"presentation\">to<\/span> <span dir=\"ltr\" role=\"presentation\">systematically <\/span><span dir=\"ltr\" role=\"presentation\">analyze bugs to sustain collaborative interactions while ensuring <\/span><span dir=\"ltr\" role=\"presentation\">that the conversation does not deviate from the debugging task <\/span><span dir=\"ltr\" role=\"presentation\">at hand. Through a within-subjects user study with 16 industry <\/span><span dir=\"ltr\" role=\"presentation\">professionals, we find that equipping R<\/span><span dir=\"ltr\" role=\"presentation\">OBIN<\/span> <span dir=\"ltr\" role=\"presentation\">to\u2014(1) leverage the <\/span><span dir=\"ltr\" role=\"presentation\">insert expansion<\/span> <span dir=\"ltr\" role=\"presentation\">interaction pattern, (2) facilitate<\/span> <span dir=\"ltr\" role=\"presentation\">turn-taking<\/span><span dir=\"ltr\" role=\"presentation\">, and <\/span><span dir=\"ltr\" role=\"presentation\">(3) utilize<\/span> <span dir=\"ltr\" role=\"presentation\">debugging strategies<\/span><span dir=\"ltr\" role=\"presentation\">\u2014leads to lowered conversation <\/span><span dir=\"ltr\" role=\"presentation\">barriers, a 2.5x improvement in bug localization and a substantial <\/span><span dir=\"ltr\" role=\"presentation\">3.5x improvement in bug resolution compared to AI-assisted <\/span><span dir=\"ltr\" role=\"presentation\">debugging in Visual Studio prior to R<\/span><span dir=\"ltr\" role=\"presentation\">OBIN<\/span><span dir=\"ltr\" role=\"presentation\">.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Despite advancements in IDE tooling, code understanding, generation, and automated repair, debugging continues to present significant challenges. Existing debugging strategies available to developers in literature are often too mechanical and rigid for day-to-day issues. Recent advances in Large Language Models (LLMs) promise practical solutions that allow for more free-form debugging strategies. While LLMs offer satisfactory [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"msr-content-type":[3],"msr-research-highlight":[246574],"research-area":[13556,13554,13560],"msr-publication-type":[193716],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[246694,255217],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1056792","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-highlight-award","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-research-area-programming-languages-software-engineering","msr-locale-en_us","msr-field-of-study-artificial-intelligence","msr-field-of-study-human-computer-interaction-4"],"msr_publishername":"","msr_edition":"","msr_affiliation":"","msr_published_date":"2024-8-1","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"Best Research Paper Award","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"file","viewUrl":"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/VLHCC__Robin__AI_assisted_Code_Debugging-14.pdf","id":"1056804","title":"_vlhcc__robin__ai_assisted_code_debugging-14","label_id":"243109","label":0}],"msr_related_uploader":"","msr_attachments":[{"id":1056804,"url":"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/VLHCC__Robin__AI_assisted_Code_Debugging-14.pdf"}],"msr-author-ordering":[{"type":"user_nicename","value":"Yasharth Bajpai","user_id":42228,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Yasharth Bajpai"},{"type":"user_nicename","value":"Bhavya Chopra","user_id":43434,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Bhavya Chopra"},{"type":"guest","value":"param-biyani","user_id":1048509,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=param-biyani"},{"type":"text","value":"Cagri Aslan","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Sumit Gulwani","user_id":33755,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Sumit Gulwani"},{"type":"text","value":"Dustin Coleman","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Chris Parnin","user_id":41985,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Chris Parnin"},{"type":"user_nicename","value":"Arjun Radhakrishna","user_id":39405,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Arjun Radhakrishna"},{"type":"user_nicename","value":"Gustavo Soares","user_id":39183,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Gustavo Soares"}],"msr_impact_theme":[],"msr_research_lab":[199565],"msr_event":[],"msr_group":[663303],"msr_project":[],"publication":[],"video":[],"download":[],"msr_publication_type":"inproceedings","related_content":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1056792"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":4,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1056792\/revisions"}],"predecessor-version":[{"id":1058673,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1056792\/revisions\/1058673"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1056792"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=1056792"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=1056792"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1056792"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=1056792"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=1056792"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=1056792"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=1056792"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=1056792"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1056792"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1056792"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=1056792"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=1056792"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=1056792"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1056792"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1056792"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}