{"id":717,"date":"2021-05-13T20:18:12","date_gmt":"2021-05-13T20:18:12","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/?post_type=pattern&#038;p=717"},"modified":"2023-10-25T07:41:45","modified_gmt":"2023-10-25T14:41:45","slug":"g15-b-request-explicit-feedback-on-selected-system-outputs","status":"publish","type":"pattern","link":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/","title":{"rendered":"Pattern 15B: Request explicit feedback on selected system&nbsp;outputs"},"content":{"rendered":"\n\n<div class=\"toolkit-heading-with-image-block\">\n\t\n\n<h2 class=\"wp-block-heading\">Pattern 15B: Request explicit feedback on selected system outputs<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png\" alt=\"\" \/><\/figure>\n\n<\/div>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"problem\">Problem<\/h4>\n\n\n\n<p>User feedback is needed to assess the system and help it improve over time.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"solution\">Solution<\/h4>\n\n\n\n<p>Implement a user-feedback mechanism that occasionally asks the user to provide explicit feedback for selected items or instances of system outputs. The system initiates the feedback interaction.<\/p>\n\n\n\n<p>Leverage user feedback for:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Evaluating the system and making decisions about deployment.<\/li>\n\n\n\n<li>Training the system so it learns and improves over time&nbsp;(see&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/learn-from-user-behavior\/\">Guideline&nbsp;13,&nbsp;<em>Learn from user behavior<\/em><\/a>,&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/update-and-adapt-cautiously\/\">Guideline&nbsp;14,&nbsp;<em>Update and adapt cautiously<\/em><\/a>,&nbsp;and&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/notify-users-about-changes\/\">Guideline&nbsp;18,&nbsp;<em>Notify users about changes<\/em><\/a>).<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"use-when\">Use When<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The nature of the system makes it impossible, difficult, or dangerous to make item-&nbsp;or instance-&nbsp;level feedback available at all times.<\/li>\n\n\n\n<li>Feedback is needed for specific features or scenarios, not for all system items or instances.<\/li>\n\n\n\n<li>Feedback is needed from a specific user group or population.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"how\">How<\/h4>\n\n\n\n<p>Decide what type of feedback to collect from the user. Options include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Binary ratings (e.g., thumbs up\/down; yes\/no; like\/dislike)<\/li>\n\n\n\n<li>Ratings on a scale (e.g., 5-star scale, smiley-&nbsp;face scale)<\/li>\n\n\n\n<li>Thoughts\/Opinions (e.g., closed-&nbsp;ended short questionnaire, open text-&nbsp;entry field)<\/li>\n<\/ul>\n\n\n\n<p>Ensure the information collected is both meaningful to the user and can be leveraged for system improvement.<\/p>\n\n\n\n<p>Design a mechanism for proactively requesting system-initiated feedback for specific system outputs, items, or instances.<\/p>\n\n\n\n<p>Determine how often to request user feedback and whether to request feedback from all users or from groups of users on a rotating basis.<\/p>\n\n\n\n<p>Consider how to convey to the user that the feedback was received, how it will be used, and whether the user experience changes immediately after giving feedback (see&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/convey-the-consequences-of-user-actions\/\">Guideline 16,&nbsp;<em>Convey the consequences of user actions<\/em><\/a>).<\/p>\n\n\n\n<p>Leverage the user feedback data as a metric of system performance (e.g., to evaluate output correctness, utility).<\/p>\n\n\n\n<p>Leverage the user feedback data for system training by working with an AI\/ML expert to decide what the model can learn from the feedback and how to integrate this data into the model\u2019s training.<\/p>\n\n\n\n<p>For open-ended feedback that may not be machine-interpretable, put a process in place for interpreting and using the feedback to improve the system.<\/p>\n\n\n\n<p>Consider how to convey to the user that the feedback was received, how it will be used, and whether the user experience changes immediately after giving feedback (see&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/convey-the-consequences-of-user-actions\/\" target=\"_blank\" rel=\"noreferrer noopener\">Guideline 16<\/a>).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"user-benefits\">User benefits<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Reduces user burden of providing feedback to the AI system.<\/li>\n\n\n\n<li>Gives the user a voice.<\/li>\n\n\n\n<li>May support personalization.<\/li>\n\n\n\n<li>May improve the experience for all system users.<\/li>\n\n\n\n<li>Feedback about system performance and user preferences can improve user experience over time (see&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/learn-from-user-behavior\/\">Guideline 13)<\/a>.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"common-pitfalls\">Common Pitfalls<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The AI system\u2019s proactive requests for user feedback disrupt the user experience (e.g., too frequent, at inopportune times).<\/li>\n\n\n\n<li>The sampling strategy for requesting user feedback introduces undesired bias in the data.<\/li>\n\n\n\n<li>It\u2019s unclear to the user that they can provide feedback to the AI system.<\/li>\n\n\n\n<li>It\u2019s unclear to the user how to provide feedback to the AI system.<\/li>\n\n\n\n<li>It\u2019s unclear to the user what the feedback options mean.<\/li>\n\n\n\n<li>It\u2019s unclear to the user what specific instance or item the feedback is about.<\/li>\n\n\n\n<li>It\u2019s unclear to the user how their feedback will be used by the AI system.<\/li>\n\n\n\n<li>In some contexts, such as driving a car, it might be inefficient, dangerous, or disruptive for the user to provide feedback to the AI system.\u200b\u200b\u200b\u200b\u200b\u200b\u200bUser feedback may be subject to reporting bias, where users with moderate views do not provide feedback. Consider mitigating reporting bias by using best practices. If reporting bias is identified in the feedback data, complement it with other data sources for system evaluation and training.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"notes\">Notes<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Explicit feedback is information the user intentionally provides for system improvement.<\/li>\n\n\n\n<li>Implicit feedback is information the system garners from user actions to improve itself.<\/li>\n<\/ul>\n","protected":false},"menu_order":152,"template":"","meta":{"toolkit_pattern_number":"15B","ep_exclude_from_search":false},"application-type":[68,61,57,96,60,99,56,55,69,29,58,30,59],"goal":[],"guideline-term":[17],"product-category":[71,70,23,64,62,26,66,28,65,31,32,63],"class_list":["post-717","pattern","type-pattern","status-publish","hentry","application-type-classification","application-type-facial-recognition","application-type-filtering-and-ranking","application-type-gen-ai","application-type-image-recognition","application-type-llm","application-type-natural-language-processing-text","application-type-natural-language-processing-voice","application-type-prediction","application-type-recommendation","application-type-route-planning","application-type-search","application-type-text-generation","guideline-term-encourage-granular-feedback","product-category-advertising","product-category-chatbot","product-category-e-commerce","product-category-email","product-category-health-and-wellness","product-category-navigation","product-category-media-entertainment","product-category-productivity","product-category-search-engine","product-category-social-networks","product-category-voice-assistants","product-category-writing-and-editing"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.8.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Pattern 15B: Request explicit feedback on selected system outputs - Microsoft HAX Toolkit<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pattern 15B: Request explicit feedback on selected system outputs - Microsoft HAX Toolkit\" \/>\n<meta property=\"og:description\" content=\"Problem User feedback is needed to assess the system and help it improve over time. Solution Implement a user-feedback mechanism that occasionally asks the user to provide explicit feedback for selected items or instances of system outputs. The system initiates the feedback interaction. Leverage user feedback for: Use When How Decide what type of feedback [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft HAX Toolkit\" \/>\n<meta property=\"article:modified_time\" content=\"2023-10-25T14:41:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data2\" content=\"Shipi Dhanorkar\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/\",\"name\":\"Pattern 15B: Request explicit feedback on selected system outputs - Microsoft HAX Toolkit\",\"isPartOf\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png\",\"datePublished\":\"2021-05-13T20:18:12+00:00\",\"dateModified\":\"2023-10-25T14:41:45+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#primaryimage\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png\",\"contentUrl\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"G15: Encourage granular feedback.\",\"item\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/encourage-granular-feedback\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Pattern 15B: Request explicit feedback on selected system&nbsp;outputs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#website\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/\",\"name\":\"Microsoft HAX Toolkit\",\"description\":\"Collaborative tools to help you create more effective and responsible human-AI experiences\",\"publisher\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#organization\",\"name\":\"Microsoft Corporation\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2022\/07\/msft-logo.png\",\"contentUrl\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2022\/07\/msft-logo.png\",\"width\":1536,\"height\":1536,\"caption\":\"Microsoft Corporation\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Pattern 15B: Request explicit feedback on selected system outputs - Microsoft HAX Toolkit","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/","og_locale":"en_US","og_type":"article","og_title":"Pattern 15B: Request explicit feedback on selected system outputs - Microsoft HAX Toolkit","og_description":"Problem User feedback is needed to assess the system and help it improve over time. Solution Implement a user-feedback mechanism that occasionally asks the user to provide explicit feedback for selected items or instances of system outputs. The system initiates the feedback interaction. Leverage user feedback for: Use When How Decide what type of feedback [&hellip;]","og_url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/","og_site_name":"Microsoft HAX Toolkit","article_modified_time":"2023-10-25T14:41:45+00:00","og_image":[{"url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes","Written by":"Shipi Dhanorkar"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/","url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/","name":"Pattern 15B: Request explicit feedback on selected system outputs - Microsoft HAX Toolkit","isPartOf":{"@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#primaryimage"},"image":{"@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#primaryimage"},"thumbnailUrl":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png","datePublished":"2021-05-13T20:18:12+00:00","dateModified":"2023-10-25T14:41:45+00:00","breadcrumb":{"@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#primaryimage","url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png","contentUrl":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2023\/05\/green-header-bar-1536x14-5-23.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/pattern\/g15-b-request-explicit-feedback-on-selected-system-outputs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/"},{"@type":"ListItem","position":2,"name":"G15: Encourage granular feedback.","item":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/guideline\/encourage-granular-feedback\/"},{"@type":"ListItem","position":3,"name":"Pattern 15B: Request explicit feedback on selected system&nbsp;outputs"}]},{"@type":"WebSite","@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#website","url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/","name":"Microsoft HAX Toolkit","description":"Collaborative tools to help you create more effective and responsible human-AI experiences","publisher":{"@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#organization","name":"Microsoft Corporation","url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#\/schema\/logo\/image\/","url":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2022\/07\/msft-logo.png","contentUrl":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/uploads\/prod\/2022\/07\/msft-logo.png","width":1536,"height":1536,"caption":"Microsoft Corporation"},"image":{"@id":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/pattern\/717","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/pattern"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/types\/pattern"}],"version-history":[{"count":10,"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/pattern\/717\/revisions"}],"predecessor-version":[{"id":10995,"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/pattern\/717\/revisions\/10995"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/media?parent=717"}],"wp:term":[{"taxonomy":"application-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/application-type?post=717"},{"taxonomy":"goal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/goal?post=717"},{"taxonomy":"guideline-term","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/guideline-term?post=717"},{"taxonomy":"product-category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/haxtoolkit\/wp-json\/wp\/v2\/product-category?post=717"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}