{"id":976521,"date":"2023-10-18T09:00:00","date_gmt":"2023-10-18T16:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=976521"},"modified":"2023-10-18T07:00:23","modified_gmt":"2023-10-18T14:00:23","slug":"understanding-the-user-how-the-enterprise-system-usability-scale-esus-aligns-with-user-reality","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/understanding-the-user-how-the-enterprise-system-usability-scale-esus-aligns-with-user-reality\/","title":{"rendered":"Understanding the user: How the Enterprise System Usability Scale aligns with user reality"},"content":{"rendered":"\n

This position research paper was presented at the <\/strong>26th<\/sup> ACM Conference on Computer-Supported Cooperative Work and Social Computing<\/strong> (opens in new tab)<\/span><\/a> (CSCW 2023), a premier venue for research on the design and use of technologies that affect groups, organizations, and communities.<\/strong><\/p>\n\n\n\n

\"Microsoft<\/figure>\n\n\n\n

In the business world, measuring success is as critical as selecting the right goals, and metrics act as a guiding compass, shaping organizational objectives. They are instrumental as businesses strategize to develop products that are likely to succeed in specific markets or among certain user groups.  <\/p>\n\n\n\n

However, businesses often overlook whether these metrics accurately reflect users\u2019 experiences and behaviors. Do they truly reflect the consumers\u2019 journey and provide a reliable evaluation of the products\u2019 place in the market? Put differently, do these metrics truly capture a product’s effectiveness and value, or are they superficial, overlooking deeper insights that could lead a business toward lasting success?<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tMicrosoft Research Blog<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"satellite\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Introducing Aurora: The first large-scale foundation model of the atmosphere<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Aurora, a new AI foundation model from Microsoft Research, can transform our ability to predict and mitigate extreme weather events and the effects of climate change by enabling faster and more accurate weather forecasts than ever before.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tLearn more\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Challenges in enterprise usability metrics research<\/h2>\n\n\n\n

In our paper, \u201cA Call to Revisit Classic Measurements for UX Evaluation (opens in new tab)<\/span><\/a>,\u201d presented at the UX Outcomes Workshop at CSCW 2023 (opens in new tab)<\/span><\/a>, we explore these questions about usability metrics\u2014which evaluate the simplicity and effectiveness of a product, service, or system for its users\u2014and their applicability to enterprise products. These metrics are vital when measuring a product’s health in the market and predicting adoption rates, user engagement, and, by extension, revenue generation. Current usability metrics in the enterprise space often fail to align with the actual user\u2019s reality when using technical enterprise products such as business analytics, data engineering, and data science software. Oftentimes, they lack methodological rigor, calling into question their generalizability and validity.<\/p>\n\n\n\n

One example is the System Usability Scale (opens in new tab)<\/span><\/a> (SUS), the most widely used usability metric. In the context of enterprise products, at least two questions used in SUS do not resonate with users\u2019 actual experiences: “I think I would like to use the system frequently” and “I think I need the support of a technical person to be able to use this product.” Because users of enterprise products are consumers, not necessarily customers, they often do not get to choose which product to use. In some cases, they are IT professionals with no one to turn to for technical assistance. This misalignment highlights the need to refine how we measure usability for enterprise products. <\/p>\n\n\n\n

Another concern is the lack of rigorous validation for metrics that reflect a product\u2019s performance. For instance, UMUX-Lite (opens in new tab)<\/span><\/a> is a popular metric for its simplicity and strong correlation with SUS. However, its scoring methodology requires that researchers use an equation consisting of a regression weight and constant to align the average scores with SUS scores. This lacks a solid theoretical foundation, which raises questions about UMUX-Lite’s ability to generalize to different contexts and respondent samples.<\/p>\n\n\n\n

The lack of standardization underscores the need for metrics that are grounded in the user\u2019s reality for the types of products being assessed and based on theoretical and empirical evidence, ensuring that they are generalizable to diverse contexts. This approach will pave the way for more reliable insights into product usability, fostering informed decisions crucial for enhancing the user experience and driving product success.<\/p>\n\n\n\n

ESUS: A reality-driven approach to usability metrics<\/h2>\n\n\n\n

Recognizing this need, we endeavored to create a new usability metric that accurately reflects the experience of enterprise product users, built on solid theory and supported by empirical evidence. Our research combines qualitative and quantitative approaches to devise a tailored usability metric for enterprise products, named the Enterprise System Usability Scale (ESUS). <\/p>\n\n\n\n

ESUS offers a number of benefits over the SUS and UMUX-Lite. It is more concise than the SUS, containing only half the questions and streamlining the evaluation process. It also eliminates the need for practitioners to use a sample-specific weight and constant, as required by UMUX-Lite, providing a more reliable measure of product usability. Moreover, ESUS demonstrates convergent validity, correlating with other usability metrics, such as SUS. Most importantly, through its conciseness and specificity, it was designed with enterprise product users in mind, providing relevant and actionable insights.  <\/p>\n\n\n\n

In Table 1 below, we offer ESUS as a step towards more accurate, reliable, and user-focused metrics for enterprise products, which are instrumental in driving well-informed decisions in improving product usability and customer satisfaction.<\/p>\n\n\n\n

ESUS Items<\/th>1<\/th>2<\/th>3<\/th>4<\/th>5<\/th><\/tr><\/thead>
How useful is [this product] to you?<\/td>Not at all useful<\/td>Slightly useful<\/td>Somewhat useful<\/td>Mostly useful<\/td>Very useful<\/td><\/tr>
How easy or hard was [this product] to use for you?<\/td>Very hard<\/td>Hard<\/td>Neutral<\/td>Easy<\/td>Very easy<\/td><\/tr>
How confident were you when using [this product]?<\/td>Not at all confident<\/td>Slightly confident<\/td>Somewhat confident<\/td>Mostly confident<\/td>Very confident<\/td><\/tr>
How well do the functions work together or do not work together in [this product]?<\/td>Does not work together at all<\/td>Does not work well together<\/td>Neutral<\/td>Works well together<\/td>Works very well together<\/td><\/tr>
How easy or hard was it to get started with [this product]?<\/td>Very hard<\/td>Hard<\/td>Neutral<\/td>Easy<\/td>Very easy<\/td><\/tr><\/tbody><\/table>
Table 1: Proposed ESUS questionnaire<\/figcaption><\/figure>\n\n\n\n

Looking ahead: Advancing precision in understanding the user<\/h2>\n\n\n\n

Moving forward, our focus is on rigorously testing and enhancing ESUS. We aim to examine its consistency over time and its effectiveness with small sample sizes. Our goal is to ensure our metrics are as robust and adaptable as the rapidly evolving enterprise product environment requires. We\u2019re committed to continuous improvement, striving for metrics that are not just accurate but also relevant and reliable, offering actionable insights for an ever-improving user experience.<\/p>\n","protected":false},"excerpt":{"rendered":"

This position research paper was presented at the 26th ACM Conference on Computer-Supported Cooperative Work and Social Computing (opens in new tab) (CSCW 2023), a premier venue for research on the design and use of technologies that affect groups, organizations, and communities. In the business world, measuring success is as critical as selecting the right […]<\/p>\n","protected":false},"author":42735,"featured_media":976542,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13563,13554,13559],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-976521","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-data-platform-analytics","msr-research-area-human-computer-interaction","msr-research-area-social-sciences","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Stephen Schneider","user_id":42945,"display_name":"Stephen Schneider","author_link":"Stephen Schneider<\/a>","is_active":false,"last_first":"Schneider, Stephen","people_section":0,"alias":"stschneider"},{"type":"user_nicename","value":"Serena Hillman","user_id":41143,"display_name":"Serena Hillman","author_link":"Serena Hillman<\/a>","is_active":false,"last_first":"Hillman, Serena","people_section":0,"alias":"sehillma"}],"msr_type":"Post","featured_image_thumbnail":"\"Microsoft","byline":"Stephen Schneider<\/a> and Serena Hillman<\/a>","formattedDate":"October 18, 2023","formattedExcerpt":"This position research paper was presented at the 26th ACM Conference on Computer-Supported Cooperative Work and Social Computing (opens in new tab) (CSCW 2023), a premier venue for research on the design and use of technologies that affect groups, organizations, and communities. In the business…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/976521"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42735"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=976521"}],"version-history":[{"count":32,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/976521\/revisions"}],"predecessor-version":[{"id":977187,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/976521\/revisions\/977187"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/976542"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=976521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=976521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=976521"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=976521"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=976521"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=976521"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=976521"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=976521"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=976521"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=976521"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=976521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}