{"id":1140810,"date":"2025-06-23T09:38:09","date_gmt":"2025-06-23T16:38:09","guid":{"rendered":""},"modified":"2025-07-09T09:57:23","modified_gmt":"2025-07-09T16:57:23","slug":"ai-testing-and-evaluation-learnings-from-science-and-industry","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/ai-testing-and-evaluation-learnings-from-science-and-industry\/","title":{"rendered":"AI Testing and Evaluation: Learnings from Science and Industry"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788.jpg\" alt=\"Illustrated headshots of Amanda Craig Deckard & Kathleen Sullivan.\" class=\"wp-image-1141309\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788.jpg 1400w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_1400x788-1280x720.jpg 1280w\" sizes=\"auto, (max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n<div class=\"wp-block-msr-podcast-container my-4\">\n\t<iframe loading=\"lazy\" src=\"https:\/\/player.blubrry.com\/?podcast_id=146460793&modern=1\" class=\"podcast-player\" frameborder=\"0\" height=\"164px\" width=\"100%\" scrolling=\"no\" title=\"Podcast Player\"><\/iframe>\n<\/div>\n\n\n\n<p>Generative AI presents a unique challenge and opportunity to reexamine governance practices for the responsible development, deployment, and use of AI. To advance thinking in this space, Microsoft has tapped into the experience and knowledge of experts across domains\u2014from genome editing to cybersecurity\u2014to investigate the role of testing and evaluation as a governance tool. <em><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/story\/ai-testing-and-evaluation-learnings-from-science-and-industry\/\">AI Testing and Evaluation: Learnings from Science and Industry<\/a>, <\/em>hosted by Microsoft Research\u2019s <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/kasull\/\">Kathleen Sullivan<\/a>, explores what the technology industry and policymakers can learn from these fields and how that might help shape the course of AI development.<\/p>\n\n\n\n<p>In this episode, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/amcraig\/\">Amanda Craig Deckard<\/a>, senior director of public policy in Microsoft&#8217;s Office of Responsible AI, joins Sullivan to detail the company\u2019s efforts to help inform AI governance discussions and decisions, including, more recently, around the role of AI testing and evaluation. Craig Deckard and Sullivan delve into the tension that exists between the risk and opportunity of technology, the similarities and differences between AI development and the fields Microsoft is studying, and the role of different stakeholders in advancing AI governance and public policy.<\/p>\n\n\n\n<div class=\"wp-block-group msr-pattern-link-list is-layout-flow wp-block-group-is-layout-flow\">\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading h5\" id=\"learn-more-1\">Learn more:<\/h2>\n\n\n\n<ul class=\"wp-block-list list-unstyled\">\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/cdn-dynmedia-1.microsoft.com\/is\/content\/microsoftcorp\/microsoft\/msc\/documents\/presentations\/CSR\/Global-Governance-Book-DIGITAL.pdf\">Global Governance: Goals and Lessons for AI<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><br>E-book | May 2024<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/learning-from-other-domains-to-advance-ai-evaluation-and-testing\/\">Learning from other domains to advance AI evaluation and testing<\/a><br>Microsoft Research Blog | June 2025<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/ai\/responsible-ai?ef_id=_k_cb05d5950e4f117c457ebda628845b7f_k_&OCID=AIDcmm1o1fzy5i_SEM__k_cb05d5950e4f117c457ebda628845b7f_k_&msclkid=cb05d5950e4f117c457ebda628845b7f\" target=\"_blank\" rel=\"noreferrer noopener\">Responsible AI: Ethical policies and practices | Microsoft AI<\/a><\/li>\n<\/ul>\n<\/div>\n\n\n\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/focus-area\/ai-and-microsoft-research\/\" target=\"_blank\" rel=\"noreferrer noopener\">AI and Microsoft Research<\/a><\/p>\n\n\n\n<div style=\"height:25px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<section class=\"wp-block-msr-subscribe-to-podcast subscribe-to-podcast\">\n\t<div class=\"subscribe-to-podcast__inner border-top border-bottom border-width-2\">\n\t\t<h2 class=\"h5 subscribe-to-podcast__heading\">\n\t\t\tSubscribe to the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>:\t\t<\/h2>\n\t\t<ul class=\"subscribe-to-podcast__list list-unstyled\">\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"black\" viewBox=\"0 0 32 32\">  <path d=\"M7.12 0c-3.937-0.011-7.131 3.183-7.12 7.12v17.76c-0.011 3.937 3.183 7.131 7.12 7.12h17.76c3.937 0.011 7.131-3.183 7.12-7.12v-17.76c0.011-3.937-3.183-7.131-7.12-7.12zM15.817 3.421c3.115 0 5.932 1.204 8.079 3.453 1.631 1.693 2.547 3.489 3.016 5.855 0.161 0.787 0.161 2.932 0.009 3.817-0.5 2.817-2.041 5.339-4.317 7.063-0.812 0.615-2.797 1.683-3.115 1.683-0.12 0-0.129-0.12-0.077-0.615 0.099-0.792 0.192-0.953 0.64-1.141 0.713-0.296 1.932-1.167 2.677-1.911 1.301-1.303 2.229-2.932 2.677-4.719 0.281-1.1 0.244-3.543-0.063-4.672-0.969-3.595-3.907-6.385-7.5-7.136-1.041-0.213-2.943-0.213-4 0-3.636 0.751-6.647 3.683-7.563 7.371-0.245 1.004-0.245 3.448 0 4.448 0.609 2.443 2.188 4.681 4.255 6.015 0.407 0.271 0.896 0.547 1.1 0.631 0.447 0.192 0.547 0.355 0.629 1.14 0.052 0.485 0.041 0.62-0.072 0.62-0.073 0-0.62-0.235-1.199-0.511l-0.052-0.041c-3.297-1.62-5.407-4.364-6.177-8.016-0.187-0.943-0.224-3.187-0.036-4.052 0.479-2.323 1.396-4.135 2.921-5.739 2.199-2.319 5.027-3.543 8.172-3.543zM16 7.172c0.541 0.005 1.068 0.052 1.473 0.14 3.715 0.828 6.344 4.543 5.833 8.229-0.203 1.489-0.713 2.709-1.619 3.844-0.448 0.573-1.537 1.532-1.729 1.532-0.032 0-0.063-0.365-0.063-0.803v-0.808l0.552-0.661c2.093-2.505 1.943-6.005-0.339-8.296-0.885-0.896-1.912-1.423-3.235-1.661-0.853-0.161-1.031-0.161-1.927-0.011-1.364 0.219-2.417 0.744-3.355 1.672-2.291 2.271-2.443 5.791-0.348 8.296l0.552 0.661v0.813c0 0.448-0.037 0.807-0.084 0.807-0.036 0-0.349-0.213-0.683-0.479l-0.047-0.016c-1.109-0.885-2.088-2.453-2.495-3.995-0.244-0.932-0.244-2.697 0.011-3.625 0.672-2.505 2.521-4.448 5.079-5.359 0.547-0.193 1.509-0.297 2.416-0.281zM15.823 11.156c0.417 0 0.828 0.084 1.131 0.24 0.645 0.339 1.183 0.989 1.385 1.677 0.62 2.104-1.609 3.948-3.631 3.005h-0.015c-0.953-0.443-1.464-1.276-1.475-2.36 0-0.979 0.541-1.828 1.484-2.328 0.297-0.156 0.709-0.235 1.125-0.235zM15.812 17.464c1.319-0.005 2.271 0.463 2.625 1.291 0.265 0.62 0.167 2.573-0.292 5.735-0.307 2.208-0.479 2.765-0.905 3.141-0.589 0.52-1.417 0.667-2.209 0.385h-0.004c-0.953-0.344-1.157-0.808-1.553-3.527-0.452-3.161-0.552-5.115-0.285-5.735 0.348-0.823 1.296-1.285 2.624-1.291z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Apple Podcasts<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M6.4 6a2.392 2.392 0 00-2.372 2.119L16 15.6l11.972-7.481A2.392 2.392 0 0025.6 6H6.4zM4 10.502V22.8a2.4 2.4 0 002.4 2.4h19.2a2.4 2.4 0 002.4-2.4V10.502l-11.365 7.102a1.2 1.2 0 01-1.27 0L4 10.502z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Email<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M12.414 4.02c-.062.012-.126.023-.18.06a.489.489 0 00-.12.675L13.149 6.3c-1.6.847-2.792 2.255-3.18 3.944h13.257c-.388-1.69-1.58-3.097-3.179-3.944l1.035-1.545a.489.489 0 00-.12-.675.492.492 0 00-.675.135l-1.14 1.68a7.423 7.423 0 00-2.55-.45c-.899 0-1.758.161-2.549.45l-1.14-1.68a.482.482 0 00-.494-.195zm1.545 3.824a.72.72 0 110 1.44.72.72 0 010-1.44zm5.278 0a.719.719 0 110 1.44.719.719 0 110-1.44zM8.44 11.204A1.44 1.44 0 007 12.644v6.718c0 .795.645 1.44 1.44 1.44.168 0 .33-.036.48-.09v-9.418a1.406 1.406 0 00-.48-.09zm1.44 0V21.76c0 .793.646 1.44 1.44 1.44h10.557c.793 0 1.44-.647 1.44-1.44V11.204H9.878zm14.876 0c-.169 0-.33.035-.48.09v9.418c.15.052.311.09.48.09a1.44 1.44 0 001.44-1.44v-6.719a1.44 1.44 0 00-1.44-1.44zM11.8 24.16v1.92a1.92 1.92 0 003.84 0v-1.92h-3.84zm5.759 0v1.92a1.92 1.92 0 003.84 0v-1.92h-3.84z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Android<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M16 4C9.383 4 4 9.383 4 16s5.383 12 12 12 12-5.383 12-12S22.617 4 16 4zm5.08 17.394a.781.781 0 01-1.086.217c-1.29-.86-3.477-1.434-5.303-1.434-1.937.002-3.389.477-3.403.482a.782.782 0 11-.494-1.484c.068-.023 1.71-.56 3.897-.562 1.826 0 4.365.492 6.171 1.696.36.24.457.725.217 1.085zm1.56-3.202a.895.895 0 01-1.234.286c-2.338-1.457-4.742-1.766-6.812-1.747-2.338.02-4.207.466-4.239.476a.895.895 0 11-.488-1.723c.145-.041 2.01-.5 4.564-.521 2.329-.02 5.23.318 7.923 1.995.419.26.547.814.286 1.234zm1.556-3.745a1.043 1.043 0 01-1.428.371c-2.725-1.6-6.039-1.94-8.339-1.942h-.033c-2.781 0-4.923.489-4.944.494a1.044 1.044 0 01-.474-2.031c.096-.023 2.385-.55 5.418-.55h.036c2.558.004 6.264.393 9.393 2.23.497.292.663.931.371 1.428z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Spotify<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M6.667 4a2.676 2.676 0 00-2.612 2.13v.003c-.036.172-.055.35-.055.534v18.666c0 .183.019.362.055.534v.003a2.676 2.676 0 002.076 2.075h.002c.172.036.35.055.534.055h18.666A2.676 2.676 0 0028 25.333V6.667a2.676 2.676 0 00-2.13-2.612h-.003A2.623 2.623 0 0025.333 4H6.667zM8 8h1.333C17.42 8 24 14.58 24 22.667V24h-2.667v-1.333c0-6.618-5.382-12-12-12H8V8zm0 5.333h1.333c5.146 0 9.334 4.188 9.334 9.334V24H16v-1.333A6.674 6.674 0 009.333 16H8v-2.667zM10 20a2 2 0 11-.001 4.001A2 2 0 0110 20z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">RSS Feed<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t<\/ul>\n\t<\/div>\n<\/section>\n\n\n<div class=\"wp-block-msr-show-more\">\n\t<div class=\"bg-neutral-100 p-5\">\n\t\t<div class=\"show-more-show-less\">\n\t\t\t<div>\n\t\t\t\t<span>\n\t\t\t\t\t\n\n<h2 class=\"wp-block-heading\" id=\"transcript\">Transcript&nbsp;<\/h2>\n\n\n\n<p>[MUSIC]&nbsp;<\/p>\n\n\n\n<p><strong>KATHLEEN SULLIVAN:<\/strong> Welcome to <em>AI Testing and Evaluation: Learnings from Science and Industry<\/em>. I&#8217;m your host, Kathleen Sullivan.<\/p>\n\n\n\n<p>As generative AI continues to advance, Microsoft has gathered a range of experts\u2014from genome editing to cybersecurity\u2014to share how their fields approach evaluation and risk assessment. Our goal is to learn from their successes and their stumbles to move the science and practice of AI testing forward. In this series, we&#8217;ll explore how these insights might help guide the future of AI development, deployment, and responsible use.<\/p>\n\n\n\n<p>[MUSIC ENDS]\u00a0<\/p>\n\n\n\n<p>For our introductory episode, I&#8217;m pleased to welcome Amanda Craig Deckard from Microsoft to discuss the company&#8217;s efforts to learn about testing in other sectors.\u00a0<\/p>\n\n\n\n<p>Amanda is senior director of public policy in the Office of Responsible AI, where she leads a team that works closely with engineers, researchers, and policy experts to help ensure AI is being developed and used responsibly. Their insights shape Microsoft&#8217;s contribution to public policy discussions on laws, norms, and standards for AI.\u00a0<\/p>\n\n\n\n<p>Amanda, welcome to the podcast.<\/p>\n\n\n\n\t\t\t\t<\/span>\n\t\t\t\t<span id=\"show-more-show-less-toggle-1\" class=\"show-more-show-less-toggleable-content\">\n\t\t\t\t\t\n\n\n\n<p><strong>AMANDA CRAIG DECKARD:<\/strong> Thank you.\u00a0<\/p>\n\n\n\n<p><strong>SULLIVAN:<\/strong> Amanda, let&#8217;s give the listeners a little bit of your background. What&#8217;s your origin story? Can you talk to us a little bit about maybe how you started in tech? And I would love to also learn a little bit more about what your team does in the Office of Responsible AI.\u00a0<\/p>\n\n\n\n<p><strong>CRAIG DECKARD:<\/strong> Sure. Thank you. I&#8217;d say my [LAUGHS] path to tech, to Microsoft, as well, was a bit, like, circuitous, maybe. You know, I thought for the longest time I was going to be a journalist. I studied forced migration. I worked in a sort of state level sort of trial court in Indiana, a legal service provider in India, just to give you a bit of a flavor.<\/p>\n\n\n\n<p>I made my way to Microsoft in 2014 and have been here since, working in cybersecurity public policy first and now in responsible AI. And the way that our Office of Responsible AI has really, sort of, structured itself is bringing together the kind of expertise to really work on defining policy and how to operationalize it at the same time.<\/p>\n\n\n\n<p>And, you know, that means that we have been working through this, you know, real challenge of defining internal policy and practice, making sure that&#8217;s deeply grounded in the work of our colleagues at Microsoft Research, and then really closely working with engineering to make sure that we have the processes, that we have the tools, to implement that policy at scale.\u00a0<\/p>\n\n\n\n<p>And I&#8217;m really drawn to these kind of hard problems where they have the character of two things being true or there&#8217;s like, you know, real tension on both sides and in particular, in the context of those kinds of problems, roles in which, like, the whole job is actually just sitting with that tension, not necessarily, like, resolving it and expecting that you&#8217;re done.<\/p>\n\n\n\n<p>And I think, really, there are two reasons why tech is so, kind of, representative of that kind of challenge that I&#8217;ve always found fascinating. You know, one is that, of course, tech is, sort of, ubiquitous. It&#8217;s really impacting so many people&#8217;s lives. But also, you know, because, as I think has become part of our vernacular now, but, you know, is not necessarily immediately intuitive, is like the fact that technology is both a tool and a weapon. And so that&#8217;s just, like, another reason why, you know, we have to continuously work through that tension and, sort of, like, sit with it, right, and even as tech evolves over time.<\/p>\n\n\n\n<p><strong>SULLIVAN:<\/strong> You bring up such great points, and this field is not black and white. I think that even underscores, you know, this notion that you highlighted that it&#8217;s impacting everyone. And, you know, to set the stage for our listeners, last year, we pulled in a bunch of experts from cybersecurity, biotech, finance, and we ran this large workshop to study how they&#8217;re thinking about governance in those playbooks. And so I&#8217;d love to understand a little bit more about what sparked that effort\u2014and, you know, there&#8217;s a piece of this which is really centered around testing\u2014and to hear from you why the focus on testing is so important.&nbsp;<\/p>\n\n\n\n<p><strong>CRAIG DECKARD:<\/strong> If I could rewind a little bit and give you a bit of history of how we even arrived at bringing these experts together, you know, we actually started on this journey in 2023. At that time, there were, like, a lot of these big questions swirling around about, you know, what did we need in terms of governance for AI? Of course, this was in the immediate aftermath of the ChatGPT sort of wave and everyone recognizing that, like, the technology was going to have a different level of impact in the near term. And so, you know, what do we need from governance? What do we need at the global level, in particular, of governance?\u00a0<\/p>\n\n\n\n<p>And so at the time, in early 2023 especially, there were a lot of attempts to sort of draw analogies to other global governance institutions in other domains. So we actually in 2023 brought together a different workshop than the one that you&#8217;re referring to specifically focused on testing last year. And we, kind of, had two big takeaways from that conversation.\u00a0<\/p>\n\n\n\n<p>One was, what are the actual functions of these institutions and how do they apply to AI? And, actually, one of the takeaways was they all sort of apply. [LAUGHS] There&#8217;s, like, a role for, you know, any of the functions, whether it be sort of driving consensus on research or building industry standards or managing, kind of, frontier risks, for thinking about how those might be needed in the AI context.\u00a0<\/p>\n\n\n\n<p>And one of the other big takeaways was that, you know, there are also limitations in these analogies. You know, each of the institutions grew up in its own, sort of, unique historical moment, like the one that we sit in with AI right now. And in each of those circumstances, they don&#8217;t exactly translate to this moment. And so, yeah, there was like this kind of, OK, we want to draw what we can from this conversation and then we also want to understand, what is also very important that&#8217;s just different for AI right now?\u00a0<\/p>\n\n\n\n<p>We published a <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/cdn-dynmedia-1.microsoft.com\/is\/content\/microsoftcorp\/microsoft\/msc\/documents\/presentations\/CSR\/Global-Governance-Book-DIGITAL.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">book with the lessons from that conversation in 2023<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. And then we actually went on a bit of a tour [LAUGHS] with that content where we had a number of roundtables actually all over the world where we gathered feedback on how those analogies were landing, how our takeaways were landing. And one of the things that we took from them was a gap that some of the participants saw in the analogies that we chose to focus on. So across multiple conversations, other domains kept being raised, like, why did you not also study pharmaceuticals? Why did you also not study cybersecurity, for example? And so that, you know, naturally got us thinking about what further lessons we could draw from <em>those<\/em> domains.&nbsp;<\/p>\n\n\n\n<p>At the same time, though, we also saw a need to, again, go deeper than what we went and really, like, focus on a narrower problem. So that&#8217;s really what led us to trying to think about a more specific problem where we could think across levels of governance and bring in some of these other domains. And, you know, testing was top of mind. Continues to be a really important topic in the AI policy conversation right now, I think, for really good reason. A lot of policymakers are focused on, you know, what we need to do to,\u00a0kind of, have there be sufficient trust, and testing is going to be a part of that\u2014really better understand risk, enable everyone to be able to make more, kind of, risk-informed decisions, right. Testing is an important component for governance and AI and, of course, in all of these other domains, as well.\u00a0<\/p>\n\n\n\n<p>So I&#8217;ll just add the other, kind of, input into the process for this second round was exploring other analogies beyond those that we, kind of, got feedback on. And one of the early, kind of, examples of another domain that would be really worthwhile to study that came to mind from, sort of, just studying the literature was genome editing. <\/p>\n\n\n\n<p>You know, genome editing was really interesting through the process of thinking about other kind of general-purpose technologies. We also arrived at nanoscience and brought those into the conversation.&nbsp;<\/p>\n\n\n\n<p><strong>SULLIVAN:<\/strong> That&#8217;s great. I mean, actually, if you could double-click,&nbsp;I mean, you just named a number of industries. I&#8217;d love to just understand which of those worlds maybe feels the closest to what we&#8217;re wrestling with, with AI and maybe which is kind of the farthest off, and what makes them stand out to you?<\/p>\n\n\n\n<p><strong>CRAIG DECKARD:<\/strong> Oh, such a good question. For this second round, we actually brought together eight different domains, right. And I think we actually thought we would come out of this conversation with some bit of clarity around, <em>Oh, if we just, sort of, take this approach for this domain or that domain, we&#8217;ll sort of have\u2014at least for now\u2014really solved part of the puzzle.<\/em> [LAUGHS] And, you know, our public policy team the day after the workshop, we had a, sort of, follow-on discussion, and the very first thing that we started with in that conversation was like, <em>OK, so which of these domains?<\/em> And fascinatingly, like, everyone was sort of like, <em>Ahh! <\/em>[LAUGHS] <em>None of them are applying perfectly<\/em>. I mean, this is also speaking to the limitations of analogies that we already acknowledged.\u00a0<\/p>\n\n\n\n<p>And also, you know, all of the experts from across these domains gave us really interesting insights into, sort of, the tradeoffs and the limitations and how they were working. None are really applying perfectly for us. But all of them do offer a thread of insight that is really useful for thinking about testing in AI, and there are some different dimensions that I think are really useful as framing for that.\u00a0<\/p>\n\n\n\n<p>I mean, one is just this horizontal-versus-vertical,<strong>&nbsp;<\/strong>kind of, difference in domains and, you know, the horizontal technology like genome editing or nanoscience&nbsp;just being inherently different and seemingly very similar to AI in that you want to be able to understand risks in the technology itself <em>and<\/em> there is just so much contextual, sort of, factor that matters in the application of those technologies for how the risk manifests that you really need to, kind of, do those two things at once\u2014of understanding the technology but then really thinking about risk and governance in the context of application versus, you know, a context like or a domain like civil aviation or nuclear technology, for example.<\/p>\n\n\n\n<p>You know, even in the workshop itself that we hosted late last year, where we brought together this second round of experts, it was really interesting. We actually started the conversation by trying to understand how those different domains defined risks, where they were able to set risk thresholds. That&#8217;s been such a part of the AI policy conversation in the last year. And, you know, it was really instructive that the more vertical domains were able to, sort of, snap to clearer answers much more quickly. [LAUGHS] But, like, the horizontal nanoscience and genome editing were not because it just depends, right. So anyway, the horizontal-vertical dimension seems like a really important one to draw from and apply to AI. <\/p>\n\n\n\n<p>The couple of others that I would offer is just, you know, thinking about the different kinds of technologies. You know, obviously, there&#8217;s some of the domains that we studied that they&#8217;re just inherently, sort of, like, physical technologies \u2026 a mix of physical and digital or virtual in a lot of cases because all of these are, of course, applying digital technology. But like, you know, there is just a difference between something like an <em>airplane<\/em> or a <em>medical device<\/em> or, you know, the more kind of virtual or intangible sort of technologies even, you know, of course, AI and some of the other like cyber and genome editing but also like, you know, financial services having some of that quality. And again, I think the thing that&#8217;s interesting to us about AI is to think about AI and risk evaluation of AI as being, you know, having a large component of that being about the kind of virtual or intangible technology. <em>And also<\/em>, you know, there is a future of robotics where we might need to think about the, kind of, physical risk evaluation kind of work, as well.<\/p>\n\n\n\n<p>And then the final thing I&#8217;d maybe say in terms of thinking about which domains have the lessons for AI that are most applicable is just how they&#8217;ve grappled with these different kind of governance questions. Things like how to turn the dial in terms of being more or less prescriptive on risk evaluation approaches, how they think about the balance of, kind of, pre-market versus post-market risk evaluation in testing, and what the tradeoffs have been there across domains has been really interesting to kind of tease out. And then also thinking about, sort of, who does what?<\/p>\n\n\n\n<p>So, you know, in each of these different domains, it was interesting to hear about, like, you know, the role of industry, the role of governments, the role of third-party experts in designing evaluations and developing standards and actually doing the work, and, kind of, having the pull through of what it means for risk and governance decisions. There were, again, there was a variety of, sort of, approaches across these domains that I think were interesting for AI.<\/p>\n\n\n\n<p><strong>SULLIVAN:<\/strong> You mentioned that there&#8217;s a number of different stakeholders to be considering across the board as we&#8217;re thinking about policy, as we&#8217;re thinking about regulation. Where can we collaborate more across industry? Is it academia? Regulators? Just, how can we move the needle faster?\u00a0<\/p>\n\n\n\n<p><strong>CRAIG DECKARD:<\/strong> I think all of the above [LAUGHTER] is needed. But it&#8217;s also really important to have all of that, kind of, expertise brought together, you know, and I think, you know, one of the things that we certainly heard from multiple of the domains, if not all of them, was that same actual interest and need and the same sort of ongoing work to try to figure that out.<\/p>\n\n\n\n<p>You know, even where there had been progress in some of the other domains with bringing together, you know, some industry stakeholders or, you know, industry and government, there was still a desire to actually do more there. Like, if there was some progress in industry and government, the need was, <em>And more kind of cross-jurisdiction government conversation<\/em>, for example. Or some progress on, you know, within the industry but needing to, like, strengthen the partnership with academia, for example. So, you know, I think it speaks to, like, the quality of your question, to be honest, that, you know, all of these domains are actually still grappling with this and still seeing the need to grow in that direction more.\u00a0<\/p>\n\n\n\n<p>What I&#8217;d say about AI today is that we have made good progress with, you know, starting to build some industry partnerships. You know, we were a founding member of the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.frontiermodelforum.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">Frontier Model Forum, or FMF<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, which has been a very useful place for us to work with some peers on really trying to bring forward some best practices that apply across our organizations. You know, there are other forums as well, like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/mlcommons.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">MLCommons<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, where we&#8217;re working with others in industry and broader, sort of, academic and civil society communities. <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/partnershiponai.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">Partnership on AI<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> is another one I think about that, kind of, fits that mold, as well, in a really positive way. And, like, there are a lot of different, sort of, governance needs to think through and where, you know, we can really think about bringing that expertise together is going to be so important.<\/p>\n\n\n\n<p>I think about almost, like, in the near to mid-term, like three issues that we need to address in the AI, kind of, policy and testing context. One is just building kind of, like, a flexible framework that allows us to really build trust while we continue to advance the science and the standards. You know, we are going to need to do both at once. And so we need a flexible framework that enables that kind of agility, and advancing the science and the standards, that <em>is <\/em>going to be something that really demands that kind of cross-discipline or cross kind of expertise group coming together to work on that\u2014researchers, academics, civil society, governments and, of course, industry.<\/p>\n\n\n\n<p>And so I think that is, actually, the second problem is, like, how do we actually build the kind of forums and ways of working together, the public-private partnership kind of efforts that allow all of that expertise to come together and fit together over time, right. Because when these are really big, broad challenges, you kind of have to break them down incrementally, make progress on them, and then bring them back together. <\/p>\n\n\n\n<p>And so I think about, like, one example that I, you know, really have been reflecting on lately is, you know, in the context of building standards, like, how do you do that, right? Again, standards are going to benefit from that whole community of expertise. And, you know, there are lots of different kinds of quote-unquote standards, though, right. You kind of have the \u201csmall <em>s<\/em>\u201d industry standards. You have the kind of \u201cbig <em>S<\/em>\u201d international standards, for example. And how do you, kind of, leverage one to accelerate the other, I think, is part of, like, how we need to work together within this ecosystem. And, like, I think what we and others have done in an organization like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/c2pa.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">C2PA [Coalition for Content Provenance and Authenticity]<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, for example, where we&#8217;ve really built an industry specification but then built on that towards an international standard effort is one example that is interesting, right, to point to.<\/p>\n\n\n\n<p>And then, you know, I actually think that bridges to the third thing that we need to do together within this whole community, which is, you know, really think again about how we manage the breadth of this challenge and opportunity of AI by thinking about this horizontal-vertical problem. And, you know, I think that&#8217;s where it&#8217;s not just the sort of tech industry, for example. It&#8217;s broader industry that&#8217;s going to be really applying this technology that needs to get involved in the conversation about not just, sort of, testing AI models, for example, but also testing how AI systems or applications are working in context. And so, yes, so much fun opportunity!&nbsp;<\/p>\n\n\n\n<p>[MUSIC]&nbsp;<\/p>\n\n\n\n<p><strong>SULLIVAN:<\/strong> Amanda, this was just fantastic. You&#8217;ve really set the stage for this podcast. And thank you so much for sharing your time and wisdom with us.\u00a0<\/p>\n\n\n\n<p><strong>CRAIG DECKARD:<\/strong> Thank you.&nbsp;<\/p>\n\n\n\n<p><strong>SULLIVAN:<\/strong> And to our listeners, we&#8217;re so glad you joined us for this conversation. An exciting lineup of episodes are on the way, and we can&#8217;t wait to have you back for the next one.\u00a0<\/p>\n\n\n\n<p>[MUSIC FADES]&nbsp;<\/p>\n\n\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t<button\n\t\t\t\tclass=\"action-trigger glyph-prepend mt-2 mb-0 show-more-show-less-toggle\"\n\t\t\t\taria-expanded=\"false\"\n\t\t\t\tdata-show-less-text=\"Show less\"\n\t\t\t\ttype=\"button\"\n\t\t\t\taria-controls=\"show-more-show-less-toggle-1\"\n\t\t\t\taria-label=\"Show more content\"\n\t\t\t\tdata-alternate-aria-label=\"Show less content\">\n\t\t\t\tShow more\t\t\t<\/button>\n\t\t<\/div>\n\t<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button is-style-outline is-style-outline--2\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/story\/ai-testing-and-evaluation-learnings-from-science-and-industry\/\">AI Testing and Evaluation podcast series<\/a><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>In the introductory episode of this new series, host Kathleen Sullivan and Senior Director Amanda Craig Deckard explore Microsoft\u2019s efforts to draw on the experience of other domains to help advance the role of AI testing and evaluation as a governance tool.<\/p>\n","protected":false},"author":43868,"featured_media":1141311,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/146460793","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Kathleen Sullivan","user_id":"40949"},{"type":"user_nicename","value":"Amanda Craig Deckard","user_id":"43899"}],"msr_hide_image_in_river":null,"footnotes":""},"categories":[240054],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,269142,243990],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[269992],"class_list":["post-1140810","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-include-in-river","msr-post-option-podcast-featured","msr-podcast-series-ai-testing-and-evaluation-learnings-from-science-and-industry"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/146460793","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Kathleen Sullivan","user_id":40949,"display_name":"Kathleen Sullivan","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/kasull\/\" aria-label=\"Visit the profile page for Kathleen Sullivan\">Kathleen Sullivan<\/a>","is_active":false,"last_first":"Sullivan, Kathleen","people_section":0,"alias":"kasull"},{"type":"user_nicename","value":"Amanda Craig Deckard","user_id":43899,"display_name":"Amanda Craig Deckard","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/amcraig\/\" aria-label=\"Visit the profile page for Amanda Craig Deckard\">Amanda Craig Deckard<\/a>","is_active":false,"last_first":"Craig Deckard, Amanda","people_section":0,"alias":"amcraig"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-960x540.jpg\" class=\"img-object-cover\" alt=\"Illustrated headshots of Amanda Craig Deckard &amp; Kathleen Sullivan.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-1024x577.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-768x433.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/06\/EP0-AI-TE_Hero_Feature_River_No_Text_1400x788.jpg 1400w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/kasull\/\" title=\"Go to researcher profile for Kathleen Sullivan\" aria-label=\"Go to researcher profile for Kathleen Sullivan\" data-bi-type=\"byline author\" data-bi-cN=\"Kathleen Sullivan\">Kathleen Sullivan<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/amcraig\/\" title=\"Go to researcher profile for Amanda Craig Deckard\" aria-label=\"Go to researcher profile for Amanda Craig Deckard\" data-bi-type=\"byline author\" data-bi-cN=\"Amanda Craig Deckard\">Amanda Craig Deckard<\/a>","formattedDate":"June 23, 2025","formattedExcerpt":"In the introductory episode of this new series, host Kathleen Sullivan and Senior Director Amanda Craig Deckard explore Microsoft\u2019s efforts to draw on the experience of other domains to help advance the role of AI testing and evaluation as a governance tool.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1140810","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43868"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1140810"}],"version-history":[{"count":70,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1140810\/revisions"}],"predecessor-version":[{"id":1144368,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1140810\/revisions\/1144368"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1141311"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1140810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1140810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1140810"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1140810"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1140810"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1140810"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1140810"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1140810"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1140810"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1140810"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1140810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}