Description<\/strong>: <\/p>\n\n\n\nA small but growing body of social scientific research has explored the many challenges of doing responsible AI in practice. Using empirical methods like ethnography, site observations, interviews, and more, this work has found that there is often a significant gap between the aspirational goals of responsible AI principles and what is currently being achieved by the frontline workers charged with realizing these principles in practice.<\/p>\n\n\n\n
These studies have identified a range of difficulties that offer important lessons for anyone seeking to develop an effective regulatory regime for AI.<\/p>\n\n\n\n
First, practitioners often confront technical limitations when seeking to assess AI systems and address possible problems. The underlying science of AI evaluation remains immature in many cases. The difficulty of reliably assessing the capabilities of frontier models provides a clear example of this problem: current methods are far from systematic and provide rather spotty coverage. As a result, those doing the work to ensure responsible development and use of such models often butt up against the limits of the current scientific understanding of the technology.<\/p>\n\n\n\n
Second, practitioners also face practical constraints that limit how well they can do their jobs. For example, developing an evaluation dataset to perform a rigorous assessment of the fairness of an AI system can be a significant and difficult undertaking, especially when it requires collecting sensitive information about demographic attributes like gender, race, etc. Similarly, adapting general purpose methods from the academic literature for practical use in evaluating specific products or services is rarely straightforward or easy. The same often holds for fairness toolkits, even those purported to be \u201cgeneral purpose.\u201d<\/p>\n\n\n\n
Third, organizational dynamics also complicate practitioners work on responsible AI. Such work often requires collaboration across teams with quite different skills and expertise, leading to cross-functional frictions. Practitioners also sometimes lack the necessary institutional support to effectively execute their duties in practice. And incentive structures don’t always align with the goals of responsible AI policies, creating difficult tensions for practitioners. This is not to even mention the challenge of keeping pace with rapid technological developments and the push to ship products and services.<\/p>\n\n\n\n
The coming wave of AI regulation needs to grapple with these challenges if it is to have its intended effects in practice. Many of the regulatory proposals currently under discussion, especially those that include requirements for evaluation (e.g., audits, impact assessments, etc.), often take for granted that actors will be able to figure out how to put in place appropriate processes, procedures, and tools to fulfill these requirements\u2014and will have the necessary institutional support to do so. The research has so far suggested this is not often the case.<\/p>\n\n\n\n
These difficulties are sometimes invoked as reasons to resist AI regulations. In this challenge, we instead call on the broader community, especially those working in law and policy and the social sciences, to collaborate with Microsoft Research to figure out how the law can be made more responsive to the known challenges of doing responsible AI in practice and those yet to be discovered. Legislation and regulation should reflect what is technically possible at the moment, what can be made possible with sufficient investment, and what is likely to remain infeasible. They should also explicitly target the practical impediments and organizational dynamics that impede efforts at responsible AI.<\/p>\n\n\n\n
To that end, this challenge seeks to (1) promote original legal scholarship that is more deeply informed by social scientific insights on the challenges of doing responsible AI in practice, (2) engage in further social scientific research on responsible AI, with a particular focus on how regulated entities are seeking to comply with existing or forthcoming laws, and (3) bring these insights into the policymaking process via direct engagement with government and civil society stakeholders.<\/p>\n\n\n\n
Ideal candidate<\/strong><\/p>\n\n\n\n