As Covid-19 vaccine rollouts began in January 2021, it became clear that the phased eligibility approach in use by the United States was causing widespread confusion. The question of eligibility has been a complex topic, with rules and qualifications differing state-by-state – and sometimes differing at county and city levels. In states like Arizona and Massachusetts, we observed eligibility differences from county to county. In Los Angeles and Chicago, eligibility criteria were different between large cities and their surrounding counties. In North Dakota, the vaccine eligibility criteria even varied between pharmacies and vaccinating locations.
Because of the technical complexity in navigating this information space, there is a healthcare equity concern as to the ability of different groups to access vaccination resources. Qualifying individuals in hard-hit communities are underserved compared to the well-connected and technically savvy. This is often due to a gap between the information that exists and the ability of people to access it – on their own terms, in a form they understand, and in their own language.
Our goal with this project was to create a conversational chatbot to streamline the determination of vaccine eligibility in the United States, aggregating policies across regions and policy updates over time. Users would interact with the bot via a simple workflow of Yes or No questions to determine their current eligibility and be directed towards authoritative sources for confirmation. We designed this bot to be accessible across a range of communication channels (e.g., web, SMS, and WhatsApp), with localizations available for communities that may otherwise be disadvantaged due to low levels of English proficiency.
Data management platform
A risk we identified right away was that inaccurate or out-of-date data could become a source of misinformation and confusion. We engaged with partners in the healthcare space – MITRE (opens in new tab) and The Fight Is In Us (opens in new tab) – to mitigate this risk, curating and auditing these criteria as they changed over the course of the pandemic.
One key decision we made in the early stages of this project was to develop it in the open. We wanted this to be an opportunity to build trust – not just with our partners and their stakeholders, but with the open-source developer community and potential hosts of adapted bots around the world. To achieve this goal, we used GitHub as the central repository of all information relating to vaccine eligibility in our target deployment context (i.e., the United States). To support rapid updates and adaptation to other contexts, we developed a simple data schema that could capture a hierarchy of regional policies using the familiar nested filesystem model.
One of our goals with this system was to make it accessible to non-developers. GitHub is traditionally a platform used for software engineering, and the partners using this platform would not necessarily be familiar with version control systems such as git developed for source code. We therefore built a Data Management Portal that uses GitHub’s API to interact with the knowledge repository automatically, on behalf of the party deploying the bot. This approach gave us the ability to build on top of GitHub’s trusted infrastructure and workflows to provide data editing tools for a non-developer audience.
Bot deployment
We deployed the Covid-19 Vaccine Eligibility Bot to thefightisinus.org (opens in new tab) on March 25th, 2021. The Fight Is In Us represents a coalition of coalitions, themselves spanning corporations, NGOs, and community organizations that have been instrumental in the promotion of Covid-19 blood therapeutics – specifically convalescent plasma infusions. With the introduction of the Vaccine Eligibility Bot, along with a complementary bot they had developed for monoclonal antibody infusion, the scope of this coalition grew from information regarding one particular blood therapeutic into a suite of bots for a range of blood therapeutics as well as vaccinations.
The Fight is In Us
One of our other partners in this process was Bing, with the goal of integrating the bot into the Bing experience. As of launch, a link to the Vaccine Eligibility Bot is shown on both the Bing Covid dashboard and the Bing search engine results page.
Vaccine Eligibility on Bing
Bot future
By this time, the Federal government had taken steps to clarify the process of determining eligibility, directing states to make all adults in the United States eligible for the vaccination by May 1st, 2021. This welcome clarity made it clear that the long-term utility of this bot would shift from the eligibility question of “Can I receive the vaccine?” to the access questions of “Where/how/when can I receive the vaccine?”
While the future is uncertain, we see near-term possibilities for the current bot that include international rules and deployments, rules for pediatric eligibility, and helping to clarify eligibility rules and recommendations for Covid-19 booster shots.
Beyond this particular bot, we anticipate that the patterns used here can be adapted for use in future crisis response and community outreach activities. Given our time constraints, we were not able to provide extensive localizations or deploy an SMS endpoint for the initial bot release. Given more lead-time, we could have reached a much broader population of underserved groups and individuals from the outset.
For now, we will continue building these capabilities as a reference implementation for a future class of “dynamic policy” bots – bots that provide a unified entry point to, and streamlined path through, a complex and ever-changing policy landscape. The open-source availability of such configurable and data-driven bots represents a comprehensive approach to policy coordination and communication, especially in times of crisis, addressing critical issues such as eligibility and access with speed, accuracy, and clarity. Most importantly, it shows a small but significant way in which we can promote more equitable distribution of essential resources, services, and support – for vaccines, healthcare, and beyond.
Key lessons and takeaways
We found great value in defining a public system for performing data management over geographically hierarchical policy rules. By leveraging known and proven techniques for managing information systems (e.g., git, pull requests, review cycles), we were able to avoid a large volume of work that would have been involved in developing a bespoke system. Organizations implementing data systems such as these should strongly consider similar approaches that leverage the transparency and workflow of platforms like GitHub.
Had we established the bot architecture and workflows prior to the outbreak of the Coivd-19 pandemic, the entire suite of bots described in this case study would have been available much sooner, with greater potential to help those in need. To that end, we recommend that governments, NGOs, and the technology community develop their policy coordination and communication tools before the next crisis occurs – and consider bots as part of the solution. We believe that open-source data management systems, with open data, interfaces, and review processes, will play an important role in managing both the tail of the Covid-19 pandemic as well as future crises, whatever and whenever they may be.
Finally, we want to express our gratitude to our partner teams at MITRE, Bing, and GeneralUI in helping to make this bot a reality.
Resilience principles in action
- Tackle the causes and consequences of systemic vulnerability
- The Covid-19 pandemic has exacerbated vulnerabilities related to the social determinants of health. Some of the groups at greatest risk from the virus are also the hardest to reach, with complex and evolving eligibility requirements representing a barrier that perpetuates healthcare inequity.
- Convene and participate in broad coalitions with a bias for action
- We are working with an external coalition convened by MITRE (opens in new tab), the coalition of coalitions represented by The Fight Is In Us (opens in new tab), and government, in addition to our internal partnership with Bing.
- Contribute a “resilience toolbox” of open tools and technologies
- Our Covid-19 eligibility bot, and reference implementation for similar bots, is available open-source on GitHub at microsoft/covid-vaccine-bot (opens in new tab) and deployed live at thefightisinus.org (opens in new tab).
- Understand the links between people, practices, and outcomes
- We have observed the practice of decentralized policy communication creating adverse outcomes for groups who are unable to access, interpret, and act on the policy information that is relevant to them, i.e., to determine eligibility and schedule and receive a complete course of vaccination.
- Democratize expert workflows and real-world evidence development
- Covid-19 is everyone’s concern, but there are not enough experts to answer everyone’s questions directly. Bots help to scale experts by absorbing the overhead of policy implementation. In many cases, the real-world actions taken by the users of bots contribute to real-world sources of evidence.
- Build trust through transparency and value-sensitive design
- We are conservative about the information captured by the bot and communicate the rationale in a clear and concise privacy statement at the initiation of each conversation. The sequence of questions aims to minimize the number of simple Yes/No answers to an eligibility determination.
- Design for transformation at the mesoscale of activity
- Our goal is not just to create a Vaccine Eligibility Bot for Covid-19 in the US, but to create a framework for use at other vaccine stages, in other counties, and for other pandemics and large-scale emergencies in general. We want to help bots become a standard part of the toolbox with which organizations coordinate and communicate information to the communities that they serve.