Grounded Reasoning and Interactive Learning (GRAIL)

Complex tasks support via interactive program synthesis

The GRAIL (Grounded Reasoning and Interactive Learning) team aims to empower people to achieve more by bringing together the advances our team has made on NL→Code neural program synthesis technologies and interactive machine teaching. Our aim is to build new classes of system that enable people to collaborate with systems to complete complex tasks. The GRAIL team breaks disciplinary boundaries with the three founding members with human-computer interaction (HCI), programming language (PL), and machine learning (ML) backgrounds. While the group participates in a variety of research activities, two primary research foci which are essential to making interactive program synthesis a reality are:

Neural program synthesis

We have built state-of-the-art NL→Code systems for database question answering (NL→SQL) that enable the translation of a natural language question to a SQL program that answers the question. Our system, RAT-SQL is the current top-1 on the Spider dataset leaderboard (opens in new tab). Our approach leverages a relation-aware self-attention mechanism to encode the database schema.

The relation-aware self-attention mechanism used in the RAT-SQL project is just one way in which we are trying to create models of structured context. In the DeepExcel project (a collaboration with MSR RiSE group and Office), for example, we are working on building multi-modal models of Excel spreadsheets to support complex tasks such as formula bug detection, table detection, and chart recommendation. As spreadsheets combine data, code, and presentation, building useful representations of their content requires combining insights in multi-modal neural representation learning and program analysis. Advances in structured representation learning and similar advances will be important to fully realize the potential of interactive program synthesis.

Interactive machine teaching

Interactive machine teaching (IMT) is a human-centered approach to building machine learning models (i.e., prediction programs). Through building integrated teaching environments in collaboration with the LUIS team, we have explored both the benefits and challenges of building IMT systems. We have identified design opportunities, machine-in-the-loop interventions and interactive techniques, that are grounded in and leverage rich human capabilities and guide the design and development of effective and efficient IMT experiences and systems. Microsoft has established itself as an innovator through its efforts in bringing machine teaching approaches to our products and we have worked to establish this leadership in the academic community.

In recent work we have built interactive machine teaching systems for image segmentation taking IMT beyond the text domain that has been the focus of the LUIS team.

Interactive program synthesis

The GRAIL team has seized the opportunity to bring these two lines of work together to create new classes of applications that enable people to collaborate with systems to complete complex tasks. Our view is that central to enabling these types of collaborative systems is interactive program synthesis. We are exploring people’s ability to communicate specifications that, while not logically complete, are adequate to communicate intent. That is, enabling program synthesis to reach to scenarios beyond example driven program synthesis of Excel’s FlashFill. We are exploring opportunities of leveraging multi-modal (including NL) and knowledge-rich specifications of how to do tasks, enabling both refinement, evolution and task decomposition. In recent work, for example, we demonstrated the potential for combining natural language explanations and demonstrations to generate web-based procedures. We are currently targeting scenarios around “conversations with data” through a broad collaboration with other groups in MSR AI and working to bringing our Neural-Program-Synthesis technology to Calendar.Help to turn textual scheduling request into programmatic constraints.