Causality and Machine Learning

At Microsoft Research, our causality research spans a broad array of topics, including: using causal insights to improve machine learning methods; adapting and scaling causal methods to leverage large-scale and high-dimensional datasets; and applying all these methods for data-driven decision making in real-world contexts.

See our publications for examples of our work.

Learning causal effects from data: Identifying causal effects is an integral part of scientific inquiry, spanning a wide range of questions such as understanding behavior in online systems, effects of social policies, or risk factors for diseases.  As computing increasingly impact all walks of life, questions of cause-and-effect are also critical for the design and data-driven evaluation of all the computer systems and applications we build.  For instance, how do algorithmic recommendations affect our purchasing decisions?  How do they affect a student’s learning outcome or a doctor’s efficacy? These are hard questions and require thinking about the counterfactual: what would have happened in a world with a different system, policy, or intervention?  Without randomized experiments and causal reasoning, correlation-based methods can lead us astray.

Causal machine learning: Machine learning based on correlational pattern recognition is insufficient for robust predictions and reliable decision-making. New approaches to machine learning based on principles of causal reasoning provide a promising path forward. Guided by joint formal reasoning over observations and auxiliary information about data collection procedures or other domain knowledge, causal machine learning methods are grounded in the stable and independent mechanisms that govern the behavior of a system being modelled. As a result, these methods promise robustness to exogenous changes and accurate modelling of counterfactual or “what-if” scenarios that are core to scientific experimentation, understanding, and decision-making.

Open challenges: At Microsoft Research, we are working on fundamental advances that combine traditional machine learning with causal inference methods. The practice of machine learning is heavily based on the ability to measure the performance of a model on a validation sample. Consistent with real-world decision-making, however, the fundamental problem of causal inference precludes the existence of a perfect analogue of out-of-sample performance for causal models, since counterfactual quantities are never observed. This opens a host of critical research challenges on evaluation of causal machine learning models and how to best formalize and integrate domain expertise into machine learning pipelines. Efforts in this direction can enhance out-of-distribution generalizability, robustness, and interpretability of machine learning models. At the same time, machine learning can help scale causal effect estimation methods to high-dimensional data and unstructured data as in text and images. As has been done for predictive systems, there is an opportunity to build accurate decision-support systems that estimate effect of interventions under real-world constraints, such as messy observed data, imperfect causal knowledge, and computational and latency constraints.

Causal tooling, libraries, and education: Complementing our core research and with the goal of broadening the use of causal methods across academia and industry, we strive to make our technologies accessible through open source tooling and libraries, such as DoWhy (opens in new tab), EconML (opens in new tab), and Azua (opens in new tab), and frequently present tutorials (opens in new tab) and seminars (opens in new tab) on new methods.