Multiworld Testing

MWT white paper (pdf) – Background for potential users of the Decision Service: machine learning methodology and system design, and how to make them fit YOUR application. Also covers past deployments and experimental evaluation. For a broad technical audience, both in product groups and in research.

Slide deck on MWT and the Decision Service.

MWT Decision Service Wiki: tutorials, guides and references.

Custom Decision Service documentation

Tutorials and lectures:

• Learning to interact (John Langford, NIPS 2013).
• Learning to Interact (Leon Bottou, MSR TechFest 2014).
• Large-Scale Machine Learning (NYU, 2013, John Langford and Yann LeCun, lectures 9-11).

Most relevant papers:

• Basic techniques of MWT with applications to news recommendation (WSDM 2011).
• MWT with estimated action selection probabilities  (NIPS 2010).
• Application of MWT to Bing Speller (WWW’14).

Background reading:

• survey on multi-armed bandits
• book on prediction & learning