A/B Integrations: 7 Lessons Learned from Enabling A/B testing as a Product Feature
Aleksander Fabijan, Benjamin Arai, Sebastian Kohlmeier, April Kwong, Andy Drake, Pavel Dmitriev
2023 International Conference on Software Engineering | May 2023
Experimentation Platform
Publications
Publications by Year
-
-
It takes a Flywheel to Fly: Kickstarting and Growing the A/B testing Momentum at Scale
Aleksander Fabijan, Benjamin Arai, Pavel Dmitriev, Lukas Vermeer
2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) | September 2021
On Post-selection Inference in A/B Testing
Alex Deng, Yicheng Li, Jiannan Lu, Vivek Ramamurthy
ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2021
-
Automated metrics calculation in a dynamic heterogeneous environment
December 2019
This is an extended abstract presented by the authors at MIT CODE conference on Nov 1, 2019.
On Heavy-user Bias in A/B Testing
Yu Wang, Somit Gupta, Jiannan Lu, Ali Mahmoudzadeh, Sophia Liu
CIKM | November 2019
Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners
ACM SIGKDD International Conference on Knowledge Discovery & Data Mining | July 2019
Top Challenges from the first Practical Online Controlled Experiments Summit
SIGKDD Explorations June 2019: Volume 21, Issue 1 | June 2019
Safe Velocity: A Practical Guide to Software Deployment at Scale using Controlled Rollout
41st International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP ’19), | May 2019
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
41st International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP ’19) | May 2019
-
Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas
Alex Deng, Ulf Knoblich, Jiannan Lu
SIGKDD International Conference on Knowledge Discovery & Data Mining | July 2018
The Anatomy of a Large-Scale Experimentation Platform
Somit Gupta, Liudmila Ulanova, Sumit Bhardwaj, Pavel Dmitriev, Paul Raff, Aleksander Fabijan
2018 IEEE International Conference on Software Architecture (ICSA) | April 2018
A note on Type S/M errors in hypothesis testing
Jiannan Lu, Yixuan Qiu, Alex Deng
British Journal of Mathematical and Statistical Psychology | March 2018, Vol 72(1): pp. 1-17
-
Beyond Success Rate: Utility as a Search Quality Metric for Online Experiments
Widad Machmouchi, Ahmed Awadallah, Imed Zitouni, Georg Buscher
2017 Conference on Information and Knowledge Management | November 2017
The Benefits of Controlled Experimentation at Scale
43th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) | August 2017
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments
Pavel Dmitriev, Somit Gupta, Dong Woo Kim, Garnet Vaz
KDD | August 2017
The Evolution of Continuous Experimentation in Software Product Development
39th International Conference on Software Engineering (ICSE ’17) | May 2017
Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions
Alex Dneg, Jiannan Lu, Jonathan Litz
ACM International Conference on Web Search and Data Mining | February 2017
-
Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing
Alex Deng, Jiannan Lu, Shouyuan Chen
IEEE International Conference on Data Science and Advanced Analytics (DSAA) | December 2016
Pitfalls of Long-Term Online Controlled Experiments
Pavel Dmitriev, Brian Frasca, Somit Gupta, Ron Kohavi, Garnet Vaz
IEEE International Conference on Big Data | December 2016
Principles for the Design of Online A/B Metrics
Widad Machmouchi, Georg Buscher
SIGIR ’16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval | July 2016
-
Tracking Users’ Clicks and Submits: Tradeoffs between User Experience and Data Loss
Ron Kohavi, David Messner, Seth Eliot, Juan M. Lavista Ferres, Randy Henne, Vignesh Kannappan, Justin Wang
November 2010
-
Online Experimentation at Microsoft
Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres, Tamir Melamed, Juan M. Lavista Ferres
July 2009
Publications by Research Area
-
A/B Integrations: 7 Lessons Learned from Enabling A/B testing as a Product Feature
Aleksander Fabijan, Benjamin Arai, Sebastian Kohlmeier, April Kwong, Andy Drake, Pavel Dmitriev
2023 International Conference on Software Engineering | May 2023
It takes a Flywheel to Fly: Kickstarting and Growing the A/B testing Momentum at Scale
Aleksander Fabijan, Benjamin Arai, Pavel Dmitriev, Lukas Vermeer
2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) | September 2021
On Post-selection Inference in A/B Testing
Alex Deng, Yicheng Li, Jiannan Lu, Vivek Ramamurthy
ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2021
Automated metrics calculation in a dynamic heterogeneous environment
December 2019
This is an extended abstract presented by the authors at MIT CODE conference on Nov 1, 2019.
On Heavy-user Bias in A/B Testing
Yu Wang, Somit Gupta, Jiannan Lu, Ali Mahmoudzadeh, Sophia Liu
CIKM | November 2019
Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners
ACM SIGKDD International Conference on Knowledge Discovery & Data Mining | July 2019
Top Challenges from the first Practical Online Controlled Experiments Summit
SIGKDD Explorations June 2019: Volume 21, Issue 1 | June 2019
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
41st International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP ’19) | May 2019
Safe Velocity: A Practical Guide to Software Deployment at Scale using Controlled Rollout
41st International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP ’19), | May 2019
Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas
Alex Deng, Ulf Knoblich, Jiannan Lu
SIGKDD International Conference on Knowledge Discovery & Data Mining | July 2018
The Anatomy of a Large-Scale Experimentation Platform
Somit Gupta, Liudmila Ulanova, Sumit Bhardwaj, Pavel Dmitriev, Paul Raff, Aleksander Fabijan
2018 IEEE International Conference on Software Architecture (ICSA) | April 2018
A note on Type S/M errors in hypothesis testing
Jiannan Lu, Yixuan Qiu, Alex Deng
British Journal of Mathematical and Statistical Psychology | March 2018, Vol 72(1): pp. 1-17
The Benefits of Controlled Experimentation at Scale
43th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) | August 2017
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments
Pavel Dmitriev, Somit Gupta, Dong Woo Kim, Garnet Vaz
KDD | August 2017
The Evolution of Continuous Experimentation in Software Product Development
39th International Conference on Software Engineering (ICSE ’17) | May 2017
Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions
Alex Dneg, Jiannan Lu, Jonathan Litz
ACM International Conference on Web Search and Data Mining | February 2017
Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing
Alex Deng, Jiannan Lu, Shouyuan Chen
IEEE International Conference on Data Science and Advanced Analytics (DSAA) | December 2016
Pitfalls of Long-Term Online Controlled Experiments
Pavel Dmitriev, Brian Frasca, Somit Gupta, Ron Kohavi, Garnet Vaz
IEEE International Conference on Big Data | December 2016
Principles for the Design of Online A/B Metrics
Widad Machmouchi, Georg Buscher
SIGIR ’16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval | July 2016
-
Tracking Users’ Clicks and Submits: Tradeoffs between User Experience and Data Loss
Ron Kohavi, David Messner, Seth Eliot, Juan M. Lavista Ferres, Randy Henne, Vignesh Kannappan, Justin Wang
November 2010
Online Experimentation at Microsoft
Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres, Tamir Melamed, Juan M. Lavista Ferres
July 2009
-
On Heavy-user Bias in A/B Testing
Yu Wang, Somit Gupta, Jiannan Lu, Ali Mahmoudzadeh, Sophia Liu
CIKM | November 2019
The Anatomy of a Large-Scale Experimentation Platform
Somit Gupta, Liudmila Ulanova, Sumit Bhardwaj, Pavel Dmitriev, Paul Raff, Aleksander Fabijan
2018 IEEE International Conference on Software Architecture (ICSA) | April 2018
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments
Pavel Dmitriev, Somit Gupta, Dong Woo Kim, Garnet Vaz
KDD | August 2017
Pitfalls of Long-Term Online Controlled Experiments
Pavel Dmitriev, Brian Frasca, Somit Gupta, Ron Kohavi, Garnet Vaz
IEEE International Conference on Big Data | December 2016
-
Beyond Success Rate: Utility as a Search Quality Metric for Online Experiments
Widad Machmouchi, Ahmed Awadallah, Imed Zitouni, Georg Buscher
2017 Conference on Information and Knowledge Management | November 2017
Principles for the Design of Online A/B Metrics
Widad Machmouchi, Georg Buscher
SIGIR ’16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval | July 2016
Publications by Type
-
A/B Integrations: 7 Lessons Learned from Enabling A/B testing as a Product Feature
Aleksander Fabijan, Benjamin Arai, Sebastian Kohlmeier, April Kwong, Andy Drake, Pavel Dmitriev
2023 International Conference on Software Engineering | May 2023
It takes a Flywheel to Fly: Kickstarting and Growing the A/B testing Momentum at Scale
Aleksander Fabijan, Benjamin Arai, Pavel Dmitriev, Lukas Vermeer
2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) | September 2021
On Post-selection Inference in A/B Testing
Alex Deng, Yicheng Li, Jiannan Lu, Vivek Ramamurthy
ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2021
On Heavy-user Bias in A/B Testing
Yu Wang, Somit Gupta, Jiannan Lu, Ali Mahmoudzadeh, Sophia Liu
CIKM | November 2019
Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners
ACM SIGKDD International Conference on Knowledge Discovery & Data Mining | July 2019
Top Challenges from the first Practical Online Controlled Experiments Summit
SIGKDD Explorations June 2019: Volume 21, Issue 1 | June 2019
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
41st International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP ’19) | May 2019
Safe Velocity: A Practical Guide to Software Deployment at Scale using Controlled Rollout
41st International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP ’19), | May 2019
Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas
Alex Deng, Ulf Knoblich, Jiannan Lu
SIGKDD International Conference on Knowledge Discovery & Data Mining | July 2018
The Anatomy of a Large-Scale Experimentation Platform
Somit Gupta, Liudmila Ulanova, Sumit Bhardwaj, Pavel Dmitriev, Paul Raff, Aleksander Fabijan
2018 IEEE International Conference on Software Architecture (ICSA) | April 2018
Beyond Success Rate: Utility as a Search Quality Metric for Online Experiments
Widad Machmouchi, Ahmed Awadallah, Imed Zitouni, Georg Buscher
2017 Conference on Information and Knowledge Management | November 2017
The Benefits of Controlled Experimentation at Scale
43th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) | August 2017
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments
Pavel Dmitriev, Somit Gupta, Dong Woo Kim, Garnet Vaz
KDD | August 2017
The Evolution of Continuous Experimentation in Software Product Development
39th International Conference on Software Engineering (ICSE ’17) | May 2017
Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions
Alex Dneg, Jiannan Lu, Jonathan Litz
ACM International Conference on Web Search and Data Mining | February 2017
Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing
Alex Deng, Jiannan Lu, Shouyuan Chen
IEEE International Conference on Data Science and Advanced Analytics (DSAA) | December 2016
Pitfalls of Long-Term Online Controlled Experiments
Pavel Dmitriev, Brian Frasca, Somit Gupta, Ron Kohavi, Garnet Vaz
IEEE International Conference on Big Data | December 2016
Principles for the Design of Online A/B Metrics
Widad Machmouchi, Georg Buscher
SIGIR ’16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval | July 2016
-
Automated metrics calculation in a dynamic heterogeneous environment
December 2019
This is an extended abstract presented by the authors at MIT CODE conference on Nov 1, 2019.
Tracking Users’ Clicks and Submits: Tradeoffs between User Experience and Data Loss
Ron Kohavi, David Messner, Seth Eliot, Juan M. Lavista Ferres, Randy Henne, Vignesh Kannappan, Justin Wang
November 2010
Online Experimentation at Microsoft
Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres, Tamir Melamed, Juan M. Lavista Ferres
July 2009
-
A note on Type S/M errors in hypothesis testing
Jiannan Lu, Yixuan Qiu, Alex Deng
British Journal of Mathematical and Statistical Psychology | March 2018, Vol 72(1): pp. 1-17