Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy
DeepTest (ICSE Workshop) | April 2025
Data&AI Science
Empowering every developer and their team to achieve more!
Publications
Publications by Year
-
Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, Wei Le
2025 International Conference on Software Engineering | April 2025
-
AdaptivePaste: Intelligent Copy-Paste in IDE
Xiaoyu Liu, Jinu Jang, Neel Sundaresan, Miltos Allamanis, Alexey Svyatkovskiy
The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) | December 2023
InferFix: End-to-End Program Repair with LLMs over Retrieval-Augmented Prompts
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, Alexey Svyatkovskiy
The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) | December 2023
-
DeepDev-PERF: A Deep Learning-Based Approach for Improving Software Performance
Spandan Garg, Roshanak Zilouchian Moghaddam, Colin Clement, Neel Sundaresan, Chen Wu
ESEC/FSE | August 2022
Exploring and Evaluating Personalized Models for Code Generation
Andrei Zlotchevski, Dawn Drain, Alexey Svyatkovskiy, Colin Clement, Neel Sundaresan, Michele Tufano
Foundations of Software Engineering | August 2022
Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Neel Sundaresan
2022 IEEE/ACM International Conference on Automation of Software Test (AST) | May 2022
DeepMerge: Learning to Merge Programs
Elizabeth Dinella, Todd Mytkowicz, Alexey Svyatkovskiy, Christian Bird, Mayur Naik, Shuvendu Lahiri
IEEE Transactions on Software Engineering (TSE’22) | May 2022
Learning to Reduce False Positives in Analytic Bug Detectors
Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, Neel Sundaresan
2022 International Conference on Software Engineering | May 2022
Methods2Test: A dataset of focal methods mapped to test cases
Michele Tufano, Shao Kun Deng, Neel Sundaresan, Alexey Svyatkovskiy
2022 International Conference on Mining Software Repositories (MSR) | May 2022
Generating Examples From CLI Usage: Can Transformers Help?
Roshanak Zilouchian Moghaddam, Spandan Garg, Colin Clement, Yevhen Mohylevskyy, Neel Sundaresan
KDD 2022 | April 2022
Learning to Complete Code with Sketches
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, Miltos Allamanis
ICLR 2022 | April 2022
Training and Evaluating a Jupyter Notebook Data Science Assistant
Shubham Chandel, Colin Clement, Guillermo Serrato, Neel Sundaresan
February 2022
-
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy
2021 Empirical Methods in Natural Language Processing | October 2021
MergeBERT: Program Merge Conflict Resolution via Neural Transformers
Alexey Svyatkovskiy, Todd Mytkowicz, Negar Ghorbani, Sarah Fakhoury, Elizabeth Dinella, Christian Bird, Neel Sundaresan, Shuvendu Lahiri
August 2021
Distilling Transformers for Neural Cross-Domain Search
Colin Clement, Chen Wu, Dawn Drain, Neel Sundaresan
August 2021
PerfLens: A Data-Driven Performance Bug Detection and Fix Platform
Spandan Garg, Roshanak Zilouchian Moghaddam, Neel Sundaresan, Chen Wu, Spandan Garg
SOAP ’21 | June 2021
Unit Test Case Generation with Transformers and Focal Context
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan
May 2021
Generating Bug-Fixes Using Pretrained Transformers
Dawn Drain, Chen Wu, Alexey Svyatkovskiy, Neel Sundaresan
PLDI 2021 | April 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang (姜大昕), Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong (YIMING), Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
February 2021
-
PyMT5: multi-mode translation of natural language and Python code with transformers
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, Neel Sundaresan
2020 Empirical Methods in Natural Language Processing | October 2020
DOI PDF Publication Publication Publication Publication Project
GraphCodeBERT: Pre-training Code Representations with Data Flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang (姜大昕), Ming Zhou
September 2020
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19) | September 2020
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
Publications by Research Area
-
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy
DeepTest (ICSE Workshop) | April 2025
Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, Wei Le
2025 International Conference on Software Engineering | April 2025
DeepDev-PERF: A Deep Learning-Based Approach for Improving Software Performance
Spandan Garg, Roshanak Zilouchian Moghaddam, Colin Clement, Neel Sundaresan, Chen Wu
ESEC/FSE | August 2022
Learning to Reduce False Positives in Analytic Bug Detectors
Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, Neel Sundaresan
2022 International Conference on Software Engineering | May 2022
Generating Examples From CLI Usage: Can Transformers Help?
Roshanak Zilouchian Moghaddam, Spandan Garg, Colin Clement, Yevhen Mohylevskyy, Neel Sundaresan
KDD 2022 | April 2022
Learning to Complete Code with Sketches
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, Miltos Allamanis
ICLR 2022 | April 2022
Training and Evaluating a Jupyter Notebook Data Science Assistant
Shubham Chandel, Colin Clement, Guillermo Serrato, Neel Sundaresan
February 2022
MergeBERT: Program Merge Conflict Resolution via Neural Transformers
Alexey Svyatkovskiy, Todd Mytkowicz, Negar Ghorbani, Sarah Fakhoury, Elizabeth Dinella, Christian Bird, Neel Sundaresan, Shuvendu Lahiri
August 2021
Distilling Transformers for Neural Cross-Domain Search
Colin Clement, Chen Wu, Dawn Drain, Neel Sundaresan
August 2021
Unit Test Case Generation with Transformers and Focal Context
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan
May 2021
Generating Bug-Fixes Using Pretrained Transformers
Dawn Drain, Chen Wu, Alexey Svyatkovskiy, Neel Sundaresan
PLDI 2021 | April 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang (姜大昕), Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong (YIMING), Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
February 2021
PyMT5: multi-mode translation of natural language and Python code with transformers
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, Neel Sundaresan
2020 Empirical Methods in Natural Language Processing | October 2020
DOI PDF Publication Publication Publication Publication Project
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19) | September 2020
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy
DeepTest (ICSE Workshop) | April 2025
Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, Wei Le
2025 International Conference on Software Engineering | April 2025
InferFix: End-to-End Program Repair with LLMs over Retrieval-Augmented Prompts
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, Alexey Svyatkovskiy
The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) | December 2023
AdaptivePaste: Intelligent Copy-Paste in IDE
Xiaoyu Liu, Jinu Jang, Neel Sundaresan, Miltos Allamanis, Alexey Svyatkovskiy
The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) | December 2023
DeepDev-PERF: A Deep Learning-Based Approach for Improving Software Performance
Spandan Garg, Roshanak Zilouchian Moghaddam, Colin Clement, Neel Sundaresan, Chen Wu
ESEC/FSE | August 2022
Exploring and Evaluating Personalized Models for Code Generation
Andrei Zlotchevski, Dawn Drain, Alexey Svyatkovskiy, Colin Clement, Neel Sundaresan, Michele Tufano
Foundations of Software Engineering | August 2022
Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Neel Sundaresan
2022 IEEE/ACM International Conference on Automation of Software Test (AST) | May 2022
Methods2Test: A dataset of focal methods mapped to test cases
Michele Tufano, Shao Kun Deng, Neel Sundaresan, Alexey Svyatkovskiy
2022 International Conference on Mining Software Repositories (MSR) | May 2022
Learning to Reduce False Positives in Analytic Bug Detectors
Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, Neel Sundaresan
2022 International Conference on Software Engineering | May 2022
DeepMerge: Learning to Merge Programs
Elizabeth Dinella, Todd Mytkowicz, Alexey Svyatkovskiy, Christian Bird, Mayur Naik, Shuvendu Lahiri
IEEE Transactions on Software Engineering (TSE’22) | May 2022
Generating Examples From CLI Usage: Can Transformers Help?
Roshanak Zilouchian Moghaddam, Spandan Garg, Colin Clement, Yevhen Mohylevskyy, Neel Sundaresan
KDD 2022 | April 2022
Learning to Complete Code with Sketches
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, Miltos Allamanis
ICLR 2022 | April 2022
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy
2021 Empirical Methods in Natural Language Processing | October 2021
MergeBERT: Program Merge Conflict Resolution via Neural Transformers
Alexey Svyatkovskiy, Todd Mytkowicz, Negar Ghorbani, Sarah Fakhoury, Elizabeth Dinella, Christian Bird, Neel Sundaresan, Shuvendu Lahiri
August 2021
PerfLens: A Data-Driven Performance Bug Detection and Fix Platform
Spandan Garg, Roshanak Zilouchian Moghaddam, Neel Sundaresan, Chen Wu, Spandan Garg
SOAP ’21 | June 2021
Unit Test Case Generation with Transformers and Focal Context
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan
May 2021
PyMT5: multi-mode translation of natural language and Python code with transformers
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, Neel Sundaresan
2020 Empirical Methods in Natural Language Processing | October 2020
DOI PDF Publication Publication Publication Publication Project
GraphCodeBERT: Pre-training Code Representations with Data Flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang (姜大昕), Ming Zhou
September 2020
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19) | September 2020
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, Wei Le
2025 International Conference on Software Engineering | April 2025
-
Exploring and Evaluating Personalized Models for Code Generation
Andrei Zlotchevski, Dawn Drain, Alexey Svyatkovskiy, Colin Clement, Neel Sundaresan, Michele Tufano
Foundations of Software Engineering | August 2022
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy
2021 Empirical Methods in Natural Language Processing | October 2021
-
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19) | September 2020
Publications by Type
-
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy
DeepTest (ICSE Workshop) | April 2025
Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, Wei Le
2025 International Conference on Software Engineering | April 2025
InferFix: End-to-End Program Repair with LLMs over Retrieval-Augmented Prompts
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, Alexey Svyatkovskiy
The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) | December 2023
AdaptivePaste: Intelligent Copy-Paste in IDE
Xiaoyu Liu, Jinu Jang, Neel Sundaresan, Miltos Allamanis, Alexey Svyatkovskiy
The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) | December 2023
DeepDev-PERF: A Deep Learning-Based Approach for Improving Software Performance
Spandan Garg, Roshanak Zilouchian Moghaddam, Colin Clement, Neel Sundaresan, Chen Wu
ESEC/FSE | August 2022
Exploring and Evaluating Personalized Models for Code Generation
Andrei Zlotchevski, Dawn Drain, Alexey Svyatkovskiy, Colin Clement, Neel Sundaresan, Michele Tufano
Foundations of Software Engineering | August 2022
Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Neel Sundaresan
2022 IEEE/ACM International Conference on Automation of Software Test (AST) | May 2022
Methods2Test: A dataset of focal methods mapped to test cases
Michele Tufano, Shao Kun Deng, Neel Sundaresan, Alexey Svyatkovskiy
2022 International Conference on Mining Software Repositories (MSR) | May 2022
Learning to Reduce False Positives in Analytic Bug Detectors
Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, Neel Sundaresan
2022 International Conference on Software Engineering | May 2022
Generating Examples From CLI Usage: Can Transformers Help?
Roshanak Zilouchian Moghaddam, Spandan Garg, Colin Clement, Yevhen Mohylevskyy, Neel Sundaresan
KDD 2022 | April 2022
Learning to Complete Code with Sketches
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, Miltos Allamanis
ICLR 2022 | April 2022
Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy
2021 Empirical Methods in Natural Language Processing | October 2021
PerfLens: A Data-Driven Performance Bug Detection and Fix Platform
Spandan Garg, Roshanak Zilouchian Moghaddam, Neel Sundaresan, Chen Wu, Spandan Garg
SOAP ’21 | June 2021
Generating Bug-Fixes Using Pretrained Transformers
Dawn Drain, Chen Wu, Alexey Svyatkovskiy, Neel Sundaresan
PLDI 2021 | April 2021
PyMT5: multi-mode translation of natural language and Python code with transformers
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, Neel Sundaresan
2020 Empirical Methods in Natural Language Processing | October 2020
DOI PDF Publication Publication Publication Publication Project
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19) | September 2020
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
MergeBERT: Program Merge Conflict Resolution via Neural Transformers
Alexey Svyatkovskiy, Todd Mytkowicz, Negar Ghorbani, Sarah Fakhoury, Elizabeth Dinella, Christian Bird, Neel Sundaresan, Shuvendu Lahiri
August 2021
Distilling Transformers for Neural Cross-Domain Search
Colin Clement, Chen Wu, Dawn Drain, Neel Sundaresan
August 2021
-
Training and Evaluating a Jupyter Notebook Data Science Assistant
Shubham Chandel, Colin Clement, Guillermo Serrato, Neel Sundaresan
February 2022
Unit Test Case Generation with Transformers and Focal Context
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan
May 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang (姜大昕), Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong (YIMING), Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
February 2021
GraphCodeBERT: Pre-training Code Representations with Data Flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang (姜大昕), Ming Zhou
September 2020
-
DeepMerge: Learning to Merge Programs
Elizabeth Dinella, Todd Mytkowicz, Alexey Svyatkovskiy, Christian Bird, Mayur Naik, Shuvendu Lahiri
IEEE Transactions on Software Engineering (TSE’22) | May 2022