Microsoft at ICML 2021

Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation

Sam Devlin, Raluca Stevenson, Ida Momennejad, Jaroslaw Rzepecki, Evelyn Zuniga, Gavin Costello, Guy Leroy, Ali Shaw, Katja Hofmann

Dataset Dynamics via Gradient Flows in Probability Space

David Alvarez-Melis, Nicolo Fusi

Exploiting structured data for learning contagious diseases under incomplete testing

Maggie Makar, Lauren R West, David C Hooper, Eric Horvitz, Erica Shenoy, John Guttag

Evolving Attention with Residual Convolutions

Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, Yunhai Tong

Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference

Shumao Zhang, Pengchuan Zhang, Thomas Hou

Acceleration via Fractal Learning Rate Schedules

Naman Agarwal, Surbhi Goel, Cyril Zhang

Boosting the Throughput and Accelerator Utilization of Specialized CNN Inference Beyond Increasing Batch Size

Jack Kosaian, Amar Phanishayee, Matthai Philipose, Debadeepta Dey, Rashmi Vinayak

Interactive Learning from Activity Description

Khanh Nguyen, Dipendra Misra, Robert E. Schapire, Miro Dudík, Patrick Shafto

Dataset Dynamics via Gradient Flows in Probability Space

David Alvarez-Melis, Nicolo Fusi

Evolving Attention with Residual Convolutions

Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, Yunhai Tong

Exploiting structured data for learning contagious diseases under incomplete testing

Maggie Makar, Lauren R West, David C Hooper, Eric Horvitz, Erica Shenoy, John Guttag

Interactive Learning from Activity Description

Khanh Nguyen, Dipendra Misra, Robert E. Schapire, Miro Dudík, Patrick Shafto

Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference

Shumao Zhang, Pengchuan Zhang, Thomas Hou

Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation

Sam Devlin, Raluca Stevenson, Ida Momennejad, Jaroslaw Rzepecki, Evelyn Zuniga, Gavin Costello, Guy Leroy, Ali Shaw, Katja Hofmann

Boosting the Throughput and Accelerator Utilization of Specialized CNN Inference Beyond Increasing Batch Size

Jack Kosaian, Amar Phanishayee, Matthai Philipose, Debadeepta Dey, Rashmi Vinayak

Acceleration via Fractal Learning Rate Schedules

Naman Agarwal, Surbhi Goel, Cyril Zhang

Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework

Wenxiao Wang, Minghao Chen, Shuai Zhao, Long Chen, Jinming Hu, Hai Liu, Deng Cai, Xiaofei He, Wei Liu

Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks

Sungryull Sohn, Sungtae Lee, Jongwook Choi, Harm van Seijen, Honglak Lee, Mehdi Fatemi

Poolingformer: Long Document Modeling with Pooling Attention

Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, Wei Chen

Network Inference and Influence Maximization from Samples

Wei Chen, Xiaoming Sun, Jialin Zhang, Zhijie Zhang

Dueling Convex Optimization

Aadirupa Saha, Tomer Koren, Yishay Mansour

Temporally Correlated Task Scheduling for Sequence Learning

Xueqing Wu, Lewen Wang, Yingce Xia, Weiqing Liu, Lijun Wu, Shufang Xie, Tao Qin, Tie-Yan Liu

Optimal Regret Algorithm for Pseudo-1d Bandit Convex Optimization

Aadirupa Saha, Nagarajan Natarajan, Praneeth Netrapalli, Prateek Jain

GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-Yan Liu, Liwei Wang

1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed

Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He

Neural Pharmacodynamic State Space Modeling

Zeshan Hussain, Rahul Gopal Krishnan, David Sontag

GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-Yan Liu, Liwei Wang

Network Inference and Influence Maximization from Samples

Wei Chen, Xiaoming Sun, Jialin Zhang, Zhijie Zhang

Neural Pharmacodynamic State Space Modeling

Zeshan Hussain, Rahul Gopal Krishnan, David Sontag

Optimal Regret Algorithm for Pseudo-1d Bandit Convex Optimization

Aadirupa Saha, Nagarajan Natarajan, Praneeth Netrapalli, Prateek Jain

Poolingformer: Long Document Modeling with Pooling Attention

Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, Wei Chen

Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks

Sungryull Sohn, Sungtae Lee, Jongwook Choi, Harm van Seijen, Honglak Lee, Mehdi Fatemi

Temporally Correlated Task Scheduling for Sequence Learning

Xueqing Wu, Lewen Wang, Yingce Xia, Weiqing Liu, Lijun Wu, Shufang Xie, Tao Qin, Tie-Yan Liu

1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed

Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He

Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework

Wenxiao Wang, Minghao Chen, Shuai Zhao, Long Chen, Jinming Hu, Hai Liu, Deng Cai, Xiaofei He, Wei Liu

SiameseXML: Siamese Networks meet Extreme Classifiers with 100M Labels

Kunal Dahiya, Ananye Agarwal, Deepak Saini, Gururaj K, Jian Jiao, Amit Singh, Sumeet Agarwal, Purushottam Kar, Manik Varma

How could Neural Networks understand Programs

Dinglan Peng, Shuxin Zheng, Yatao Li, Guolin Ke, Di He, Tie-Yan Liu

Joint Online Learning and Decision-making via Dual Mirror Descent

Alfonso Lobos, Paul Grigas, Zheng Wen

ChaCha for Online AutoML

Qingyun Wu, Chi Wang, John Langford, Paul Mineiro, Marco Rossi

How could Neural Networks understand Programs

Dinglan Peng, Shuxin Zheng, Yatao Li, Guolin Ke, Di He, Tie-Yan Liu

Incentivizing Compliance with Algorithmic Instruments

Dung Ngo, Logan Stapleton, Vasilis Syrgkanis, Steven Wu

SiameseXML: Siamese Networks meet Extreme Classifiers with 100M Labels

Kunal Dahiya, Ananye Agarwal, Deepak Saini, Gururaj K, Jian Jiao, Amit Singh, Sumeet Agarwal, Purushottam Kar, Manik Varma

TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL

Clément Romac, Rémy Portelas, Katja Hofmann, Pierre-Yves Oudeyer

Joint Online Learning and Decision-making via Dual Mirror Descent

Alfonso Lobos, Paul Grigas, Zheng Wen

ChaCha for Online AutoML

Qingyun Wu, Chi Wang, John Langford, Paul Mineiro, Marco Rossi

The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks

Bohan Wang, Qi Meng, Wei Chen, Tie-Yan Liu

Confidence-Budget Matching for Sequential Budgeted Learning

Yonathan Efroni, Nadav Merlis, Aadirupa Saha, Shie Mannor

Adversarial Dueling Bandits

Aadirupa Saha, Tomer Koren, Yishay Mansour

TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL

Clément Romac, Rémy Portelas, Katja Hofmann, Pierre-Yves Oudeyer

Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks

Greg Yang, Edward J. Hu

Interaction-Grounded Learning

Tengyang Xie, John Langford, Paul Mineiro, Ida Momennejad

Tensor Programs IIb: Architectural Universality Of Neural Tangent Kernel Training Dynamics

Greg Yang, Etai Littwin

Incentivizing Compliance with Algorithmic Instruments

Dung Ngo, Logan Stapleton, Vasilis Syrgkanis, Steven Wu

Multi-layered Network Exploration via Random Walks: From Offline Optimization to Online Learning

Xutong Liu, Jinhang Zuo, Xiaowei Chen, Wei Chen, John C. S. Lui

Provably Correct Optimization and Exploration with Non-linear Policies

Fei Feng, Wotao Yin, Alekh Agarwal, Lin Yang

Safe Reinforcement Learning Using Advantage-Based Intervention

Nolan Wagener, Byron Boots, Ching-An Cheng

Multi-layered Network Exploration via Random Walks: From Offline Optimization to Online Learning

Xutong Liu, Jinhang Zuo, Xiaowei Chen, Wei Chen, John C. S. Lui

Provably Correct Optimization and Exploration with Non-linear Policies

Fei Feng, Wotao Yin, Alekh Agarwal, Lin Yang

Safe Reinforcement Learning Using Advantage-Based Intervention

Nolan Wagener, Byron Boots, Ching-An Cheng

The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks

Bohan Wang, Qi Meng, Wei Chen, Tie-Yan Liu

Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks

Greg Yang, Edward J. Hu

Interaction-Grounded Learning

Tengyang Xie, John Langford, Paul Mineiro, Ida Momennejad

Tensor Programs IIb: Architectural Universality Of Neural Tangent Kernel Training Dynamics

Greg Yang, Etai Littwin

Memory-Efficient Pipeline-Parallel DNN Training

Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia

Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition

Shengyang Sun, Jiaxin Shi, Andrew Wilson, Roger Grosse

Off-Policy Confidence Sequences

Nikos Karampatziakis, Paul Mineiro, Aaditya Ramdas

Quantum algorithms for reinforcement learning with a generative model

Daochen Wang, Aarthi Sundaram, Robin Kothari, Ashish Kapoor, Martin Roetteler

Accuracy, Interpretability, and Differential Privacy via Explainable Boosting

Harsha Nori, Rich Caruana, Zhiqi Bu, Judy Hanwen Shen, Janardhan (Jana) Kulkarni

Domain Generalization using Causal Matching

Divyat Mahajan, Shruti Tople, Amit Sharma

EL-Attention: Memory Efficient Lossless Attention for Generation

Yu Yan, Jiusheng Chen, Weizhen Qi, Nikhil Bhendawade, Yeyun Gong, Nan Duan, Ruofei Zhang

Accuracy, Interpretability, and Differential Privacy via Explainable Boosting

Harsha Nori, Rich Caruana, Zhiqi Bu, Judy Hanwen Shen, Janardhan (Jana) Kulkarni

Differentially Private Correlation Clustering

Mark Bun, Marek Eliás, Janardhan (Jana) Kulkarni

Domain Generalization using Causal Matching

Divyat Mahajan, Shruti Tople, Amit Sharma

EL-Attention: Memory Efficient Lossless Attention for Generation

Yu Yan, Jiusheng Chen, Weizhen Qi, Nikhil Bhendawade, Yeyun Gong, Nan Duan, Ruofei Zhang

Memory-Efficient Pipeline-Parallel DNN Training

Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia

Off-Policy Confidence Sequences

Nikos Karampatziakis, Paul Mineiro, Aaditya Ramdas

Quantum algorithms for reinforcement learning with a generative model

Daochen Wang, Aarthi Sundaram, Robin Kothari, Ashish Kapoor, Martin Roetteler

Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition

Shengyang Sun, Jiaxin Shi, Andrew Wilson, Roger Grosse

Statistical Estimation from Dependent Data

Vardis Kandiros, Yuval Dagan, Nishanth Dikkala, Surbhi Goel, Constantinos Daskalakis

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang

Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning

Luisa Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, Shimon Whiteson

Decomposed Mutual Information Estimation for Contrastive Representation Learning

Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman, Remi Tachet des Combes

Grey-box Extraction of Natural Language Models

Santiago Zanella-Béguelin, Shruti Tople, Andrew Paverd, Boris Köpf

WILDS: A Benchmark of in-the-Wild Distribution Shifts

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, Percy Liang

Large Scale Private Learning via Low-rank Reparametrization

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, Tie-Yan Liu

Integrated Defense for Resilient Graph Matching

Jiaxiang Ren, Zijie Zhang, Jiayin Jin, Xin Zhao, Sixing Wu, Yang Zhou, Yelong Shen, Tianshi Che, Ruoming Jin, Dejing Dou, Yelong Shen

Differentially Private Correlation Clustering

Mark Bun, Marek Eliás, Janardhan (Jana) Kulkarni

Graph Neural Networks Inspired by Classical Iterative Algorithms

Yongyi Yang, Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, David Wipf

Correlation Clustering in Constant Many Parallel Rounds

Vincent Cohen-Addad, Silvio Lattanzi, Slobodan Mitrović, Ashkan Norouzi-Fard, Nikos Parotsidis, Jakub Tarnawski

Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons

Bohang Zhang, Tianle Cai, Zhou Lu, Di He, Liwei Wang

Finding k in Latent k-polytope

Chiranjib Bhattacharyya, Ravi Kannan, Amit Kumar

On the Problem of Underranking in Group-Fair Ranking

Sruthi Gorantla, Amit Deshpande, Anand Louis

Correlation Clustering in Constant Many Parallel Rounds

Vincent Cohen-Addad, Silvio Lattanzi, Slobodan Mitrović, Ashkan Norouzi-Fard, Nikos Parotsidis, Jakub Tarnawski

Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning

Luisa Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, Shimon Whiteson

Finding k in Latent k-polytope

Chiranjib Bhattacharyya, Ravi Kannan, Amit Kumar

Graph Neural Networks Inspired by Classical Iterative Algorithms

Yongyi Yang, Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, David Wipf

Grey-box Extraction of Natural Language Models

Santiago Zanella-Béguelin, Shruti Tople, Andrew Paverd, Boris Köpf

Integrated Defense for Resilient Graph Matching

Jiaxiang Ren, Zijie Zhang, Jiayin Jin, Xin Zhao, Sixing Wu, Yang Zhou, Yelong Shen, Tianshi Che, Ruoming Jin, Dejing Dou, Yelong Shen

Large Scale Private Learning via Low-rank Reparametrization

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, Tie-Yan Liu

On the Problem of Underranking in Group-Fair Ranking

Sruthi Gorantla, Amit Deshpande, Anand Louis

Online Learning with Optimism and Delay

Genevieve Flaspohler, Francesco Orabona, Judah Cohen, Soukayna Mouatadid, Miruna Oprescu, Paulo Orenstein, Lester Mackey

Statistical Estimation from Dependent Data

Vardis Kandiros, Yuval Dagan, Nishanth Dikkala, Surbhi Goel, Constantinos Daskalakis

Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons

Bohang Zhang, Tianle Cai, Zhou Lu, Di He, Liwei Wang

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang

WILDS: A Benchmark of in-the-Wild Distribution Shifts

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, Percy Liang

Decomposed Mutual Information Estimation for Contrastive Representation Learning

Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman, Remi Tachet des Combes

BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining

Weizhen Qi, Yeyun Gong, Jian Jiao, Yu Yan, Dayiheng Liu, Wei Chen, Kewen Tang, Houqiang Li, Jiusheng Chen, Ruofei Zhang, Ming Zhou, Nan Duan

BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining

Weizhen Qi, Yeyun Gong, Jian Jiao, Yu Yan, Dayiheng Liu, Wei Chen, Kewen Tang, Houqiang Li, Jiusheng Chen, Ruofei Zhang, Ming Zhou, Nan Duan

Online Learning with Optimism and Delay

Genevieve Flaspohler, Francesco Orabona, Judah Cohen, Soukayna Mouatadid, Miruna Oprescu, Paulo Orenstein, Lester Mackey

A Unified Bayesian Framework for Discriminative and Generative Continual Learning

Abhishek Kumar, Sunabha Chatterjee, Piyush Rai

Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs

Shruthi Bannur, Ozan Oktay, Melanie Bernhardt, Anton Schwaighofer, Rajesh Jena, Besmira Nushi, Sharan Wadhwani, Aditya Nori, Kal Natarajan, Shazad Ashraf, Javier Alvarez-Valle, Daniel Coelho de Castro