Bridging Discrete and Backpropagation: Straight-Through and Beyond
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, Jianfeng Gao
NeurIPS 2023 (Oral) | December 2023
Xiaodong Liu
Principal Researcher
Publications
Publications by Year
-
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, Tuo Zhao
ICLR 2024 | November 2023
Fast-ELECTRA for Efficient Pre-training
Chengyu Dong, Liyuan Liu, Hao Cheng, Jingbo Shang, Jianfeng Gao, Xiaodong Liu
ICLR 2024 | October 2023
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
ACL 2023 | July 2023
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Hao Cheng, Hao Fang, Xiaodong Liu, Jianfeng Gao
ACL 2023 | July 2023
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL 2023 | June 2023
Fine-tuning large neural language models for biomedical natural language processing
Rob Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
Patterns | June 2023, Vol 4(4)
Chain-of-Skills: A Configurable Model for Open-domain Question Answering
Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
ACL 2023 | May 2023
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
-
Efficient Long Sequence Modeling via State Space Augmented Transformer
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, Jianfeng Gao
December 2022
Preprint
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
EMNLP 2022 | October 2022
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, Xuedong Huang
IJCAI 2022 | July 2022
human parity result on CommonsenseQA
Knowledge-Rich Self-Supervision for Biomedical Entity Linking
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
EMNLP 2022 | May 2022
Open Domain Question Answering with A Unified Knowledge Interface
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
ACL 2022 | May 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
April 2022
arXiv:2204.06644
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
ICLR 2022 | April 2022
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan Awadalla, Ruofei Zhang, Tuo Zhao, Jianfeng Gao
ICLR 2022 | April 2022
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki, Xiaodong Liu, Weizhu Chen, Jianfeng Gao
NeurIPS 2021 | March 2022
-
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
Subhabrata (Subho) Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Chris Meek, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2021 | December 2021
Taming Sparsely Activated Transformer with Stochastic Experts.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, Jianfeng Gao, Hany Hassan Awadalla
October 2021
Very Deep Transformers for Neural Machine Translation
Xiaodong Liu, Kevin Duh, Liyuan Liu, Jianfeng Gao
October 2021
ARCH: Efficient Adversarial Regularized Training with Caching
Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Targeted Adversarial Training for Natural Language Understanding
Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi
NAACL | June 2021
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Wei Chen
ACL-IJCNLP 2021 | May 2021
DeBERTa: Decoding-Enhanced BERT with Disentangled Attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, Wei Chen
2021 International Conference on Learning Representations | May 2021
Under review
Token-wise Curriculum Learning for Neural Machine Translation
Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Wei Chen, Jianfeng Gao, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | March 2021
-
Reader-Guided Passage Reranking for Open-Domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | December 2020
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Wei Chen, Jianfeng Gao
ACL-IJCNLP 2021 | December 2020
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, Jianfeng Gao
October 2020
Generation-Augmented Retrieval for Open-domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | September 2020
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji
EMNLP 2021 | August 2020
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
August 2020
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Bailin Wang, Richard Shin, Xiaodong Liu, Alex Polozov, Matthew Richardson
ACL 2020 | June 2020
Adversarial Training for Large Neural Language Models
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, Jianfeng Gao
April 2020
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
Eighth International Conference on Learning Representations (ICLR) | April 2020
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao
April 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
February 2020
-
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain.
Yichong Xu, Xiaodong Liu, Chunyuan Li, Hoifung Poon, Jianfeng Gao
BioNLP 2019 | August 2019
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, Jianfeng Gao
Proc. of ACL | June 2019
Multi-Task Deep Neural Networks for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
ACL 2019 | June 2019
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
April 2019
Cyclical Annealing Schedule: A Simple Approach to Mitigate KL Vanishing
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin
NAACL 2019 | February 2019
-
Dynamic Fusion Networks for Machine Reading Comprehension
Yichong Xu, JJ (Jingjing) Liu, Jianfeng Gao, Yelong Shen, Xiaodong Liu
February 2018
-
Stochastic Answer Networks for Machine Reading Comprehension
Xiaodong Liu, Yelong Shen, Jianfeng Gao, Kevin Duh
MSR-TR-2017-49 | December 2017
-
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang
May 2015
Publications by Research Area
-
Bridging Discrete and Backpropagation: Straight-Through and Beyond
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, Jianfeng Gao
NeurIPS 2023 (Oral) | December 2023
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, Tuo Zhao
ICLR 2024 | November 2023
Fast-ELECTRA for Efficient Pre-training
Chengyu Dong, Liyuan Liu, Hao Cheng, Jingbo Shang, Jianfeng Gao, Xiaodong Liu
ICLR 2024 | October 2023
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
ACL 2023 | July 2023
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Hao Cheng, Hao Fang, Xiaodong Liu, Jianfeng Gao
ACL 2023 | July 2023
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL 2023 | June 2023
Fine-tuning large neural language models for biomedical natural language processing
Rob Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
Patterns | June 2023, Vol 4(4)
Chain-of-Skills: A Configurable Model for Open-domain Question Answering
Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
ACL 2023 | May 2023
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
Efficient Long Sequence Modeling via State Space Augmented Transformer
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, Jianfeng Gao
December 2022
Preprint
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
EMNLP 2022 | October 2022
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, Xuedong Huang
IJCAI 2022 | July 2022
human parity result on CommonsenseQA
Knowledge-Rich Self-Supervision for Biomedical Entity Linking
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
EMNLP 2022 | May 2022
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
April 2022
arXiv:2204.06644
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
ICLR 2022 | April 2022
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan Awadalla, Ruofei Zhang, Tuo Zhao, Jianfeng Gao
ICLR 2022 | April 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki, Xiaodong Liu, Weizhu Chen, Jianfeng Gao
NeurIPS 2021 | March 2022
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
Subhabrata (Subho) Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Chris Meek, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2021 | December 2021
Taming Sparsely Activated Transformer with Stochastic Experts.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, Jianfeng Gao, Hany Hassan Awadalla
October 2021
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Targeted Adversarial Training for Natural Language Understanding
Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi
NAACL | June 2021
DeBERTa: Decoding-Enhanced BERT with Disentangled Attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, Wei Chen
2021 International Conference on Learning Representations | May 2021
Under review
Reader-Guided Passage Reranking for Open-Domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | December 2020
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Wei Chen, Jianfeng Gao
ACL-IJCNLP 2021 | December 2020
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, Jianfeng Gao
October 2020
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji
EMNLP 2021 | August 2020
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Bailin Wang, Richard Shin, Xiaodong Liu, Alex Polozov, Matthew Richardson
ACL 2020 | June 2020
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao
April 2020
Adversarial Training for Large Neural Language Models
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, Jianfeng Gao
April 2020
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
Eighth International Conference on Learning Representations (ICLR) | April 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
February 2020
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain.
Yichong Xu, Xiaodong Liu, Chunyuan Li, Hoifung Poon, Jianfeng Gao
BioNLP 2019 | August 2019
Multi-Task Deep Neural Networks for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
ACL 2019 | June 2019
Cyclical Annealing Schedule: A Simple Approach to Mitigate KL Vanishing
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin
NAACL 2019 | February 2019
Dynamic Fusion Networks for Machine Reading Comprehension
Yichong Xu, JJ (Jingjing) Liu, Jianfeng Gao, Yelong Shen, Xiaodong Liu
February 2018
Stochastic Answer Networks for Machine Reading Comprehension
Xiaodong Liu, Yelong Shen, Jianfeng Gao, Kevin Duh
MSR-TR-2017-49 | December 2017
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang
May 2015
-
Bridging Discrete and Backpropagation: Straight-Through and Beyond
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, Jianfeng Gao
NeurIPS 2023 (Oral) | December 2023
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
Eighth International Conference on Learning Representations (ICLR) | April 2020
-
Fine-tuning large neural language models for biomedical natural language processing
Rob Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
Patterns | June 2023, Vol 4(4)
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
August 2020
DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain.
Yichong Xu, Xiaodong Liu, Chunyuan Li, Hoifung Poon, Jianfeng Gao
BioNLP 2019 | August 2019
-
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
ACL 2023 | July 2023
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Hao Cheng, Hao Fang, Xiaodong Liu, Jianfeng Gao
ACL 2023 | July 2023
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
Efficient Long Sequence Modeling via State Space Augmented Transformer
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, Jianfeng Gao
December 2022
Preprint
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
EMNLP 2022 | October 2022
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, Xuedong Huang
IJCAI 2022 | July 2022
human parity result on CommonsenseQA
Knowledge-Rich Self-Supervision for Biomedical Entity Linking
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
EMNLP 2022 | May 2022
Open Domain Question Answering with A Unified Knowledge Interface
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
ACL 2022 | May 2022
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
April 2022
arXiv:2204.06644
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan Awadalla, Ruofei Zhang, Tuo Zhao, Jianfeng Gao
ICLR 2022 | April 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
Subhabrata (Subho) Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Chris Meek, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2021 | December 2021
Taming Sparsely Activated Transformer with Stochastic Experts.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, Jianfeng Gao, Hany Hassan Awadalla
October 2021
Very Deep Transformers for Neural Machine Translation
Xiaodong Liu, Kevin Duh, Liyuan Liu, Jianfeng Gao
October 2021
ARCH: Efficient Adversarial Regularized Training with Caching
Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Wei Chen
ACL-IJCNLP 2021 | May 2021
Token-wise Curriculum Learning for Neural Machine Translation
Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Wei Chen, Jianfeng Gao, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | March 2021
Reader-Guided Passage Reranking for Open-Domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | December 2020
Generation-Augmented Retrieval for Open-domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | September 2020
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji
EMNLP 2021 | August 2020
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Bailin Wang, Richard Shin, Xiaodong Liu, Alex Polozov, Matthew Richardson
ACL 2020 | June 2020
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao
April 2020
Adversarial Training for Large Neural Language Models
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, Jianfeng Gao
April 2020
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
Eighth International Conference on Learning Representations (ICLR) | April 2020
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, Jianfeng Gao
Proc. of ACL | June 2019
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
April 2019
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang
May 2015
-
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Wei Chen
ACL-IJCNLP 2021 | May 2021
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, Jianfeng Gao
October 2020
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
Eighth International Conference on Learning Representations (ICLR) | April 2020
-
Targeted Adversarial Training for Natural Language Understanding
Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi
NAACL | June 2021
Publications by Type
-
Bridging Discrete and Backpropagation: Straight-Through and Beyond
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, Jianfeng Gao
NeurIPS 2023 (Oral) | December 2023
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, Tuo Zhao
ICLR 2024 | November 2023
Fast-ELECTRA for Efficient Pre-training
Chengyu Dong, Liyuan Liu, Hao Cheng, Jingbo Shang, Jianfeng Gao, Xiaodong Liu
ICLR 2024 | October 2023
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
ACL 2023 | July 2023
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Hao Cheng, Hao Fang, Xiaodong Liu, Jianfeng Gao
ACL 2023 | July 2023
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL 2023 | June 2023
Chain-of-Skills: A Configurable Model for Open-domain Question Answering
Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
ACL 2023 | May 2023
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
EMNLP 2022 | October 2022
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, Xuedong Huang
IJCAI 2022 | July 2022
human parity result on CommonsenseQA
Knowledge-Rich Self-Supervision for Biomedical Entity Linking
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
EMNLP 2022 | May 2022
Open Domain Question Answering with A Unified Knowledge Interface
Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao
ACL 2022 | May 2022
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
ICLR 2022 | April 2022
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan Awadalla, Ruofei Zhang, Tuo Zhao, Jianfeng Gao
ICLR 2022 | April 2022
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki, Xiaodong Liu, Weizhu Chen, Jianfeng Gao
NeurIPS 2021 | March 2022
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding
Subhabrata (Subho) Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Chris Meek, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2021 | December 2021
Taming Sparsely Activated Transformer with Stochastic Experts.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, Jianfeng Gao, Hany Hassan Awadalla
October 2021
ARCH: Efficient Adversarial Regularized Training with Caching
Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | September 2021
Targeted Adversarial Training for Natural Language Understanding
Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi
NAACL | June 2021
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Wei Chen
ACL-IJCNLP 2021 | May 2021
DeBERTa: Decoding-Enhanced BERT with Disentangled Attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, Wei Chen
2021 International Conference on Learning Representations | May 2021
Under review
Token-wise Curriculum Learning for Neural Machine Translation
Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Wei Chen, Jianfeng Gao, Tuo Zhao
2021 Empirical Methods in Natural Language Processing | March 2021
Reader-Guided Passage Reranking for Open-Domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | December 2020
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Wei Chen, Jianfeng Gao
ACL-IJCNLP 2021 | December 2020
Generation-Augmented Retrieval for Open-domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Wei Chen
ACL-IJCNLP 2021 | September 2020
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji
EMNLP 2021 | August 2020
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Bailin Wang, Richard Shin, Xiaodong Liu, Alex Polozov, Matthew Richardson
ACL 2020 | June 2020
On the Variance of the Adaptive Learning Rate and Beyond
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
Eighth International Conference on Learning Representations (ICLR) | April 2020
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
DoubleTransfer at MEDIQA 2019: Multi-Source Transfer Learning for Natural Language Understanding in the Medical Domain.
Yichong Xu, Xiaodong Liu, Chunyuan Li, Hoifung Poon, Jianfeng Gao
BioNLP 2019 | August 2019
Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, Jianfeng Gao
Proc. of ACL | June 2019
Multi-Task Deep Neural Networks for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
ACL 2019 | June 2019
Cyclical Annealing Schedule: A Simple Approach to Mitigate KL Vanishing
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin
NAACL 2019 | February 2019
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang
May 2015
-
Fine-tuning large neural language models for biomedical natural language processing
Rob Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
Patterns | June 2023, Vol 4(4)
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
-
Efficient Long Sequence Modeling via State Space Augmented Transformer
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, Jianfeng Gao
December 2022
Preprint
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao
April 2022
arXiv:2204.06644
Very Deep Transformers for Neural Machine Translation
Xiaodong Liu, Kevin Duh, Liyuan Liu, Jianfeng Gao
October 2021
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao
April 2019
-
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, Jianfeng Gao
October 2020
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
August 2020
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao
April 2020
Adversarial Training for Large Neural Language Models
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, Jianfeng Gao
April 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
February 2020
Dynamic Fusion Networks for Machine Reading Comprehension
Yichong Xu, JJ (Jingjing) Liu, Jianfeng Gao, Yelong Shen, Xiaodong Liu
February 2018
-
Stochastic Answer Networks for Machine Reading Comprehension
Xiaodong Liu, Yelong Shen, Jianfeng Gao, Kevin Duh
MSR-TR-2017-49 | December 2017