Li Dong

REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models

Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang

December 2024

Publication

Differential Transformer

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei

MSR-TR-2024-42 | October 2024

Published by Microsoft

Publication

Direct Preference Knowledge Distillation for Large Language Models

Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei

June 2024

DOI Publication Publication

BIOCLIP: A Vision Foundation Model for the Tree of Life

Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Carlyn, Li Dong, W. Dahdul, Charles Stewart, Tanya Y. Berger-Wolf, Wei-Lun Chao, Yu Su

CVPR 2024 | June 2024

CVPR 2024 Best Student Paper Award

DOI Publication

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei

NeurIPS 2024 | May 2024

DOI Publication Publication

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei

ICLR 2024 | May 2024

DOI Publication PDF

Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei

NeurIPS 2024 | April 2024

Publication

Towards Optimal Learning of Language Models

Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei

February 2024

DOI Publication Publication

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei

February 2024

Work in progress

DOI Publication Publication

BitNet: Scaling 1-bit Transformers for Large Language Models

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei

October 2023

DOI Publication

Augmenting Language Models with Long-Term Memory

Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei

NeurIPS 2023 | October 2023

Publication

Kosmos-2.5: A Multimodal Literate Model

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei

September 2023

Preprint Project

Retentive Network: A Successor to Transformer for Large Language Models

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei

August 2023

Publication

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei

July 2023

DOI PDF Video

Semi-Offline Reinforcement Learning for Optimized Text Generation

Changyu Chen, Xiting Wang, Yiqiao Jin, Victor Ye Dong, Li Dong, Jim Cao, Yi Liu, Rui Yan

International Conference on Machine Learning (ICML) | July 2023

Publication

Knowledge Distillation of Large Language Models

Yuxian Gu, Li Dong, Furu Wei, Minlie Huang

ICLR 2024 | June 2023

DOI PDF

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

NeurIPS 2023 | March 2023

Publication

AdaPrompt: Adaptive Model Training for Prompt-based NLP

Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang

Findings of Empirical Methods in Natural Language Processing (EMNLP), Abu Dhabi, the United Arab Emirates, 2022. | December 2022

Publication

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song

ACL 2023 | October 2022

Work in progress

Publication

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed

August 2022

Publication

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

ACL 2022 | May 2022

Publication

Controllable Natural Language Generation with Contrastive Prefixes

Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen

ACL 2022 | May 2022

Publication

Knowledge Neurons in Pretrained Transformers

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei

ACL 2022 | May 2022

Publication

StableMoE: Stable Routing Strategy for Mixture of Experts

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

ACL 2022 | May 2022

Publication

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei

May 2022

BEiT: BERT Pre-Training of Image Transformers

Hangbo Bao, Li Dong, Songhao Piao, Furu Wei

ICLR 2022 | April 2022

Publication

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

WMT | November 2021

Publication Publication Project

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

EMNLP 2021 | October 2021

Publication

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

EMNLP 2021 | October 2021

Publication

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

EMNLP 2021 | September 2021

Publication Project

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

ACL-IJCNLP 2021 | July 2021

Publication Github

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

ACL 2022 | June 2021

Publication Project

Learning to Sample Replacements for ELECTRA Pre-Training

Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

arXiv: Computation and Language | June 2021

Publication Publication Publication Project

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

June 2021

Publication Publication Project

Consistency Regularization for Cross-Lingual Fine-Tuning

Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

ACL-IJCNLP 2021 | June 2021

Publication

Memory-Efficient Differentiable Transformer Architecture Search

Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen

ACL-IJCNLP 2021 | May 2021

Publication

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

ACL-IJCNLP 2021 | December 2020

PDF Publication

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.

Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei

December 2020

Publication Publication Publication Project

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao

ECCV | August 2020

Publication Video Github Project Project Project

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou

July 2020

Publication Publication Github

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Yaru Hao, Li Dong, Furu Wei, Ke Xu

AAAI 2021 | April 2020

AAAI 2021 Best Paper Runner Up

Publication PDF

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

February 2020

Publication Github Project

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou

NeurIPS 2020 | February 2020

Publication Project

Unified Language Model Pre-training for Natural Language Understanding and Generation

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019

Publication Github Project

REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models

Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang

December 2024

Publication

Differential Transformer

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei

MSR-TR-2024-42 | October 2024

Published by Microsoft

Publication

Direct Preference Knowledge Distillation for Large Language Models

Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei

June 2024

DOI Publication Publication

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei

NeurIPS 2024 | May 2024

DOI Publication Publication

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei

ICLR 2024 | May 2024

DOI Publication PDF

Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei

NeurIPS 2024 | April 2024

Publication

Towards Optimal Learning of Language Models

Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei

February 2024

DOI Publication Publication

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei

February 2024

Work in progress

DOI Publication Publication

BitNet: Scaling 1-bit Transformers for Large Language Models

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei

October 2023

DOI Publication

Augmenting Language Models with Long-Term Memory

Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei

NeurIPS 2023 | October 2023

Publication

Kosmos-2.5: A Multimodal Literate Model

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei

September 2023

Preprint Project

Retentive Network: A Successor to Transformer for Large Language Models

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei

August 2023

Publication

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei

July 2023

DOI PDF Video

Semi-Offline Reinforcement Learning for Optimized Text Generation

Changyu Chen, Xiting Wang, Yiqiao Jin, Victor Ye Dong, Li Dong, Jim Cao, Yi Liu, Rui Yan

International Conference on Machine Learning (ICML) | July 2023

Publication

Knowledge Distillation of Large Language Models

Yuxian Gu, Li Dong, Furu Wei, Minlie Huang

ICLR 2024 | June 2023

DOI PDF

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

NeurIPS 2023 | March 2023

Publication

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song

ACL 2023 | October 2022

Work in progress

Publication

BEiT: BERT Pre-Training of Image Transformers

Hangbo Bao, Li Dong, Songhao Piao, Furu Wei

ICLR 2022 | April 2022

Publication

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

WMT | November 2021

Publication Publication Project

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

EMNLP 2021 | September 2021

Publication Project

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

ACL-IJCNLP 2021 | July 2021

Publication Github

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

ACL 2022 | June 2021

Publication Project

Learning to Sample Replacements for ELECTRA Pre-Training

Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

arXiv: Computation and Language | June 2021

Publication Publication Publication Project

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

June 2021

Publication Publication Project

Consistency Regularization for Cross-Lingual Fine-Tuning

Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

ACL-IJCNLP 2021 | June 2021

Publication

Memory-Efficient Differentiable Transformer Architecture Search

Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen

ACL-IJCNLP 2021 | May 2021

Publication

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.

Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei

December 2020

Publication Publication Publication Project

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

ACL-IJCNLP 2021 | December 2020

PDF Publication

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao

ECCV | August 2020

Publication Video Github Project Project Project

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou

July 2020

Publication Publication Github

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Yaru Hao, Li Dong, Furu Wei, Ke Xu

AAAI 2021 | April 2020

AAAI 2021 Best Paper Runner Up

Publication PDF

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

February 2020

Publication Github Project

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou

NeurIPS 2020 | February 2020

Publication Project

Unified Language Model Pre-training for Natural Language Understanding and Generation

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019

Publication Github Project

BIOCLIP: A Vision Foundation Model for the Tree of Life

Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Carlyn, Li Dong, W. Dahdul, Charles Stewart, Tanya Y. Berger-Wolf, Wei-Lun Chao, Yu Su

CVPR 2024 | June 2024

CVPR 2024 Best Student Paper Award

DOI Publication

Kosmos-2.5: A Multimodal Literate Model

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei

September 2023

Preprint Project

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

NeurIPS 2023 | March 2023

Publication

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed

August 2022

Publication

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao

ECCV | August 2020

Publication Video Github Project Project Project

Kosmos-2.5: A Multimodal Literate Model

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei

September 2023

Preprint Project

AdaPrompt: Adaptive Model Training for Prompt-based NLP

Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang

Findings of Empirical Methods in Natural Language Processing (EMNLP), Abu Dhabi, the United Arab Emirates, 2022. | December 2022

Publication

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song

ACL 2023 | October 2022

Work in progress

Publication

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei

May 2022

StableMoE: Stable Routing Strategy for Mixture of Experts

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

ACL 2022 | May 2022

Publication

Knowledge Neurons in Pretrained Transformers

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei

ACL 2022 | May 2022

Publication

Controllable Natural Language Generation with Contrastive Prefixes

Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen

ACL 2022 | May 2022

Publication

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

ACL 2022 | May 2022

Publication

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

WMT | November 2021

Publication Publication Project

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

EMNLP 2021 | October 2021

Publication

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

EMNLP 2021 | October 2021

Publication

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

EMNLP 2021 | September 2021

Publication Project

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

ACL 2022 | June 2021

Publication Project

Learning to Sample Replacements for ELECTRA Pre-Training

Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

arXiv: Computation and Language | June 2021

Publication Publication Publication Project

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

June 2021

Publication Publication Project

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

ACL-IJCNLP 2021 | June 2021

Publication

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.

Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei

December 2020

Publication Publication Publication Project

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Yaru Hao, Li Dong, Furu Wei, Ke Xu

AAAI 2021 | April 2020

AAAI 2021 Best Paper Runner Up

Publication PDF

Unified Language Model Pre-training for Natural Language Understanding and Generation

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019

Publication Github Project

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Yaru Hao, Li Dong, Furu Wei, Ke Xu

AAAI 2021 | April 2020

AAAI 2021 Best Paper Runner Up

Publication PDF

REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models

Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang

December 2024

Publication

Kosmos-2.5: A Multimodal Literate Model

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei

September 2023

Preprint Project

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed

August 2022

Publication

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei

May 2022

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

June 2021

Publication Publication Project

Direct Preference Knowledge Distillation for Large Language Models

Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei

June 2024

DOI Publication Publication

Towards Optimal Learning of Language Models

Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei

February 2024

DOI Publication Publication

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei

February 2024

Work in progress

DOI Publication Publication

BitNet: Scaling 1-bit Transformers for Large Language Models

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei

October 2023

DOI Publication

Retentive Network: A Successor to Transformer for Large Language Models

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei

August 2023

Publication

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei

July 2023

DOI PDF Video

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.

Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei

December 2020

Publication Publication Publication Project

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou

July 2020

Publication Publication Github

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

February 2020

Publication Github Project

Differential Transformer

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei

MSR-TR-2024-42 | October 2024

Published by Microsoft

Publication

BIOCLIP: A Vision Foundation Model for the Tree of Life

Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Carlyn, Li Dong, W. Dahdul, Charles Stewart, Tanya Y. Berger-Wolf, Wei-Lun Chao, Yu Su

CVPR 2024 | June 2024

CVPR 2024 Best Student Paper Award

DOI Publication

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei

NeurIPS 2024 | May 2024

DOI Publication Publication

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei

ICLR 2024 | May 2024

DOI Publication PDF

Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei

NeurIPS 2024 | April 2024

Publication

Augmenting Language Models with Long-Term Memory

Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei

NeurIPS 2023 | October 2023

Publication

Semi-Offline Reinforcement Learning for Optimized Text Generation

Changyu Chen, Xiting Wang, Yiqiao Jin, Victor Ye Dong, Li Dong, Jim Cao, Yi Liu, Rui Yan

International Conference on Machine Learning (ICML) | July 2023

Publication

Knowledge Distillation of Large Language Models

Yuxian Gu, Li Dong, Furu Wei, Minlie Huang

ICLR 2024 | June 2023

DOI PDF

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

NeurIPS 2023 | March 2023

Publication

AdaPrompt: Adaptive Model Training for Prompt-based NLP

Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, Yue Zhang

Findings of Empirical Methods in Natural Language Processing (EMNLP), Abu Dhabi, the United Arab Emirates, 2022. | December 2022

Publication

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song

ACL 2023 | October 2022

Work in progress

Publication

StableMoE: Stable Routing Strategy for Mixture of Experts

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

ACL 2022 | May 2022

Publication

Knowledge Neurons in Pretrained Transformers

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei

ACL 2022 | May 2022

Publication

Controllable Natural Language Generation with Contrastive Prefixes

Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen

ACL 2022 | May 2022

Publication

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

ACL 2022 | May 2022

Publication

BEiT: BERT Pre-Training of Image Transformers

Hangbo Bao, Li Dong, Songhao Piao, Furu Wei

ICLR 2022 | April 2022

Publication

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.

Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

WMT | November 2021

Publication Publication Project

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

EMNLP 2021 | October 2021

Publication

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

EMNLP 2021 | October 2021

Publication

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

EMNLP 2021 | September 2021

Publication Project

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

ACL-IJCNLP 2021 | July 2021

Publication Github

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

ACL 2022 | June 2021

Publication Project

Learning to Sample Replacements for ELECTRA Pre-Training

Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

Consistency Regularization for Cross-Lingual Fine-Tuning

Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

ACL-IJCNLP 2021 | June 2021

PDF

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

ACL-IJCNLP 2021 | June 2021

Publication

Memory-Efficient Differentiable Transformer Architecture Search

Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen

ACL-IJCNLP 2021 | May 2021

Publication

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

ACL-IJCNLP 2021 | December 2020

PDF Publication

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao

ECCV | August 2020

Publication Video Github Project Project Project

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Yaru Hao, Li Dong, Furu Wei, Ke Xu

AAAI 2021 | April 2020

AAAI 2021 Best Paper Runner Up

Publication PDF

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou

NeurIPS 2020 | February 2020

Publication Project

Unified Language Model Pre-training for Natural Language Understanding and Generation

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019

Publication Github Project

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.

Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

arXiv: Computation and Language | June 2021

Publication Publication Publication Project