DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Yu Cheng, Wei Chen, Zhangyang Wang, Ahmed Awadallah
ACL 2023 | May 2023
Reducing AI’s Carbon Footprint
Pursuing more efficient AI
Publications
Publications
Publications by Year
-
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression
Mengnan Du, Subhabrata (Subho) Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Awadallah
EACL | April 2023
-
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
-
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
Subhabrata (Subho) Mukherjee, Ahmed Awadallah, Jianfeng Gao
ArXiv | June 2021
arXiv:2106.04563
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
-
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
Probabilistic Neural Architecture Search
Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi
February 2019
Publications by Research Area
-
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Yu Cheng, Wei Chen, Zhangyang Wang, Ahmed Awadallah
ACL 2023 | May 2023
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
Probabilistic Neural Architecture Search
Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi
February 2019
-
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression
Mengnan Du, Subhabrata (Subho) Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Awadallah
EACL | April 2023
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
Subhabrata (Subho) Mukherjee, Ahmed Awadallah, Jianfeng Gao
ArXiv | June 2021
arXiv:2106.04563
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
-
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
Publications by Type
-
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Yu Cheng, Wei Chen, Zhangyang Wang, Ahmed Awadallah
ACL 2023 | May 2023
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression
Mengnan Du, Subhabrata (Subho) Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Awadallah
EACL | April 2023
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
Subhabrata (Subho) Mukherjee, Ahmed Awadallah, Jianfeng Gao
ArXiv | June 2021
arXiv:2106.04563
-
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
Probabilistic Neural Architecture Search
Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi
February 2019