Optimization in Deep Learning

Qi Meng*, Shuxin Zheng*, Huishuai Zhang, Wei Chen, Zhi-Ming Ma, and Tie-Yan Liu, G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space, In Proceedings of the 7th International Conference of Learning Representations (ICLR), 2019

Xufang Luo, Qi Meng, Wei Chen, Yunhong Wang, and Tie-Yan Liu, Path-BN: Towards Effective Batch Normalization in the Path Space for ReLU Networks, In the Conference on Uncertainty in Artificial Intelligence (UAI), 2021

Huishuai Zhang, Da Yu, Mingyang Yi, Wei Chen and Tie-Yan Liu, Stabilize Deep ResNet with A Sharp Scaling Factor τ, Machine Learning Journal, August 2021

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang and Tie-Yan Liu, On Layer Normalization in the Transformer Architecture, ICML, July 2020

Shicong Cen, Huishuai Zhang, Yuejie Chi, Wei Chen and Tie-Yan Liu, Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data, IEEE Transactions on Signal Processing | June 2020, Vol 68

Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang and Vahid Tarokh, SGD Converges to Global Minimum in Deep Learning via Star-convex Path, International Conference on Learning Representations (ICLR), May 2019

Huishuai Zhang, Wei Chen and Tie-Yan Liu, On the local Hessian in back-propagation, Neural Information Processing Systems (NeurIPS), 2018