![\"Figure](\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/05\/Robust-Language-F1_Site_05_2019_1400x788.png\")
<\/a>Figure 1: Architecture of the MT-DNN student model.<\/p><\/div>\n
In \u201cImproving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding<\/a>\u201d, researchers Xiaodong Liu and Jianfeng Gao of Microsoft Research and Pengcheng He and Weizhu Chen of Microsoft Dynamics 365 AI compressed multiple ensembled models into a single Multi-Task Deep Neural Network (MT-DNN) via knowledge distillation for learning robust and universal text representations across multiple NLU tasks. The ensemble model was utilized, in an offline manner, to produce a set of soft targets<\/em> for each task in the training dataset. These generated soft targets provide more information per training sample than the hard targets and less variance in the gradient between training samples. For example, the sentiment of the sentence \u201cI really enjoyed the conversation with Tom\u201d has a small chance of being classified as negative. But the sentence \u201cTom and I had an interesting conversation\u201d can be either positive or negative, depending on its context, if available, leading to a high entropy of the soft targets assigned by the teacher. Then, a single MT-DNN (student) was trained via multi-task learning with the help of the teachers by using both the soft targets and correct targets across different tasks. Empirical experiments show that the distilled MT-DNN outperforms the original MT-DNN and obtained a new state-of-the-art result on the General Language Understanding Evaluation (GLUE) benchmark.<\/p>\n The student model is a MT-DNN model proposed by Microsoft in 2019 that incorporates a pre-trained bidirectional transformer language model, known as BERT, developed by Google AI. Figure 1 illustrates the architecture of the student model. The lower layers are shared across all tasks while the top layers are task-specific. The input X<\/em> (either a sentence or a set of sentences) is first represented as a sequence of embedding vectors, one for each word, in l1<\/sub><\/em>. Then the Transformer encoder captures the contextual information for each word and generates the shared contextual embedding vectors in l2<\/sub><\/em>. Finally, for each task, additional task-specific layers generate task-specific representations, followed by operations necessary for classification, similarity scoring, or relevance ranking. The shared layers of MT-DNN are initialized by using BERT, then trained jointly via multi-task learning (MTL).<\/p>\n The process of knowledge distillation for MTL is illustrated in Figure 2. A set of tasks where there is task-specific labeled training data is picked. Then, for each task, an ensemble of different neural nets (teacher) is trained. The teacher is used to generate a set of soft targets for each task-specific training sample. Given the soft targets of the training datasets across multiple tasks, a single MT-DNN (student) is trained using multi-task learning, except that if task $t$ has a teacher, the task-specific loss is the average of two objective functions\u2014one for the correct targets and the other for the soft targets assigned by the teacher.<\/p>\n Figure 2: Knowledge distillation for multi-task learning<\/p><\/div>\n The authors compared the distilled MT-DNN with BERT and MT-DNN. As shown in Figure 2, the MT-DNN outperforms BERT by a substantial margin, and the distilled MT-DNN further realizes a significant improvement over MT-DNN in terms of overall score on GLUE. On the GLUE diagnostics data, which is a manually-curated test set for the analysis of system performance, similar observations are found: both MT-DNN and distilled MT-DNN outperform BERT. All these demonstrate that language representation learned via distilled MT-DNN is more robust and universal than BERT\/MT-DNN. Detailed results can be found at the GLUE leaderboard<\/a>.<\/p>\nThe MT-DNN student model<\/h3>\n
Multi-task knowledge distillation<\/h3>\n
<\/a>Results<\/h3>\n