{"id":788843,"date":"2021-10-26T23:14:21","date_gmt":"2021-10-27T06:14:21","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=788843"},"modified":"2021-11-21T02:09:13","modified_gmt":"2021-11-21T10:09:13","slug":"language-speech-pretraining","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/language-speech-pretraining\/","title":{"rendered":"Language-speech Pretraining"},"content":{"rendered":"<section class=\"mb-3 moray-highlight\">\n\t<div class=\"card-img-overlay mx-lg-0\">\n\t\t<div class=\"card-background bg-gray-200 has-background- card-background--full-bleed\">\n\t\t\t\t\t<\/div>\n\t\t<!-- Foreground -->\n\t\t<div class=\"card-foreground d-flex mt-md-n5 my-lg-5 px-g px-lg-0\">\n\t\t\t<!-- Container -->\n\t\t\t<div class=\"container d-flex mt-md-n5 my-lg-5 align-self-center\">\n\t\t\t\t<!-- Card wrapper -->\n\t\t\t\t<div class=\"w-100 w-lg-col-5\">\n\t\t\t\t\t<!-- Card -->\n\t\t\t\t\t<div class=\"card material-md-card py-5 px-md-5\">\n\t\t\t\t\t\t<div class=\"card-body \">\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n<h1 id=\"language-speech-pretraining\">Language-speech Pretraining<\/h1>\n\n\n\n<p><\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n<p>We are working on pre-trained language model, including new pre-training method, pre-trained model compression, pre-training for other tasks such as speech and music.<\/p>\n\n\n\n\n\n<ul class=\"wp-block-list\" type=\"1\"><li>Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu<em>, MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training<\/em>, <strong>ACL<\/strong>\u00a02021. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2106.05630.pdf\">Paper<\/a>]<\/li><li>Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, Tie-Yan Liu, <em>NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search<\/em>, KDD 2021. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2105.14444.pdf\">Paper<\/a>]<\/li><li>Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, Tao Qin,\u00a0<em>SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint<\/em>,\u00a0<strong>AAAI<\/strong>\u00a02021. [<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/songmass-automatic-song-writing-with-pre-training-and-alignment-constraint\/\">Paper<\/a>]<\/li><li>Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu,\u00a0<em>MPNet: Masked and Permuted Pre-training for Language Understanding<\/em>,\u00a0<strong>NeurIPS<\/strong>\u00a02020. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2004.09297.pdf\">Paper<\/a>] [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwMTA3MzM4Nw==&mid=2649451850&idx=1&sn=1680d06b76f29027f01e68ee82e397e9&chksm=82c084ceb5b70dd8366f296309d486dcc3feb3d16f887df980aa8d30104b14c216a67778f7a1&scene=21#wechat_redirect\">Blog<\/a>] [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/MPNet\">Code@Github<\/a>]<\/li><li>Kaitao Song, Hao Sun, Xu Tan, Tao Qin, Jianfeng Lu, Hongzhi Liu, Tie-Yan Liu,\u00a0<em>LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning<\/em>, arXiv 2020. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2004.12817.pdf\">Paper<\/a>]<\/li><li>Hao Sun, Xu Tan, Jun-Wei Gan, Sheng Zhao, Dongxu Han, Hongzhi Liu, Tao Qin, and Tie-Yan Liu,\u00a0<em>Knowledge Distillation from BERT in Pre-training and Fine-tuning for Polyphone Disambiguation,<\/em>\u00a0<strong>ASRU<\/strong>\u00a02019. [<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/ieeexplore.ieee.org\/abstract\/document\/9003918\">Paper<\/a>]<\/li><li>Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu,\u00a0<em>MASS: Masked Sequence to Sequence Pre-training for Language Generation<\/em>,\u00a0<strong>ICML<\/strong>\u00a02019.\u00a0[<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/1905.02450.pdf\">Paper<\/a>][<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/MASS\">Code@Github<\/a>][<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/mp.weixin.qq.com\/s\/7yCnAHk6x0ICtEwBKxXpOw\">Article<\/a>][<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/introducing-mass-a-pre-training-method-that-outperforms-bert-and-gpt-in-sequence-to-sequence-language-generation-tasks\/\">Blog<\/a>]<\/li><\/ul>\n\n\n","protected":false},"excerpt":{"rendered":"<p>We are working on pre-trained language model, including new pre-training method, pre-trained model compression, pre-training for other tasks such as speech and music. Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu, MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training, ACL\u00a02021. [Paper] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-788843","msr-project","type-msr-project","status-publish","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Rui Wang","user_id":39880,"people_section":"Section name 0","alias":"ruiwa"}],"msr_research_lab":[199560],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/788843"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":2,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/788843\/revisions"}],"predecessor-version":[{"id":798514,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/788843\/revisions\/798514"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=788843"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=788843"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=788843"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=788843"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=788843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}