AI Music

Muzic is a research project on AI music that empowers music understanding and generation with deep learning and artificial intelligence. Muzic is pronounced as [ˈmjuːzeik] and ‘谬贼客’ (in Chinese). Besides the logo in image version (see above), Muzic also has a logo in video version (you can click here (opens in new tab) to watch).

We summarize the scope of our Muzic project in the following figure:

The current work in Muzic (opens in new tab) include:

• Music Understanding
  • Symbolic Music Understanding: MusicBERT (opens in new tab)
  • Automatic Lyrics Transcription: PDAugment (opens in new tab)
• Music Generation
  • Song Writing: SongMASS (opens in new tab)
  • Lyric Generation: DeepRapper (opens in new tab)
  • Lyric-to-Melody Generation: TeleMelody (opens in new tab), ReLyMe (opens in new tab), Re-Creation of Creations (opens in new tab)
  • Accompaniment Generation: PopMAG (opens in new tab)
  • Singing Voice Synthesis: HiFiSinger (opens in new tab)

We release the code of our research work in https://github.com/microsoft/muzic (opens in new tab).

You can find some music samples generated by our systems from this page: https://ai-muzic.github.io/ (opens in new tab).

Thanks Microsoft Band (opens in new tab) for the great support on this project!

Our Papers

• Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation, Ang Lv, Xu Tan, Tao Qin, Tie-Yan Liu, Rui Yan, arXiv 2022. [Paper (opens in new tab)]
• ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships, Chen Zhang, LuChin Chang, Songruoyao Wu, Xu Tan, Tao Qin, Tie-Yan Liu, Kejun Zhang, ACM Multimedia 2022. [Paper (opens in new tab)]
• TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method, Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiangyang Li, Tao Qin, Tie-Yan Liu, arXiv 2021. [Paper (opens in new tab)] [Demo (opens in new tab)]
• PDAugment: Data Augmentation by Pitch and Duration Adjustments for Automatic Lyrics Transcription, Chen Zhang, Jiaxing Yu, Luchin Chang, Xu Tan, Jiawei Chen, Tao Qin, Kejun Zhang, ISMIR 2022. [Paper (opens in new tab)]
• MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training, Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu, ACL 2021. [Paper (opens in new tab)]
• DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling, Lanqing Xue, Kaitao Song, Duocai Wu, Xu Tan, Nevin L. Zhang, Tao Qin, Wei-Qiang Zhang, Tie-Yan Liu, ACL 2021. [Paper (opens in new tab)] [Demo (opens in new tab)]
• SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint, Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, Tao Qin, AAAI 2021. [Paper (opens in new tab)] [Demo (opens in new tab)]
• PopMAG: Pop Music Accompaniment Generation, Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, Tie-Yan Liu, ACM Multimedia 2020. [Paper (opens in new tab)] [Demo (opens in new tab)]
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis, Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, Tie-Yan Liu, arXiv 2020. [Paper (opens in new tab)] [Demo (opens in new tab)]
• DeepSinger: Singing Voice Synthesis with Data Mined From the Web, Yi Ren, Xu Tan, Tao Qin, Jian Luan, Zhou Zhao, Tie-Yan Liu, KDD 2020. [Paper (opens in new tab)] [Demo (opens in new tab)]
• XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System, Peiling Lu, Jie Wu, Jian Luan, Xu Tan, Li Zhou, INTERSPEECH 2020. [Paper (opens in new tab)] [Demo (opens in new tab)]