{"id":170910,"date":"2012-02-21T23:25:22","date_gmt":"2012-02-21T23:25:22","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/project\/turning-a-monolingual-speaker-into-multi-lingual-speaker\/"},"modified":"2016-06-07T15:56:20","modified_gmt":"2016-06-07T22:56:20","slug":"turning-a-monolingual-speaker-into-multi-lingual-speaker","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/turning-a-monolingual-speaker-into-multi-lingual-speaker\/","title":{"rendered":"Turning a Monolingual Speaker Into Multi-Lingual Speaker"},"content":{"rendered":"<p class=\"asset-content\">Voice user interface needs to output responses in Text-To-Speech (TTS) synthesized speech. Sometimes it is even more desirable to have the response in mixed languages, For example, in a foreign country, it would be convenient if a user of car-navigation system who is not fluent in that particular foreign language could hear instructions in mixed-codes, such as entities like street names synthesized in the local language and routing directions in the user\u2019s native language.<\/p>\n<p>Voice user interface needs to output responses in Text-To-Speech (TTS) synthesized speech. Sometimes it is even more desirable to have the response in mixed languages,\u00a0For example,\u00a0in a foreign country, it would be convenient if a user of car-navigation system who is not fluent in that particular foreign language could hear instructions in mixed-codes,\u00a0such as\u00a0entities like street names synthesized in the local language and routing directions in the user\u2019s native language. The mixed-coded TTS can be easily built by a truly bilingual speaker. However, it is usually difficult to find such a talent. We demo a new approach in turning monolingual TTS into multi-lingual TTS. Out of a speaker\u2019s monolingual recordings, our algorithm can render speech sentences of different languages for building mixed-coded, bilingual TTS systems. We have recordings of 26 languages which are used to build our TTS of corresponding languages. By using the new approach, we can synthesize any mixed language pair out of the 26 languages.<\/p>\n\t\t\t<div class=\"moray-accordion\" data-show=\"false\">\n\t\t\t\t\t\t\t<div class=\"clearfix\">\n\t\t\t\t\t<div\n\t\t\t\t\t\tclass=\"btn-group align-items-center mb-g float-sm-right\"\n\t\t\t\t\t\trole=\"group\"\n\t\t\t\t\t\tdata-mount=\"collapse-controls\"\n\t\t\t\t\t\tdata-bi-aN=\"accordion-collapse-controls\"\n\t\t\t\t\t\tdata-target=\"#accordion-1\">\n\t\t\t\t\t\t<button class=\"btn btn-link m-0\" aria-disabled=\"false\" aria-pressed=\"false\" data-action=\"expand\" type=\"button\" data-bi-cN=\"Expand all\">Expand all<\/button>\n\t\t\t\t\t\t<span aria-hidden=\"true\"> | <\/span>\n\t\t\t\t\t\t<button class=\"btn btn-link m-0\" aria-disabled=\"false\" aria-pressed=\"false\" data-action=\"collapse\" type=\"button\" data-bi-cN=\"Collapse all\">Collapse all<\/button>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<ul class=\"accordion\" id=\"accordion-1\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-2\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-2\">\n\t\tDemo 1\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-2\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p><strong>Demo 1: Synthesizing Web Search Result<\/strong><\/p>\n<p>&#8220;Driving directions to Beijing Railway Station. Head south on \u4e2d\u5173\u6751(Zhong Guan Cun)\u5357\u5927\u8857(Nan Da Jie), then toward \u5927\u6167\u5bfa\u8def(Da Hui Si Lu), turn left at \u767d\u77f3\u65b0\u6865(Bai Shi Xin Qiao), continue onto \u897f\u76f4\u95e8(Xi Zhi Men)\u5916\u5927\u8857(Wai Da Jie)\u3002&#8221; <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-demo1.wav\" target=\"_self\">Click to Play<\/a><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-3\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-3\">\n\t\tDemo 2\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-3\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p><strong>Demo 2: Translating Rick Rashid\u2019s English Speech to Many Languages <\/strong><\/p>\n<p><span id=\"c6af8267-c5d9-4f18-bec0-75c5d74692a5\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imagec6af8267-c5d9-4f18-bec0-75c5d74692a5\" class=\"alignleft\" style=\"height: 49px; width: 43px;\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-rick_picture.jpg\" alt=\"\" \/><\/span>Rick is\u00a0a native <strong>English<\/strong> speaker. Here is a sample of the recordings\u00a0by\u00a0his public speech.&#8220;You know I I never when I first came to Microsoft I would have never imagined that we would be doing that and research here. <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-100174.wav\" target=\"_self\">Click to Play<\/a><\/p>\n<p>Translating to <strong>Chinese<\/strong>&#8220;\u4f60\u77e5\u9053\u6211\u4ece\u6765\u6ca1\u6709\u5f53\u6211\u7b2c\u4e00\u6b21\u6765\u5230\u5fae\u8f6f\uff0c\u6211\u4ece\u6765\u6ca1\u6709\u60f3\u8c61\uff0c\u6211\u4eec\u5c06\u5728\u8fd9\u91cc\u505a\u548c\u7814\u7a76\u3002&#8221; <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-chinese.wav\" target=\"_self\">Click to Play<\/a><\/p>\n<p>Translating to <strong>Spanish<\/strong>&#8220;Sabes que nunca, cuando llegu\u00e9 por primera vez a Microsoft nunca habr\u00eda podido imaginar que \u00edbamos a estar haciendo eso y la investigaci\u00f3n aqu\u00ed.&#8221; <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-spanish.wav\" target=\"_self\">Click to Play<\/a><\/p>\n<p>Translating to <strong>Italian<\/strong>&#8220;Tu sai che io mai quando sono venuto a Microsoft non avrei mai immaginare che avremmo fatto questo e la ricerca qui.&#8221; <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-italian.wav\" target=\"_self\">Click to Play<\/a><\/p>\n<p>Rick&#8217;s\u00a0<strong>Spanish<\/strong> Speech:&#8220;Bienvenido a TechFest 2012, donde hoy se podr\u00e1 ver de primera mano c\u00f3mo Microsoft Research est\u00e1 estudiando las tendencias tecnol\u00f3gicas clave que definir\u00e1n el siglo 21.&#8221;<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-rick_mx_00000.wav\" target=\"_self\">Click to Play<\/a><\/p>\n<p>Rick&#8217;s\u00a0<b>Chinese<\/b> Speech:&#8220;\u65b0\u5e74\u597d\uff0c\u6b22\u8fce\u5927\u5bb6\u6765\u5fae\u8f6f\u4e9a\u6d32\u7814\u7a76\u9662\uff0c\u6211\u4e3a\u5927\u5bb6\u8bfb\u4e00\u9996\u8bd7\uff0c\u6211\u7684\u4e2d\u6587\u8bf4\u5f97\u4e0d\u5927\u597d\uff0c\u8bf7\u5404\u4f4d\u591a\u591a\u6279\u8bc4\u548c\u6307\u6559\u3002\u674e\u767d\u7684\u4e0b\u6c5f\u9675\u671d\u8f9e\u767d\u5e1d\u5f69\u4e91\u95f4\uff0c\u5343\u91cc\u6c5f\u9675\u4e00\u65e5\u8fd8\u3002\u4e24\u5cb8\u733f\u58f0\u557c\u4e0d\u4f4f\uff0c\u8f7b\u821f\u5df2\u8fc7\u4e07\u91cd\u5c71\u3002\u8c22\u8c22\u5927\u5bb6\uff01&#8221;\u00a0\u00a0<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-rick_mandarin.wav\" target=\"_self\">Click to Play<\/a><\/p>\n<p>Rick&#8217;s\u00a0<strong>Italian<\/strong> Speech:&#8220;A partire dal prossimo mese, cominceremo un book club in italiano che prender\u00e0 in considerazione libri di scrittori italiani contemporanei. Il primo incontro si terr\u00e0 marted\u00ec 6 marzo alle 5 pomeridiane nell&#8217;aula 306 della Casa Italiana Zerilli-Marim\u00f2, per discutere sia questioni di logistica che la scelta dei libri che saranno considerati. Il libro che sara discusso il 5 marzo sara &#8216;Che La festa cominci&#8217; di Niccol\u00f2 Ammaniti.&#8221; <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-rick_it_new.wav\" target=\"_self\">Click to Play<\/a><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-4\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-4\">\n\t\tDemo 3\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-4\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p><b>Demo 3:\u00a0 <\/b><strong>Craig&#8217;s 3D Avatar with Lip-synced TTS Voices<\/strong><\/p>\n<p>Craig is our Chief Research & Strategy officer Craig Mundie. He is a native English speaker.\u00a0He\u00a0doesn&#8217;t speak Chinese, but he is speaking Chinese!\u00a0\u00a0<a title=\"\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-intro.wav\" target=\"_self\">A Short Introduction<\/a><\/p>\n<p><span id=\"571e43bc-a5b6-4bfa-8618-1d81d6a9bdc3\" class=\"ImageBlock fn\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-craigengv3.wmv\" target=\"_self\"><img decoding=\"async\" id=\"Image571e43bc-a5b6-4bfa-8618-1d81d6a9bdc3\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-craig_eng.jpg\" alt=\"\" \/><\/a><\/span><\/p>\n<p>With English TTS\u00a0 <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-craigengv3.wmv\" target=\"_self\"><b>Click to Play<\/b><\/a><\/p>\n<p><span id=\"bddd8ed9-ffff-41f3-9bbc-455778f1f269\" class=\"ImageBlock fn\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-craigchnv2.wmv\" target=\"_self\"><img loading=\"lazy\" decoding=\"async\" id=\"Image6f5571da-338c-49f0-a20a-5b884d24cb3b\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-craig_chn.jpg\" alt=\"\" width=\"252\" height=\"213\" \/><\/a><\/span><\/p>\n<p>With Chinese TTS <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-craigchnv2.wmv\" target=\"_self\"><b>Click to Play<\/b><\/a><b><\/b><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-5\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-5\">\n\t\tDemo 4\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-5\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p><b>Demo 4<\/b><\/p>\n<p>Kit&#8217;s recorded\u00a0Chinese speech. <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-fig.13_top.wav\" target=\"_self\">Click to Play<\/a><span id=\"eef395f5-7c1f-4664-8281-58b86bee44f9\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imageeef395f5-7c1f-4664-8281-58b86bee44f9\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-top.jpg\" alt=\"\" \/><span id=\"ImageCaptioneef395f5-7c1f-4664-8281-58b86bee44f9\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>The same sentence synthesized\u00a0by\u00a0TTS built only with his English data. <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-fig.13_bottom.wav\" target=\"_self\">Click to Play<\/a> <span id=\"ed32f430-3866-4bcb-be1d-5ee0c9a91729\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Imageed32f430-3866-4bcb-be1d-5ee0c9a91729\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-bottom.jpg\" alt=\"\" \/><span id=\"ImageCaptioned32f430-3866-4bcb-be1d-5ee0c9a91729\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>Synthesized\u00a0semantically unpredictable sentences (SUS)<\/p>\n<p>\u9694\u4ee3\u7684\u540e\u536b\u8981\u8fdd\u5baa\u540d\u8a89\u7684\u67e5\u8be2.\u00a0\u00a0\u00a0 <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-sus1.wav\" target=\"_self\">Click to Play<\/a>\u5bf9\u5916\u7684\u81ea\u6cbb\u8981\u7ec4\u88c5\u8d1f\u7684\u5f8c\u6bcd.\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-sus2.wav\" target=\"_self\">Click to Play<\/a>\u53e4\u5178\u7684\u7b7e\u8bc1\u8981\u6d89\u53ca\u7eff\u7684\u7cd6\u5c3f\u75c5.\u00a0\u00a0\u00a0 <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-sus3.wav\" target=\"_self\">Click to Play<\/a>\u957f\u671f\u7684\u4f5c\u6218\u8981\u62a5\u544a\u5e73\u5747\u7684\u5b9d\u5ea7.\u00a0\u00a0\u00a0 <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-sus4.wav\" target=\"_self\">Click to Play<\/a>\u6b63\u9762\u7684\u6c14\u8c61\u8981\u906d\u9047\u6bd2\u7684\u53f7\u53ec.\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-sus5.wav\" target=\"_self\">Click to Play<\/a><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-6\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-6\">\n\t\tAlgorithm\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-6\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p><b>Algorithm<\/b><\/p>\n<p><b><\/b><span id=\"20d71215-5954-4941-8e0c-a36f7c98fbb7\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image20d71215-5954-4941-8e0c-a36f7c98fbb7\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mixedlangtts-algorithm.jpg\" alt=\"\" \/><\/span><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t\t\n","protected":false},"excerpt":{"rendered":"<p>Voice user interface needs to output responses in Text-To-Speech (TTS) synthesized speech. Sometimes it is even more desirable to have the response in mixed languages, For example, in a foreign country, it would be convenient if a user of car-navigation system who is not fluent in that particular foreign language could hear instructions in mixed-codes, [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13554],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-170910","msr-project","type-msr-project","status-publish","hentry","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2012-02-21","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[],"msr_research_lab":[199560],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170910"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170910\/revisions"}],"predecessor-version":[{"id":216271,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/170910\/revisions\/216271"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=170910"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=170910"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=170910"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=170910"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=170910"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}