Deploying our new ‘game changing’ Interpreter agent in our meetings at Microsoft

The new Interpreter agent in Microsoft Teams meetings is transforming the lives of our employees who must speak a second language at work.

We in Microsoft Digital, our internal IT organization, have enabled Interpreter across Microsoft, providing our employees with live, real-time interpretation in their meetings.

Interpreter is especially helpful because it goes beyond regular interpretation.

With the speaker’s consent, it can simulate their own voice—helping create a more personal, natural, and inclusive experience. The result is meetings where every participant speaks and hears everything spoken in their preferred language.

Equally as important, speakers no longer need to struggle through sharing their thoughts in another language—they are free to speak their mind in their own language.

“I can think and speak at the speed of my first language,” says Masato Esaka, a business program manager at Microsoft, and one of the early adopters of Interpreter. “I can speak smoothly and articulate my thoughts clearly without worrying about what I sound like in English.”

Interpreter is enabling our employees to communicate with each other more effectively.

“Hearing my voice speaking Japanese for the first time was surreal,” says Petra Glattbach, a senior business program manager in Microsoft Digital. “This agent is going to completely change the way we—and Microsoft customers—have multilingual meetings.”

When conversations happen outside someone’s native language, it can be challenging to communicate clearly. For global teams, that can lead to feeling less included and not fully understood in meetings.

“Historically, human interpreters have bridged the language gap in multilingual meeting scenarios, whether in person or online,” says Harin Lee, a principal product manager in the Teams product group. “However, as technology has advanced, it’s made more sense to translate using tools. It’s too difficult—and expensive—to think about having human interpreters in place for every multilingual meeting scenario that plays out in any global organization. There are too many meetings, too many languages, and not enough resources.”

Recent advances in technology have certainly helped. Many collaboration-focused apps—including Microsoft Teams—offer dynamic translation of meeting transcripts, but this didn’t resolve all the pain points. Key struggles in multilingual meetings persist:

• Traditional translation tools often feel impersonal and disconnected for participants to feel fully included.
• Relying on translation tools or interpreters can introduce technical difficulties such as poor audio quality or delays in translation. 
• Some participants may feel less confident speaking in a non-native language, leading to reduced participation and engagement. This can result in valuable insights and perspectives being overlooked.

{Learn more about our public preview of Interpreter.}

Introducing Interpreter in Microsoft Teams

Interpreter enables participants in Teams meetings speaking different languages to converse and hear the meeting in their preferred language with the help of an AI-based interpreter.

“It’s like being on the floor at the United Nations, but as the words are spoken in a myriad of languages, every person can easily understand everyone else,” Glattbach says. “Languages from around the world are spoken, except with Interpreter, the AI listens and speaks instead of human interpreters. If I speak French and a Japanese speaker is in the same meeting, they can set Interpreter to hear what I say in Japanese, in near real time, in my voice!”

The agent uses advanced text-to-speech (TTS) and speech-to-text (STT) capabilities developed using Microsoft Azure AI services to provide interpretation services that leverage AI-driven interpretation capabilities. The Microsoft Teams product group responsible for Interpreter has been developing the agent and testing it internally at Microsoft. It’s a perfect example of our Customer Zero approach.

As Customer Zero, Microsoft Digital is the first adopter of almost all Microsoft products. Being Customer Zero means a deep partnership between our end users and our product engineering teams to envision the right experiences, co-develop innovative solutions, and both listen to and act on insights we gather from our employees. We work together to stay grounded in the way our employees use our products every day, so your employees can benefit from our insights.

Glattbach is appreciative of the Customer Zero experience with Interpreter.

“I’ve never said ‘wow’ so many times in a single meeting,” she says, recalling her first time in a Teams meeting where Interpreter was being used. “Hearing multiple people speaking several different languages in the same meeting and the AI voicing over each of them in my language was incredible.”

Glattbach, Chanda Jensen, a senior product manager, and our full team in Microsoft Digital are responsible for deployment, testing, change management, and adoption at Microsoft. They help evangelize the products to their fellow employees so they can benefit from using them.

When the Interpreter is enabled at the tenant level, participants can use the agent where and when they desire in the meeting. Teams notifies participants when Interpreter is enabled, and participants can opt to turn it on for themself as well. When they do, they’re prompted to select their preferred language from a dropdown menu on their first time using this agent. For subsequent meetings, the system defaults to the language used in previous meetings if supported by AI interpretation, ensuring a seamless and intuitive experience for participants.

One it’s enabled, Interpreter allows participants to choose the language they want to hear the meeting in. The AI models, leveraging TTS and STT technologies hosted in Azure AI Services, deliver near-real-time speech-to-speech translation through an Interpreter in the meeting. Optional voice simulation provides an audio replication of the original speech, using the speaker’s voice. This means that the spoken words are translated into the chosen language with a small delay, replicating the voice of the original speaker and maintaining a conversational, natural language interaction for all meeting participants.

Participants can adjust the volume of the Interpreter in relation to the original audio of the other participants, which allows for a customized experience for each participant depending on their needs.

Turning Interpreter on

Providing governance and control

Giving everyone the ability to hear their own voice speak another language is our goal with the Interpreter, but we’re also aware of potential concerns over impersonation and using biometric data, like someone’s voice.

“Privacy and impersonation are legitimate concerns,” says Tori Lang, a senior product manager in Microsoft Digital. “We’ve worked hard to ensure the proper controls are in place to let both the user and organization decide what they’re willing—and not willing—to share.”

Lang’s team has worked with governance and security experts within Microsoft Digital to examine and address concerns about Interpreter and address those concerns with controls and transparency in the Teams app. Some of these controls include:

• User Consent. Before simulating the voice of the speaker, consent must be obtained. If consent is not given, the default voice provided by the platform will be used.
• Admin controls. Administrators have the ability to manage and configure Interpreter settings. They can enable or disable the agent for all participants or specific individuals during a meeting.
• Notifications. Participants are notified when Interpreter is turned on or off. This ensures transparency and keeps everyone informed about the status of the interpretation.

“I would say our goal is to remove language barriers and enhance meeting inclusivity more broadly,” Lang says.

Our team in Microsoft Digital worked closely with representatives of our works councils and product engineering teams to ensure compliance and address concerns related to data privacy and user control.

“This partnership has created a mutual understanding of features and controls and has helped the product team to strike a balance between ease of use and compliance,” Jensen says.

Changing the game with Interpreter

Now that we’ve deployed it the full company here at Microsoft, our next step is to let employees know it’s there and to show them how powerful it is.

“Interpreter really is a game-changer,” Glattbach says. “It fundamentally transforms how users participate and engage in Teams meetings where their preferred language isn’t spoken. It democratizes language for all users at Microsoft and creates an inclusive and supportive meeting environment.”

In addition to these general scenarios, the Teams product group has identified many scenarios within Microsoft that have been transformed, or can be transformed, with this new level of inclusivity.

“Interpreter helps users communicate without barriers,” says Ritika Gupta, a principal group program manager for the Microsoft Teams product group. “It’s a cost-effective option for multilingual support that applies across so many scenarios. We’re just beginning to uncover the specific use cases here at Microsoft.”

Gupta lists several use cases to which Microsoft users have applied Interpreter throughout the pilot and test deployment phases:

• Enabling expression and comfort. In primarily bilingual workforces in countries or regions like India, China, and Japan, many employees speak and understand English but are more comfortable when they can speak and hear in their preferred language, where expression and nuanced understanding are easier.
• Connecting with customers, stakeholders, and partners in non-English speaking markets and regions.
• Enabling agents handling support and sales meetings to support multiple geographies and languages.
• Supporting executives communicating and connecting with their globally dispersed team members.

Jensen, who worked closely with the product to lead our deployment of Interpreter across the company, believes we’re barely scratching the surface of how Interpreter can impact Microsoft and its customers.

“This agent is going to shape the way we communicate at Microsoft, connecting colleagues, partners, and customers worldwide so that everyone is able to speak in their preferred language.  In the foreseeable future, I can imagine not being able to hold Teams meetings without Interpreter,” she says. “It is going to quickly become a critical part of the way we communicate as a global organization.”

Jensen and her team at Microsoft Digital are responsible for Customer Zero testing, feedback, and deployment for Microsoft employees. The team worked with product engineering on deploying Interpreter in a ringed model to Microsoft employees.

Investing in multilingual communication

The Customer Zero effort between the Teams product group and Microsoft Digital is constantly evolving and expanding. The team has deployed Interpreter to the full company and is currently working on extending Interpreter to include more languages and optimizing translation performance in the underlying machine learning models.

“AI has completely changed the playing field in multilingual communications,” Lee says. “This is a defining moment in communications, and we’re committed to ensuring that every participant in any Teams meeting around the world is able to express themselves and feel understood.”