[Editor’s note: This content was written to highlight a particular event or moment in time. Although that moment has passed, we’re republishing it here so you can see what our thinking and experience was like at the time.]

Managing financial transactions can be a tricky business.

When Microsoft employees and external payees run into invoice or payment problems, they turn to our expert support agents for help. But how do those agents know what’s working for their customers and what isn’t?

“As a customer-obsessed organization, we simply want to know how well we’re serving our customers,” says Jovalene Teo, senior technical solution manager for the Center of Innovation within Finance Operations. “When you come to us for help, are you getting the help you need?”

Finance Operations knew feedback and user insights were crucial to a great customer experience, so they created opportunities to harvest data at every stage of a user’s support journey. But not everyone has the time or inclination to share their thoughts. For users who don’t provide feedback, the team has developed a way to unlock those insights through machine learning sentiment analysis.

[Find out how Microsoft is reimagining employee support with Azure. Learn how Microsoft rebooted its internal app support with Dynamics 365. See how Microsoft is streamlining its global customer call center system with Azure]

Making it easy for users to share their experience

Our procure-to-pay (P2P) process covers everything from onboarding new suppliers to invoicing and payments. Both internal employees and external suppliers engage with the P2P process, so a robust support structure ensures that people without a finance background can deal with any issues promptly.

Finance Operations wanted to standardize the user feedback experience across every support team that touches the P2P process. That would help our frontline agents serve users better and provide deeper, business-wide insights into pain points and support trends.

“The idea came up that we should build a system where all these teams work through the same platform,” says Dhwani Kamdar, senior product manager for Financial Experience Applications within Finance Operations. “That way, they can talk to each other, and every single ticket or inquiry that comes into Microsoft Finance is tracked.”

The conventional feedback process used external surveys where employees or suppliers initiated a support ticket by contacting the team via email. Once the ticket was complete, users received a link to an external site where they could provide feedback. That extra step was a serious barrier because it asked users to interrupt their workflow by traveling to an entirely different website.

“We thought about how we could integrate feedback into a CRM experience that agents can see and interpret as they’re working on the tickets,” says Rajiv Maheshwari, principal software engineering manager for Financial Experience Applications. “So we went from after-the-fact tracking to more dynamic, interaction-focused insights available to agents in real time.”

Their solution evolved into a new tool that integrated user feedback into the flow of email communications during the support ticket process. As a result, they could incorporate the results into UniFinance, a platform based on Dynamics 365 that 25 Microsoft Finance teams use for support and feedback management.

Integrated, dynamic feedback powers meaningful conversations

Finance Operations worked to implement best practices that would make providing feedback more natural to users. Through collaboration with 25 team leads who manage 850 support agents, the engineers worked to ensure that the tool’s interface was stylish, accessible by design, and smoothly integrated into the email support process.

First, they updated the depth of possible responses. What started as a binary “satisfied” or “dissatisfied” choice evolved into a five-point satisfaction scale. That provided a wider array of feedback and aligned with the highest industry standards.

Next, the team needed to integrate feedback into ongoing support conversations. Fortunately, existing Microsoft Outlook features provided the basis for dynamic, in-email feedback.

The tool collects two different categories of user feedback. The first is known as “active feedback.”

This tool enables the user to provide their feedback more readily and easily and for us to collect those insights. So instead of just post-mortem feedback, we collect insights during the conversation so we can actually work with it to improve our service.

––Jovalene Teo, senior technical solution manager, Center of Innovation, Microsoft Finance Operations

Active feedback provides the opportunity for ongoing sentiment collection from the end user. In the signature line of every support email, users can click on one of five icons—with faces ranging from angry to overjoyed—indicating how they feel about their experience within the flow of communication.

UniFinance routes that user feedback through Microsoft Dynamics 365 CRM, making it available to support agents so they can understand the customer’s level of satisfaction and whether any course corrections are necessary. If a user submits feedback on the lower end of the spectrum, the CRM automatically keeps the ticket open, allowing the agent to offer further support.

“This tool enables the user to provide their feedback more readily and easily and for us to collect those insights,” Teo says. “So instead of just post-mortem feedback, we collect insights during the conversation so we can actually work with it to improve our service.”

When the support ticket is closed, the system requests a more extensive response, known as “closure feedback.” At this stage, users receive a final email featuring an Outlook actionable card that asks for an overall assessment on a five-point scale as well as a field prompting a textual response.

Closure feedback represents the end user’s final sentiment on the ticket’s resolution. Their input contributes to the support team’s key performance metric: customer satisfaction (CSAT).

On top of providing concrete guidance for frontline support agents, aggregating feedback data through Dynamics 365 allows the Finance Operations team to collect, analyze, and surface insights to business owners throughout the group.

Since the implementation of the tool, the participation rate for users providing feedback has soared. Before in-email feedback, the response rate was just 0.01 percent. That rate has now skyrocketed to 8 percent.

Completing the picture with machine learning

But what about the other 92 percent of cases? Finance Operations knew valuable insights were hiding in those interactions as well.

“Actual customer feedback is more common when the experience is on the high end or the low end, so either I’m super happy as a customer and I give you five stars, or I’m very frustrated and I give you one,” Maheshwari says. “But we were missing out on the middle of the spectrum, so we didn’t have the insights to improve that experience.”

Even when the customer doesn’t give feedback, the machine is giving us some insight. So we can still course-correct and make sure we’re doing our best to serve our customers..

––Dhwani Kamdar, senior product manager, Financial Experience Applications, Microsoft Finance Operations

Their solution leveraged machine learning to analyze the sentiment of user emails. Within the tool, Microsoft Azure Cognitive Services compares users’ language against established, extensively trained positive and negative sentiment language models. Then the system returns a positive or negative sentiment score to the support agent.

“Even when the customer doesn’t give feedback, the machine is giving us some insight,” Kamdar says. “So we can still course-correct and make sure we’re doing our best to serve our customers.”

Interpreting language and tone is complicated, so support agents use these sentiment scores cautiously. Even if a message receives a positive score, they still review it carefully. But when the system throws a negative score, agents know they should pay special attention to that message and potentially modify their approach.

Between dynamic user feedback and machine learning sentiment analysis, support agents now have powerful tools in their corner to help them provide excellent service.

Intelligent outcomes in finance

UniFinance has become a powerful engine for improving the finance support experience, providing both frontline and business-level insights.

“Just having sentiment analysis enabled wasn’t the only goal,” Kamdar says. “The goal was to use that data for building Power BIs and executive-level dashboards to see where we are and calculate customer satisfaction. So there are a lot of discussions around the data analytics and insights to create those dashboards and make sure we don’t stop at building the five smileys.”

The team is already piloting ways to use machine learning beyond sentiment analysis. Dynamic categorization is an exciting new feature that engages Azure Cognitive Services to scan user messages for keywords.

It associates those keywords with primary topics like “invoicing” and then breaks them down into increasingly granular subtopics such as “invoice payment failure.” Over time, the goal is to identify trends, build predictability, and even automate responses, saving finance professionals time to focus on more significant challenges.

In the meantime, UniFinance’s email feedback feature continues to improve the user experience through streamlined customer feedback, powerful data insights, and machine learning.

Here are some key lessons to keep in mind if you’re planning to rework the support process at your company:

• Start by reimagining the possibilities: When you work in a traditional field, moving out of the status quo is the goal.
• Think in terms of the customer experience because that’s where you’ll find your innovations.
• Design matters: Never underestimate how aesthetics can drive results.
• Keep an open mind: The more you talk to users and the more demos you run, the better and faster your work will be.
• When you’re working with machine learning and text, start small with specific scenarios, find success, then build on it.
• Having a diverse team is very advantageous because they bring skills, languages, and insights you won’t find elsewhere.

• Learn how Microsoft is integrating ServiceNow with Microsoft Azure Monitor.
• Find out how Microsoft is modernizing its support experience with ServiceNow.
• Learn how Microsoft is enabling a modern support experience.

Tags: AI and Machine Learning, Dynamics 365, Microsoft Azure