Next generation document and image processing with GPT

One of the most game-changing updates in AI Builder is the new multi-modal content processing capability, which allows businesses to handle various data types —including text, document, and images—within a single natural language instruction. This integration greatly simplifies the process of crafting AI actions for intelligent document processing. Unlike traditional machine learning models that require extensive training and expertise, AI Builder’s multi-modal processing can be set up using natural language, and without training data.

Structured JSON outputs now generally available

With AI Builder’s new structured JSON output feature, businesses gain a more deterministic and reliable way to handle generated content. Unlike traditional outputs that might vary in format and structure, JSON outputs are now generated with a consistent predefined schema that you define at design time, ensuring that every result aligns precisely with expected data formats. This determinism in content generation means your automated workflows will produce predictable, structured data every time, and those results will be readily integrated as variables in your Power Automate workflow for downstream processing. This feature is especially beneficial when integrating AI Builder outputs into other systems, as it reduces the need for additional data cleaning or transformation. Whether feeding data into a CRM, database, or custom application, you can ensure that data is generated in a structured and predictable way. 

Dataverse grounding is now generally available

The new Dataverse grounding capability empowers Generative AI models with up-to-date, contextually relevant information from your organization’s own data. By employing Retrieval-Augmented Generation (RAG), AI Builder integrates directly with Dataverse to pull in real-time, business-specific knowledge, giving models a deeper understanding of your unique context.

This RAG-enhanced approach allows AI models to dynamically retrieve and incorporate relevant data from Dataverse during processing, enabling Q&A scenarios with accurate responses that are also highly contextualized to your organization. For instance, models can instantly access customer histories, product information, and up-to-the-minute operational data, enriching outputs with knowledge directly grounded in your business reality.

As part of this feature becoming generally available, the capability has been enhanced to support multiple tables, including tables with large amounts of data records.

Model selection: GPT-4o and GPT-4o Mini

AI Builder offers flexibility in model selection, allowing users to choose between the powerful GPT-4o and the cost-effective GPT-4o Mini. This adaptability enables customization based on specific project requirements and resource considerations. 

Prompt library

The prompt library feature provides a collection of pre-designed prompts, serving as templates to expedite the creation of AI models. This resource accelerates development and ensures best practices are followed in prompt engineering. Users can modify templates to suit specific needs, adjusting language, tone, and detail to match organizational standards and requirements. The templates cover key areas like document extraction, data transformation, and content generation, making it easy for users to find the right starting point for their intelligent automation goals. 

Use cases to get started

With AI Builder, you don’t need to be a coding expert to leverage the power of AI in your daily operations. By creating prompts that can extract information from documents, classify emails, and even analyze images, you can empower your team to work smarter, not harder. AI Builder works hand-in-hand with other Power Platform tools like Power Automate, Copilot Studio and Power Apps, enabling users to create intelligent workflows and applications that are customized to meet the needs of any business.

For example, let’s say your team frequently processes customer inquiries. Using AI Builder and Power Automate, you can design a workflow that analyzes the content of incoming emails, categorizes them by urgency or topic, and even suggests responses. This not only saves time but also keeps communication consistent and efficient. By setting up this automated flow, businesses can stay on top of customer interactions and ensure timely responses.

Are you ready to bring AI to your organization?

The AI Builder Documentation provides a comprehensive guide on setting up AI Builder, configuring models, and integrating it with tools like Power Automate and Power Apps.

For step-by-step tutorials, the Microsoft Learn AI Builder Learning Path is an excellent place to start. It covers everything from foundational concepts to advanced use cases, helping users of all levels become proficient in leveraging AI Builder to automate and optimize workflows.

For guidance and samples on using the multi-modal content processing and other prompts with Power Apps, you can use the Creative AI Kit in App Source or GitHub. The new samples that use multi-modal prompts include AI Image Diff, AI Describe Image, AI Fields from Image and AI Generate QnA.

Join our Ignite sessions and read about other exciting announcements for Microsoft Power Platform:

• Automate e-mail, doc, and image processing with GPT in Power Automate with Gwenael Bego and Maria Gutierrez
• Revolutionize the way you work with Power Automate and AI with Joe Fernandez and Heather Orta-Olmo
• Unlock innovation with practical strategies in Power Automate with Jonathan Kendall
• See our Dataverse Ignite session (virtual and in person): Explore Ignite 2024: Unlock enterprise knowledge with Dataverse – Microsoft Power Platform Blog

The post Boost productivity: Automate emails, documents, and images with Microsoft AI Builder appeared first on Microsoft Power Platform Blog.

Keep reading to find out more about how to craft Power Automate actions that will transform your business processes with the power of GPT-4, including the ability to automate GPT Prompts with enterprise knowledge stored in Dataverse, generate structured outputs with JSON formatting and adjust GPT model temperature settings.

Use the power of GPT to process content using your enterprise knowledge

Prompts are the natural language instructions that users provide to Large Language Models (LLMs) to produce a specific response. With AI Builder’s “Create text with GPT using a prompt” action, you can use prompts to make AI powered functions that can process and generate text content dynamically with GPT. Prompts for example can include instructions in natural language that will guide your cloud flow or desktop flow to accurately process an incoming e-mail, analyze the content of a document or automate insights extraction from your data. Today, we are adding more features to Prompts to enable users to connect their enterprise knowledge and generate trusted outputs.

Enterprise knowledge is crucial for building effective prompts because it ensures contextual relevance, accuracy, and precision. Understanding the specific domain language and business context allows for crafting prompts that are aligned with the company’s goals and processes. This leads to streamlined workflows, consistent outputs, and enhanced user experiences by providing tailored and intuitive interactions.

In the Prompt Builder interface, a new option to add data sources is now available, allowing you to select a Dataverse table and its related entities as knowledge associated with your prompt. The fields selected will be used by the Generative AI model to expand its knowledge base with current enterprise specific data that will help provide the most contextual response to the prompt.

The new interface allows you to filter the data to add to your knowledge source, and to reference the associated knowledge inline within your natural language instruction. This will unlock many new enterprise specific actions, like for example generating summaries of account interactions based on Dataverse data, classifying content based on custom categories stored in Dataverse, or suggesting the most appropriate actions to respond to a customer query based on historical data stored in Dataverse. Learn more about prompt data grounding in our documentation.

GPT-4 Model selection and temperature settings

Another new feature in Prompts is the ability to select which model to use to process a given action, and to control the Temperature associated with the given model. Using the model selector, users can select to opt-in to use GPT-4o (rolling out in the coming weeks), GPT-4 (gpt-4-32k-0613 available today) or GPT 3.5 (gpt-3.5-turbo-0613 default model) for individual prompts. This allows for greater flexibility to optimize costs and performance associated to Prompts based on what tasks will be performed by the prompt and the context window required to perform the task.

Alongside the model selector, a slider has also been added to control the temperature of the Generative AI model, allowing you to adjust the model to be more deterministic or more creative to perform a given task. Learn more about model selection and parameters in our documentation.

JSON output formatting for reliable end-to-end automation

By default, prompts generate a text output allowing you to automate the process of creating text-based data that can be used across an end-to-end workflow. To optimize how to parse and re-use text data created by GPT, we’re introducing a new feature that allows you to generate a structured text output formatted in JSON. This new capability will allow you to ensure that the generated data always follows the same structured schema, so that you can reliably use it in downstream actions in your flow. This will be particularly useful when you want GPT to generate structured content like a project schedule, or extract data from a document like the different fields associated to an invoice or purchase order.

To trigger structured output formatting, you can select JSON as an output format in the Prompt Builder. The target JSON structure will be auto-detected based on the prompt instructions provided, but you can also edit the schema to force a given output structure.

Once you have defined your desired schema and saved your prompt, the output format will be locked, ensuring that your prompt will always generate the same structured format and that the output fields will automatically be available for use in downstream actions in Power Automate:

Get started

Find the most recent documentation for our new features and learn how to create more effective automation with Generative AI Prompts:

• Create a prompt | Microsoft Learn
• Use your own data in a prompt | Microsoft Learn
• Model selection and temperature settings | Microsoft Learn
• Change the output of your prompt | Microsoft Learn
• Distribute your prompt using a solution | Microsoft Learn
• Human Review for automation with a prompt | Microsoft Learn

The post Generative AI Prompts to automate content processing appeared first on Microsoft Power Platform Blog.

What are the benefits of Invoice Customization?

AI Builder is already equipped with a pre-configured AI model capable of extracting standard fields from invoices. This model is finely tuned to identify typical invoice components such as invoice ID, invoice date, amount due, and various other standard field types. If your invoices include custom fields beyond the standard ones, you’ll need to create a specialized Document Processing AI model and label a training dataset with sample documents. This process trains the AI model to identify and extract these extra fields.

Instead of starting from scratch and labeling every field for extraction, you can now opt to use an existing invoice model as your starting point:

This method allows you to efficiently extract standard invoice fields and simplifies the training phase, concentrating only on adding the extra fields required to develop your custom invoice processing model. Customers like Ashley Francis, a Certified Public Accountant (CPA) with The Francis Group practice in Seattle, are already benefiting from this new solution to transform traditional processes, increase efficiency, and save time.

“Finding out about all the things in the Power Platform has been amazing. Like this is technology that I didn’t even know existed. I want so much more time to learn everything on the Power Platform and AI because it can do so much for my profession.”

– Ashley Francis, Certified Public Accountant, The Francis Group

Read the full story: Certified Public Accountant transforms accounting processes using Power Automate 

What are the benefits of Document field types?

In an end-to-end workflow for document processing, it’s crucial to extract not just the field’s name and value but also its type and format (such as a number, date, checkbox, table, or text field). The newly introduced typed fields feature enables the extraction of fields identified as numbers and dates, allowing you to specify their anticipated format while training a custom document processing model. This enhancement aids in standardizing the output of document processing and boosts the precision of document extraction performance.

Get started with invoice customization and field types

Leverage these additional resources to learn more about these new document processing features:

• Create a document processing custom model in AI Builder | Microsoft Learn
• Extract dates and numbers from documents | Microsoft Learn
• Start using AI Builder

The post Custom processing for invoice and document field types generally available appeared first on Microsoft Power Platform Blog.

What are Prompts? 

Prompts are the instructions that we can give to large language models (LLMs) like GPT to perform specific tasks. The more precise the instructions we give, the better GPT performs at executing the tasks. To allow companies to create effective prompts we’ve created a new tool, Prompt Builder, a low-code prompt engineering interface that is seamlessly integrated in Power Automate. 
Customers are already excited about the perspectives that Generative AI can bring to Power Automate. Epiq, a global leader in technology-enabled services, has set a new standard with its innovative use of Generative AI by applying it to transform their HR operations. “With AI Builder in Power Automate, we added a model that cleans up formatting of new employee submission forms. This allows us to process more efficiently at scale – and it works flawlessly.” says Colt Coan, Senior Manager of Rapid Development, Epiq. Want le learn more? Read Epiq’s customer story article.

How to build a Prompt 

To access Prompt Builder, start by adding the “Create text with GPT using a prompt” action from the AI Builder connector:  

You’ll then be able to use Prompt Builder, AI Builder’s new low-code prompt engineering interface, to create your instructions to GPT that will be executed during the workflow. 
We’ve added examples of Prompts to help you get started. Common scenarios include: 

• Classify Text: E-mail classification and case routing. 
• Summarize Text: Summarization of e-mails, contracts or meeting notes. 
• Extract action points from text: Identify upsell/cross-sell opportunities and next best actions. 
• Extract information from text: Understand customer requests to automate sales quotes and proposals. 
• Respond to a complaint: Draft e-mail responses to fast-track support case management. 

When creating a prompt, you’ll be able to add dynamic values in the instructions so that the Prompt can automatically be executed over new data, for example an incoming e-mail. 


In the body of the prompt, you’ll be able to craft your GPT instructions, add multiple dynamic values, define the output format you want, and test your prompt with sample data:

Once you’re happy with the way your prompt performs, you can save your prompt for future use. You’ll be able to access all your saved Prompts along with a library of pre-built Prompts from the new AI Prompts page. Prompts can be shared with other users, added to solutions for cross-environment deployment, and invoked across other products like Power Apps or Copilot Studio. To learn more about how to create effective Prompts, refer to our new Prompt engineering guide.

Human in the loop validation 

Content generated by AI can include mistakes and surprises, so it’s important to make sure it is accurate and appropriate before using it. To facilitate this process, you can include GPT-generated text in an approval flow that allows you to validate and edit the content before using it: 

Approval workflows to validate GPT content are highly recommended but no longer mandatory now that the feature is generally available. 

Available in the United States, Australia, UK, and Europe  

GPT Prompts and Prompt Builder will be generally available this week across the United States, Australia, the United Kingdom, and Europe. 
GPT Prompts currently leverage the GPT-3.5-Turbo model from Azure’s OpenAI Service, meaning your prompts and outputs: 

• are NOT available to other customers. 
• are NOT available to OpenAI. 
• are NOT used to improve OpenAI models. 
• are NOT used to improve any Microsoft or 3rd party products or services. 
• are NOT used for automatically improving Azure OpenAI models for your use in your resource
   

Please reference the Azure OpenAI Service data, privacy and security documentation for details on how your data will be processed, used and stored with GPT Prompts. 

Get Started

• Try Prompt Builder: https://aka.ms/tryprompts  
• Prompt Engineering Guide: https://aka.ms/promptguide 
• Documentation: Prompts Overview | Microsoft Learn
• Learning Videos: Overview Video | GPT Prompts with Excel Video 
• Customer Story: Epiq – Revolutionizing HR operations with Power Platform + Generative AI 
• Pricing for GPT Prompts can be found in the Microsoft Power Platform Licensing Guide 
• Governance: Learn about administrating Generative AI features and cross region availability  

The post AI Builder GPT Prompts are generally available appeared first on Microsoft Power Platform Blog.