Next generation document and image processing with GPT

One of the most game-changing updates in AI Builder is the new multi-modal content processing capability, which allows businesses to handle various data types —including text, document, and images—within a single natural language instruction. This integration greatly simplifies the process of crafting AI actions for intelligent document processing. Unlike traditional machine learning models that require extensive training and expertise, AI Builder’s multi-modal processing can be set up using natural language, and without training data.

Structured JSON outputs now generally available

With AI Builder’s new structured JSON output feature, businesses gain a more deterministic and reliable way to handle generated content. Unlike traditional outputs that might vary in format and structure, JSON outputs are now generated with a consistent predefined schema that you define at design time, ensuring that every result aligns precisely with expected data formats. This determinism in content generation means your automated workflows will produce predictable, structured data every time, and those results will be readily integrated as variables in your Power Automate workflow for downstream processing. This feature is especially beneficial when integrating AI Builder outputs into other systems, as it reduces the need for additional data cleaning or transformation. Whether feeding data into a CRM, database, or custom application, you can ensure that data is generated in a structured and predictable way. 

Dataverse grounding is now generally available

The new Dataverse grounding capability empowers Generative AI models with up-to-date, contextually relevant information from your organization’s own data. By employing Retrieval-Augmented Generation (RAG), AI Builder integrates directly with Dataverse to pull in real-time, business-specific knowledge, giving models a deeper understanding of your unique context.

This RAG-enhanced approach allows AI models to dynamically retrieve and incorporate relevant data from Dataverse during processing, enabling Q&A scenarios with accurate responses that are also highly contextualized to your organization. For instance, models can instantly access customer histories, product information, and up-to-the-minute operational data, enriching outputs with knowledge directly grounded in your business reality.

As part of this feature becoming generally available, the capability has been enhanced to support multiple tables, including tables with large amounts of data records.

Model selection: GPT-4o and GPT-4o Mini

AI Builder offers flexibility in model selection, allowing users to choose between the powerful GPT-4o and the cost-effective GPT-4o Mini. This adaptability enables customization based on specific project requirements and resource considerations. 

Prompt library

The prompt library feature provides a collection of pre-designed prompts, serving as templates to expedite the creation of AI models. This resource accelerates development and ensures best practices are followed in prompt engineering. Users can modify templates to suit specific needs, adjusting language, tone, and detail to match organizational standards and requirements. The templates cover key areas like document extraction, data transformation, and content generation, making it easy for users to find the right starting point for their intelligent automation goals. 

Use cases to get started

With AI Builder, you don’t need to be a coding expert to leverage the power of AI in your daily operations. By creating prompts that can extract information from documents, classify emails, and even analyze images, you can empower your team to work smarter, not harder. AI Builder works hand-in-hand with other Power Platform tools like Power Automate, Copilot Studio and Power Apps, enabling users to create intelligent workflows and applications that are customized to meet the needs of any business.

For example, let’s say your team frequently processes customer inquiries. Using AI Builder and Power Automate, you can design a workflow that analyzes the content of incoming emails, categorizes them by urgency or topic, and even suggests responses. This not only saves time but also keeps communication consistent and efficient. By setting up this automated flow, businesses can stay on top of customer interactions and ensure timely responses.

Are you ready to bring AI to your organization?

The AI Builder Documentation provides a comprehensive guide on setting up AI Builder, configuring models, and integrating it with tools like Power Automate and Power Apps.

For step-by-step tutorials, the Microsoft Learn AI Builder Learning Path is an excellent place to start. It covers everything from foundational concepts to advanced use cases, helping users of all levels become proficient in leveraging AI Builder to automate and optimize workflows.

For guidance and samples on using the multi-modal content processing and other prompts with Power Apps, you can use the Creative AI Kit in App Source or GitHub. The new samples that use multi-modal prompts include AI Image Diff, AI Describe Image, AI Fields from Image and AI Generate QnA.

Join our Ignite sessions and read about other exciting announcements for Microsoft Power Platform:

• Automate e-mail, doc, and image processing with GPT in Power Automate with Gwenael Bego and Maria Gutierrez
• Revolutionize the way you work with Power Automate and AI with Joe Fernandez and Heather Orta-Olmo
• Unlock innovation with practical strategies in Power Automate with Jonathan Kendall
• See our Dataverse Ignite session (virtual and in person): Explore Ignite 2024: Unlock enterprise knowledge with Dataverse – Microsoft Power Platform Blog

The post Boost productivity: Automate emails, documents, and images with Microsoft AI Builder appeared first on Microsoft Power Platform Blog.

Keep reading to find out more about how to craft Power Automate actions that will transform your business processes with the power of GPT-4, including the ability to automate GPT Prompts with enterprise knowledge stored in Dataverse, generate structured outputs with JSON formatting and adjust GPT model temperature settings.

Use the power of GPT to process content using your enterprise knowledge

Prompts are the natural language instructions that users provide to Large Language Models (LLMs) to produce a specific response. With AI Builder’s “Create text with GPT using a prompt” action, you can use prompts to make AI powered functions that can process and generate text content dynamically with GPT. Prompts for example can include instructions in natural language that will guide your cloud flow or desktop flow to accurately process an incoming e-mail, analyze the content of a document or automate insights extraction from your data. Today, we are adding more features to Prompts to enable users to connect their enterprise knowledge and generate trusted outputs.

Enterprise knowledge is crucial for building effective prompts because it ensures contextual relevance, accuracy, and precision. Understanding the specific domain language and business context allows for crafting prompts that are aligned with the company’s goals and processes. This leads to streamlined workflows, consistent outputs, and enhanced user experiences by providing tailored and intuitive interactions.

In the Prompt Builder interface, a new option to add data sources is now available, allowing you to select a Dataverse table and its related entities as knowledge associated with your prompt. The fields selected will be used by the Generative AI model to expand its knowledge base with current enterprise specific data that will help provide the most contextual response to the prompt.

The new interface allows you to filter the data to add to your knowledge source, and to reference the associated knowledge inline within your natural language instruction. This will unlock many new enterprise specific actions, like for example generating summaries of account interactions based on Dataverse data, classifying content based on custom categories stored in Dataverse, or suggesting the most appropriate actions to respond to a customer query based on historical data stored in Dataverse. Learn more about prompt data grounding in our documentation.

GPT-4 Model selection and temperature settings

Another new feature in Prompts is the ability to select which model to use to process a given action, and to control the Temperature associated with the given model. Using the model selector, users can select to opt-in to use GPT-4o (rolling out in the coming weeks), GPT-4 (gpt-4-32k-0613 available today) or GPT 3.5 (gpt-3.5-turbo-0613 default model) for individual prompts. This allows for greater flexibility to optimize costs and performance associated to Prompts based on what tasks will be performed by the prompt and the context window required to perform the task.

Alongside the model selector, a slider has also been added to control the temperature of the Generative AI model, allowing you to adjust the model to be more deterministic or more creative to perform a given task. Learn more about model selection and parameters in our documentation.

JSON output formatting for reliable end-to-end automation

By default, prompts generate a text output allowing you to automate the process of creating text-based data that can be used across an end-to-end workflow. To optimize how to parse and re-use text data created by GPT, we’re introducing a new feature that allows you to generate a structured text output formatted in JSON. This new capability will allow you to ensure that the generated data always follows the same structured schema, so that you can reliably use it in downstream actions in your flow. This will be particularly useful when you want GPT to generate structured content like a project schedule, or extract data from a document like the different fields associated to an invoice or purchase order.

To trigger structured output formatting, you can select JSON as an output format in the Prompt Builder. The target JSON structure will be auto-detected based on the prompt instructions provided, but you can also edit the schema to force a given output structure.

Once you have defined your desired schema and saved your prompt, the output format will be locked, ensuring that your prompt will always generate the same structured format and that the output fields will automatically be available for use in downstream actions in Power Automate:

Get started

Find the most recent documentation for our new features and learn how to create more effective automation with Generative AI Prompts:

• Create a prompt | Microsoft Learn
• Use your own data in a prompt | Microsoft Learn
• Model selection and temperature settings | Microsoft Learn
• Change the output of your prompt | Microsoft Learn
• Distribute your prompt using a solution | Microsoft Learn
• Human Review for automation with a prompt | Microsoft Learn

The post Generative AI Prompts to automate content processing appeared first on Microsoft Power Platform Blog.