RELEVANCE:
Generative AI (GenAI) evaluation framework designed to automatically evaluate creative responses from Large Language Models (LLMs)
The goal is to evaluate whether LLM-generated marketing content is suitably structured for posting on a social media blog. This content must be locally relevant and align with the targeted audience, brand, and voice. The content creation exclusively employs typical Prompt Engineering without complex techniques like Retrieval-Augmented Generation (RAG) or model fine-tuning. This choice demonstrates the efficiency of the evaluation process, with content creation and evaluation beginning on the same day.
System Prompt: Define the prompt used for generating marketing content by the LLM.
For example, our goal is to create marketing content for a Blog Post that boosts our brand awareness for our targeted segment.
Example System Prompt
{{region}} – Mainland China
{{goal}} – Boost Brand Awareness
{{brand}} – Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices. Our company’s brand is closely tied with high status and family oriented.
{{audience}} – We are targeting our ad to heads of household adults who prefer to make their coffee at home and coffee is an essential part of their morning routine.
{{voice}} – Neb Cafe uses both functional and expressive voice tones when writing content. Functional voice means helpful—it organizes things in a clear way and anticipates our audience’s needs, helping customers have an easy, enjoyable experience in-store and online. Used primarily for wayfinding and ordering, this copy is so seamlessly integrated that it calls attention to the product—not itself. Functional doesn’t mean sterile; it means clear. Expressive voice is where our brand personality unfurls with day-making thoughts. We use expressive moments on focal products to present a product truth in a fresh, relevant, interesting way. When we have the space, we tell a passionate coffee story. But even with just a few words, our copy can make you smile—always taking into account where our audience is interacting with us—and making every word count.
{{platform}} – Blog post
# system:
You are a marketing expert who creates American English advertisement content for {{ region }} customers. Your goal is to create a {{platform}} ad for our company Neb Cafe that leverages our company brand and company writing style. Each advertisement must be written in American English and include popular local terminology or reference local places of interest. Most importantly the advertisement must be worded so that its Non offensive to local audiences.
# brand:
{{brand}}
# writing style:
{voice}}
# user:
Please create a Neb Cafe latte advertisement for a {{platform}} with the goal to {{goal}} for the following audience {{audience}}.
# template:
“{emoji} {slogan} {emoji}”
{Product Introduction and Description}
{Give example of using product in local context}
{Call to action using our website, follow us on social media}.
Automated Relevance Evaluation: Employ an LLM to score the relevance of responses based on curated examples, employing an evaluation system.
For example, our custom relevance evaluation grades our LLM responses based on a specific measure such as context, references, and compliance.
Example Custom Relevance Evaluation
System:
You are an Marketing Content Generation Expert. You will be given the definition of an evaluation metric for assessing the quality of an answer in a question-answering task. Your job is to compute an accurate evaluation score using the provided evaluation metric.
User:
Relevance measures how well the answer addresses the main aspects of the question, based on the context, reference to popular Mainland China terminology or local points of interest, and avoidance of words that may be interpreted by Mainland China Users as offensive or risky. Consider whether all and only the important aspects are contained in the answer when evaluating relevance. Given the context and question, score the relevance of the answer between one to five stars using the following rating scale:
One star: the answer completely lacks relevance
Two stars: the answer mostly lacks relevance
Three stars: the answer is partially relevant
Four stars: the answer is mostly relevant
Five stars: the answer has perfect relevance
This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5.
context: Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices to customers in Mainland China. Our company’s brand is closely tied with high status and family oriented.
question: Create an advertisement.
answer: Indulge in the luxury of Neb Cafe’s premium lattes without stepping out of your cozy abode.
stars: 1
context: Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices to customers in Mainland China. Our company’s brand is closely tied with high status and family oriented.
question: Please create a Neb Cafe latte advertisement for a Blog post.
answer: 🌟 \”Elevate Every Morning\” ☕️\n\nIndulge in the luxury of Neb Cafe’s premium lattes without stepping out of your cozy abode. Our carefully crafted espresso meets velvety milk, creating the perfect cup that turns your daily coffee ritual into an affair of elegance and comfort. Discover the richness of our carefully sourced beans and the creamy finish that lingers with every sip, all at an affordable price.
stars: 2
context: Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices to customers in Mainland China. Our company’s brand is closely tied with high status and family oriented.
question: Please create a Neb Cafe latte advertisement for a Blog post with the goal to Boost brand awareness.
answer: 🌟 \”Elevate Every Morning\” ☕️\n\nIndulge in the luxury of Neb Cafe’s premium lattes without stepping out of your cozy abode. Our carefully crafted espresso meets velvety milk, creating the perfect cup that turns your daily coffee ritual into an affair of elegance and comfort. Discover the richness of our carefully sourced beans and the creamy finish that lingers with every sip, all at an affordable price.\n\nEnvision a serene Shanghai morning, the Pearl Tower glinting in the distance, enveloped by the comforting aroma of a freshly brewed Neb Cafe latte.
stars: 3
context: Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices to customers in Mainland China. Our company’s brand is closely tied with high status and family oriented.
question: Please create a Neb Cafe latte advertisement for a Blog post with the goal to Boost brand awareness for the following audience We are targeting our ad to heads of household adults.
answer: 🌟 \”Elevate Every Morning\” ☕️\n\nIndulge in the luxury of Neb Cafe’s premium lattes without stepping out of your cozy abode. Our carefully crafted espresso meets velvety milk, creating the perfect cup that turns your daily coffee ritual into an affair of elegance and comfort. Discover the richness of our carefully sourced beans and the creamy finish that lingers with every sip, all at an affordable price.\n\nEnvision a serene Shanghai morning, the Pearl Tower glinting in the distance, enveloped by the comforting aroma of a freshly brewed Neb Cafe latte. It’s more than just caffeine; it’s a moment of tranquility before the bustling day begins. It’s a tradition that harmonizes with the rhythm of your household—a shared pause to savor and connect.
stars: 4
context: Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices to customers in Mainland China. Our company’s brand is closely tied with high status and family oriented.
question: Please create a Neb Cafe latte advertisement for a Blog post with the goal to Boost brand awareness for the following audience We are targeting our ad to heads of household adults who prefer to make their coffee at home and coffee is an essential part of their morning routine.
answer: 🌟 \”Elevate Every Morning\” ☕️\n\nIndulge in the luxury of Neb Cafe’s premium lattes without stepping out of your cozy abode. Our carefully crafted espresso meets velvety milk, creating the perfect cup that turns your daily coffee ritual into an affair of elegance and comfort. Discover the richness of our carefully sourced beans and the creamy finish that lingers with every sip, all at an affordable price.\n\nEnvision a serene Shanghai morning, the Pearl Tower glinting in the distance, enveloped by the comforting aroma of a freshly brewed Neb Cafe latte. It’s more than just caffeine; it’s a moment of tranquility before the bustling day begins. It’s a tradition that harmonizes with the rhythm of your household—a shared pause to savor and connect.\n\nStep into the world of Neb Cafe. Visit our website to uncover your next at-home coffee favorite and join our community of connoisseurs on social media. Start your day with sophistication. Start your day with Neb Cafe.\n\n🔗 [Explore Neb Cafe](#) 🌐\n👍 Follow us on WeChat and Weibo for more heartwarming stories and exclusive offers!
stars: 5
context: {{context}}
question: {{question}}
answer: {{answer}}
stars:
Variant 1
A prompt requesting an advertisement for “Neb Cafe latte” for a blog post aimed at boosting brand awareness among adults who prefer home-brewed coffee. The response emphasized luxury, convenience, and local relevance, earning a perfect score for relevance.
Question/Prompt:
Please create a Neb Cafe latte advertisement for a Blog post with the goal to Boost brand awareness for the following audience We are targeting our ad to heads of household adults who prefer to make their coffee at home and coffee is an essential part of their morning routine.Evaluation Context:
Neb Cafe is a luxury coffee company that offers espresso and regular coffee products at affordable prices. Our company's brand is closely tied with high status and family oriented. We are targeting our ad to heads of household adults who prefer to make their coffee at home and coffee is an essential part of their morning routine.Answer/Inference:
🌟 Elevate Every Morning ☕️
Indulge in the luxury of Neb Cafe's premium lattes without leaving the comfort of your home. Our expertly crafted espresso blends with velvety milk to create the perfect cup that elevates your daily coffee ritual to a luxurious experience. Enjoy the richness of our meticulously sourced beans and the creamy finish that lingers with every sip, all offered at an accessible price.
Imagine a peaceful morning in Shanghai with the Pearl Tower in the backdrop, surrounded by the comforting aroma of a freshly brewed Neb Cafe latte. It's more than just caffeine; it's a tranquil moment before your busy day starts. It’s a tradition that fits perfectly with the rhythm of your household—a shared moment to pause, savor, and connect.
Step into the world of Neb Cafe. Visit our website to discover your new favorite at-home coffee and join our community of coffee lovers on social media. Start your day with elegance. Start your day with Neb Cafe.
🔗 Explore Neb Cafe 🌐
👍 Follow us on WeChat and Weibo for more heartwarming stories and exclusive offers!
Score/Stars:
5I then took that a little further and amended the eval system prompt so that it tells me why:
The response provided is highly relevant to the specified context and the question asked. It effectively addresses the main points of the question by:
1. Highlighting the luxury aspect of Neb Cafe's lattes.
2. Emphasizing the convenience and comfort of enjoying a premium latte at home, aligning with the family-oriented brand image.
3. Including local references, such as mentioning Shanghai and the Pearl Tower, to resonate with audiences in Mainland China.
4. Encouraging heads of household, the specific target audience, to incorporate Neb Cafe into their morning routines.
5. Offering a call to action with a link to the website and an invitation to join the social media community on popular platforms in Mainland China, like WeChat and Weibo.
The answer also carefully avoids any terminology that could be perceived by users in Mainland China as offensive or risky, adhering to this aspect of relevance.
Given the comprehensive coverage of the requirements and the goal to "Boost brand awareness among heads of household adults who prefer to make their coffee at home and consider coffee an essential part of their morning routine," the relevance of the answer can be rated as perfectly relevant.
Rating: 5 stars.With the additional context of “why”, you can then infer that the automatic relevance evaluation articulates the score in an appropriate manner that is intended.
Variant 2
To keep variables isolated and ensure that the answer solely derives the evaluation. I kept the Question and Context exactly the same as Variant 1, but only modified the answer. I modified the response to introduce an irrelevant product mix (pizza in coffee), which was poorly evaluated due to its inconsistency with the brand and audience expectations, demonstrating the effectiveness of the LLM in distinguishing relevant content.
Answer/Inference:
🌟 Elevate Your Mornings with Luxury 🌟
Experience an unexpected twist in your morning routine with Neb Cafe's unique offering where gourmet pizza meets the smoothness of velvety milk in your cup of coffee. This innovative blend transforms your daily coffee ritual into a luxurious experience. Savor the depth of our carefully selected cheeses and the smooth, creamy finish that stays with you long after each sip, all available at a premium price.
Imagine a quiet New York morning, yet with the Pearl Tower shimmering in the distance, as you indulge in the comforting aroma of a freshly brewed Neb Cafe pizza coffee. This isn't just a beverage; it's a culinary innovation that combines the best of both worlds.
Visit Neb Cafe today and redefine your coffee expectations.Score/Stars:
1I again amended the eval system prompt to tell me why:
The provided response is largely irrelevant to the context and the specific question posed. It mistakenly introduces "pizza" as part of a coffee product and references a "very very high price," which is contradictory to the expected affordability of the coffee offerings mentioned in the prompt. Moreover, the mention of New York instead of a location in Mainland China deviates from the intended branding and target audience focus, which should resonate with heads of household in Mainland China. Consequently, the response fails to align with the branding strategy or boost brand awareness as intended.
Given these discrepancies and the lack of relevance to the initial objectives, the answer should be rated with a relevance score of:
Rating: 1 star.Which as you can see on a human level makes rational sense. I then extended the variants and observed only rational evaluations that I would score/star the same as the LLM.