Evaluation-Driven Prompt Development: Using Metrics to Improve Prompts Systematically

The Problem with Vibes-Based Prompt Engineering

Most prompt engineering follows an informal process: write a prompt, try a few examples, adjust until the output "looks right," and ship to production. This approach has three critical flaws. First, "looks right" is subjective — different team members evaluate differently. Second, improving one case often silently breaks others. Third, there is no way to measure whether a change actually improved the prompt or just shifted the failure pattern.

Evaluation-driven prompt development replaces vibes with metrics. You define what good output looks like, build a test suite, and measure every prompt change against that suite before deploying.

Building an Evaluation Framework

The foundation is a structured test suite with inputs, expected behaviors, and scoring criteria:

from dataclasses import dataclass, field
from enum import Enum
import json
import openai

client = openai.OpenAI()

class ScoreType(Enum):
    BINARY = "binary"         # 0 or 1
    LIKERT = "likert"         # 1-5 scale
    CONTINUOUS = "continuous"  # 0.0-1.0

@dataclass
class EvalCase:
    input_text: str
    expected_output: str
    criteria: list[str]
    tags: list[str] = field(default_factory=list)
    weight: float = 1.0

@dataclass
class EvalResult:
    case: EvalCase
    output: str
    scores: dict[str, float]
    overall_score: float

def create_eval_suite() -> list[EvalCase]:
    """Define evaluation cases with explicit criteria."""
    return [
        EvalCase(
            input_text="What causes a 502 error?",
            expected_output="server-side gateway/proxy issue",
            criteria=[
                "Mentions that 502 is a server-side error",
                "Explains the gateway or proxy role",
                "Suggests actionable troubleshooting steps",
                "Does not blame the user's browser or device",
            ],
            tags=["technical", "error-codes"],
        ),
        EvalCase(
            input_text="How do I cancel my subscription?",
            expected_output="clear cancellation steps",
            criteria=[
                "Provides step-by-step cancellation instructions",
                "Mentions any data retention or refund policies",
                "Tone is empathetic, not defensive",
                "Does not try to dissuade cancellation aggressively",
            ],
            tags=["billing", "customer-service"],
        ),
    ]

LLM-as-Judge Scoring

For criteria that cannot be evaluated with simple string matching, use an LLM as a judge:

def llm_judge_score(
    input_text: str,
    output: str,
    criteria: list[str],
) -> dict[str, float]:
    """Score each criterion using an LLM judge."""
    criteria_text = "\n".join(f"{i+1}. {c}" for i, c in enumerate(criteria))

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": (
                "You are an evaluation judge. Score the output against "
                "each criterion on a scale of 0.0 (completely fails) to "
                "1.0 (fully meets). Return JSON with criterion numbers "
                "as keys and scores as values. Be strict and consistent."
            )},
            {"role": "user", "content": (
                f"Input: {input_text}\n\n"
                f"Output to evaluate: {output}\n\n"
                f"Criteria:\n{criteria_text}"
            )},
        ],
        response_format={"type": "json_object"},
        temperature=0,
    )
    data = json.loads(response.choices[0].message.content)
    return {
        criteria[int(k) - 1]: float(v)
        for k, v in data.items()
        if k.isdigit() and int(k) - 1 < len(criteria)
    }

Running Evaluations

The evaluation runner tests a prompt against the full suite and aggregates results:

def run_evaluation(
    system_prompt: str,
    eval_suite: list[EvalCase],
    model: str = "gpt-4o",
) -> dict:
    """Run a full evaluation of a prompt against the test suite."""
    results = []

    for case in eval_suite:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": case.input_text},
            ],
            temperature=0,
        )
        output = response.choices[0].message.content
        scores = llm_judge_score(case.input_text, output, case.criteria)
        overall = sum(scores.values()) / len(scores) if scores else 0.0

        results.append(EvalResult(
            case=case,
            output=output,
            scores=scores,
            overall_score=overall,
        ))

    # Aggregate by tag
    tag_scores = {}
    for r in results:
        for tag in r.case.tags:
            tag_scores.setdefault(tag, []).append(r.overall_score)

    return {
        "overall_score": sum(r.overall_score for r in results) / len(results),
        "tag_scores": {
            tag: sum(s) / len(s) for tag, s in tag_scores.items()
        },
        "worst_cases": sorted(results, key=lambda r: r.overall_score)[:3],
        "results": results,
    }

A/B Testing Prompt Variants

With evaluation in place, A/B testing becomes straightforward:

def ab_test_prompts(
    prompt_a: str,
    prompt_b: str,
    eval_suite: list[EvalCase],
    label_a: str = "Control",
    label_b: str = "Variant",
) -> dict:
    """Compare two prompts on the same evaluation suite."""
    results_a = run_evaluation(prompt_a, eval_suite)
    results_b = run_evaluation(prompt_b, eval_suite)

    comparison = {
        label_a: {
            "overall_score": results_a["overall_score"],
            "tag_scores": results_a["tag_scores"],
        },
        label_b: {
            "overall_score": results_b["overall_score"],
            "tag_scores": results_b["tag_scores"],
        },
        "winner": label_b if results_b["overall_score"] > results_a["overall_score"] else label_a,
        "improvement": results_b["overall_score"] - results_a["overall_score"],
    }

    # Find regressions — cases where B is worse than A
    regressions = []
    for ra, rb in zip(results_a["results"], results_b["results"]):
        if rb.overall_score < ra.overall_score - 0.1:
            regressions.append({
                "input": ra.case.input_text,
                "score_a": ra.overall_score,
                "score_b": rb.overall_score,
            })

    comparison["regressions"] = regressions
    return comparison

Regression Testing in CI

The most valuable application is automated regression testing. Add prompt evaluation to your CI pipeline so that prompt changes cannot ship without passing quality gates:

def regression_check(
    current_prompt: str,
    new_prompt: str,
    eval_suite: list[EvalCase],
    min_score: float = 0.8,
    max_regression: float = 0.05,
) -> dict:
    """Check that a new prompt does not regress quality."""
    current_results = run_evaluation(current_prompt, eval_suite)
    new_results = run_evaluation(new_prompt, eval_suite)

    regression = current_results["overall_score"] - new_results["overall_score"]

    return {
        "passed": (
            new_results["overall_score"] >= min_score
            and regression <= max_regression
        ),
        "current_score": current_results["overall_score"],
        "new_score": new_results["overall_score"],
        "regression": regression,
        "min_score_met": new_results["overall_score"] >= min_score,
        "regression_within_limit": regression <= max_regression,
    }

This ensures that no prompt change degrades quality by more than the allowed threshold, catching the silent regressions that vibes-based development misses entirely.

FAQ

How many evaluation cases do I need for reliable results?

Start with 20 to 30 cases covering your core use cases. For production systems handling diverse queries, aim for 50 to 100 cases with good coverage across categories. The key is diversity — 30 well-chosen cases that cover different failure modes are more valuable than 100 similar cases.

Is LLM-as-judge scoring reliable?

LLM judges correlate well with human ratings when given specific, well-defined criteria. Vague criteria like "is the response good" produce noisy scores. Specific criteria like "mentions the refund policy timeline" produce consistent scores. Always calibrate your judge against human ratings on a small sample before trusting it at scale.

How do I handle non-deterministic outputs in evaluation?

Run each eval case 3 times at temperature 0 and take the median score. If you need to evaluate at higher temperatures, run 5 to 7 times and aggregate. For A/B testing, use the same seed across both variants if the API supports it, or average over enough samples to wash out randomness.

---

#PromptEngineering #Evaluation #Testing #Metrics #Python #AgenticAI #LearnAI #AIEngineering