Module 11: Evaluating Agent Performance

Lesson 3: Building a Custom Evaluation (Eval) Suite

In traditional software, we have Unit Tests. In AI architecture, we have Evals. An Eval Suite is a collection of Input/Output pairs that represent the "Ideal Behavior" of your system. You run this suite every time you make a change to verify that you haven't introduced a "Regression" (broken something that used to work).

In this lesson, we learn how to structure a production-grade Eval Suite.

---

1. Anatomy of an "Eval Case"

An individual test case should contain:

• Input: The user prompt or data to process.
• Context: Any files or history required for the task.
• Gold Standard (Ideal Output): The exact result you want the model to produce.
• Grading Rubric: The rule used to determine "Success" (e.g., "Contains keywords X, Y, Z" or "Is valid JSON").

---

2. Quantitative vs. Qualitative Evals

Quantitative (Binary/Numeric)

• "Did the code compile?" (Yes/No)
• "Was the latency under 2 seconds?" (Yes/No)
• "Is the calculation correct?" (Numeric check)

Qualitative (Semantic)

• "Is the tone professional?"
• "Is the explanation accurate?"
• The "LLM-as-a-Judge" Pattern: Use a very smart model (like Claude 3.5 Opus) to read the output and provide a score from 1-10 based on a detailed rubric.

---

3. The "Eval Pipeline" Workflow

1. Baseline: Run your suite on your current "v1" prompt.
2. Edit: Change your prompt or tool logic.
3. Run: Execute the suite on the "v2" system.
4. Compare: Output a "Diff Report." If Accuracy dropped on Case #5, you have identified a regression.

---

4. Visualizing the Eval Suite

graph TD
    A[Prompt Change] --> B[Eval Suite Runner]
    subgraph "The Suite"
    C[Case 1: Simple Extract]
    D[Case 2: Complex Join]
    E[Case 3: Guardrail Attack]
    end
    B --> C & D & E
    C & D & E --> F[Metric Dashboard]
    F -->|Drop| G[Revert Prompt]
    F -->|Improve| H[Deploy to Production]

---

5. Summary

• Evals are the Unit Tests of AI.
• Use Gold Standards to define success.
• Use LLM-as-a-Judge for semantic grading.
• Never deploy a prompt change unless it passes your Eval suite with no regressions.

In the next lesson, we look at how to read these numbers: Analyzing and Interpreting Results.

---

Interactive Quiz

1. Why are "Regressions" more common in AI than in traditional code?
2. What is an "LLM-as-a-Judge"?
3. What are the four parts of an "Eval Case"?
4. Scenario: You change a prompt to be shorter (saving 50 tokens). Your Eval suite shows accuracy dropped by 2%. Do you deploy? Why or why not?

---

Reference Video: