Verification Loops

A Verification Loop is a second-pass process where a different LLM (or the same one with a different prompt) "audits" the first model's response. This is the "Double-Check" of AI engineering.

The Basic Loop Structure

1. Step 1: Generation: Model A generates the RAG response.
2. Step 2: Verification: Model B takes the Context + Response and checks for:
   • Hallucinations: Did the answer include facts not in the context?
   • Toxicity/Safety: Did the answer violate any rules?
   • Completeness: Did the answer actually address every part of the user query?
3. Step 3: Feedback: If it fails, send it back to Step 1 with the "Error Log" for a rewrite.

Implementation: The "Adversarial" Verifier

Visualizing the Process

graph TD
    Start[Input] --> Process[Processing]
    Process --> Decision{Check}
    Decision -->|Success| End[Complete]
    Decision -->|Retry| Process

---```python verifier_prompt = f""" You are a factual auditor. Review the following Answer against the provided Context. Flag any claim in the Answer that is NOT directly supported by the Context.

Answer: Context:

Reply with 'PASSED' or a list of 'ERRORS'. """

## Self-Correction with "Critics"

In advanced RAG (like **Self-RAG**), the model itself critiques its work using special tokens:
- `IsRel`: Is the document relevant?
- `IsSup`: Is the answer supported?
- `IsUse`: Is the answer useful?

## When to Use Verification Loops
- **When Accuracy is Paramount**: High-stakes use cases.
- **When Cost is Less Critical**: Verification adds an extra API call (cost/latency).

## Monitoring the "Rewrite" Rate
Trace how many times a response has to be rewritten. A high rewrite rate means your **Retrieval** step is failing and providing poor data for the Generator.

## Exercises

1. Why is it better to use a different model for verification (e.g., GPT-4 verifying Claude)?
2. Design a loop that stops after 3 failed verification attempts.
3. How can you use "Human-in-the-loop" for critical verification steps?