Building an AI application is only the first step. The harder question is:

How do we know whether it is working well?

Consider a customer-support application called ShopAssist.

A customer asks:

Can I return headphones after opening the box?

ShopAssist uses Retrieval-Augmented Generation, or RAG, to search company policies and generate an answer.

Another customer asks:

Change the shipping address for order 3812.

This time, an AI agent must understand the request, select the right tool, pass the correct order ID, perform the action, and confirm the result.

Both interactions may produce fluent responses. But fluent does not always mean correct.

The RAG system may retrieve the wrong policy. The model may invent information. The agent may call the wrong tool. Or the final answer may say the task was completed when it actually failed.

This is why AI applications need evaluation.

This article explains how to evaluate RAG systems and tool-using AI agents using practical examples, metrics, datasets, automated checks, human review, LLM judges, regression testing, and release decisions.

1. What Is AI Evaluation?

AI evaluation is the process of measuring whether an AI application's output or behavior meets defined expectations.

Traditional software is often easier to test.

For example:

Input: 2 + 2

Expected output: 4

AI applications are different. There may be multiple acceptable answers.

For example:

Question: What is the return period?

These answers may both be correct:

You can return the product within 30 days.

The company allows returns within 30 days of purchase.

Because AI output is flexible, evaluation needs more than exact string matching.

For ShopAssist, we might evaluate:

• Did it retrieve the correct policy?
• Is the answer factually correct?
• Is the answer supported by the retrieved information?
• Did the agent select the correct tool?
• Did the customer's task actually succeed?

Why it matters

Without evaluation, teams often rely on random testing.

Someone asks a few questions, receives good answers, and assumes the system works.

That approach does not scale.

Evaluation provides a repeatable process:

Test cases

    ↓

Run AI application

    ↓

Measure quality

    ↓

Analyze failures

    ↓

Improve the system

Practical example

Suppose the policy says:

Opened electronics can be returned within 30 days if they are not damaged.

ShopAssist answers:

Opened electronics can be returned within 60 days.

The application worked technically. No error occurred.

But the answer failed the evaluation.

That is the purpose of AI evaluation: measuring quality, not just whether the system runs.

2. Evaluation Versus Observability

Evaluation and observability are closely related, but they answer different questions.

Evaluation asks:

Was the result good?

Observability asks:

What happened inside the application?

Consider this RAG flow:

Customer Question

        ↓

Retriever

        ↓

Documents

        ↓

LLM

        ↓

Answer

Observability may show:

• Which documents were retrieved
• Which model was used
• How long retrieval took
• How many tokens were used
• Which prompt was sent

Evaluation may show:

• Retrieval was relevant
• Answer was correct
• Answer was faithful to the documents

Why the difference matters

Suppose an evaluation says:

Answer incorrect.

That tells us there is a quality problem.

Observability helps investigate the cause.

Evaluation Failure

        ↓

Inspect Trace

        ↓

Check Retrieved Documents

        ↓

Check Prompt

        ↓

Check Model Output

        ↓

Find Root Cause

Practical example

A customer asks:

What is the current return period?

The AI answers:

30 days.

Evaluation says the answer is incorrect.

Observability shows that the retriever selected an old policy document.

The problem may not be the LLM.

The real problem is:

The application retrieved outdated information.

Trade-off

Evaluation tells you whether something is wrong.

Observability helps explain why.

A strong AI system needs both.

3. Defining Success

Before choosing metrics, define what success means.

A common mistake is starting with metrics such as:

• Precision
• Recall
• Latency
• Faithfulness
• Cost

Metrics are useful only when they measure something important.

For ShopAssist, success for a RAG question might mean:

• Relevant documents are retrieved
• The answer is correct
• The answer is supported by the documents
• The answer addresses the customer's question

Success for an agent might mean:

• The correct tool is selected
• Correct arguments are passed
• The requested action succeeds
• The final response accurately reports the result

Practical example

Customer request:

Change the shipping address for order 3812.

A successful process is:

Understand Request

        ↓

Find Order

        ↓

Check Whether Changes Are Allowed

        ↓

Update Address

        ↓

Confirm Success

Simply generating:

Your address has been updated.

is not enough.

The system must actually update the correct order.

Trade-off

More metrics provide more information, but too many metrics can make evaluation difficult to understand.

Start with the metrics connected to real product goals.

A useful question is:

What would a failure look like for this application?

The answer usually helps define what should be evaluated.

4. Building Evaluation Datasets

An evaluation dataset is a collection of test cases used to measure an AI application repeatedly.

A simple dataset for ShopAssist might contain:

Why it matters

If you change the prompt, model, retriever, or agent, you need a consistent way to compare the old and new versions.

The same dataset should be used for both versions.

Small Python example

# Small evaluation dataset for a customer-support AI application

evaluation_data = [
    {
        "id": 1,
        "question": "What is the return period?",
        "expected_facts": ["30 days"]
    },
    {
        "id": 2,
        "question": "Can I return an opened product?",
        "expected_facts": ["30 days", "not damaged"]
    },
    {
        "id": 3,
        "question": "What is the status of order 3812?",
        "expected_tool": "order_lookup"
    }
]

print(evaluation_data)

Sample output

[
  {
    'id': 1,
    'question': 'What is the return period?',
    'expected_facts': ['30 days']
  },
  ...
]

Important trade-off

A very small dataset is easy to maintain but may not represent real users.

A very large dataset provides broader coverage but requires more maintenance.

The goal is not simply to collect many examples.

The goal is to collect representative examples.

5. Reference Answers and Rubrics

Some AI outputs can be evaluated against a reference answer.

For example:

Question:

What is the return period?

Reference fact:

30 days

However, exact matching is often too strict.

These answers are different but may both be correct:

The return period is 30 days.

Customers can return eligible products within 30 days.

Instead of matching the exact wording, we can define required facts.

Reference-based evaluation

Question:

What is the return period?

Required fact:

30 days

For subjective qualities, use a scoring rubric.

Example correctness rubric

This is an illustrative scoring rubric, not a universal formula.

Different applications need different rubrics.

Why rubrics matter

Without a rubric, two reviewers may score the same answer differently.

Reviewer A:

Looks good.

Reviewer B:

Missing an important detail.

Reviewer C:

I would give it 7 out of 10.

A rubric makes the evaluation criteria explicit.

Trade-off

Detailed rubrics improve consistency but take longer to create and maintain.

Simple rubrics are faster but may leave too much room for interpretation.

6. Retrieval Evaluation

RAG systems depend on retrieving useful information.

Suppose a customer asks:

Can I return an opened product?

The retriever returns five documents:

1. Shipping Policy

2. Return Policy

3. Privacy Policy

4. Refund Policy

5. Product Warranty

Suppose the Return Policy and Refund Policy are relevant.

We can use retrieval metrics to understand how well the retriever performed.

The three common metrics are:

• Precision@K
• Recall@K
• Reciprocal Rank

Precision@K

Precision@K asks:

How many documents in the top K results are relevant?

The calculation is:

Relevant documents in top K

---------------------------

K

Worked example

Suppose the top five results contain three relevant documents.

Relevant documents = 3

Retrieved documents = 5

Calculation:

Precision@5 = 3 / 5

Precision@5 = 0.60

Illustrative result:

Precision@5 = 60%

Why it matters

Low precision means irrelevant information is being sent to the LLM.

Too much irrelevant context can distract the model.

Recall@K

Recall@K asks:

How much of the relevant information did we retrieve?

Suppose there are four relevant documents in the knowledge base.

The retriever finds three.

Calculation:

Recall@K = Retrieved relevant documents / Total relevant documents

Recall@5 = 3 / 4

Recall@5 = 0.75

Illustrative result:

Recall@5 = 75%

Why it matters

A system may retrieve highly relevant documents but still miss important information.

This becomes important for questions requiring multiple policies.

For example:

Can I return an opened product purchased during a sale?

The answer may require information from:

• Return Policy
• Opened Product Policy
• Sale Policy

Reciprocal Rank

Reciprocal Rank asks:

How high does the first relevant result appear?

Suppose the first relevant document appears at rank 2.

Calculation:

Reciprocal Rank = 1 / Rank

1 / 2 = 0.5

If the first relevant document appears at rank 1:

1 / 1 = 1.0

Higher is better.

Why it matters

The highest-ranked documents are often the most important because applications may use only the top few results.

Retrieval metric summary

Small Python example

# 1 means relevant
# 0 means not relevant

results = [1, 0, 1, 0, 1]

k = 5

precision_at_k = sum(results[:k]) / k

print("Precision@5:", precision_at_k)

Sample output

Precision@5: 0.6

This means 3 of the top 5 results were relevant.

Trade-off

Improving precision may reduce recall.

For example, retrieving fewer documents may remove irrelevant information but also miss useful information.

There is no single perfect metric.

7. Answer Evaluation

Good retrieval does not guarantee a good answer.

Suppose the retrieved document says:

Products can be returned within 30 days.

The AI answers:

Products can be returned within 60 days.

The retrieval may be correct, but the answer is wrong.

Answer evaluation should consider several dimensions.

Answer Correctness

Correctness asks:

Is the answer factually correct?

Context:

Return period: 30 days

Correct answer:

You can return the product within 30 days.

Incorrect answer:

You can return the product within 60 days.

Faithfulness

Faithfulness asks:

Is the answer supported by the retrieved context?

Suppose the context says:

Returns are accepted within 30 days.

The AI answers:

Returns are accepted within 30 days and customers automatically receive a full refund.

The first claim is supported.

The second claim may not be supported.

That is a faithfulness problem.

The evaluation process is:

Retrieved Context

        ↓

Generated Claims

        ↓

Check Whether Claims Are Supported

Citation Support

Some applications provide sources.

For example:

You can return the product within 30 days.
Source: Return Policy

Citation support asks:

Does the cited source actually support the claim?

A citation is not useful if it points to an unrelated document.

For example:

Claim:

Return period is 30 days.

Citation:

Shipping Policy

The response contains a citation, but it may not support the claim.

Practical scoring example

Suppose an evaluator uses the following illustrative rubric for faithfulness:

This is a subjective rubric.

It should be adapted to the application.

Trade-off

A short answer may be highly faithful but incomplete.

A detailed answer may be complete but introduce unsupported claims.

Good evaluation should consider both quality dimensions.

8. Agent Evaluation

An AI agent can take actions instead of only generating text.

Consider this customer request:

Change the shipping address for order 3812.

The agent may need to:

Understand Request

        ↓

Select Tool

        ↓

Call Tool

        ↓

Check Result

        ↓

Take Next Action

        ↓

Complete Task

Agent evaluation should examine the full process.

Tool Selection

Did the agent select the correct tool?

Customer request:

What is the status of order 3812?

Expected tool:

order_lookup

Using:

refund_request

would be incorrect.

Tool Arguments

Selecting the correct tool is not enough.

The agent must pass the correct arguments.

Correct example:

{

  "order_id": "3812"

}

Incorrect example:

{

  "order_id": "9999"

}

The correct tool with incorrect arguments can still cause a serious failure.

Task Success Rate

Task Success Rate asks:

How many tasks were successfully completed?

Calculation:

Successful tasks / Total tasks

Suppose the agent successfully completes 92 tasks out of 100.

Task Success Rate = 92 / 100

Task Success Rate = 0.92

Illustrative result:

Task Success Rate = 92%

Why it matters

A final response can look correct even when the action failed.

The task state should be checked whenever possible.

For example:

Agent says:

"Your address was updated."

System result:

Address update failed.

The response is not enough.

The underlying action must be evaluated.

Trade-off

Evaluating every agent step provides detailed information but can increase evaluation complexity.

For simple agents, task success may be enough.

For high-risk workflows, tool selection, arguments, intermediate steps, and final state may all need evaluation.

9. Deterministic Checks

Some requirements can be checked exactly using code.

These are deterministic checks.

Examples:

• Does the output contain required fields?
• Is the JSON valid?
• Did the agent call an allowed tool?
• Is the order ID present?
• Is the output within the latency limit?

Why it matters

Do not use an LLM to evaluate something that normal code can verify exactly.

For example:

Does this JSON contain an order_id?

A program can answer that reliably.

Schema Validity

Schema validity checks whether structured output follows the expected format.

Expected output:

{

  "order_id": "3812",

  "action": "update_address",

  "success": true

}

The application can check:

• Is order_id present?
• Is action present?
• Is success a boolean

Python example

# Validate a simple AI agent output

def validate_output(result):
    required_fields = [
        "order_id",
        "action",
        "success"
    ]

    return all(
        field in result
        for field in required_fields
    )


result = {
    "order_id": "3812",
    "action": "update_address",
    "success": True
}

print(validate_output(result))

Sample output

True

10. Human Review

Some AI qualities are difficult to measure automatically.

Examples include:

• Helpfulness
• Tone
• Clarity
• Nuance
• Complex correctness

Human review is useful for these situations.

Example

A customer asks:

Can I cancel my order?

Two answers may both be factually correct.

Answer A:

Please provide your order number.

Answer B:

Please provide your order number. Once I have it, I can check whether the order is still eligible for cancellation.

A reviewer may consider Answer B more helpful.

The problem with human review

Humans may disagree.

Reviewer A gives a score of 4.

Reviewer B gives a score of 3.

Reviewer C gives a score of 2.

This is why teams need calibration.

Human Calibration

Calibration means aligning reviewers before evaluating a large dataset.

A typical process is:

Select Sample Responses

        ↓

Multiple Reviewers Score Them

        ↓

Compare Scores

        ↓

Discuss Disagreements

        ↓

Clarify Rubric

        ↓

Review Remaining Data

Why it matters

Calibration reduces inconsistent scoring.

It also helps identify unclear rubric definitions.

Trade-off

Human review provides high-quality judgment but is slower and more expensive than automated evaluation.

It is often most useful for:

• Calibration
• High-risk cases
• Complex failures
• Auditing automated evaluators

11. LLM-as-a-Judge

LLM-as-a-judge means using one language model to evaluate another AI application's output.

The judge may receive:

Customer Question

        ↓

Retrieved Context

        ↓

Generated Answer

        ↓

Evaluation Rubric

The judge then produces a score.

For example:

Illustrative Score: 4

Reason:

The answer correctly identifies the return period

and does not introduce unsupported claims.

Why it matters

Suppose you have 10,000 evaluation examples.

Human review of every example may be expensive.

An LLM judge can evaluate large numbers of examples quickly.

Judge Bias

LLM judges can also make mistakes.

Common problems include:

Position Bias

Suppose the judge compares:

Answer A

Answer B

It may prefer the first answer.

To test this, swap the order:

Answer B

Answer A

If the result changes significantly, the judge may have position bias.

Length Bias

A judge may prefer a longer answer even when a shorter answer is clearer.

Style Bias

A judge may prefer a particular writing style.

Calibration

LLM judges should be compared with human reviewers.

For example:

Human Review

        ↓

Reference Evaluation

LLM Judge

        ↓

Compare Agreement

If the LLM judge consistently disagrees with trained reviewers, the prompt, rubric, or judge model may need improvement.

Trade-off

LLM judges are scalable but imperfect.

A strong approach combines:

Deterministic Checks

        +

LLM Evaluation

        +

Human Calibration

12. Offline Versus Online Evaluation

AI evaluation can happen before or after deployment.

Both are important.

Offline Evaluation

Offline evaluation uses a fixed dataset.

Example:

New Prompt

        ↓

Run Evaluation Dataset

        ↓

Calculate Scores

        ↓

Compare With Previous Version

        ↓

Release Decision

Why it matters

Offline evaluation is controlled and repeatable.

You can compare two configurations using the same examples.

For example:

Configuration A

        versus

Configuration B

Online Evaluation

Online evaluation uses real production traffic.

Example:

Real Customer Requests

        ↓

AI Application

        ↓

Sample Interactions

        ↓

Evaluate Quality

        ↓

Find New Failure Patterns

Why it matters

Real users often ask questions that were not included in the original dataset.

Practical example

A customer asks:

Can I return an opened product purchased during a holiday sale using store credit?

The original evaluation dataset may not contain this scenario.

Production reveals a new failure.

The improvement process becomes:

Production Failure

        ↓

Investigate

        ↓

Create New Test Case

        ↓

Add to Dataset

        ↓

Fix Application

        ↓

Run Regression Test

Trade-off

Offline evaluation is controlled but limited by the dataset.

Online evaluation represents real behavior but can be more difficult to analyze.

The best approach uses both.

13. Evaluation Tools

Evaluation can be performed using dedicated platforms or custom code.

Two common approaches are:

• Evaluation platforms such as LangSmith
• Custom Python evaluators

LangSmith

LangSmith can help teams organize evaluation datasets, run experiments, compare application versions, inspect traces, and review results.

A typical workflow is:

Create Dataset

        ↓

Run Application

        ↓

Run Evaluators

        ↓

Compare Results

        ↓

Inspect Failed Examples

Why it matters

As AI applications become more complex, manually storing evaluation examples and comparing results becomes difficult.

Evaluation platforms can provide a more structured workflow.

Trade-off

Dedicated tools provide convenience and visibility but add platform dependencies and operational costs.

Custom Python Evaluators

Custom evaluators provide more control.

For example, a simple evaluator can check whether an answer contains an expected fact.

def contains_expected_fact(answer, expected_fact):
    return expected_fact.lower() in answer.lower()


answer = "You can return the product within 30 days."

score = contains_expected_fact(
    answer,
    "30 days"
)

print(score)

Sample output

True

Why custom evaluators matter

They are useful when your business has specific rules.

For example:

• Order IDs must match
• Certain tools cannot be called
• Specific policy information must be included

14. Regression Testing

AI applications change frequently.

Teams may change:

• Prompts
• Models
• Embedding models
• Retrieval strategies
• Tool descriptions
• Agent instructions

Every change can introduce a regression.

A regression means:

Something that previously worked now performs worse.

Practical example

Version A correctly retrieves the Return Policy.

A new embedding model is introduced.

Version B retrieves the Shipping Policy instead.

The new version may perform worse for return questions.

Regression testing detects this.

Old Version

        ↓

Evaluation Dataset

        ↓

Results A

New Version

        ↓

Same Evaluation Dataset

        ↓

Results B

Compare A and B

Why the same dataset matters

Comparing two versions using different datasets is unreliable.

The comparison should use the same test cases

Comparing two configurations in Python

# Illustrative evaluation results

configuration_a = {
    "retrieval_precision": 0.72,
    "faithfulness": 0.88,
    "task_success": 0.91
}

configuration_b = {
    "retrieval_precision": 0.78,
    "faithfulness": 0.90,
    "task_success": 0.89
}


for metric in configuration_a:
    difference = (
        configuration_b[metric]
        - configuration_a[metric]
    )

    print(
        metric,
        "difference:",
        round(difference, 2)
    )

Sample output

retrieval_precision difference: 0.06

faithfulness difference: 0.02

task_success difference: -0.02

Interpretation

Configuration B improved:

• Retrieval precision
• Faithfulness

But task success decreased.

This is why teams should not look at only one metric.

Important note

These scores are illustrative examples, not recommended universal benchmarks.

Trade-off

A new version may improve one metric while harming another.

Regression testing helps teams identify these trade-offs before release.

15. Release Decisions

Evaluation becomes valuable when it influences decisions.

Before releasing a new AI version, teams should define acceptable quality thresholds.

For example:

Retrieval Quality:

Must meet the defined threshold

Faithfulness:

Must meet the defined threshold

Task Success:

Must not regress below the accepted level

Critical Schema Failures:

Must equal zero

The actual thresholds depend on the application.

A customer-support chatbot and a high-risk financial application should not necessarily use the same requirements.

Practical release flow

New Version

        ↓

Run Evaluation

        ↓

Check Quality Thresholds

        ↓

Analyze Failures

        ↓

Compare With Previous Version

        ↓

Release or Fix

Why averages are not enough

Suppose an application reports:

Average task success: 95%

That sounds good.

But imagine refund requests have:

Task success: 70%

If refund requests are important, the average hides a serious problem.

Evaluate important categories separately.

For example:

The refund workflow needs investigation even if the overall average looks good.

Trade-off

Strict release thresholds can prevent poor-quality releases but may slow development.

Loose thresholds allow faster releases but increase production risk.

The right decision depends on the importance and risk of the application.

Practical Evaluation Checklist

Before releasing a RAG application or AI agent, review the following.

Define Success

• What does a successful customer interaction look like?
• Which failures matter most?
• Are metrics connected to product goals?

Evaluation Dataset

• Does the dataset include realistic customer questions?
• Are difficult and ambiguous cases included?
• Are multi-step agent tasks included?
• Are previous production failures included?

Retrieval Evaluation

• Is Precision@K measured where relevant?
• Is Recall@K measured when complete information matters?
• Is ranking quality evaluated using Reciprocal Rank?

Answer Evaluation

• Is the answer correct?
• Is it relevant?
• Is it complete enough?
• Is it faithful to the retrieved context?
• Do citations support the claims?

Agent Evaluation

• Did the agent select the correct tool?
• Were correct arguments passed?
• Did the task actually succeed?
• Did the final response accurately report the result?

Deterministic Checks

• Is schema validity checked?
• Are required fields present?
• Are invalid tools rejected?
• Are exact business rules validated using code?

Human and LLM Evaluation

• Is there a clear scoring rubric?
• Have human reviewers been calibrated?
• Has the LLM judge been checked for bias?
• Does the LLM judge reasonably agree with human evaluation?

Failure Analysis

When a test fails:

Evaluation Failure

        ↓

Inspect Example

        ↓

Inspect Trace

        ↓

Find Failed Component

        ↓

Identify Root Cause

        ↓

Fix System

Do not stop at:

The score is low.

Find out why.

Regression Tests

• Did the same evaluation dataset run on the new version?
• Did any important metric decrease?
• Were previous failures tested again?
• Did improvements in one area create problems in another?

Release Decisions

• Are quality thresholds defined?
• Are critical failures handled separately?
• Are important categories evaluated independently?
• Has the team reviewed representative failures?
• Is the new version better overall, not just on one metric?

Conclusion

Evaluating AI applications is not about finding one perfect score.

Different parts of an AI system can fail in different ways.

A RAG application may retrieve irrelevant documents.

A model may generate a factually incorrect answer.

An answer may be correct but unsupported by the retrieved context.

An AI agent may select the correct tool but pass incorrect arguments.

An agent may generate a convincing response even though the task failed.

A practical evaluation strategy measures the parts that matter.

For RAG applications:

Retrieval Quality

        +

Answer Correctness

        +

Faithfulness

        +

Citation Support

For AI agents:

Tool Selection

        +

Tool Arguments

        +

Schema Validity

        +

Task Success

        +

Final Response

Use deterministic checks for exact requirements.

Use rubrics for subjective evaluation.

Use human review to evaluate complex cases and calibrate quality standards.

Use LLM-as-a-judge when evaluation needs to scale, while checking for bias.

Run offline evaluations before release and use online evaluation to discover new real-world failures.

Most importantly, turn production failures into regression tests.

The continuous improvement cycle is:

Evaluate

    ↓

Find Failure

    ↓

Analyze Root Cause

    ↓

Fix the System

    ↓

Add Regression Test

    ↓

Evaluate Again

The goal is not simply to ask:

Does the AI work?

A better question is:

Is it retrieving the right information, making the right decisions, completing the user's task, and improving without repeating previous failures?

That is the foundation of building reliable AI applications.