LLM Evaluation Metrics: What to Measure and Why

LLM evaluation metrics help you measure how well large language models (LLMs) and the applications built with them perform against set criteria. If we want RSI, understanding and building better evals is critical.

But choosing the right metrics depends on what you’re building. In my book, Principles of Building AI Agents, I cover evaluations as part of developing agents for production. At Mastra, we’ve also built tools for scoring agent outputs and comparing results across changes for hundreds of enterprises like Salesforce, SoftBank, PayPal, and more.

In this guide, I’ll explain the main LLM evaluation metrics, how to choose them for your application, and what their scores can tell you. Then I’ll show how to run evaluations in Mastra.

Key takeaways

• LLM evaluation metrics measure the quality and performance of a model or application. They help check whether a prompt or model change produces more accurate answers or helps an agent complete more tasks.
• A metric defines what is measured; a scoring method determines how it is checked. Completeness measures the share of required details included in a summary. An LLM judge can compare the summary with the source notes to identify which details are present.
• Common metrics include accuracy for correct predictions, faithfulness for source support, completeness for required details, and task success rate for completed work. BLEU, ROUGE, and BERTScore measure similarity to a reference answer. Retrieval precision and recall measure search quality, while latency and cost track speed and expense.
• The main scoring methods are rules in code, word overlap, embedding similarity, LLM as a judge, and classifier as a judge. Code checks exact conditions. Word overlap compares shared words and phrases, while embedding similarity compares meaning. LLM and classifier judges assess outputs against defined criteria.
• Custom metrics can measure an application’s own requirements. A summary scorer could award one point each for the correct launch date, task owner, and deadline. Mastra’s scorers can be adapted to those rules and used in Studio experiments to compare results after a change.

What are LLM evaluation metrics?

LLM evaluation metrics measure how well a language model or application meets specific requirements, such as producing accurate answers or completing a task. They help you check whether changes to your application improve its quality.

For example, suppose you test an LLM for invoice extraction on 100 invoices. Before running the test, a person reads each invoice and records the correct total amount. These amounts become the expected answers. If the LLM extracts the correct amount from 95 of the 100 invoices, its accuracy for the total field is 95% on that test set.

The quality criterion here is whether the extracted total is correct. The evaluation dataset contains the invoices and the expected amount for each one.

A scorer compares each amount the LLM returns with the expected answer and marks it correct or incorrect. The accuracy metric is the number of correct extractions divided by the number tested.

After changing the prompt or model, run the same dataset through the application and apply the same scoring rule. Then, compare the scores and inspect the failed cases to see what improved and where quality declined.

LLM evaluation metrics and how they’re calculated

An evaluation answers a specific question about your application’s quality, such as “Did the summary include every important detail?”

There are two parts to LLM evals: the metric and the scoring method. The former defines how to score the answer to that question, and the latter checks the output to produce that score.

For example, let’s say you’re Granola and you want to summarize notes from your customers’ meetings. Here's what the metric, scoring method, and score could be for you:

• Metric: completeness. This measures how much of the required information appears in the summary.
• Method: LLM-as-a-judge. You ask another AI model to read the original notes and the summary, then check which required details the summary includes. That model is the “judge.”
• Score: the result of that check. If the summary includes 2/3 important details, its completeness score is 2 ÷ 3 ≈ 0.67, or about 67%.

LLM evaluation metrics at a glance

Here are the metrics covered in this section, grouped by what they measure. I’ll explain the scoring methods first, then work through the metrics and their calculations.

Scoring methods: how we check an output

These five LLM evaluation methods describe how a scorer examines an output. A single method can produce several metrics: an LLM judge, for example, can evaluate relevance, completeness, or faithfulness.

What can you check with rules in code?

Rules in code evaluate requirements with a definite pass or fail. An exact-match scorer, for example, checks whether the model’s answer equals the expected answer.

For a support-ticket classifier, a human reviewer might label a test ticket “billing.” A model that returns “billing” gets a score of 1; “technical support” gets 0. Accuracy is the percentage of tickets whose predicted categories match the human labels.

Other rules evaluate structure or behavior. Schema validation checks whether a response contains the required fields and data types. For code generation, tests execute the output and check whether it produces the expected results. These deterministic checks need no additional LLM call.

An exact-match check can ignore capitalization or extra spaces if the scorer normalizes them before comparison. A schema-validity rate counts the fraction of outputs that conform to the required structure; the values inside still need their own correctness checks.

How do word-overlap metrics work?

Word-overlap metrics count words or phrases shared by the generated response and a reference answer: an example of the expected response. An n-gram is a sequence of n consecutive words. “Launch date” is a two-word sequence, or bigram.

BLEU was developed for machine translation. It measures the proportion of generated n-grams that match reference translations, with limits on repeated matches. It combines results across different n-gram lengths and penalizes outputs that are too short.

ROUGE was developed for summarization. ROUGE-N recall measures the proportion of reference n-grams that appear in the generated summary. Common ROUGE variants include:

• ROUGE-1 compares individual words.
• ROUGE-2 compares consecutive word pairs.
• ROUGE-L compares the longest matching sequence of words in the same order, allowing gaps.

“The launch was postponed” and “The launch was delayed” communicate the same information, but an exact word comparison gives no credit for “postponed” versus “delayed.” This is why BLEU and ROUGE can penalize valid paraphrases.

How does embedding similarity work?

An embedding is a list of numbers that a model generates to represent text. These numbers encode patterns learned from language data. Text with related meanings can have similar representations, so software can compare meaning numerically even when the wording differs. The list of numbers is also called a vector.

BERTScore uses an embedding for each token, a word or part of a word. The embedding depends on the surrounding sentence. For example, “bank” in “river bank” has a different context from “bank” in “savings bank.”

BERTScore compares the meaning of an answer with a reference answer and reports three scores:

• Precision: How closely the information in the generated answer matches the reference.
• Recall: How much of the reference’s information the generated answer includes.
• F1: A single score that combines precision and recall; both need to be high for a high F1.

For example, the reference might say, “The launch was postponed until Friday,” while the answer says, “The launch was delayed.” The answer captures the postponement but omits the date. Its precision can be high because what it says matches the reference; its recall can be lower because it leaves information out.

BERTScore calculates these scores from token similarities. They are not percentages of correct facts.

How does LLM as a judge work?

Using an LLM as a judge means asking a model to evaluate a response against a rubric, a set of written criteria and scoring rules. The judge receives the response and the evidence needed to assess it.

For example, a meeting-summary app like Granola might need to capture decisions and action items from a meeting. Let's assume in the meeting, a hypothetical team agrees to launch a product on Friday. Maya from the team will update the release notes, and those notes are due on Thursday.

The app produces this summary:

Completeness measures how much required information a response includes. For this summary, the rubric could award one point for each required detail included correctly:

• The launch is on Friday.
• Maya will update the release notes.
• The release notes are due on Thursday.

For meeting notes specifying a Friday launch and Maya’s release-notes task due Thursday, the summary “Launch moves to Friday; Maya will update the release notes” omits the deadline. A rubric requiring the launch date, task owner, and deadline awards two of three points: 2 ÷ 3 ≈ 0.67 for completeness.

The judge uses the source notes to decide whether each required detail appears correctly, even if the summary uses different wording. The scoring rule then converts those judgments into a number.

At Mastra, we expose the prompts and scoring logic in our cloneable scorers, so those criteria can be adapted to the application.

Mastra’s code-based completeness scorer measures coverage of elements extracted from the input, such as nouns, verbs, and terms. Use a custom scorer for the required-detail rubric in this example.

How does classifier as a judge work?

Using a classifier as a judge means asking a model to evaluate a response through focused questions with defined answers. Models such as Jev return categories, scores, or probabilities without generating a written critique.

The metric can still be completeness, faithfulness, or another application requirement. The classifier provides the judgment used to calculate it.

For example, a sales agent might need to propose a next step. Give the classifier the customer’s message and the draft reply, then ask which next step the reply proposes. A scoring rule could award:

• 1 for proposing a meeting.
• 0.4 for asking the customer to reply.
• 0 when neither is present.

A reply classified as proposing a meeting receives a score of 1. That measures whether it meets the next-step requirement. Checking whether it quotes the correct price requires a separate judgment.

Mastra’s createClassifierScorer converts a classifier’s answer into an evaluation score between 0 and 1. Choice questions use the score mapping you define. Ordered ratings are normalized to that range, while Boolean questions return the estimated probability that a condition is true.

For a Boolean question such as “Does the reply include a next step?”, a score of 0.8 means an estimated 80% probability that it does. It does not mean the reply is 80% complete.

For simple checks, classifier judges can reduce evaluation cost and latency, as explored in Mastra’s Agents Hour episode on Jev. Check their judgments against human labels. An LLM judge may be a better fit when evaluation requires several reasoning steps or a written explanation.

Evaluation metrics: what the scores measure

The examples above introduced exact match, BLEU, ROUGE, BERTScore, and completeness while explaining how scorers work. The remaining metrics measure classification, answer quality, retrieval, agent actions, language prediction, speed, and cost. For each, I’ll explain what it measures and how the score is calculated.

How are accuracy, precision, recall, and F1 calculated for classification?

For classification tasks, these metrics count correct and incorrect predictions. The scoring method compares the model’s labels with labels supplied by a reviewer.

Suppose a test set contains 20 support tickets, six of which concern billing. The model labels five tickets as billing. Four of those are correct, one is wrongly included, and two billing tickets are missed. The other 13 tickets are correctly labeled as non-billing.

• Accuracy: The fraction of all predictions that are correct. Here, (4 + 13) ÷ 20 = 85%.
• Precision: How many tickets labeled “billing” belong in that category. Here, 4 ÷ 5 = 80%.
• Recall: How many of the actual billing tickets the model finds. Here, 4 ÷ 6 ≈ 67%.
• F1: A combined score balancing precision and recall. The formula is 2 × precision × recall ÷ (precision + recall), giving approximately 0.73.

For several categories, macro F1 averages each category’s F1 with equal weight. Micro F1 combines the prediction counts across categories before calculating the score, so more common categories contribute more.

How do you measure answer correctness?

Correctness measures whether an answer agrees with the expected facts. Suppose a customer’s meeting notes in Granola say Maya must finish the release notes by Thursday. When the user asks when those notes are due, “Thursday” is correct and “Friday” is incorrect.

A simple correctness metric awards 1 for a correct answer and 0 for an incorrect answer. The average across test questions gives the fraction answered correctly.

For meeting notes that assign Maya to finish the release notes by Thursday, a rule-based scorer can compare an extracted deadline field with “Thursday.” An LLM judge or human reviewer can check whether a free-text answer gives that same deadline.

Longer answers can be evaluated claim by claim. Ragas Factual Correctness, for example, compares claims in the response with a reference answer and calculates precision, recall, or F1.

How do you measure answer relevance?

Relevance measures whether a response addresses the user’s question. Suppose a user asks a meeting-notes assistant, “When does Maya need to finish the release notes?” A response describing the product’s new onboarding features is off-topic because it does not answer the deadline question, even if those descriptions are accurate.

In Mastra, our answer relevancy scorer uses an LLM to split the response into statements and judge each statement’s relevance. When all judgments are “yes” or “no,” the score is relevant statements divided by total statements. “Unsure” judgments receive partial credit. For example, three relevant statements and one irrelevant statement produce 3 ÷ 4 = 0.75 at the default scale.

How do you measure faithfulness, groundedness, and hallucinations?

Faithfulness measures factual consistency: whether the response’s claims are supported by the source material. A common calculation, used by Ragas, is supported claims ÷ total claims. An LLM checks each claim against the supplied context.

Suppose a customer’s meeting notes in Granola say, “Launch is Friday. Maya will update the release notes by Thursday.” The generated summary says, “Launch is Friday. Maya will update the release notes. The customer approved the release.” The first two summary claims are supported by the notes; customer approval is not. A faithfulness metric counting these as three claims gives 2 ÷ 3 ≈ 0.67.

Groundedness also measures support from supplied evidence. Its calculation depends on the evaluator: Microsoft’s groundedness evaluator, for example, uses a one-to-five scale.

A claim-level hallucination rate counts unsupported claims divided by all claims. For a response containing one unsupported claim out of three, the rate is 1 ÷ 3 ≈ 33%. A response-level rate counts the fraction of responses containing at least one unsupported claim.

Faithfulness checks whether the included claims are supported; completeness checks whether required information is missing. If meeting notes say Maya must write the release notes by Thursday, a summary saying only “Maya will write the release notes” preserves a supported fact but omits the deadline. It is faithful to the notes but incomplete under a rubric that requires deadlines.

How do you measure retrieval quality?

In retrieval augmented generation (RAG), an application searches a collection of documents, then gives the text it finds to an LLM to help answer a question. That search step is called retrieval. The retrieved text becomes the model’s context: the information it can use when writing its answer.

Retrieval metrics evaluate the search results. A document is relevant if it contains information that helps answer the question. To evaluate a search, a reviewer or an LLM judge labels which documents are relevant.

Suppose someone asks a workplace assistant, “What is our product launch plan?” A reviewer identifies four relevant documents: the launch schedule, release checklist, documentation plan, and customer announcement. The search returns the first three, plus a cafeteria menu and an expense policy. It finds three useful documents, adds two unrelated ones, and misses the customer announcement.

Precision and recall count different things. The k in precision@k or recall@k means we evaluate only the first k search results. Here, k is five.

• Precision@5: How much of what we found is useful? Three of the five returned documents help answer the launch-plan question, so precision is 3 ÷ 5 = 60%. The cafeteria menu and expense policy lower this score.
• Recall@5: How much of the useful material did we find? The collection contains four known relevant launch documents, but the first five results include only three. Recall is 3 ÷ 4 = 75%. The missing customer announcement lowers this score.

Both use the number of relevant documents found. Precision divides by the number returned; recall divides by the number known to be relevant.

nDCG measures whether the most useful results appear first. Its full name is normalized discounted cumulative gain. It gives credit for relevance, reduces that credit for results further down the list, then divides by the score of the best possible ordering. A score of 1 means the ranking matches that ideal. It can also account for degrees of relevance, such as “somewhat useful” versus “highly useful.”

For a launch-plan search that returns three relevant documents and two unrelated ones, moving the three relevant documents to the top improves nDCG. Precision@5 and recall@5 stay the same because the set of five documents has not changed.

Mastra’s context precision scorer also rewards useful results appearing earlier. It uses a calculation called mean average precision (MAP): at each relevant result, it calculates precision for the results seen so far, then averages those values. Irrelevant results near the top lower the score because they reduce precision at the relevant results below them.

How do you measure task success and tool use?

Task success measures whether an agent completes the requested work. Suppose a user asks an agent, “Create a task for Maya to finish the release notes by Thursday.” Success means the task is saved with Maya assigned and Thursday set as the deadline.

For a task-creation agent, a test can compare the saved task’s description, assignee, and due date with the user’s request. Across a test set, successful tasks ÷ attempted tasks gives the task success rate. In an illustrative run with 18 successful tasks out of 20, the rate is 90%.

Tool-call accuracy measures whether the agent selects the expected or appropriate tool. Mastra supports both code-based and LLM-based tool-call scorers: code can check an exact tool name or sequence, while a judge can assess whether the choice fits the request. Calling the task-creation tool and creating the correctly assigned task are separate checks.

What does perplexity measure?

Perplexity measures how well a language model predicts a text sample. A token is a word or part of a word. For each token already present in the sample, we measure the probability the model assigns to it given the preceding text. Lower perplexity means the model assigned higher probabilities to the tokens that actually appear.

For a simplified example, treat “Launch on Friday” as three tokens. Model A assigns a probability of 50% to each actual token; Model B assigns 25% to each. Their perplexities are 2 and 4 respectively. Model A’s lower score means it predicts this text better.

The general calculation uses negative log-likelihood, a penalty that grows when the model assigns a low probability to the actual token. Perplexity is the exponential of the average penalty. When every actual token has the same probability, as in the example, this simplifies to 1 ÷ probability.

Perplexity is useful for evaluating language prediction during model development. Its value depends on tokenization and available context; it does not directly measure whether an agent completes a user’s task.

How do you measure latency and cost?

Latency measures how long a request takes. Time to first token measures the wait before a streamed answer begins; end-to-end latency includes the full response and any tool calls. The p95 latency is the duration at or below which 95% of measured requests finish.

Cost per successful task measures the spend needed to deliver a completed outcome. It is total cost across all attempted tasks ÷ successful tasks. If 100 attempts cost $40 and 80 succeed, the cost is $40 ÷ 80 = $0.50 per success. Failed attempts and retries contribute to that cost.

Metrics by application

How to know whether your evaluation scores are trustworthy

Human evaluation lets you check an LLM judge by comparing its judgments with a reviewer’s assessment of the same outputs.

Define criteria and establish trusted labels

For a meeting-summary app such as Granola, a human reviewer checks the summary against the meeting notes and records which required details are missing.

Suppose the meeting notes say, “Maya will update the release notes by Thursday,” but the summary says only, “Maya will update the release notes.” If deadlines are required, the summary is incomplete.

A judge that awards full marks has missed that omission.

Human reviewers need to agree on what the rubric requires. Human feedback can help refine the judge’s instructions, but validation needs separate human-labeled examples that were not used to develop the judge. Otherwise, the test may only show that it handles examples it was tuned on.

Inspect false passes and false failures

Suppose a reviewer checks 100 meeting summaries. Twenty omit at least one required detail, and 80 include everything required. In this illustrative example, an LLM judge flags 12 of the incomplete summaries, misses eight, and wrongly flags two complete summaries.

A false pass happens when the judge accepts an incomplete summary: eight cases here. A false failure happens when it rejects a complete summary: two cases.

The matrix groups the judge’s decisions by whether they agree with the human reviewer.

In this case, the judge’s accuracy is 90% as it correctly flags 12 incomplete summaries and passes 78 complete ones. Its failure-detection recall is 60%, because it catches only 12 of the 20 incomplete summaries. The remaining eight incomplete summaries pass, so the judge misses 40% of the failures.

Check bias and repeatability

The MT-Bench and Chatbot Arena study found several biases in the LLM judges it tested:

• Position bias: A judge favors an answer because of where it appears in a comparison. Swapping two answers can change the winner.
• Verbosity bias: A longer answer receives a better judgment even when it adds no useful information.
• Self-enhancement bias: A model favors answers it generated itself.

Repeated grading of identical inputs shows how much the scores vary.

Account for uncertainty

Suppose a reviewer checks 50 meeting summaries and finds no missing required details. The observed failure rate is 0%, but a small sample can miss occasional failures.

For 50 independent, representative tests with zero failures and the same underlying failure probability, an exact one-sided 95% confidence bound puts the upper limit at about 5.8%. A system with a 5.8% failure rate still has about a 5% chance of passing all 50 tests.

Zero failures in that sample therefore cannot rule out a failure rate of several percent.

How to run LLM evaluations in Mastra Studio

Mastra Studio lets you run test cases against an agent and compare its scores after a change. A dataset stores the cases; an experiment runs them and records the outputs. Scorers measure output quality.

This workflow starts with an agent and scorers configured in your Mastra project, with storage enabled for datasets and results.

1. Create a dataset

In Studio, open Datasets and select Create Dataset. Give it a name, then use Add Item in the Items tab to add test cases.

Each item contains an input and, optionally, ground truth: the expected answer used by scorers that need a reference.

For example, a translation dataset can pair the input “Hello” with the expected Spanish answer “Hola.” Studio also supports importing cases from CSV or JSON.

A translation dataset in Mastra Studio.

2. Run an experiment with scorers

From the dataset, select Run Experiment. Choose the agent or workflow you want to test and the scorers that will grade its outputs.

An experiment can use multiple scorers, so each response can receive separate scores for different requirements.

The Experiments tab lists your runs. Opening a run shows individual test results, scores, and execution traces, so you can inspect the response behind a score.

3. Observability for your LLM eval metrics

Mastra’s metrics dashboard tracks scorer performance over time alongside token usage, estimated model costs, latency, and errors. The dashboard requires an observability store that supports metrics. This helps you check whether better evaluation scores come with slower responses or higher costs.

Studio brings metrics, traces, and logs together. Use the dashboard to spot trends, then inspect the related traces and logs to see what happened during a run.

4. Compare results after a change

When comparing models or prompt versions, run each experiment against the same dataset version with the same scorers. In Experiments, select Compare and choose the original run and the new one.

Studio shows the average for each scorer and the difference between runs. Its per-item comparison shows how each test case’s score changed, letting you identify which responses improved or declined.

Evaluate your agents with Mastra

I hope you now have a clearer idea of which LLM evaluation metrics fit your application and what their scores mean. You’ve also seen how to check an LLM judge and compare results across changes.

What I like about evals is that they give you a way to test your ideas. A new prompt or model might look promising, but running it against the same cases shows which problems it fixes and which ones remain.

We’ve built Mastra’s evaluation tools to help you do this with your own agents. Try evals in Mastra to set up your scorers and compare your first experiments in Studio.

Frequently asked questions

What is the best metric for evaluating an LLM?

The best metric depends on the application. Accuracy measures how often a support-ticket classifier assigns the correct category. For an agent that creates tasks, task success rate checks whether the record was saved with the requested assignee and deadline.

Can you evaluate LLM outputs without reference answers?

Yes. You don’t always need someone to write the ideal answer first. For a meeting summary, you can compare its claims with the original notes to check for unsupported information. You can also check whether an output follows a required format. Those checks still need clear criteria or evidence to score against.

What is a good LLM evaluation score?

A good score depends on the metric, its calculation, and the application’s requirements. A completeness score of 0.8 could mean a summary includes four of five required details. If the missing detail is an action item’s deadline, the summary may still fail to give the reader what they need.

Are BLEU and ROUGE useful for LLM applications?

Yes, especially when comparing translations or summaries with human-written references. BLEU and ROUGE measure overlap in wording, so they can miss a good paraphrase. Use them to measure that similarity and check factual accuracy separately.

How do you evaluate an LLM judge?

Compare the judge’s decisions with human reviews of the same outputs, using examples that weren’t used to tune its instructions. For a meeting-summary judge, check which incomplete summaries it accepts and which complete ones it rejects. Repeat the check on identical outputs to assess consistency.