Context Precision Scorer

The createContextPrecisionScorer() function creates a scorer that evaluates how relevant and well-positioned retrieved context pieces are for generating expected outputs. It uses Mean Average Precision (MAP) to reward systems that place relevant context earlier in the sequence.

It is especially useful for these use cases:

RAG System Evaluation

Ideal for evaluating retrieved context in RAG pipelines where:

• Context ordering matters for model performance
• You need to measure retrieval quality beyond simple relevance
• Early relevant context is more valuable than later relevant context

Context Window Optimization

Use when optimizing context selection for:

• Limited context windows
• Token budget constraints
• Multi-step reasoning tasks

Parameters

model:

MastraModelConfig

The language model to use for evaluating context relevance

options:

ContextPrecisionMetricOptions

Configuration options for the scorer

Note: Either context or contextExtractor must be provided. If both are provided, contextExtractor takes precedence.

.run() Returns

score:

number

Mean Average Precision score between 0 and scale (default 0-1)

reason:

string

Human-readable explanation of the context precision evaluation

Scoring Details

Mean Average Precision (MAP)

Context Precision uses Mean Average Precision to evaluate both relevance and positioning:

1. Context Evaluation: Each context piece is classified as relevant or irrelevant for generating the expected output
2. Precision Calculation: For each relevant context at position i, precision = relevant_items_so_far / (i + 1)
3. Average Precision: Sum all precision values and divide by total relevant items
4. Final Score: Multiply by scale factor and round to 2 decimals

Scoring Formula

MAP = (Σ Precision@k) / R

Where:
- Precision@k = (relevant items in positions 1...k) / k
- R = total number of relevant items
- Only calculated at positions where relevant items appear

Score Interpretation

• 0.9-1.0: Excellent precision - all relevant context early in sequence
• 0.7-0.8: Good precision - most relevant context well-positioned
• 0.4-0.6: Moderate precision - relevant context mixed with irrelevant
• 0.1-0.3: Poor precision - little relevant context or poorly positioned
• 0.0: No relevant context found

Reason analysis

The reason field explains:

• Which context pieces were deemed relevant/irrelevant
• How positioning affected the MAP calculation
• Specific relevance criteria used in evaluation

Optimization insights

Use results to:

• Improve retrieval: Filter out irrelevant context before ranking
• Optimize ranking: Ensure relevant context appears early
• Tune chunk size: Balance context detail vs. relevance precision
• Evaluate embeddings: Test different embedding models for better retrieval

Example Calculation

Given context: [relevant, irrelevant, relevant, irrelevant]

• Position 0: Relevant → Precision = 1/1 = 1.0
• Position 1: Skip (irrelevant)
• Position 2: Relevant → Precision = 2/3 = 0.67
• Position 3: Skip (irrelevant)

MAP = (1.0 + 0.67) / 2 = 0.835 ≈ 0.83

Scorer configuration

Dynamic context extraction

const scorer = createContextPrecisionScorer({
  model: "openai/gpt-4o-mini",
  options: {
    contextExtractor: (input, output) => {
      // Extract context dynamically based on the query
      const query = input?.inputMessages?.[0]?.content || "";

      // Example: Retrieve from a vector database
      const searchResults = vectorDB.search(query, { limit: 10 });
      return searchResults.map((result) => result.content);
    },
    scale: 1,
  },
});

Large context evaluation

const scorer = createContextPrecisionScorer({
  model: "openai/gpt-4o-mini",
  options: {
    context: [
      // Simulate retrieved documents from vector database
      "Document 1: Highly relevant content...",
      "Document 2: Somewhat related content...",
      "Document 3: Tangentially related...",
      "Document 4: Not relevant...",
      "Document 5: Highly relevant content...",
      // ... up to dozens of context pieces
    ],
  },
});

Examples

High precision example

This example shows perfect context precision where all relevant context appears early:

import { createContextPrecisionScorer } from "@mastra/evals";

const scorer = createContextPrecisionScorer({
  model: "openai/gpt-4o-mini",
  options: {
    context: [
      "Photosynthesis is the process by which plants convert sunlight, carbon dioxide, and water into glucose and oxygen.",
      "The process occurs in the chloroplasts of plant cells, specifically in the thylakoids.",
      "Light-dependent reactions happen in the thylakoid membranes, while the Calvin cycle occurs in the stroma.",
    ],
    scale: 1,
  },
});

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        id: "1",
        role: "user",
        content: "How does photosynthesis work in plants?",
      },
    ],
  },
  output: [
    {
      id: "2",
      role: "assistant",
      content:
        "Photosynthesis is the process where plants convert sunlight, CO2, and water into glucose and oxygen using chloroplasts.",
    },
  ],
});

console.log(result);
// Output:
// {
//   score: 1.0,
//   reason: "The score is 1.0 because all context pieces are highly relevant to explaining photosynthesis and are optimally ordered to support the expected output."
// }

Mixed precision example

This example shows moderate precision with both relevant and irrelevant context:

import { createContextPrecisionScorer } from "@mastra/evals";

const scorer = createContextPrecisionScorer({
  model: "openai/gpt-4o-mini",
  options: {
    context: [
      "Regular exercise improves cardiovascular health by strengthening the heart muscle.",
      "A balanced diet should include fruits, vegetables, and whole grains.",
      "Physical activity releases endorphins which improve mood and reduce stress.",
      "The average person should drink 8 glasses of water per day.",
      "Exercise also helps maintain healthy body weight and muscle mass.",
    ],
    scale: 1,
  },
});

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        id: "1",
        role: "user",
        content: "What are the mental and physical benefits of exercise?",
      },
    ],
  },
  output: [
    {
      id: "2",
      role: "assistant",
      content:
        "Exercise provides cardiovascular benefits, improves mood through endorphin release, and helps maintain healthy body composition.",
    },
  ],
});

console.log(result);
// Output:
// {
//   score: 0.72,
//   reason: "The score is 0.72 because contexts 1, 3, and 5 are relevant to exercise benefits, but irrelevant contexts about diet and hydration reduce the precision score."
// }

Low precision example

This example shows poor context precision with mostly irrelevant context:

import { createContextPrecisionScorer } from "@mastra/evals";

const scorer = createContextPrecisionScorer({
  model: "openai/gpt-4o-mini",
  options: {
    context: [
      "The weather forecast shows sunny skies this weekend.",
      "Coffee is one of the world's most popular beverages.",
      "Machine learning requires large amounts of training data.",
      "Cats typically sleep 12-16 hours per day.",
      "The capital of France is Paris.",
    ],
    scale: 1,
  },
});

const result = await scorer.run({
  input: {
    inputMessages: [
      {
        id: "1",
        role: "user",
        content: "How does photosynthesis work?",
      },
    ],
  },
  output: [
    {
      id: "2",
      role: "assistant",
      content:
        "Photosynthesis is the process by which plants convert sunlight into energy using chlorophyll.",
    },
  ],
});

console.log(result);
// Output:
// {
//   score: 0.0,
//   reason: "The score is 0.0 because none of the retrieved context pieces are relevant to explaining photosynthesis."
// }

Comparison with Context Relevance

Choose the right scorer for your needs:

Use Case	Context Relevance	Context Precision
RAG evaluation	When usage matters	When ranking matters
Context quality	Nuanced levels	Binary relevance
Missing detection	✓ Identifies gaps	✗ Not evaluated
Usage tracking	✓ Tracks utilization	✗ Not considered
Position sensitivity	✗ Position agnostic	✓ Rewards early placement

Related

• Answer Relevancy Scorer - Evaluates if answers address the question
• Faithfulness Scorer - Measures answer groundedness in context
• Custom Scorers - Creating your own evaluation metrics