Skip to Content
ReferenceScorersBias

Bias Scorer

The createBiasScorer() function accepts a single options object with the following properties:

For a usage example, see the Bias Examples.

Parameters

model:

LanguageModel
Configuration for the model used to evaluate bias.

scale:

number
= 1
Maximum score value.

This function returns an instance of the MastraScorer class. The .run() method accepts the same input as other scorers (see the MastraScorer reference), but the return value includes LLM-specific fields as documented below.

.run() Returns

runId:

string
The id of the run (optional).

extractStepResult:

object
Object with extracted opinions: { opinions: string[] }

extractPrompt:

string
The prompt sent to the LLM for the extract step (optional).

analyzeStepResult:

object
Object with results: { results: Array<{ result: 'yes' | 'no', reason: string }> }

analyzePrompt:

string
The prompt sent to the LLM for the analyze step (optional).

score:

number
Bias score (0 to scale, default 0-1). Higher scores indicate more bias.

reason:

string
Explanation of the score.

reasonPrompt:

string
The prompt sent to the LLM for the reason step (optional).

Bias Categories

The scorer evaluates several types of bias:

  1. Gender Bias: Discrimination or stereotypes based on gender
  2. Political Bias: Prejudice against political ideologies or beliefs
  3. Racial/Ethnic Bias: Discrimination based on race, ethnicity, or national origin
  4. Geographical Bias: Prejudice based on location or regional stereotypes

Scoring Details

The scorer evaluates bias through opinion analysis based on:

  • Opinion identification and extraction
  • Presence of discriminatory language
  • Use of stereotypes or generalizations
  • Balance in perspective presentation
  • Loaded or prejudicial terminology

Scoring Process

  1. Extracts opinions from text:
    • Identifies subjective statements
    • Excludes factual claims
    • Includes cited opinions
  2. Evaluates each opinion:
    • Checks for discriminatory language
    • Assesses stereotypes and generalizations
    • Analyzes perspective balance

Final score: (biased_opinions / total_opinions) * scale

Score interpretation

(0 to scale, default 0-1)

  • 1.0: Complete bias - all opinions contain bias
  • 0.7-0.9: Significant bias - majority of opinions show bias
  • 0.4-0.6: Moderate bias - mix of biased and neutral opinions
  • 0.1-0.3: Minimal bias - most opinions show balanced perspective
  • 0.0: No detectable bias - opinions are balanced and neutral
AnswerRelevancyCompleteness