Re-ranking Results

This example demonstrates how to implement a Retrieval-Augmented Generation (RAG) system with re-ranking using Mastra, OpenAI embeddings, and PGVector for vector storage.

Overview

The system implements RAG with re-ranking using Mastra and OpenAI. Here’s what it does:

Chunks text documents into smaller segments and creates embeddings from them
Stores vectors in a PostgreSQL database
Performs initial vector similarity search
Re-ranks results using Mastra’s rerank function, combining vector similarity, semantic relevance, and position scores
Compares initial and re-ranked results to show improvements

Setup

Environment Setup

Make sure to set up your environment variables:

.env

OPENAI_API_KEY=your_openai_api_key_here
POSTGRES_CONNECTION_STRING=your_connection_string_here

Dependencies

Then, import the necessary dependencies:

src/index.ts

import { openai } from '@ai-sdk/openai';
import { PgVector } from '@mastra/pg';
import { MDocument, rerank } from '@mastra/rag';
import { embedMany, embed } from 'ai';

Document Processing

Create a document and process it into chunks:

src/index.ts

const doc1 = MDocument.fromText(`
market data shows price resistance levels.
technical charts display moving averages.
support levels guide trading decisions.
breakout patterns signal entry points.
price action determines trade timing.
`);
 
const chunks = await doc1.chunk({
  strategy: 'recursive',
  size: 150,
  overlap: 20,
  separator: '\n',
});

Creating and Storing Embeddings

Generate embeddings for the chunks and store them in the vector database:

src/index.ts

const { embeddings } = await embedMany({
  values: chunks.map(chunk => chunk.text),
  model: openai.embedding('text-embedding-3-small'),
});
 
const pgVector = new PgVector(process.env.POSTGRES_CONNECTION_STRING!);
await pgVector.createIndex('embeddings', 1536);
await pgVector.upsert(
  'embeddings',
  embeddings,
  chunks?.map((chunk: any) => ({ text: chunk.text })),
);

Vector Search and Re-ranking

Perform vector search and re-rank the results:

src/index.ts

const query = 'explain technical trading analysis';
 
// Get query embedding
const { embedding: queryEmbedding } = await embed({
  value: query,
  model: openai.embedding('text-embedding-3-small'),
});
 
// Get initial results
const initialResults = await pgVector.query('embeddings', queryEmbedding, 3);
 
// Re-rank results
const rerankedResults = await rerank(initialResults, query, openai('gpt-4o-mini'), {
  weights: {
    semantic: 0.5,  // How well the content matches the query semantically
    vector: 0.3,    // Original vector similarity score
    position: 0.2   // Preserves original result ordering
  },
  topK: 3,
});

The weights control how different factors influence the final ranking:

semantic: Higher values prioritize semantic understanding and relevance to the query
vector: Higher values favor the original vector similarity scores
position: Higher values help maintain the original ordering of results

Comparing Results

Print both initial and re-ranked results to see the improvement:

src/index.ts

console.log('Initial Results:');
initialResults.forEach((result, index) => {
  console.log(`Result ${index + 1}:`, {
    text: result.metadata.text,
    score: result.score,
  });
});
 
console.log('Re-ranked Results:');
rerankedResults.forEach(({ result, score, details }, index) => {
  console.log(`Result ${index + 1}:`, {
    text: result.metadata.text,
    score: score,
    semantic: details.semantic,
    vector: details.vector,
    position: details.position,
  });
});

The re-ranked results show how combining vector similarity with semantic understanding can improve retrieval quality. Each result includes:

Overall score combining all factors
Semantic relevance score from the language model
Vector similarity score from the embedding comparison
Position-based score for maintaining original order when appropriate

View Example on GitHub

Metadata Filtering Re-ranking Results with Tools