RAG chatbot: a complete guide for TypeScript developers

Meta title: RAG Chatbot: A Complete Guide for TypeScript Developers

Meta description: Learn how to build a RAG chatbot in TypeScript, from embeddings and chunking to retrieval, evals, and deployment with Mastra.

Author: Aron Schuhmann

Reviewer: Sam Bhagwat

RAG chatbot: a complete guide for TypeScript developers

If you’re building a chatbot or assistant, your users expect the same level of quality and interactivity they’re used to from ChatGPT and Gemini. They want answers grounded in your actual data, and if they start seeing generic LLM completions trained on last year's internet, they’re going to be frustrated.

If you have a large document corpus, it’s worth adding RAG to your chatbot. That means retrieving relevant documents at query time and feeding them to a language model as context, so every response cites real sources instead of guessing.

According to the Menlo Ventures 2025 State of Generative AI in the Enterprise, RAG is the second most common customization technique in gen AI deployments after prompt engineering now uses some form of retrieval-augmented generation. The pattern has moved from research curiosity to production default because it solves the two problems teams hit first: AI hallucinations and stale knowledge.

This guide covers the full build, from embedding and chunking concepts through vector storage, retrieval pipelines, chat interfaces, testing, and deployment. You will walk away with a working architecture you can ship in TypeScript.

What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation (RAG) is an architecture pattern where you fetch relevant information from an external knowledge base and inject it into a large language model’s prompt before generating a response. The foundation model does not rely solely on its training data. Instead, it reasons over documents you control.

In a typical RAG pipeline, two things happen for every user query. First, a retrieval step searches your knowledge base for chunks that are semantically similar to the question. Second, a generation step passes those chunks as context to the LLM, which produces an answer grounded in the retrieved material.

Why RAG matters for chatbot use cases

If you are building a customer-facing chatbot, you need answers that reflect your product docs, your policies, and your latest release notes. A vanilla LLM cannot do that without retrieval because its training data is frozen at a cutoff date and does not include your proprietary content.

RAG gives you a way to keep the model current without retraining. You update the knowledge base, and the next query automatically retrieves the new information. For rag based chatbot applications, this means support answers stay accurate, internal tools reference the right runbooks, and documentation assistants always cite the latest version.

How RAG differs from semantic search and fine-tuning

You might wonder why you would not just fine-tune the model or use semantic search on its own. The three approaches solve different problems, and understanding the tradeoffs helps you pick the right one.

Fine-tuning teaches the model how to speak. RAG teaches it what to say. In most chatbot scenarios you want both accurate content and natural language output. Natural language processing benchmarks consistently show that retrieval-augmented models outperform frozen models on domain-specific Q&A, which is why RAG is the more practical starting point.

Core RAG concepts to understand

Before you write any retrieval code, you need a working mental model of three primitives: embeddings, chunks, and vector indexes. These are the building blocks every rag chatbot depends on.

Embedding

An embedding is a numerical vector that captures the semantic meaning of a piece of text. Embedding models are trained on NLP tasks across large corpora, which is what lets them map semantically similar phrases to nearby points in vector space. When you embed the sentence "How do I reset my password?" and the documentation paragraph about password resets, those two vectors end up close together in high-dimensional space. That proximity is what makes retrieval work.

You generate embeddings using a model like OpenAI's text-embedding-ada-002 or text-embedding-3-small. Open-source alternatives on HuggingFace like all-MiniLM-L6-v2 work well for lower-cost setups. On Mastra in TypeScript, the call looks like this:

import { embed } from "ai";
 
const { embedding } = await embed({
  model: "openai/text-embedding-3-small",
 
  value: "How do I reset my password?",
});

The returned vector (typically 1,536 dimensions) is what you store and search against.

Chunking

Your source documents are rarely the right size for embedding. A 40-page PDF embedded as a single vector produces a low-quality representation because too much meaning gets compressed into one point. Chunking splits documents into smaller segments so each embedding captures a focused idea.

Common chunking strategies include:

• Sentence splitting: Break on periods or sentence boundaries. Simple and effective for FAQ-style content.

• Fixed-size windows: Split every N tokens with a chunk overlap of M tokens. Good for long-form prose where sentence boundaries are not natural breakpoints.

• Recursive character splitting: Split by paragraph, then sentence, then character, until each chunk is under a target size. This is the default in most frameworks, including LangChain and Mastra.

The right chunk size depends on your content. Smaller chunks (100 to 200 tokens) improve retrieval precision. Larger chunks (400 to 600 tokens) preserve more context per result. Start with 200 to 300 tokens and adjust based on retrieval eval scores.

Vector storage and indexing

Once you have embeddings, you need somewhere to store them and a way to search them quickly. A vector database (or a relational database with vector extensions) handles both.

Popular options for TypeScript teams include:

• Postgres with pgvector: Adds vector columns and similarity search to your existing database. No new infrastructure if you already run Postgres.

• Pinecone: A managed vector database with built-in filtering and metadata support.

• ChromaDB: Open-source vector database with hybrid keyword search (keyword plus vector). LangChain integrates natively with ChromaDB, Pinecone, and pgvector through a unified retriever interface.

For indexing, you will typically choose between HNSW (hierarchical navigable small world) and IVFFlat. HNSW is faster at query time and handles incremental inserts well, which makes it the default for most production rag chat applications.

Benefits of RAG-powered chatbots

You might already have an intuition for why retrieval-augmented generation is useful. Here is what it concretely gives you in production.

Current, source-grounded responses

Your chatbot answers from documents you control. When your team ships a new API version, you re-embed the updated docs and the chatbot starts citing them on the next query. No retraining, no waiting for the next model release.

Reduced hallucinations and enhanced user trust

By constraining the model to reason over retrieved context, you dramatically reduce the surface area for hallucination. Users see citations pointing back to real source material, which builds trust. In customer-facing applications, this is the difference between a chatbot people use once and one they rely on daily.

More developer control without retraining the model

RAG keeps the foundation model frozen and moves the knowledge into a layer you manage directly. You can version your knowledge base, A/B test different chunking strategies, and swap embedding models without touching the LLM. That separation of concerns is what makes the pattern so practical for teams that need to iterate quickly.

Project setup and prerequisites

You will build this rag chatbot using a TypeScript stack: Next.js for the frontend and API routes, Mastra for agent orchestration and RAG, Drizzle ORM for database access, and Postgres with pgvector for vector storage.

Environment and dependencies

Start by scaffolding a Next.js project and installing the required packages:

npx create-next-app@latest rag-chatbot
cd rag-chatbot
pnpm add @mastra/core @mastra/rag @mastra/pg drizzle-orm @neondatabase/serverless
pnpm add -D drizzle-kit

The @mastra/rag package provides chunking, embedding, and vector store utilities. The @mastra/pg package provides the pgvector store. Mastra ties it together with agent orchestration and model routing across hundreds of models across dozens of providers through a single interface.

Database and schema setup

You need a Postgres instance with the pgvector extension enabled. Neon, Supabase, and Railway all support pgvector out of the box. Create two tables:

1. resources: Stores the original source content (the full document or paragraph before chunking).

2. embeddings: Stores each chunk's text, its vector, and a foreign key back to the parent resource.

Your embeddings schema should include an HNSW index on the vector column for fast cosine similarity queries.

API keys and model access

You will need an API key for your embedding and chat models. If you are using OpenAI, set OPENAI_API_KEY in your .env file. For Pinecone, add PINECONE_API_KEY. For in your .env file. Mastra supports hundreds of models across dozens of providers through a single interface, so you can swap to Anthropic, Google, or a self-hosted model later without rewriting your retrieval logic. See the Mastra RAG docs at mastra.ai/docs/rag/overview for the full setup guide.

Building the knowledge base

With your project scaffolded, the next step is to ingest documents, chunk them, generate embeddings, and store everything in your vector database.

Load and chunk your documents

Your ingestion pipeline starts by reading source material into memory. Most knowledge bases contain unstructured data: PDF files, markdown docs, HTML pages, and plain-text logs. For each format you need a loader that extracts raw text before chunking. The key decision is your chunking strategy.

A simple sentence-based chunker in TypeScript looks like this:

const generateChunks = (input: string): string[] => {
  return input
 
    .trim()
 
    .split(".")
 
    .filter((chunk) => chunk.trim() !== "");
};

For production use, consider a recursive splitter that respects paragraph and heading boundaries. This preserves more semantic coherence per chunk.

Generate and store embeddings

Once you have chunks, embed them in batch using Mastra’s embedMany function:

import { embedMany } from "ai";
 
export const generateEmbeddings = async (value: string) => {
  const chunks = generateChunks(value);
 
  const { embeddings } = await embedMany({
    model: "openai/text-embedding-3-small",
 
    values: chunks,
  });
 
  return embeddings.map((e, i) => ({ content: chunks[i], embedding: e }));
};

Then insert the results into your embeddings table with the resourceId linking each chunk back to its source document.

Index document chunks for retrieval

After insertion, verify that your HNSW index is active. Run a test similarity query against a known chunk to confirm that the index returns results in the expected order. This sanity check catches schema misconfigurations before you wire up the chat interface.

Implementing retrieval

Your knowledge base is populated. Now you need a function that takes a user query, embeds it, and returns the most relevant chunks.

Dense retrieval

Dense retrieval means you embed the query, then run vector search against your index using cosine similarity (the HNSW index handles speed at scale). A typical implementation in Drizzle ORM with pgvector looks like this:

import { cosineDistance, desc, gt, sql } from "drizzle-orm";
 
export const findRelevantContent = async (userQuery: string) => {
  const queryEmbedding = await generateEmbedding(userQuery);
 
  const similarity = sql<number>`1 - (\${cosineDistance(
 
embeddings.embedding,
 
queryEmbedding
 
)})`;
 
  return db
 
    .select({ content: embeddings.content, similarity })
 
    .from(embeddings)
 
    .where(gt(similarity, 0.5))
 
    .orderBy(desc(similarity))
 
    .limit(4);
};

The 0.5 threshold filters out low-relevance noise. The limit(4) keeps the context window manageable. Both values are tunable.

Reranking retrieved results

Dense retrieval alone can surface chunks that are semantically adjacent but not directly relevant to the question. A reranker (like Cohere Rerank or a cross-encoder machine learning model) rescores the top-k results using the full query-chunk pair, not just vector distance.

The typical flow is:

1. Retrieve the top 10 to 20 chunks by cosine similarity.

2. Pass those chunks plus the query to a reranker.

3. Take the top 3 to 5 reranked results as your final context.

Reranking adds latency (50 to 200ms depending on the model) but meaningfully improves answer accuracy, especially when your knowledge base contains overlapping or similar content. LangChain, LlamaIndex, and Mastra all provide reranker integrations you can drop into an existing pipeline.

Testing retrieval quality before wiring up the chat

Before you connect retrieval to the LLM, test it in isolation. Write a set of 10 to 20 representative queries and manually evaluate whether the retrieved chunks contain the information needed to answer each one. Track two metrics:

• Recall@k: What fraction of relevant chunks appear in the top k results?

• Precision@k: What fraction of the top k results are actually relevant?

If recall is low, your chunks may be too large or your embedding model may not fit your domain. If precision is low, consider adding a reranker or tightening your similarity threshold.

Building the chat interface and response pipeline

With retrieval working and tested, you can wire it into a conversational interface.

Search query generation from conversation history

In a multi-turn conversation, the user’s latest message often lacks context. Pull the full chat history into a context window and synthesize a standalone search query from it before calling the retriever. "What about the pricing?" makes no sense without knowing the previous messages discussed your Pro plan. Your pipeline should synthesize a standalone search query from the full conversation history before calling the retriever.

You can do this with a lightweight LLM call that takes the conversation and outputs a search query, or by concatenating the last two to three user messages into a single retrieval input.

Document retrieval and context injection

Once you have a search query, call your findRelevantContent function to get relevant chunks. Format those chunks into a context block and inject them into the system prompt:

const context = relevantChunks
 
  .map((c) => c.content)
 
  .join("\n\n");
 
const systemPrompt = `You are a helpful assistant. Answer using only the provided context.
 
If no relevant context is available, say "I don't have that information."
 
Context:
 
\${context}`;

This constrains the model to reason over your documents instead of its training data. Good prompt engineering in the system prompt, such as specifying tone, citation format, and fallback behavior, determines how well the model uses that context.

Response and citation generation

To build user trust, your chatbot should cite its sources. Include the chunk's source document ID or title in the context block, and instruct the model to reference those sources in its answer. This turns a generic response into a verifiable one.

Improving UX with multi-step tool calls

Instead of hardcoding the retrieval into every request, you can give the model a retrieval tool and let it decide when to search. Mastra supports this pattern with the tool function:

const tools = {
  getInformation: tool({
    description: "Search the knowledge base for relevant information",
 
    inputSchema: z.object({
      question: z.string().describe("the search query"),
    }),
 
    execute: async ({ question }) => findRelevantContent(question),
  }),
};

With stopWhen: stepCountIs(5), the model can call the tool, receive results, and then generate a response in one streaming turn. This is how you build a rag chatbot that feels like a conversation, not a search box.

Building a RAG chatbot with Mastra

Mastra

If you want to skip the boilerplate of wiring retrieval, memory, and deployment yourself, Mastra gives you agents, workflows, memory, evals, and observability in one framework, with a built-in model router (AI SDK providers are supported too, if you want them).

Defining a typed agent with tools and memory

You define a Mastra agent with instructions, a model, and tools in one declaration. Attach a retrieval tool that calls your vector store, and the framework handles conversation memory, tool execution, and multi-step reasoning out of the box. Mastra's agent definition is fully typed, so your IDE catches schema mismatches between tool inputs and retrieval outputs at compile time rather than at runtime.

Wiring retrieval into a Mastra workflow

For pipelines where you need explicit control (chunk, embed, retrieve, rerank, generate), use createWorkflow to define the pipeline. Chain steps with .then() for sequential execution, use .branch() with condition functions for conditional paths, run steps in parallel with .parallel(), and call .commit() to finalize the workflow. Durable state persists if a step suspends for human review, so your retrieval pipeline can pause for quality checks without losing context.

Deploying to Vercel, Cloudflare, or a standalone server

Mastra ships deployers for Vercel, Cloudflare Workers, Netlify, and a standalone Hono server. For self-hosted setups, Docker is supported for containerized deployment. Pick the target that matches your infrastructure and deploy with a single command. The framework supports model routing across hundreds of models across dozens of providers through a single interface, so you can swap embedding or chat models without rewriting your retrieval logic. Traces capture every step of a workflow or agent run, from the initial query embedding through retrieval scoring to the final generated response, giving you full visibility into production behavior.

Get started with Mastra to build your first RAG chatbot.

Testing, observability, and debugging your RAG pipeline

A working demo is not a production system. You need evals, traces, and guardrails before you hand your rag chatbot to real users.

Evaluating retrieval and answer quality

Set up automated evals that run on every code change. Track at least three metrics:

• Retrieval relevance: Do the top-k chunks contain the answer? Use a labeled dataset of query-answer pairs.

• Answer faithfulness: Does the generated answer stay within the bounds of the retrieved context? Score this with an LLM judge.

• Answer completeness: Does the response address the full question, or does it only cover part of it?

Tracing agent runs end to end

When a user reports a bad answer, you need to trace the full path: what query was generated, which chunks were retrieved, what context was injected, and what the model produced. Mastra ships built-in tracing and observability hooks that log each step of a workflow or agent run, so you can pinpoint whether the issue was retrieval, context formatting, or model reasoning.

Guardrails for prompt injection and output sanitization

Your chatbot accepts arbitrary user input, which means it is a prompt injection surface. Add guardrails at two points:

• Input: Validate and sanitize user messages before they reach the model. Reject or flag messages that attempt to override system instructions.

• Output: Check the model's response for PII leakage, off-topic content, or attempts to reveal the system prompt.

These checks add a few milliseconds per request and prevent the kinds of failures that erode user trust overnight.

RAG chatbot patterns and rag architecture variants

Not every rag based chatbot uses the same retrieval pattern. Your choice of architecture depends on the complexity of your queries, the size of your knowledge base, and your latency budget. LangChain’s ecosystem, including LangGraph for graph-based orchestration, covers several of these patterns in Python; the TypeScript equivalents are Mastra and LlamaIndex.TS.

The 7 common RAG types explained

Choosing the right pattern for your use case

Start with naive RAG. If retrieval precision is too low, add a reranker or hybrid keyword search pass (advanced RAG). If you need the model to decide what to search for, give it a retrieval tool (agentic RAG). Only reach for graph RAG or self-RAG when your domain complexity demands it, because each layer adds latency and implementation cost.

Most TypeScript teams building their first rag chat application land on advanced or agentic RAG. Those two patterns cover the majority of production use cases without requiring a knowledge graph or custom training loop.

You now have a complete blueprint for building a rag chatbot in TypeScript: from embedding and chunking through retrieval, reranking, chat interfaces, and production hardening with evals and guardrails. The pattern is straightforward, but the details (chunk size, similarity thresholds, reranker selection) are where production quality lives. Start simple, measure retrieval quality early, and iterate from there.