Speech-to-Speech capabilities in Mastra

Introduction

Speech-to-Speech (STS) in Mastra provides a standardized interface for real-time interactions across multiple providers. STS enables continuous bidirectional audio communication through listening to events from Realtime models. Unlike separate TTS and STT operations, STS maintains an open connection that processes speech continuously in both directions.

Configuration

• apiKey: Your OpenAI API key. Falls back to the OPENAI_API_KEY environment variable.
• model: The model ID to use for real-time voice interactions (e.g., gpt-5.1-realtime).
• speaker: The default voice ID for speech synthesis. This allows you to specify which voice to use for the speech output.

const voice = new OpenAIRealtimeVoice({
  apiKey: 'your-openai-api-key',
  model: 'gpt-5.1-realtime',
  speaker: 'alloy', // Default voice
})

// If using default settings the configuration can be simplified to:
const voice = new OpenAIRealtimeVoice()

Using STS

import { Agent } from '@mastra/core/agent'
import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'

const agent = new Agent({
  id: 'agent',
  name: 'OpenAI Realtime Agent',
  instructions: `You are a helpful assistant with real-time voice capabilities.`,
  model: 'openai/gpt-5.5',
  voice: new OpenAIRealtimeVoice(),
})

// Connect to the voice service
await agent.voice.connect()

// Listen for agent audio responses
agent.voice.on('speaker', ({ audio }) => {
  playAudio(audio)
})

// Initiate the conversation
await agent.voice.speak('How can I help you today?')

// Send continuous audio from the microphone
const micStream = getMicrophoneStream()
await agent.voice.send(micStream)

For a broader overview of voice providers on agents, see Voice in Mastra.

Use tools in realtime sessions

Realtime voice providers can use tools configured on the agent. Add the tools to the Agent definition, then connect and send audio through the voice provider:

import { Agent } from '@mastra/core/agent'
import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
import { calculate, search } from '../tools'

export const agent = new Agent({
  id: 'speech-to-speech-agent',
  name: 'Speech-to-Speech Agent',
  instructions: 'You are a helpful assistant with speech-to-speech capabilities.',
  model: 'openai/gpt-5.5',
  tools: {
    search,
    calculate,
  },
  voice: new OpenAIRealtimeVoice(),
})

Listen for realtime events

Realtime voice providers emit events you can use to update your UI, play assistant audio, log transcriptions, and handle errors:

agent.voice.on('speaking', ({ audio }) => {
  playAudio(audio)
})

agent.voice.on('writing', ({ text, role }) => {
  console.log(`${role}: ${text}`)
})

agent.voice.on('error', error => {
  console.error('Voice error:', error)
})

Event names and payloads vary by provider. Check the provider section below or the provider reference for the full event list.

Per-session voice instances

A static voice instance is shared across every request. This works for one-shot text-to-speech, but real-time and speech-to-speech providers store session state such as the WebSocket connection, tools, instructions, and request context. If one agent handles several live sessions at once, a shared instance can let one session overwrite another session's state.

Provide voice as a resolver when each live session needs its own voice instance. Mastra runs the resolver on each getVoice() call and returns a fresh instance for that request context:

import { Agent } from '@mastra/core/agent'
import { RequestContext } from '@mastra/core/request-context'
import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'

export const agent = new Agent({
  id: 'support-line',
  name: 'Support Line',
  instructions: ({ requestContext }) => `Help user ${requestContext.get('user')}.`,
  model: 'openai/gpt-5.5',
  voice: ({ requestContext }) =>
    new OpenAIRealtimeVoice({
      apiKey: requestContext.get('apiKey'),
    }),
})

const requestContext = new RequestContext()
requestContext.set('user', 'user-123')
requestContext.set('apiKey', process.env.OPENAI_API_KEY)

const voice = await agent.getVoice({ requestContext })
await voice.connect()

When you use a resolver:

• Each call to getVoice() returns a new instance, so concurrent sessions don't share state.
• Mastra doesn't add tools or instructions to a resolver instance. Configure them inside the resolver or on the provider.
• You own the returned instance lifecycle, so call disconnect() or close() when the session ends.

The agent.voice getter has no request context, so it throws when voice is a resolver. Use agent.getVoice({ requestContext }) instead.

Google Gemini Live (Realtime)

import { Agent } from '@mastra/core/agent'
import { GeminiLiveVoice } from '@mastra/voice-google-gemini-live'
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'

const agent = new Agent({
  id: 'agent',
  name: 'Gemini Live Agent',
  instructions: 'You are a helpful assistant with real-time voice capabilities.',
  // Model used for text generation; voice provider handles realtime audio
  model: 'openai/gpt-5.5',
  voice: new GeminiLiveVoice({
    apiKey: process.env.GOOGLE_API_KEY,
    model: 'gemini-2.0-flash-exp',
    speaker: 'Puck',
    debug: true,
    // Vertex AI option:
    // vertexAI: true,
    // project: 'your-gcp-project',
    // location: 'us-central1',
    // serviceAccountKeyFile: '/path/to/service-account.json',
  }),
})

await agent.voice.connect()

agent.voice.on('speaker', ({ audio }) => {
  playAudio(audio)
})

agent.voice.on('writing', ({ role, text }) => {
  console.log(`${role}: ${text}`)
})

await agent.voice.speak('How can I help you today?')

const micStream = getMicrophoneStream()
await agent.voice.send(micStream)

Note:

• Live API requires GOOGLE_API_KEY. Vertex AI requires project/location and service account credentials.
• Events: speaker (audio stream), writing (text), turnComplete, usage, and error.

AWS Nova Sonic (Realtime)

import { Agent } from '@mastra/core/agent'
import { NovaSonicVoice } from '@mastra/voice-aws-nova-sonic'
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'

const agent = new Agent({
  id: 'agent',
  name: 'Nova Sonic Agent',
  instructions: 'You are a helpful assistant with real-time voice capabilities.',
  // Model used for text generation; voice provider handles realtime audio
  model: 'openai/gpt-5.5',
  voice: new NovaSonicVoice({
    region: 'us-east-1',
    speaker: 'matthew',
    // Static credentials are optional. The default AWS credential provider
    // chain is used when none are passed.
  }),
})

await agent.voice.connect()

// Assistant audio is emitted as 16-bit PCM on the `speaking` event
agent.voice.on('speaking', ({ audioData }) => {
  if (audioData) playAudio(audioData)
})

agent.voice.on('writing', ({ role, text }) => {
  console.log(`${role}: ${text}`)
})

await agent.voice.speak('How can I help you today?')

const micStream = getMicrophoneStream()
await agent.voice.send(micStream)

Note:

• Available regions: us-east-1, us-west-2, and ap-northeast-1.
• Authenticates through the standard AWS credential provider chain. Pass credentials to override.
• Events: speaking (Int16Array audio), writing (text with generationStage), toolCall, interrupt, turnComplete, usage, session, and error.

Inworld Realtime

import { Agent } from '@mastra/core/agent'
import { InworldRealtimeVoice } from '@mastra/voice-inworld'
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'

const agent = new Agent({
  id: 'agent',
  name: 'Inworld Realtime Agent',
  instructions: 'You are a helpful assistant with real-time voice capabilities.',
  // Model used for text generation; voice provider handles realtime audio
  model: 'openai/gpt-5.5',
  voice: new InworldRealtimeVoice({
    apiKey: process.env.INWORLD_API_KEY,
    model: 'inworld/models/gemma-4-26b-a4b-it',
    speaker: 'Sarah',
    // Typed Inworld realtime knobs (semantic VAD, playback speed, etc.)
    // session: {
    //   audio: {
    //     output: { speed: 1.1 },
    //     input: { turn_detection: { type: 'semantic_vad', eagerness: 'high' } },
    //   },
    // },
  }),
})

await agent.voice.connect()

agent.voice.on('speaker', stream => {
  playAudio(stream)
})

agent.voice.on('writing', ({ role, text }) => {
  console.log(`${role}: ${text}`)
})

await agent.voice.speak('How can I help you today?')

const micStream = getMicrophoneStream()
await agent.voice.send(micStream)

Note:

• Requires INWORLD_API_KEY. Inworld API keys ship pre-Basic-encoded — paste them verbatim.
• The WebSocket URL appends a client-generated ?key=...&protocol=realtime. The model is configured via the initial session.update, not in the URL.
• Inworld's wire protocol is the OpenAI Realtime GA spec, so event names match @mastra/voice-openai-realtime.
• Typed Inworld realtime knobs (MCP tool routing, semantic VAD eagerness, playback speed, transcription model, output modalities, …) are exposed through the session constructor field; an untyped providerData escape hatch is also deep-merged for forward compatibility with new Inworld features.
• Events: speaker (PCM audio stream), speaking (audio Buffer per delta), writing (text), conversation.item.added, conversation.item.done, function_call.arguments, tool-call-start, tool-call-result, and error.