OpenAI

Use OpenAI with Mastra in three ways:

• Agent tools let an agent call OpenAI APIs, for example to create images or manage files and vector stores, through Mastra Connect.
• Voice adds text-to-speech and speech-to-text with OpenAIVoice.
• Realtime adds speech-to-speech conversations with OpenAIRealtimeVoice.

To use an OpenAI model as the agent's language model, see OpenAI models instead.

Connect

Before you start, complete the Connect setup to install @mastra/connect and configure your platform project.

Agent tools

Give agents tools to create responses, chat completions, embeddings, images, and moderations, and manage files, batches, and vector stores in OpenAI.

Connect an account with the openai provider ID. Run this command from the directory that contains your .env file:

mastra connect add openai

Then pass tools() to the agent and select the provider:

src/mastra/agents/openai-agent.ts

import { Agent } from '@mastra/core/agent'
import { tools } from '@mastra/connect'

export const openaiAgent = new Agent({
  id: 'openai-agent',
  name: 'OpenAI agent',
  instructions:
    'You help people create responses, chat completions, embeddings, images, and moderations, and manage files, batches, and vector stores in OpenAI.',
  model: '__GATEWAY_OPENAI_MODEL_LUNA__',
  tools: tools({ providers: ['openai'] }),
})

The agent gets 26 OpenAI tools, with keys such as openai_add_vector_store_file. See OpenAI tools for the full list, and the tools() parameters to restrict tools or require approval before a call runs.

Voice

The OpenAIVoice class in Mastra provides text-to-speech and speech-to-text capabilities using OpenAI's models.

Usage example

import { OpenAIVoice } from '@mastra/voice-openai'

// Initialize with default configuration using environment variables
const voice = new OpenAIVoice()

// Or initialize with specific configuration
const voiceWithConfig = new OpenAIVoice({
  speechModel: {
    name: 'tts-1-hd',
    apiKey: 'your-openai-api-key',
  },
  listeningModel: {
    name: 'whisper-1',
    apiKey: 'your-openai-api-key',
  },
  speaker: 'alloy', // Default voice
})

// Convert text to speech
const audioStream = await voice.speak('Hello, how can I help you?', {
  speaker: 'nova', // Override default voice
  speed: 1.2, // Adjust speech speed
})

// Convert speech to text
const text = await voice.listen(audioStream, {
  filetype: 'mp3',
})

Configuration

Constructor options

speechModel?:

OpenAIConfig

= { name: 'tts-1' }

Configuration for text-to-speech synthesis.

OpenAIConfig

name?:

'tts-1' | 'tts-1-hd' | 'whisper-1'

Model name. Use 'tts-1-hd' for higher quality audio.

apiKey?:

string

OpenAI API key. Falls back to OPENAI_API_KEY environment variable.

listeningModel?:

OpenAIConfig

= { name: 'whisper-1' }

Configuration for speech-to-text recognition.

OpenAIConfig

name?:

'tts-1' | 'tts-1-hd' | 'whisper-1'

Model name. Use 'tts-1-hd' for higher quality audio.

apiKey?:

string

OpenAI API key. Falls back to OPENAI_API_KEY environment variable.

speaker?:

OpenAIVoiceId

= 'alloy'

Default voice ID for speech synthesis.

Methods

speak()

Converts text to speech using OpenAI's text-to-speech models.

input:

string | NodeJS.ReadableStream

Text or text stream to convert to speech.

options?:

Options

Configuration options.

Options

speaker?:

OpenAIVoiceId

Voice ID to use for speech synthesis.

speed?:

number

Speech speed multiplier.

Returns: Promise<NodeJS.ReadableStream>

listen()

Transcribes audio using OpenAI's Whisper model.

audioStream:

NodeJS.ReadableStream

Audio stream to transcribe.

options?:

Options

Configuration options.

Options

filetype?:

string

Audio format of the input stream.

Returns: Promise<string>

getSpeakers()

Returns an array of available voice options, where each node contains:

voiceId:

string

Unique identifier for the voice

Notes

• API keys can be provided via constructor options or the OPENAI_API_KEY environment variable
• The tts-1-hd model provides higher quality audio but may have slower processing times
• Speech recognition supports multiple audio formats including mp3, wav, and webm

Realtime

The OpenAIRealtimeVoice class provides real-time voice interaction capabilities using OpenAI's WebSocket-based API. It supports real time speech to speech, voice activity detection, and event-based audio streaming.

Usage example

import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'

// Initialize with default configuration using environment variables
const voice = new OpenAIRealtimeVoice()

// Or initialize with specific configuration
const voiceWithConfig = new OpenAIRealtimeVoice({
  apiKey: 'your-openai-api-key',
  model: 'gpt-realtime-2.1-mini',
  speaker: 'alloy', // Default voice
})

voiceWithConfig.updateSession({
  turn_detection: {
    type: 'server_vad',
    threshold: 0.6,
    silence_duration_ms: 1200,
  },
})

// Establish connection
await voice.connect()

// Set up event listeners
voice.on('speaker', ({ audio }) => {
  // Handle audio data (Int16Array) pcm format by default
  playAudio(audio)
})

voice.on('writing', ({ text, role }) => {
  // Handle transcribed text
  console.log(`${role}: ${text}`)
})

// Convert text to speech
await voice.speak('Hello, how can I help you today?', {
  speaker: 'echo', // Override default voice
})

// Process audio input
const microphoneStream = getMicrophoneStream()
await voice.send(microphoneStream)

// When done, disconnect
voice.connect()

Configuration

Constructor options

model?:

string

= 'gpt-realtime-2.1-mini'

The model ID to use for real-time voice interactions.

apiKey?:

string

OpenAI API key. Falls back to OPENAI_API_KEY environment variable.

speaker?:

string

= 'alloy'

Default voice ID for speech synthesis.

Voice Activity Detection (VAD) Configuration

type?:

string

= 'server_vad'

Type of VAD to use. Server-side VAD provides better accuracy.

threshold?:

number

= 0.5

Speech detection sensitivity (0.0-1.0).

prefix_padding_ms?:

number

= 1000

Milliseconds of audio to include before speech is detected.

silence_duration_ms?:

number

= 1000

Milliseconds of silence before ending a turn.

Methods

connect()

Establishes a connection to the OpenAI Realtime service. Must be called before using speak, listen, or send functions.

returns:

Promise<void>

Promise that resolves when the connection is established.

speak()

Emits a speaking event using the configured voice model. Can accept either a string or a readable stream as input.

input:

string | NodeJS.ReadableStream

Text or text stream to convert to speech.

options?:

Options

Configuration options.

Options

speaker?:

string

Voice ID to use for this specific speech request.

Returns: Promise<void>

listen()

Processes audio input for speech recognition. Takes a readable stream of audio data and emits a 'listening' event with the transcribed text.

audioData:

NodeJS.ReadableStream

Audio stream to transcribe.

Returns: Promise<void>

send()

Streams audio data in real-time to the OpenAI service for continuous audio streaming scenarios like live microphone input.

audioData:

NodeJS.ReadableStream

Audio stream to send to the service.

Returns: Promise<void>

updateConfig()

Updates the session configuration for the voice instance. This can modify voice settings and turn detection. It can also modify other parameters.

sessionConfig:

Realtime.SessionConfig

New session configuration to apply.

Returns: void

addTools()

Adds a set of tools to the voice instance. Tools allow the model to perform additional actions during conversations. When OpenAIRealtimeVoice is added to an Agent, any tools configured for the Agent will automatically be available to the voice interface.

tools?:

ToolsInput

Tools configuration to equip.

Returns: void

close()

Disconnects from the OpenAI Realtime session and cleans up resources. Should be called when you're done with the voice instance.

Returns: void

sendEvent()

Sends a raw client event to the OpenAI Realtime session. Use this for session control that has no dedicated method, such as adding conversation items. Events sent before the session is created are queued and sent once it is.

type:

string

OpenAI Realtime client event type, such as conversation.item.create.

data?:

Record<string, unknown>

= {}

Event payload sent alongside the type.

Returns: void

getSpeakers()

Returns a list of available voice speakers.

Returns: Promise<Array<{ voiceId: string; [key: string]: any }>>

on()

Registers an event listener for voice events.

event:

string

Name of the event to listen for.

callback:

Function

Function to call when the event occurs.

Returns: void

off()

Removes a previously registered event listener.

event:

string

Name of the event to stop listening to.

callback:

Function

The specific callback function to remove.

Returns: void

Events

The OpenAIRealtimeVoice class emits the following events:

speaking:

event

Emitted when audio data is received from the model. Callback receives { audio: Int16Array }.

writing:

event

Emitted when transcribed text is available. Callback receives { text: string, role: string }.

error:

event

Emitted when an error occurs. Callback receives the error object.

OpenAI Realtime Events

Every server event received from the OpenAI Realtime API is also emitted with the openAIRealtime: prefix, using the OpenAI server event type as the suffix. The callback receives the complete event payload.

voice.on('openAIRealtime:rate_limits.updated', event => {
  console.log(event.rate_limits)
})

Socket Events

open:

event

Emitted when the WebSocket connection to OpenAI opens.

close:

event

Emitted when the WebSocket connection closes, including when OpenAI closes it. Callback receives { code: number, reason: string }.

Available voices

The following voice options are available:

• alloy: Neutral and balanced
• ash: Clear and precise
• ballad: Melodic and smooth
• coral: Warm and friendly
• echo: Resonant and deep
• sage: Calm and thoughtful
• shimmer: Bright and energetic
• verse: Versatile and expressive

Notes

• API keys can be provided via constructor options or the OPENAI_API_KEY environment variable
• The OpenAI Realtime Voice API uses WebSockets for real-time communication
• Server-side Voice Activity Detection (VAD) provides better accuracy for speech detection
• All audio data is processed as Int16Array format
• The voice instance must be connected with connect() before using other methods
• Always call close() when done to properly clean up resources
• Memory management is handled by OpenAI Realtime API