voice.send()

The send() method streams audio data in real-time to voice providers for continuous processing. This method is essential for real-time speech-to-speech conversations, allowing you to send microphone input directly to the AI service.

Usage Example


import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
import Speaker from "@mastra/node-speaker";
import { getMicrophoneStream } from "@mastra/node-audio";
 
const speaker = new Speaker({
  sampleRate: 24100,  // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
  channels: 1,        // Mono audio output (as opposed to stereo which would be 2)
  bitDepth: 16,       // Bit depth for audio quality - CD quality standard (16-bit resolution)
});
 
 
// Initialize a real-time voice provider
const voice = new OpenAIRealtimeVoice({
  realtimeConfig: {
    model: "gpt-4o-mini-realtime",
    apiKey: process.env.OPENAI_API_KEY,
  },
});
 
// Connect to the real-time service
await voice.connect();
 
// Set up event listeners for responses
voice.on("writing", ({ text, role }) => {
  console.log(`${role}: ${text}`);
});
 
voice.on("speaker", (stream) => {
  stream.pipe(speaker)
});
 
// Get microphone stream (implementation depends on your environment)
const microphoneStream = getMicrophoneStream();
 
// Send audio data to the voice provider
await voice.send(microphoneStream);
 
// You can also send audio data as Int16Array
const audioBuffer = getAudioBuffer(); // Assume this returns Int16Array
await voice.send(audioBuffer);

Parameters

audioData:

NodeJS.ReadableStream | Int16Array

Audio data to send to the voice provider. Can be a readable stream (like a microphone stream) or an Int16Array of audio samples.

Return Value

Returns a Promise<void> that resolves when the audio data has been accepted by the voice provider.

Notes

This method is only implemented by real-time voice providers that support speech-to-speech capabilities
If called on a voice provider that doesn’t support this functionality, it will log a warning and resolve immediately
You must call connect() before using send() to establish the WebSocket connection
The audio format requirements depend on the specific voice provider
For continuous conversation, you typically call send() to transmit user audio, then answer() to trigger the AI response
The provider will typically emit ‘writing’ events with transcribed text as it processes the audio
When the AI responds, the provider will emit ‘speaking’ events with the audio response