Full-duplex voice agents in React Native

OpenAI's GPT-Live (gpt-live-1) is a full-duplex voice model. It listens while it speaks and decides for itself when to take a turn, when to back off, and when someone has interrupted it. There is no turn detection to configure and no barge-in to tune. Conversations feel closer to talking with a person.

@synervoz/openai-live-toolkit puts that model in a React Native app. One provider and two hooks handle the Live session, microphone and speaker I/O, platform echo cancellation, tool calls into your app, and a live transcript of both sides. Optional on-device speaker isolation keeps other voices in the room from reaching the model. It's built on the Switchboard SDK.

▶ Watch the demo on Loom (3½ min)

The example app running on an iPhone. It opens with talking over the model mid-answer, which GPT-Live handles without any barge-in settings. Then "what time is it?" and "make the background dark blue" run through two useTool hooks while the conversation continues. It closes on speaker isolation with someone else talking nearby: with it off, the transcript picks up the other voice, and with it on, only yours.

GPT-Live makes the conversation itself easy: speech goes up, speech comes back, and the model handles the turn-taking in between. What it gives you is a connection and an audio stream. Building an app on it still means solving:

• Capture and playback. Pulling audio off the microphone and playing the reply through the right audio route, at matching sample rates, in real time. The model streams audio continuously, silence included, so playback has to keep pace.
• The microphone hearing the speaker. A full-duplex model never stops listening. On speakerphone it hears its own voice the whole time it talks, unless acoustic echo cancellation removes it first.
• Tool calls. GPT-Live doesn't call functions. It hands work to a separate Responses model, which does, and it only hands off work it knows your app can do.
• A transcript. Both sides arrive as interleaved fragments that need grouping into lines.
• Staying connected. The session has to survive the screen locking and the app going to the background.

This toolkit is that layer.

Wrap your app, call the hook, register a tool, press the button.

import { Text, TouchableOpacity } from 'react-native'
import {
    OpenAILiveToolkitProvider,
    useOpenAILiveToolkit,
    useTool,
} from '@synervoz/openai-live-toolkit'

export default function App() {
    return (
        <OpenAILiveToolkitProvider instructions="You are a terse, friendly voice assistant.">
            <Screen />
        </OpenAILiveToolkitProvider>
    )
}

function Screen() {
    const { isRunning, start, stop } = useOpenAILiveToolkit()

    // Something the agent can actually do. Ask it the time.
    useTool({
        name: 'get_time',
        description: 'Get the current time.',
        handler: () => ({ time: new Date().toLocaleTimeString() }),
    })

    return (
        <TouchableOpacity onPress={isRunning ? stop : start}>
            <Text>{isRunning ? 'Stop' : 'Start talking'}</Text>
        </TouchableOpacity>
    )
}

That is a working voice agent, tool call included. start() requests the microphone, builds the audio graph with echo cancellation on, and opens the session. useTool tells the model the tool exists and routes the call to your handler; the return value goes back automatically. Talk over the agent and GPT-Live handles the interruption itself.

Because GPT-Live listens all the time, everything the microphone picks up is you as far as the model is concerned. A podcast playing, a colleague on a call at the next desk, a TV in the background: the model hears a second voice talking continuously, decides you haven't finished, and stops taking turns. In our tests with a competing talker, the model answered about one prompt in ten.

Speaker isolation fixes that on the device. ai-coustics Quail separates the main talker from the rest before the audio leaves the phone, so the model hears only the person holding it. With it on, the model answered six to ten prompts in ten in the same tests.

mic → mono → 48→16 kHz → Quail → 16→48 kHz → OpenAI.Live → speaker

It's a runtime switch, applied to the live session:

const { speakerIsolation } = useOpenAILiveToolkit()
speakerIsolation.setEnabled(true)

Speaker isolation is optional and iOS only. It needs an ai-coustics license key and the AICoustics Switchboard extension, which isn't published yet and has no Android build. Without it the graph runs mic → mono → OpenAI.Live → speaker and speakerIsolation.available is false. See Getting started.

useTool scales up to anything your app can do. It takes JSON Schema parameters and can change state or call your backend. Register a set_background_color tool and "make the background dark blue" repaints the screen. Handlers run while the conversation continues, so a slow one doesn't freeze the voice. See Tools for how delegation works, dynamic tool sets, and what happens when a handler throws.

• The OpenAI key stays on the backend. By default the session runs over WebRTC, and the Switchboard backend creates it with your Switchboard app's OpenAI key, so the app ships without one. The websocket transport needs the key in the app.
• Private Switchboard builds. The OpenAI.Live node isn't in a public Switchboard release yet, so pod install and the first Android build fetch a build from s3://switchboard-sdk and need AWS credentials that can read it. The WebRTC transport isn't in S3 yet either; see Getting started for using a local SDK build.
• Billing. GPT-Live bills per second of session, including while the app is backgrounded. Stop the engine when the conversation ends, or set idleTimeoutMs to close the session during silences and reopen it when the user speaks (Idle mode).
• iOS 14. The WebRTC transport needs it. Android needs API 24 and an app built with NDK 29 (Android setup).
• Physical device with AICoustics. Its framework has no simulator slice. Everything else runs in the simulator on Apple silicon Macs.

• Getting started covers install, frameworks, speaker isolation, iOS and Android setup, and credentials.
• Tools covers useTool, delegation, dynamic tool sets, and tool errors.
• API reference covers the provider, the useOpenAILiveToolkit() hook, every export, and the error codes.
• Example app is a complete screen with a transcript, the isolation toggle, and two tools.