Most voice AI architectures chain together three distinct external APIs: speech-to-text, an LLM, and text-to-speech. That setup means three separate network round trips, three vendors, extra points of failure, and a frustrating 2–3 second latency before the agent responds, not to mention shipping raw audio off to third-party servers.
In this guide, you’ll build a voice agent that eliminates every single one of those network hops. Speech recognition, reasoning, and synthesis all execute right inside the browser tab using Transformers.js v4 powered by WebGPU (with an automatic fallback to WASM when WebGPU isn’t available). Everything- audio inputs, transcripts, and model outputs remains completely local.
Thanks to WebGPU, modern browsers are finally fast enough to run a compact Whisper model alongside a quantized LLM for real-time conversational flow. We’ll cover how to stitch these three stages together, gracefully handle lower-spec hardware, and manage system state so your agent never talks over itself.
SpeechSynthesis API, strictly filtered to local, on-device voicesBy the end, you’ll have a working agent and a clear idea of what running everything locally means for latency, model size, and browser support.
See the Pen
Real-time voice agent by Miracle Jude (@JudeIV)
on CodePen.
Launch the app in a WebGPU-capable desktop browser like Chrome or Edge. The initial load downloads around 400MB of model weights, which are cached locally for future runs. Grant microphone access, wait for the models to load, and start speaking.
Although the pipeline relies on three distinct components, they all operate within a single unified runtime powered by Transformers.js.
Transformers.js brings Hugging Face’s core libraries directly into JavaScript. Built on ONNX Runtime Web, version 4 uses WebGPU as its primary acceleration engine with a WASM fallback. Because both Whisper and the LLM share the exact same pipeline() API, you get to skip the headache of managing separate engines or disparate API abstractions.
SpeechSynthesis APISpeechSynthesis speaks, preventing the agent from hearing and transcribing its own voiceThis loop control is more critical than it looks. A voice agent isn’t just three independent stages; it’s a closed loop with a strict state gate at each transition. Mic capture must pause before TTS begins, and TTS must fully wrap up before the mic reopens. If you miss this step, the agent ends up listening to itself, and the entire flow breaks down fast.
WebGPU handles the heavy lifting for both Whisper and the LLM, giving you direct access to GPU power right inside the browser. That compute is what makes running a local language model at conversational speeds possible without a server backend. If a user’s device lacks WebGPU support, Transformers.js automatically degrades to WASM, so both models still function, just at a slower pace. It’s important to understand how much slower before going live, which we’ll break down with real benchmark numbers next.
Whisper takes care of transcription, but this stage can be tricky on lower-end hardware. Picking a model that is too large causes high latency, while picking one that is too small leads to transcription errors.
Transformers.js exposes Whisper through the automatic-speech-recognition pipeline. Pass device: 'webgpu' explicitly in the options to enforce GPU execution rather than letting the runtime guess:
import { pipeline } from '@huggingface/transformers';
const transcriber = await pipeline(
'automatic-speech-recognition',
'onnx-community/whisper-base',
{
device: 'webgpu',
dtype: 'q4', // quantized weights, smaller download, faster inference
}
);
The dtype option is what makes this setup practical for the web. Standard full-precision Whisper weights balloon into hundreds of megabytes, but when quantized to 4-bit (q4), a model like whisper-base drops to roughly 40MB, small enough to fetch quickly and cache in the browser for instant reloads.
Whisper comes in five standard sizes: tiny, base, small, medium, and large. For real-time conversational agents, stick strictly to tiny or base. Anything larger introduces too much processing latency on consumer devices, breaking the flow of natural conversation.
| Model | Params | Quantized size (q4) | Accuracy | Real-time on mid-range GPU |
|---|---|---|---|---|
| tiny | 39M | ~15MB | Noticeably rougher on accents, background noise | Yes, comfortably |
| base | 74M | ~40MB | Solid for clear speech, single speaker | Yes, with headroom |
| small | 244M | ~120MB | Meaningfully better | Borderline on laptops without a discrete GPU |
Start with the base model. It delivers solid transcription accuracy without the heavy load times or compute costs of the small model. If you need to target lower-end devices, fall back to the tiny model and provide clear UI feedback so users understand why accuracy might take a slight hit.
Whisper isn’t built for native stream processing out of the box; it expects fixed-length audio buffers. To make it feel real-time, you need to chunk the incoming mic stream before passing it to the pipeline. A rolling audio buffer solves this: slice short audio segments, run each through Whisper, and maintain an overlapping context window so you don’t chop words in half mid-sentence.
Learn more →// chunker-worklet.js — runs on the audio rendering thread, not the main thread
class ChunkerProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.buffer = [];
this.chunkSize = 3000 * (16000 / 1000); // 3s at 16kHz
}
process(inputs) {
const input = inputs[0][0];
if (!input) return true;
this.buffer.push(...input);
if (this.buffer.length >= this.chunkSize) {
this.port.postMessage(new Float32Array(this.buffer));
this.buffer = [];
}
return true; // keep the processor alive
}
}
registerProcessor('chunker-processor', ChunkerProcessor);
This worklet runs in its own file on the dedicated audio rendering thread, a high-priority background thread reserved by the browser for low-latency audio tasks. Because it operates entirely off the main thread, it won’t block UI rendering or network requests, which is critical when timing matters. The main thread simply registers the worklet and listens for completed audio chunks over a message channel:
// main thread
const audioContext = new AudioContext({ sampleRate: 16000 });
await audioContext.audioWorklet.addModule('chunker-worklet.js');
const source = audioContext.createMediaStreamSource(micStream);
const chunker = new AudioWorkletNode(audioContext, 'chunker-processor');
chunker.port.onmessage = ({ data }) => {
transcribeChunk(data); // hand the chunk off to the Whisper worker
};
source.connect(chunker);
This setup is intentional: the audio rendering thread handles raw capture, the Whisper worker processes transcription, and the main thread manages UI state. Because each runs on its own thread, they won’t block one another during heavy workloads.
There is one drawback to a strict fixed-window approach: a hard 3-second boundary might slice a word in half, causing Whisper to hallucinate or mis-transcribe both fragments. The rolling buffer’s overlap helps smooth this out, but for a production-ready agent, it is better to use energy-based Voice Activity Detection (VAD). VAD segments audio dynamically at natural speech pauses rather than relying on a rigid timer, which is the exact approach used in the demo at the top.
Two things will paralyze your UI thread: running heavy model inference directly on it, and updating the DOM with every partial transcript token.
Run the transcriber inside a Web Worker. Transformers.js runs smoothly off the main thread, and moving it there is essential for building real-time, responsive agents. Without offloading inference to a worker, every single transcription step will cause the UI to stutter and freeze during live conversations.
// worker.js
self.onmessage = async ({ data }) => {
const result = await transcriber(data, {
return_timestamps: false,
});
self.postMessage({ text: result.text });
};
The worker only manages the model. It doesn’t touch the DOM or handle state; it just takes audio in and sends text out. This keeps the main thread free to update the user interface:
// main thread
const worker = new Worker('worker.js');
worker.onmessage = ({ data }) => {
updateTranscriptUI(data.text); // cheap DOM update, not a re-render of the whole tree
};
function transcribeChunk(audio) {
worker.postMessage(audio);
}
Make UI updates as lightweight as possible. In React, target individual text nodes or localized component states rather than triggering a re-render of the entire conversation list. If partial transcription tokens are streaming in rapidly, batch your state updates to keep frame rates smooth.
Now that you have clean text, this stage feels much more familiar: load a model, pass in a system prompt, and stream the generated tokens back. The main challenge is handling all of this without a server backend and without freezing the browser tab while a multi-hundred-megabyte model loads into memory.
You’ll use the same pipeline() API as Whisper, but with a different task type:
import { pipeline } from '@huggingface/transformers';
const generator = await pipeline(
'text-generation',
'onnx-community/Qwen2.5-0.5B-Instruct',
{
device: 'webgpu',
dtype: 'q4',
}
);
A 0.5B parameter model won’t write your unit tests, but it’s fast enough for back-and-forth conversation, which is what actually counts here. Quantized to 4-bit, it sits at around 350MB, a heavy initial download, but one the browser caches for subsequent loads.
If your target hardware can handle a larger model, something like Qwen2.5-1.5B-Instruct delivers noticeably better responses, though you will trade off longer load times and higher VRAM consumption.
Keep a close eye on total memory footprint. Holding both Whisper and the LLM in WebGPU simultaneously requires roughly 1 to 1.5 GB of VRAM. Desktop GPUs won’t even blink at that, but a low-end mobile browser under strict memory limits will outright kill the WebGPU context, which manifests as the agent silently dying mid-session rather than throwing a clean, catchable error. If you’re targeting mobile devices, that VRAM ceiling is a far bigger constraint than the initial download size.
Keep the conversation history and feed it through the model’s chat template rather than hand-rolling a prompt string:
const messages = [
{ role: 'system', content: 'You are a concise voice assistant. Keep responses short and conversational.' },
...conversationHistory,
{ role: 'user', content: transcript },
];
The system prompt is especially important here. If a voice agent replies with long paragraphs or lists, TTS will read everything, and users will stop listening before it’s done. Make sure to tell the model to keep responses short.
Just like with Whisper, run this in a Web Worker, not on the main thread. Even though a 0.5B model is small for an LLM, generating text still takes time. If the tab freezes, even for a second, it ruins the feel of a live conversation.
// llm-worker.js
import { TextStreamer } from '@huggingface/transformers';
self.onmessage = async ({ data }) => {
const streamer = new TextStreamer(tokenizer, {
skip_prompt: true,
callback_function: (token) => {
self.postMessage({ type: 'token', token });
},
});
await generator(data.messages, {
max_new_tokens: 200,
streamer,
});
self.postMessage({ type: 'done' });
};
TextStreamer helps make the experience feel real-time instead of waiting for a long response. Each token is sent to the main thread as soon as it’s generated, so the UI can show it right away. More importantly, TTS can start speaking the first sentence before the model finishes the whole response.
// main thread
const llmWorker = new Worker('llm-worker.js');
let currentSentence = '';
llmWorker.onmessage = ({ data }) => {
if (data.type === 'token') {
currentSentence += data.token;
updateResponseUI(currentSentence);
// Hand off complete sentences to TTS as they finish,
// don't wait for the full response
if (/[.!?]\s*$/.test(currentSentence)) {
speakSentence(currentSentence);
currentSentence = '';
}
}
};
The sentence-boundary check is important. Without it, TTS waits for the full response before speaking, and any latency savings from running locally are lost while waiting for the LLM to finish.
WebGPU isn’t available everywhere yet. Safari support is still improving, and many users have older hardware or GPUs that don’t support compute shaders in the browser. Transformers.js handles this by falling back to WASM if WebGPU isn’t available, but this fallback comes with a performance cost.
Detecting support takes one extra step people usually skip. Checking that navigator.gpu exists isn’t enough, since some devices expose the API but still fail to return a usable adapter. Request one and confirm it’s there before committing to WebGPU:
async function getDevice() {
if (!navigator.gpu) return 'wasm';
const adapter = await navigator.gpu.requestAdapter().catch(() => null);
return adapter ? 'webgpu' : 'wasm';
}
Then pass the result into the pipeline instead of guessing inline:
const generator = await pipeline(
'text-generation',
'onnx-community/Qwen2.5-0.5B-Instruct',
{
device: await getDevice(),
dtype: 'q4',
}
);
Resolving the device up front lets you set expectations before the model even loads, rather than letting the user discover the slowdown mid-conversation.
The difference in performance is clear. With WebGPU, the 0.5B model produces about 20 to 30 tokens per second, but with WASM, that drops to just 1 to 3 tokens per second. This speed might be fine for text generation, but for a voice agent, where quick responses are important, WASM is usually too slow to feel natural. You can either use a smaller model with WASM or let users know in the interface that their device may be slower.
This is where the ‘fully local’ setup can fail if you’re not careful. SpeechSynthesis is a built-in browser API, so there’s no download or model to load. However, not every voice it offers runs on your device; some use a network service, and the API doesn’t clearly show which ones are local.
When you call speechSynthesis.getVoices(), you’ll see both local and cloud-based voices, sometimes from the same vendor and with similar names. On Chrome, Google’s higher-quality voices use the network, while the local ones sound more robotic and are less prominently listed.
The API actually tells you which is which; you have to check:
function getLocalVoices() {
const voices = speechSynthesis.getVoices();
return voices.filter((voice) => voice.localService === true);
}
The localService flag is what matters. If you ignore it, your agent might claim to be fully local but actually send generated text to a third party as soon as it starts speaking.
function pickVoice(preferredLang = 'en-US') {
const localVoices = getLocalVoices();
// Prefer a voice matching the target language, fall back to any local voice
const match = localVoices.find((v) => v.lang === preferredLang);
return match || localVoices[0] || null;
}
Voice options depend on the user’s OS and browser, so always check for null and have a backup plan. If needed, let users know if their browser doesn’t have a usable local voice.
Once you’ve got a local voice, speaking is the easy part:
function speakSentence(text, voice) {
const utterance = new SpeechSynthesisUtterance(text);
utterance.voice = voice;
utterance.rate = 1.05; // a slightly faster rate sounds more natural for short responses
return new Promise((resolve) => {
utterance.onend = resolve;
speechSynthesis.speak(utterance);
});
}
Wrapping this in a promise that resolves when onend fires is important. You need to know exactly when speech finishes so you can safely reopen the mic.
Because the LLM worker streams completed sentences out as they are generated, multiple calls to speak() can pile up far faster than the browser can physically speak them.
While the Web Speech API’s SpeechSynthesis interface maintains an internal utterance queue by default, relying on browser-native queuing makes handling real-time interruptions messy and unpredictable. It is much better to manage an explicit application-level queue yourself.
class SpeechQueue {
constructor(voice) {
this.voice = voice;
this.queue = [];
this.speaking = false;
}
add(text) {
this.queue.push(text);
if (!this.speaking) this.next();
}
async next() {
if (this.queue.length === 0) {
this.speaking = false;
return;
}
this.speaking = true;
const text = this.queue.shift();
await speakSentence(text, this.voice);
this.next();
}
clear() {
this.queue = [];
speechSynthesis.cancel();
this.speaking = false;
}
}
Managing your own queue lets you use a single clear() method to handle interruptions, instead of dealing with the browser’s internal queue state.
Voice agents often get interrupted. If the user starts talking before the response is finished, the agent should stop mid-sentence instead of continuing to speak over the user.
function handleUserSpeaking() {
speechQueue.clear(); // stop TTS immediately
llmWorker.postMessage({ type: 'abort' }); // stop generation too, no point finishing a response nobody will hear
}
The LLM worker actually needs to respect that abort message; otherwise, it keeps generating tokens into a queue nobody’s draining:
// llm-worker.js
let aborted = false;
self.onmessage = async ({ data }) => {
if (data.type === 'abort') {
aborted = true;
return;
}
aborted = false;
const streamer = new TextStreamer(tokenizer, {
skip_prompt: true,
callback_function: (token) => {
if (aborted) return;
self.postMessage({ type: 'token', token });
},
});
await generator(data.messages, { max_new_tokens: 200, streamer });
};
This doesn’t fully cancel generation; the model still computes tokens in the background, but it stops sending them. A better solution would use an AbortController if supported, but just silencing the output is enough to make interruptions feel instant for users.
So far, each stage works in isolation: Whisper transcribes, the LLM generates, and SpeechSynthesis speaks. But if you chain them together without a central state machine, the microphone can easily stay active while the agent is talking, causing it to transcribe its own voice and feed those outputs back to the LLM as user input.
At any given moment, a voice agent only needs three core states:
LISTENING: The mic is open, Whisper is actively transcribing, and the system is waiting for the user to finish talkingTHINKING: The mic is muted, the transcript has been handed off to the LLM, and tokens are actively streaming backSPEAKING: The mic stays muted, SpeechSynthesis is playing back audio, and the sentence queue is drainingThe transitions are more important than the states themselves. LISTENING should only switch to THINKING when the user has actually stopped talking, not just paused to take a breath. THINKING should cut over to SPEAKING as soon as the very first sentence is ready, rather than waiting for the entire LLM response to complete. Finally, SPEAKING returns to LISTENING only after the playback queue is completely empty.
const STATES = {
LISTENING: 'listening',
THINKING: 'thinking',
SPEAKING: 'speaking',
};
class ConversationController {
constructor() {
this.state = STATES.LISTENING;
}
transition(next) {
console.log(`${this.state} -> ${next}`);
this.state = next;
}
}
The state machine is simple, but just tracking the state isn’t enough. To keep the mic quiet during SPEAKING, you need to actively enforce access control on the input stream based on the current state, rather than just logging transitions.
The microphone should only pass audio to Whisper while the controller is explicitly in the LISTENING state. In every other state, audio buffers still arrive from the hardware, but they must be immediately dropped before reaching the transcription worker:
chunker.port.onmessage = ({ data }) => {
if (controller.state !== STATES.LISTENING) return;
transcribeChunk(data);
};
That single guard clause prevents the agent from hearing itself. Without it, SpeechSynthesis output plays through the speakers, the mic picks it up, Whisper transcribes it, and the LLM ends up responding to its own voice. On laptops with speakers positioned right next to the internal microphone, this feedback loop triggers instantly and can be brutal to debug if you don’t block the input at the gate.
Attach state transitions directly to the event handlers where the work actually happens, rather than relying on a separate coordinator polling for status changes:
// End of user speech detected (silence threshold hit)
function onUserFinishedSpeaking(transcript) {
controller.transition(STATES.THINKING);
llmWorker.postMessage({ messages: buildMessages(transcript) });
}
// First sentence ready from the LLM
function onFirstSentenceReady(sentence) {
controller.transition(STATES.SPEAKING);
speechQueue.add(sentence);
}
// Speech queue fully drained
function onSpeechQueueEmpty() {
controller.transition(STATES.LISTENING);
}
onSpeechQueueEmpty should be triggered directly by the queue itself, never by an arbitrary timer or guess. Update the SpeechQueue class from the TTS section to fire a callback whenever next() finds no remaining text to process:
async next() {
if (this.queue.length === 0) {
this.speaking = false;
this.onEmpty?.(); // notify the controller
return;
}
this.speaking = true;
const text = this.queue.shift();
await speakSentence(text, this.voice);
this.next();
}
The interruption mechanism from the TTS section slots directly into this flow. If the user begins speaking while the controller is in the THINKING or SPEAKING state, treat it as a deliberate user interrupt rather than background noise to throw away.
function onUserStartedSpeaking() {
if (controller.state === STATES.LISTENING) return; // already listening, nothing to interrupt
speechQueue.clear();
llmWorker.postMessage({ type: 'abort' });
controller.transition(STATES.LISTENING);
}
Detecting when the user starts speaking while the mic is “off” presents a separate challenge, since Whisper isn’t running during the THINKING or SPEAKING states. A lightweight Voice Activity Detector (VAD) that monitors raw audio energy levels, rather than running full transcription, works perfectly here. It doesn’t need to parse words; it simply flags when sound crosses a defined volume threshold.
With the state machine controlling the mic and every transition bound to an actual runtime event rather than an arbitrary timer, the loop runs reliably: the agent listens, thinks, speaks, and cleanly hands control back at the exact right moment.
Every piece detailed above comes together in the accompanying CodePen demo: the VAD worklet, both Web Workers, the custom speech queue, and the state machine, all organized inside a single reference file.
To try it out, open the demo in a WebGPU-capable browser (such as desktop Chrome or Edge). The initial run downloads roughly 400MB of model weights, which are cached locally for subsequent visits. Grant microphone permissions, wait for the models to finish loading into VRAM, and start talking.
See the Pen
Real-time voice agent by Miracle Jude (@JudeIV)
on CodePen.
Two things differ from the code in this article, both deliberate. The workers are created from Blob URLs rather than separate files, since CodePen serves a single document. And the demo uses energy-based voice activity detection to segment utterances instead of fixed 3-second chunks, which is more reliable for a live conversation than transcribing on a timer.
The entire pipeline runs locally on your device: Whisper handles transcription, a quantized LLM generates responses, and SpeechSynthesis provides playback, all powered by Transformers.js with WebGPU acceleration (and WASM as a CPU fallback). No audio or text ever leaves the browser tab.
The core tradeoff here is between local capability, complete privacy, and operational latency:
Before choosing an in-browser WebGPU pipeline over a traditional server-based backend, make sure you clearly weigh those hardware requirements against your application’s privacy and latency thresholds.
Learn how to use browser AI APIs for summarization, translation, and language detection with practical examples and implementation tips.
Build a live task board with Topcoat and Rust, using reactive signals, server-rendered search, and WebSocket updates.
Learn how to build and install three Claude Code mods to monitor usage, protect secrets, and track changes.
I tested Jev against popular LLMs to compare speed, cost, and accuracy on real bank support messages.
Hey there, want to help make our blog better?
Join LogRocket’s Content Advisory Board. You’ll help inform the type of content we create and get access to exclusive meetups, social accreditation, and swag.
Sign up now
