How to build a real time voice AI agent in the browser

Most voice AI architectures chain together three distinct external APIs: speech-to-text, an LLM, and text-to-speech. That setup means three separate network round trips, three vendors, extra points of failure, and a frustrating 2–3 second latency before the agent responds, not to mention shipping raw audio off to third-party servers.

In this guide, you’ll build a voice agent that eliminates every single one of those network hops. Speech recognition, reasoning, and synthesis all execute right inside the browser tab using Transformers.js v4 powered by WebGPU (with an automatic fallback to WASM when WebGPU isn’t available). Everything- audio inputs, transcripts, and model outputs remains completely local.

Thanks to WebGPU, modern browsers are finally fast enough to run a compact Whisper model alongside a quantized LLM for real-time conversational flow. We’ll cover how to stitch these three stages together, gracefully handle lower-spec hardware, and manage system state so your agent never talks over itself.

What does this voice AI agent’s tech stack include?

• STT: A quantized Whisper model executed directly via the Transformers.js pipeline API
• LLM: A compact, quantized language model running on the exact same runtime and backend
• TTS: The browser’s built-in SpeechSynthesis API, strictly filtered to local, on-device voices

By the end, you’ll have a working agent and a clear idea of what running everything locally means for latency, model size, and browser support.

See the Pen
Real-time voice agent by Miracle Jude (@JudeIV)
on CodePen.

Launch the app in a WebGPU-capable desktop browser like Chrome or Edge. The initial load downloads around 400MB of model weights, which are cached locally for future runs. Grant microphone access, wait for the models to load, and start speaking.

How does Transformers.js run speech-to-text and an LLM in the same browser tab?

Although the pipeline relies on three distinct components, they all operate within a single unified runtime powered by Transformers.js.

Transformers.js brings Hugging Face’s core libraries directly into JavaScript. Built on ONNX Runtime Web, version 4 uses WebGPU as its primary acceleration engine with a WASM fallback. Because both Whisper and the LLM share the exact same pipeline() API, you get to skip the headache of managing separate engines or disparate API abstractions.

The end-to-end flow

1. Mic capture: Raw audio is captured via the Web Audio API, sliced into short buffers, and pushed to Whisper
2. Real-time transcription: Whisper transcribes incoming chunks, streaming interim text updates while the user speaks
3. Turn transition: Once the user stops talking (triggered via silence detection or an explicit cutoff), the full transcript is passed to the local LLM
4. Token generation & speech: The LLM streams tokens, handing off completed sentences to the SpeechSynthesis API
5. Feedback loop protection: The microphone remains muted while SpeechSynthesis speaks, preventing the agent from hearing and transcribing its own voice

This loop control is more critical than it looks. A voice agent isn’t just three independent stages; it’s a closed loop with a strict state gate at each transition. Mic capture must pause before TTS begins, and TTS must fully wrap up before the mic reopens. If you miss this step, the agent ends up listening to itself, and the entire flow breaks down fast.

WebGPU handles the heavy lifting for both Whisper and the LLM, giving you direct access to GPU power right inside the browser. That compute is what makes running a local language model at conversational speeds possible without a server backend. If a user’s device lacks WebGPU support, Transformers.js automatically degrades to WASM, so both models still function, just at a slower pace. It’s important to understand how much slower before going live, which we’ll break down with real benchmark numbers next.

How do you run Whisper speech-to-text in the browser?

Whisper takes care of transcription, but this stage can be tricky on lower-end hardware. Picking a model that is too large causes high latency, while picking one that is too small leads to transcription errors.

Loading the model

Transformers.js exposes Whisper through the automatic-speech-recognition pipeline. Pass device: 'webgpu' explicitly in the options to enforce GPU execution rather than letting the runtime guess:

import { pipeline } from '@huggingface/transformers';

const transcriber = await pipeline(
  'automatic-speech-recognition',
  'onnx-community/whisper-base',
  {
    device: 'webgpu',
    dtype: 'q4', // quantized weights, smaller download, faster inference
  }
);

The dtype option is what makes this setup practical for the web. Standard full-precision Whisper weights balloon into hundreds of megabytes, but when quantized to 4-bit (q4), a model like whisper-base drops to roughly 40MB, small enough to fetch quickly and cache in the browser for instant reloads.

Selecting a model size

Whisper comes in five standard sizes: tiny, base, small, medium, and large. For real-time conversational agents, stick strictly to tiny or base. Anything larger introduces too much processing latency on consumer devices, breaking the flow of natural conversation.

Model	Params	Quantized size (q4)	Accuracy	Real-time on mid-range GPU
tiny	39M	~15MB	Noticeably rougher on accents, background noise	Yes, comfortably
base	74M	~40MB	Solid for clear speech, single speaker	Yes, with headroom
small	244M	~120MB	Meaningfully better	Borderline on laptops without a discrete GPU

Start with the base model. It delivers solid transcription accuracy without the heavy load times or compute costs of the small model. If you need to target lower-end devices, fall back to the tiny model and provide clear UI feedback so users understand why accuracy might take a slight hit.

Chunking mic audio

Whisper isn’t built for native stream processing out of the box; it expects fixed-length audio buffers. To make it feel real-time, you need to chunk the incoming mic stream before passing it to the pipeline. A rolling audio buffer solves this: slice short audio segments, run each through Whisper, and maintain an overlapping context window so you don’t chop words in half mid-sentence.

// chunker-worklet.js — runs on the audio rendering thread, not the main thread
class ChunkerProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
    this.buffer = [];
    this.chunkSize = 3000 * (16000 / 1000); // 3s at 16kHz
  }

  process(inputs) {
    const input = inputs[0][0];
    if (!input) return true;

    this.buffer.push(...input);

    if (this.buffer.length >= this.chunkSize) {
      this.port.postMessage(new Float32Array(this.buffer));
      this.buffer = [];
    }

    return true; // keep the processor alive
  }
}

registerProcessor('chunker-processor', ChunkerProcessor);

This worklet runs in its own file on the dedicated audio rendering thread, a high-priority background thread reserved by the browser for low-latency audio tasks. Because it operates entirely off the main thread, it won’t block UI rendering or network requests, which is critical when timing matters. The main thread simply registers the worklet and listens for completed audio chunks over a message channel:

// main thread
const audioContext = new AudioContext({ sampleRate: 16000 });
await audioContext.audioWorklet.addModule('chunker-worklet.js');

const source = audioContext.createMediaStreamSource(micStream);
const chunker = new AudioWorkletNode(audioContext, 'chunker-processor');

chunker.port.onmessage = ({ data }) => {
  transcribeChunk(data); // hand the chunk off to the Whisper worker
};

source.connect(chunker);

This setup is intentional: the audio rendering thread handles raw capture, the Whisper worker processes transcription, and the main thread manages UI state. Because each runs on its own thread, they won’t block one another during heavy workloads.

There is one drawback to a strict fixed-window approach: a hard 3-second boundary might slice a word in half, causing Whisper to hallucinate or mis-transcribe both fragments. The rolling buffer’s overlap helps smooth this out, but for a production-ready agent, it is better to use energy-based Voice Activity Detection (VAD). VAD segments audio dynamically at natural speech pauses rather than relying on a rigid timer, which is the exact approach used in the demo at the top.

Streaming interim transcripts without freezing the UI

Two things will paralyze your UI thread: running heavy model inference directly on it, and updating the DOM with every partial transcript token.

Run the transcriber inside a Web Worker. Transformers.js runs smoothly off the main thread, and moving it there is essential for building real-time, responsive agents. Without offloading inference to a worker, every single transcription step will cause the UI to stutter and freeze during live conversations.

// worker.js
self.onmessage = async ({ data }) => {
  const result = await transcriber(data, {
    return_timestamps: false,
  });
  self.postMessage({ text: result.text });
};

The worker only manages the model. It doesn’t touch the DOM or handle state; it just takes audio in and sends text out. This keeps the main thread free to update the user interface:

// main thread
const worker = new Worker('worker.js');

worker.onmessage = ({ data }) => {
  updateTranscriptUI(data.text); // cheap DOM update, not a re-render of the whole tree
};

function transcribeChunk(audio) {
  worker.postMessage(audio);
}

Make UI updates as lightweight as possible. In React, target individual text nodes or localized component states rather than triggering a re-render of the entire conversation list. If partial transcription tokens are streaming in rapidly, batch your state updates to keep frame rates smooth.

How do you run a local LLM in the browser for voice responses?

Now that you have clean text, this stage feels much more familiar: load a model, pass in a system prompt, and stream the generated tokens back. The main challenge is handling all of this without a server backend and without freezing the browser tab while a multi-hundred-megabyte model loads into memory.

Loading the model

You’ll use the same pipeline() API as Whisper, but with a different task type:

import { pipeline } from '@huggingface/transformers';

const generator = await pipeline(
  'text-generation',
  'onnx-community/Qwen2.5-0.5B-Instruct',
  {
    device: 'webgpu',
    dtype: 'q4',
  }
);

A 0.5B parameter model won’t write your unit tests, but it’s fast enough for back-and-forth conversation, which is what actually counts here. Quantized to 4-bit, it sits at around 350MB, a heavy initial download, but one the browser caches for subsequent loads.

If your target hardware can handle a larger model, something like Qwen2.5-1.5B-Instruct delivers noticeably better responses, though you will trade off longer load times and higher VRAM consumption.

Keep a close eye on total memory footprint. Holding both Whisper and the LLM in WebGPU simultaneously requires roughly 1 to 1.5 GB of VRAM. Desktop GPUs won’t even blink at that, but a low-end mobile browser under strict memory limits will outright kill the WebGPU context, which manifests as the agent silently dying mid-session rather than throwing a clean, catchable error. If you’re targeting mobile devices, that VRAM ceiling is a far bigger constraint than the initial download size.

Building the prompt

Keep the conversation history and feed it through the model’s chat template rather than hand-rolling a prompt string:

const messages = [
  { role: 'system', content: 'You are a concise voice assistant. Keep responses short and conversational.' },
  ...conversationHistory,
  { role: 'user', content: transcript },
];

The system prompt is especially important here. If a voice agent replies with long paragraphs or lists, TTS will read everything, and users will stop listening before it’s done. Make sure to tell the model to keep responses short.

Streaming tokens without blocking the main thread

Just like with Whisper, run this in a Web Worker, not on the main thread. Even though a 0.5B model is small for an LLM, generating text still takes time. If the tab freezes, even for a second, it ruins the feel of a live conversation.

// llm-worker.js
import { TextStreamer } from '@huggingface/transformers';

self.onmessage = async ({ data }) => {
  const streamer = new TextStreamer(tokenizer, {
    skip_prompt: true,
    callback_function: (token) => {
      self.postMessage({ type: 'token', token });
    },
  });

  await generator(data.messages, {
    max_new_tokens: 200,
    streamer,
  });

  self.postMessage({ type: 'done' });
};

TextStreamer helps make the experience feel real-time instead of waiting for a long response. Each token is sent to the main thread as soon as it’s generated, so the UI can show it right away. More importantly, TTS can start speaking the first sentence before the model finishes the whole response.

// main thread
const llmWorker = new Worker('llm-worker.js');
let currentSentence = '';

llmWorker.onmessage = ({ data }) => {
  if (data.type === 'token') {
    currentSentence += data.token;
    updateResponseUI(currentSentence);

    // Hand off complete sentences to TTS as they finish,
    // don't wait for the full response
    if (/[.!?]\s*$/.test(currentSentence)) {
      speakSentence(currentSentence);
      currentSentence = '';
    }
  }
};

The sentence-boundary check is important. Without it, TTS waits for the full response before speaking, and any latency savings from running locally are lost while waiting for the LLM to finish.

The WASM fallback

WebGPU isn’t available everywhere yet. Safari support is still improving, and many users have older hardware or GPUs that don’t support compute shaders in the browser. Transformers.js handles this by falling back to WASM if WebGPU isn’t available, but this fallback comes with a performance cost.

Detecting support takes one extra step people usually skip. Checking that navigator.gpu exists isn’t enough, since some devices expose the API but still fail to return a usable adapter. Request one and confirm it’s there before committing to WebGPU:

async function getDevice() {
  if (!navigator.gpu) return 'wasm';
  const adapter = await navigator.gpu.requestAdapter().catch(() => null);
  return adapter ? 'webgpu' : 'wasm';
}

Then pass the result into the pipeline instead of guessing inline:

const generator = await pipeline(
  'text-generation',
  'onnx-community/Qwen2.5-0.5B-Instruct',
  {
    device: await getDevice(),
    dtype: 'q4',
  }
);

Resolving the device up front lets you set expectations before the model even loads, rather than letting the user discover the slowdown mid-conversation.

The difference in performance is clear. With WebGPU, the 0.5B model produces about 20 to 30 tokens per second, but with WASM, that drops to just 1 to 3 tokens per second. This speed might be fine for text generation, but for a voice agent, where quick responses are important, WASM is usually too slow to feel natural. You can either use a smaller model with WASM or let users know in the interface that their device may be slower.

How do you keep browser text-to-speech fully local?

This is where the ‘fully local’ setup can fail if you’re not careful. SpeechSynthesis is a built-in browser API, so there’s no download or model to load. However, not every voice it offers runs on your device; some use a network service, and the API doesn’t clearly show which ones are local.

The voice filtering problem

When you call speechSynthesis.getVoices(), you’ll see both local and cloud-based voices, sometimes from the same vendor and with similar names. On Chrome, Google’s higher-quality voices use the network, while the local ones sound more robotic and are less prominently listed.

The API actually tells you which is which; you have to check:

function getLocalVoices() {
  const voices = speechSynthesis.getVoices();
  return voices.filter((voice) => voice.localService === true);
}

The localService flag is what matters. If you ignore it, your agent might claim to be fully local but actually send generated text to a third party as soon as it starts speaking.

function pickVoice(preferredLang = 'en-US') {
  const localVoices = getLocalVoices();

  // Prefer a voice matching the target language, fall back to any local voice
  const match = localVoices.find((v) => v.lang === preferredLang);
  return match || localVoices[0] || null;
}

Voice options depend on the user’s OS and browser, so always check for null and have a backup plan. If needed, let users know if their browser doesn’t have a usable local voice.

Speaking a sentence

Once you’ve got a local voice, speaking is the easy part:

function speakSentence(text, voice) {
  const utterance = new SpeechSynthesisUtterance(text);
  utterance.voice = voice;
  utterance.rate = 1.05; // a slightly faster rate sounds more natural for short responses
  return new Promise((resolve) => {
    utterance.onend = resolve;
    speechSynthesis.speak(utterance);
  });
}

Wrapping this in a promise that resolves when onend fires is important. You need to know exactly when speech finishes so you can safely reopen the mic.

Queue management

Because the LLM worker streams completed sentences out as they are generated, multiple calls to speak() can pile up far faster than the browser can physically speak them.

While the Web Speech API’s SpeechSynthesis interface maintains an internal utterance queue by default, relying on browser-native queuing makes handling real-time interruptions messy and unpredictable. It is much better to manage an explicit application-level queue yourself.

class SpeechQueue {
  constructor(voice) {
    this.voice = voice;
    this.queue = [];
    this.speaking = false;
  }

  add(text) {
    this.queue.push(text);
    if (!this.speaking) this.next();
  }

  async next() {
    if (this.queue.length === 0) {
      this.speaking = false;
      return;
    }
    this.speaking = true;
    const text = this.queue.shift();
    await speakSentence(text, this.voice);
    this.next();
  }

  clear() {
    this.queue = [];
    speechSynthesis.cancel();
    this.speaking = false;
  }
}

Managing your own queue lets you use a single clear() method to handle interruptions, instead of dealing with the browser’s internal queue state.

Handling interruptions

Voice agents often get interrupted. If the user starts talking before the response is finished, the agent should stop mid-sentence instead of continuing to speak over the user.

function handleUserSpeaking() {
  speechQueue.clear(); // stop TTS immediately

  llmWorker.postMessage({ type: 'abort' }); // stop generation too, no point finishing a response nobody will hear
}

The LLM worker actually needs to respect that abort message; otherwise, it keeps generating tokens into a queue nobody’s draining:

// llm-worker.js
let aborted = false;

self.onmessage = async ({ data }) => {
  if (data.type === 'abort') {
    aborted = true;
    return;
  }

  aborted = false;
  const streamer = new TextStreamer(tokenizer, {
    skip_prompt: true,
    callback_function: (token) => {
      if (aborted) return;
      self.postMessage({ type: 'token', token });
    },
  });

  await generator(data.messages, { max_new_tokens: 200, streamer });
};

This doesn’t fully cancel generation; the model still computes tokens in the background, but it stops sending them. A better solution would use an AbortController if supported, but just silencing the output is enough to make interruptions feel instant for users.

How do you prevent a voice agent from hearing itself?

So far, each stage works in isolation: Whisper transcribes, the LLM generates, and SpeechSynthesis speaks. But if you chain them together without a central state machine, the microphone can easily stay active while the agent is talking, causing it to transcribe its own voice and feed those outputs back to the LLM as user input.

The three states

At any given moment, a voice agent only needs three core states:

• LISTENING: The mic is open, Whisper is actively transcribing, and the system is waiting for the user to finish talking
• THINKING: The mic is muted, the transcript has been handed off to the LLM, and tokens are actively streaming back
• SPEAKING: The mic stays muted, SpeechSynthesis is playing back audio, and the sentence queue is draining

The transitions are more important than the states themselves. LISTENING should only switch to THINKING when the user has actually stopped talking, not just paused to take a breath. THINKING should cut over to SPEAKING as soon as the very first sentence is ready, rather than waiting for the entire LLM response to complete. Finally, SPEAKING returns to LISTENING only after the playback queue is completely empty.

const STATES = {
  LISTENING: 'listening',
  THINKING: 'thinking',
  SPEAKING: 'speaking',
};

class ConversationController {
  constructor() {
    this.state = STATES.LISTENING;
  }

  transition(next) {
    console.log(`${this.state} -> ${next}`);
    this.state = next;
  }
}

The state machine is simple, but just tracking the state isn’t enough. To keep the mic quiet during SPEAKING, you need to actively enforce access control on the input stream based on the current state, rather than just logging transitions.

Gating the mic

The microphone should only pass audio to Whisper while the controller is explicitly in the LISTENING state. In every other state, audio buffers still arrive from the hardware, but they must be immediately dropped before reaching the transcription worker:

chunker.port.onmessage = ({ data }) => {
  if (controller.state !== STATES.LISTENING) return;
  transcribeChunk(data);
};

That single guard clause prevents the agent from hearing itself. Without it, SpeechSynthesis output plays through the speakers, the mic picks it up, Whisper transcribes it, and the LLM ends up responding to its own voice. On laptops with speakers positioned right next to the internal microphone, this feedback loop triggers instantly and can be brutal to debug if you don’t block the input at the gate.

Wiring the transitions

Attach state transitions directly to the event handlers where the work actually happens, rather than relying on a separate coordinator polling for status changes:

// End of user speech detected (silence threshold hit)
function onUserFinishedSpeaking(transcript) {
  controller.transition(STATES.THINKING);
  llmWorker.postMessage({ messages: buildMessages(transcript) });
}

// First sentence ready from the LLM
function onFirstSentenceReady(sentence) {
  controller.transition(STATES.SPEAKING);
  speechQueue.add(sentence);
}

// Speech queue fully drained
function onSpeechQueueEmpty() {
  controller.transition(STATES.LISTENING);
}

onSpeechQueueEmpty should be triggered directly by the queue itself, never by an arbitrary timer or guess. Update the SpeechQueue class from the TTS section to fire a callback whenever next() finds no remaining text to process:

async next() {
  if (this.queue.length === 0) {
    this.speaking = false;
    this.onEmpty?.(); // notify the controller
    return;
  }
  this.speaking = true;
  const text = this.queue.shift();
  await speakSentence(text, this.voice);
  this.next();
}

Handling interruptions within the state machine

The interruption mechanism from the TTS section slots directly into this flow. If the user begins speaking while the controller is in the THINKING or SPEAKING state, treat it as a deliberate user interrupt rather than background noise to throw away.

function onUserStartedSpeaking() {
  if (controller.state === STATES.LISTENING) return; // already listening, nothing to interrupt

  speechQueue.clear();
  llmWorker.postMessage({ type: 'abort' });
  controller.transition(STATES.LISTENING);
}

Detecting when the user starts speaking while the mic is “off” presents a separate challenge, since Whisper isn’t running during the THINKING or SPEAKING states. A lightweight Voice Activity Detector (VAD) that monitors raw audio energy levels, rather than running full transcription, works perfectly here. It doesn’t need to parse words; it simply flags when sound crosses a defined volume threshold.

With the state machine controlling the mic and every transition bound to an actual runtime event rather than an arbitrary timer, the loop runs reliably: the agent listens, thinks, speaks, and cleanly hands control back at the exact right moment.

Where can you see the complete voice AI agent implementation?

Every piece detailed above comes together in the accompanying CodePen demo: the VAD worklet, both Web Workers, the custom speech queue, and the state machine, all organized inside a single reference file.

To try it out, open the demo in a WebGPU-capable browser (such as desktop Chrome or Edge). The initial run downloads roughly 400MB of model weights, which are cached locally for subsequent visits. Grant microphone permissions, wait for the models to finish loading into VRAM, and start talking.

See the Pen
Real-time voice agent by Miracle Jude (@JudeIV)
on CodePen.

Two things differ from the code in this article, both deliberate. The workers are created from Blob URLs rather than separate files, since CodePen serves a single document. And the demo uses energy-based voice activity detection to segment utterances instead of fixed 3-second chunks, which is more reliable for a live conversation than transcribing on a timer.

Conclusion

The entire pipeline runs locally on your device: Whisper handles transcription, a quantized LLM generates responses, and SpeechSynthesis provides playback, all powered by Transformers.js with WebGPU acceleration (and WASM as a CPU fallback). No audio or text ever leaves the browser tab.

The core tradeoff here is between local capability, complete privacy, and operational latency:

• Capability: A 0.5B or 1.5B parameter model cannot match a server-hosted, GPT-4-class API in reasoning quality or zero-shot breadth
• Privacy: Zero network requests mean zero data leakage, making this architecture completely compliant by default and functional offline after the initial model load
• Latency: While WebGPU delivers snappy inference on dedicated hardware, the WASM fallback runs roughly ten times slower, which, for a conversational voice agent, is the difference between a natural back-and-forth and an unusable, frustrating delay

Before choosing an in-browser WebGPU pipeline over a traditional server-based backend, make sure you clearly weigh those hardware requirements against your application’s privacy and latency thresholds.