Skip to main content
This guide walks through a reference integration that connects OpenAI GPT-Live — a full-duplex speech-to-speech model — to a HeyGen LiveAvatar, with a custom frontend that renders the model’s tool calls as animated overlays on top of the avatar’s video. Out of the box it’s a Japanese tutor: it teaches you words out loud, puts a term card (term · pronunciation · meaning) on screen as it says them, and periodically shrinks itself to the corner to review everything taught so far. The persona is two Markdown files and each visual is one tool plus one composition — swap those and it’s any other voice agent with a face. By the end you’ll have:
  • A LITE Mode avatar session whose voice is generated by GPT-Live, not by LiveAvatar’s built-in stack.
  • A Node orchestrator bridging GPT-Live’s continuous audio stream into the avatar’s media server.
  • Tool calls landing in the browser as Hyperframes overlays layered over the avatar’s video.

liveavatar-gpt-live-demos

Reference implementation for this guide. pnpm workspace, TypeScript, MIT licensed.

​
Prerequisites

You do not need your own avatar. With no avatar configured, the server picks the first active public avatar and logs which one it chose.

​
Quickstart

Skip the manual setup — this repo is written for coding agents. Install Agent Skills and hand it off:
Then ask your agent (Claude Code, Cursor, Codex, etc.) to run the GPT-Live language tutor demo. The liveavatar-demo skill clones the repo, installs dependencies, and writes your API keys to .env for you — it’ll only ask you for the keys themselves and to confirm the avatar talks in your browser.
Only the server reads .env — the browser never holds an API key or a session token. It gets a LiveKit token to watch the avatar and a websocket for mic audio (up) and transcripts + visuals (down).
If something’s off on a fresh clone — no greeting, no audio, a silent avatar — the repo’s README troubleshooting section maps each symptom to its cause.

​
Architecture

GPT-Live and LiveAvatar make different assumptions about conversation. GPT-Live is full-duplex: one continuous audio stream each way, with the model managing turn-taking itself — it interrupts, yields, and barges in on its own. LiveAvatar’s session protocol is turn-based (agent.speak / agent.interrupt). The orchestrator in server/ bridges the two without manufacturing turn boundaries: it treats the avatar as a pure audio-to-face renderer fed one never-ending utterance. The key is starting the LITE session server-side and bare — no agent configuration. The start response returns livekit_url + livekit_client_token for the browser to watch, and ws_url: a direct websocket into the avatar’s media server. That socket is the avatar’s ear. The protocol over it is public and small (LITE Mode events): agent.speak carries PCM16 24kHz audio chunks, agent.interrupt clears the buffer, session.keep_alive holds the session open. A session, end to end:
  1. Start. The browser posts to /api/session/start. The server mints the LITE session, connects to the media server and to GPT-Live, and returns the LiveKit credentials.
  2. Watch. The browser joins the LiveKit room and attaches the avatar’s video and audio tracks — already lip-synced. Avatar audio deliberately never travels over the browser websocket.
  3. Talk. The mic streams continuously to the server as 24kHz PCM16 — no push-to-talk, no browser-side voice activity detection. Turn detection belongs to the model, which hears the same audio with the conversation as context.
  4. Speak. Every GPT-Live audio chunk is appended to the avatar’s buffer as it arrives. The silence GPT-Live omits between utterances is reconstructed from chunk timestamps and forwarded as real audio (server/src/audio.ts) — otherwise the mouth runs ahead of the voice.
  5. Interrupt. Barge-in is two-step: a user turn opening only starts a watch, and the avatar’s buffer is cleared only if the model actually stops producing audio. Acting on the turn alone would cut the avatar off on every “mm-hmm”.
The full rationale — silence reconstruction math, barge-in timing, session lifecycle, API quirks the bridge absorbs — lives in the repo’s ARCHITECTURE.md.
LiveAvatar LITE is a pure audio-to-face renderer with no LLM layer — tool calling never touches it. The tool path (GPT-Live → orchestrator → browser overlay) is exactly the gap this integration fills, which is also why it doesn’t use the LiveAvatar Web SDK: the media-server ws_url only comes back from a server-side session start.

​
From tool call to overlay

The frontend is not a video player with a chat log — it’s a stage. Tool calls made by the model surface in the browser as transparent Hyperframes compositions (self-contained animated HTML pages) layered over the avatar’s video. Nothing is composited into the stream itself; the video is untouched underneath. The event path has four hops:
1

The live model delegates

The live model holds no tools — it’s told it cannot draw on screen. When a visual is wanted it delegates the turn to its backend Responses model, which does hold the tools (shared/tools.ts). The Responses model answers in words and calls e.g. show_term_card in the same reply: the reply text is injected back into the live session and voiced by the avatar, while the tool call surfaces on the orchestrator’s socket. That’s why the avatar keeps talking as the card lands.
2

The orchestrator validates and dispatches

server/src/tools.ts treats model output as untrusted: it validates the call, clamps every string, and forwards a single typed message to the browser:
The wire protocol both sides speak is typed once, in shared/messages.ts.
3

The browser switches on the widget

web/src/overlays/index.ts is a single switch over widget — one renderer module per widget. Each renderer plays its Hyperframes composition (web/public/overlays/*.html) in a transparent layer over the video.Staging is per-widget and client-decided — never a tool argument. The term card is a lower-third over the full-frame avatar; the recap panel sets picture-in-picture, shrinking the avatar to the corner while the panel is up. Keeping layout out of the tool schema keeps the model from arguing with the product about staging.
4

Session state stays on the server

Every term card shown is recorded server-side, deduped, in teaching order. The recap tool (show_learned_words) renders from that store — the model asks for the recap and supplies only the heading, so the word list is never the model’s to misremember, invent, or drop.
You can iterate on overlays without burning session minutes — from the browser console:

​
Next steps

What ships here is a starter, not a deployment. Three directions to take it:

​
Replace the persona

The Japanese tutor is two Markdown files, and they’re the intended customization point:
  • server/prompts/instructions.md — who the avatar is, what it teaches, how it behaves.
  • server/prompts/greeting.md — how it opens. An empty file means “say nothing, let the user speak first”.
Edit them and restart the server — no code changes. The delegation and tool mechanics live separately in server/src/prompts.ts, so a new persona can’t accidentally break the tool path. Swap the voice with GPT_LIVE_VOICE, the avatar with LIVEAVATAR_AVATAR_ID.

​
Add a widget and its tool

Each visual is one tool plus one composition. Adding one is three small edits:
  1. shared/tools.ts — add the tool schema and its argument types. Keep its required parameter names distinct from every other tool’s (the bridge uses them to identify calls).
  2. server/src/tools.ts — add a dispatch case mapping validated args to a widget message.
  3. web/src/overlays/ — add a renderer and, if it plays a new composition, the Hyperframes page under web/public/overlays/ (copy term-card.html as the template).
Then tell the model when to use it in RESPONSES_INSTRUCTIONS (server/src/prompts.ts) — a tool in the registry is callable, not called. Test the rendering with window.__ui(...) before involving the model. The repo’s AGENTS.md carries the full recipe, including the composition gotchas.

​
Productionize

Deliberately left out of the starter, in rough priority order:
  1. Auth. Gate /api/session/start behind your login, and hand the browser a short-lived credential for the websocket upgrade — a signed ticket scoped to one session id.
  2. Hide upstream error bodies. The starter passes them through verbatim so a fresh clone is debuggable; a public deployment shouldn’t tell a prober about your credit and concurrency limits.
  3. Browser-side ducking. Drop the avatar’s playback volume the moment the local mic gets loud, before the server-side interrupt confirms — makes barge-in feel instant.
  4. Audible-time scheduling. Transcripts arrive seconds ahead of the avatar’s voice. Anything that must land when words are heard (synced captions, timed visuals) needs a playback-position estimate, not arrival time.
  5. Observability and scale. Per-session structured logs, delegation latency timing, usage warnings from session.usage.updated; then multi-pod session routing, reconnect paths, rate limiting.
Sessions are billable from the moment they start, so whatever you add, keep the teardown paths intact: every exit must reach stopSession, and the idle/max-duration watchdogs are the backstop, not the plan.