BYO LiveKit Agent - LiveAvatar Documentation

In the Custom LiveKit Agent guide, LiveAvatar provisions the LiveKit room and you send an agent into it. This guide covers the other case: you own the LiveKit room, and the LiveAvatar avatar joins as a participant. This guide walks through a Python starter that does exactly that.

liveavatar-starter-livekit-agent-python

Reference implementation for this guide. Demo entrypoint: src/byo_livekit_demo.py.

This guide is best suited for developers who already run a LiveKit deployment and need control beyond what the LiveKit avatar plugin provides. The plugin handles the orchestration between your LiveKit agent and the avatar participant out of the box — start there first. Reach for the BYO flow below only if your use case needs more control over room lifecycle, dispatch, recording, or worker fleet.

Prerequisites

Before you start, make sure you have:

A LiveAvatar account with an API key — sign up at app.liveavatar.com.
An AVATAR_ID from your account. The default in the starter repo is sandbox-compatible; swap it for one of your own avatars when going to production.
A LiveKit Cloud project with LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. Sign up at cloud.livekit.io if you do not have one.
Python 3.11+ and uv (or pip) to install the starter’s dependencies.
The LiveKit CLI (brew install livekit-cli, then lk cloud auth) for deploying the worker to LiveKit Cloud.

Unlike the Custom LiveKit Agent flow, LIVEKIT_URL is required here — you own the room, so you tell LiveAvatar where to join.

Quickstart

Clone the reference implementation and run it against the LiveAvatar API and your own LiveKit project.

git clone https://github.com/heygen-com/liveavatar-starter-livekit-agent-python
cd liveavatar-starter-livekit-agent-python
cp .env.example .env.local

Set the following in .env.local:

LIVEAVATAR_API_KEY=...   # from app.liveavatar.com
AVATAR_ID=...            # any avatar in your account; sandbox-compatible by default
LIVEKIT_URL=wss://<your-project>.livekit.cloud
LIVEKIT_API_KEY=...      # your LK Cloud project — used for the room AND inference
LIVEKIT_API_SECRET=...
IS_SANDBOX=true          # default; switch to false for production avatars / billed minutes

Install and run two terminals:

uv sync

# Terminal 1 — register the worker against your LK project
python src/worker.py dev

# Terminal 2 — drive a session
python src/byo_livekit_demo.py

A new browser tab should pop up with the viewer connected to your LiveKit room. Try speaking to the avatar.

Both terminals load credentials from .env.local via load_dotenv. The dev worker registers against whichever LK project LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET point to — make sure those match the project you intend to dispatch into.

src/worker.py exposes three CLI subcommands inherited from LiveKit Agents:

Subcommand	Purpose
`dev`	Local development — registers the worker against your LK project, hot-reloads on file changes. Use this in Terminal 1 above.
`start`	Production entrypoint baked into the Dockerfile (`CMD`). LK Cloud runs this in the deployed worker.
`download-files`	Downloads VAD + turn-detector model weights. Run at Docker build time so cold-starts in LK Cloud are fast.

To verify the worker is actually registered before running the dispatcher, check lk agent status (after deploy) or look for a registered worker log line in Terminal 1. Behind the scenes, the dispatcher mints room tokens, asks LiveAvatar to join your room as a participant, and dispatches your worker into the room by name. The next sections break down the full flow.

Understanding the flow

You mint two LiveKit tokens. One for the LiveAvatar avatar, one for the viewer.
LiveAvatar joins your room. Using the minted avatar token, LiveAvatar dispatches its avatar into your room as a participant.
Your agent joins your room. Your worker is dispatched into the same room by name, alongside the avatar and the viewer.
The worker drives the connection. Using the WebSocket returned when the session starts, the worker controls the avatar — running STT → LLM → TTS and streaming audio for lip-sync.

Ownership at a glance

Concern	Owned by
LiveKit room	You (your LK Cloud project)
Room participant-minutes	You
Avatar + viewer tokens	You (you mint both)
Agent worker fleet	You (LK Cloud or self-hosted)
Avatar minutes / credits	You (your LiveAvatar account)
Inference billing (STT / LLM / TTS)	You (your LK Cloud project)

Understanding the code

When we run python src/byo_livekit_demo.py the dispatcher does the following:

Mint room + tokens against your LK project. Generates a fresh room name, then mints separate tokens for the avatar participant and the browser viewer.
Create a LiveAvatar session pinned to your room. Calls create_session_token with livekit_config={livekit_url, livekit_room, livekit_client_token=avatar_token} so LiveAvatar’s avatar joins your room.
Start the session. Calls start_session to get the avatar media-server WebSocket URL (ws_url).
Dispatch the agent into the room. Calls AgentDispatchService.create_dispatch(agent_name=AGENT_NAME, room=room_name, metadata=json.dumps({ws_url, session_id})). LK Cloud routes the dispatch to a registered worker.
Open the viewer. Serves viewer/index.html locally and opens a browser tab with LIVEKIT_URL + viewer token preloaded.
Worker accepts the dispatch. Reads metadata, opens the LiveAvatar WebSocket, runs STT → LLM → TTS, and tees synthesized audio to the WebSocket for lip-sync.
Clean up on exit. The worker registers a shutdown callback that calls stop_session on the LiveAvatar API whenever the job ends.

agent_name dispatch is not automatic dispatch. The worker must register under a name (@server.rtc_session(agent_name=AGENT_NAME) in worker.py) and the dispatcher must call create_dispatch with the same name. Workers using LiveKit’s default auto-dispatch won’t receive these jobs.

livekit_config.livekit_client_token is the avatar’s token, not the viewer’s. The avatar uses it to join your room as a participant. Mint the viewer token separately for the browser client.

File structure

The starter is split into a backend (the dispatcher + the worker) and a small frontend used to visualize the session locally. Backend (src/):

File	Responsibility
`liveavatar_client.py`	Makes API calls to LiveAvatar (create session, start session, stop session).
`avatar_ws.py`	Maintains the WebSocket connection to LiveAvatar and forwards audio frames for lip-sync.
`agent.py`	Defines the agent that handles the conversation and tees synthesized audio to the LiveAvatar WebSocket.
`pipeline.py`	Builds the STT → LLM → TTS pipeline, wires up room + session observability, and exposes `mute_agent_audio_on_publish` (silences the agent’s own LK audio track so the viewer only hears the avatar — otherwise audio doubles).
`worker.py`	Defines the `AgentServer` worker that registers under `AGENT_NAME` and accepts dispatched jobs. Reads `ws_url` + `session_id` from `ctx.job.metadata`.
`byo_livekit_demo.py`	Demo dispatcher — mints tokens, calls the LiveAvatar API, dispatches the worker, and opens the local viewer.

Frontend (viewer/): We ship a minimal frontend so you can see the avatar and talk to it locally without standing up your own client. Treat it as a reference, not as production frontend code.

File	Responsibility
`index.html`	Vanilla LiveKit JS client. Connects to your LK room with the viewer token, publishes the user’s microphone, and renders the avatar’s audio + video.

The avatar joins the room under the participant identity "avatar" (set as AVATAR_IDENTITY in byo_livekit_demo.py). When building your own frontend, subscribe to that identity’s audio + video tracks to render the avatar.

Customizing the agent

The most common things you’ll want to tweak after the quickstart:

What	Where
`AGENT_NAME`	`worker.py` — must match the `agent_name` passed to `create_dispatch`. Rename from the default `my-agent` before deploying.
System prompt / agent persona	`agent.py` — the `LiveAvatarAgent` definition.
LLM model, STT provider, TTS voice	`pipeline.py` — `build_session` configures all three.
Audio input options (noise cancellation, etc.)	`pipeline.py` — `build_room_options`.
Logging / observability hooks	`pipeline.py` — `wire_room_observability`, `wire_session_observability`.

Productionizing

This demo is wired up for fast local iteration. When productionizing, move dispatch logic to a backend service, deploy the worker to LiveKit Cloud, and tighten up session lifecycle.

Improvements beyond the demo

A few things in the starter exist purely for local convenience. We recommend the following swaps:

Build your own frontend. Replace the local viewer HTTP server and auto-launched browser tab with a real client that joins your LK room using the viewer token.
Move dispatch to a backend service. byo_livekit_demo.py mints LK tokens and calls the LiveAvatar API client-side because it’s a single-process demo. In production, that logic belongs on a backend that authenticates the user, mints the viewer token, calls create_session_token + start_session + create_dispatch, and returns only {room_url, viewer_token} to the browser. Never ship LIVEKIT_API_SECRET, LIVEAVATAR_API_KEY, or the avatar token to the browser.
Deploy the worker to LiveKit Cloud. python src/worker.py dev is for local iteration only. The repo ships a Dockerfile and livekit.toml for lk agent deploy (see below).
Rename AGENT_NAME. The default my-agent collides across LK Cloud accounts and makes lk agent logs painful. Pick something project-specific (e.g. support-agent, tutor-bot) before deploying.
Guard against leaked sessions. The worker registers an add_shutdown_callback that calls stop_session whenever the job ends. This is the load-bearing cleanup path in production — your backend dispatcher should fire-and-forget after create_dispatch and let the worker handle lifecycle. As a safety net for runners that crash without it (SIGKILL, OOM), set max_session_duration server-side when calling create_session_token so LiveAvatar will close abandoned sessions on its own.
Manual termination. To end a session early (e.g. user clicks “end call”), call stop_session(session_id) directly from your backend. The worker’s shutdown callback will fire when the job is torn down.

The Ctrl-C cleanup in byo_livekit_demo.py is belt-and-suspenders for the demo. Do not copy it into a backend dispatcher service. A backend dispatcher should exit immediately after create_dispatch and let the worker’s shutdown callback close the LiveAvatar session.

Deploying the worker to LiveKit Cloud

The repo ships a Dockerfile and livekit.toml for lk agent deploy.

# First-time setup — writes subdomain + agent id back into livekit.toml
lk agent create --secrets-file .env.local

# Subsequent updates
lk agent deploy

# Inspect
lk agent status
lk agent logs

lk agent create modifies livekit.toml in place — it fills [project].subdomain and [agent].id. Commit the diff, otherwise teammates and CI won’t be able to redeploy the same agent.

The Dockerfile runs python src/worker.py download-files at build time to bake VAD + turn-detector model weights into the image. This is what makes cold-starts in LK Cloud fast. Don’t strip it.

Don’t bake LIVEKIT_* env vars into the image. LK Cloud injects LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET at runtime. Only ship application-level secrets (LIVEAVATAR_API_KEY, etc.) via --secrets-file.

After deploy, drop the python src/worker.py dev terminal entirely. Dispatches go straight to the cloud-managed worker fleet.

Troubleshooting

Symptom	Likely cause	Fix
Avatar never joins the room	`livekit_config` missing or wrong (bad `livekit_url`, room name mismatch, expired/invalid avatar token)	Verify all three fields in `livekit_config`. Confirm the avatar token is for the same room name passed in `livekit_room`. Check token TTL.
Worker registered but no jobs arrive	`agent_name` mismatch between worker and dispatcher, or worker uses default auto-dispatch	`@server.rtc_session(agent_name=...)` on the worker must equal the `agent_name` in `create_dispatch`. Named dispatch and auto-dispatch are mutually exclusive.
`start_session` returned but `ws_url` is empty	Session failed to allocate a media server (sandbox quota, region, billing)	Inspect the response body. Re-check `IS_SANDBOX` + `AVATAR_ID` compatibility. Retry.
Avatar visible but silent (no lip-sync)	Worker never opened the LiveAvatar WebSocket, or `tts_node` not teeing audio to it	Confirm `meta["ws_url"]` parsed correctly from job metadata. Check worker logs for `AvatarWebSocket` connect errors.
Viewer hears doubled / echoed audio	Agent audio track not muted on publish	`mute_agent_audio_on_publish(ctx.room)` must be called in the worker entrypoint. The avatar is the audio source — the agent’s own track has to be silenced.
`lk agent deploy` fails on first run	`lk agent create` was skipped — `livekit.toml` still has empty `subdomain` / `id`	Run `lk agent create --secrets-file .env.local` first. Commit the resulting `livekit.toml` diff.
Session leaks after worker crash (SIGKILL)	Shutdown callback can’t run on SIGKILL	Set `max_session_duration` on `create_session_token` as a TTL safety net.

liveavatar-starter-livekit-agent-python

​Prerequisites

​Quickstart

​Understanding the flow

​Ownership at a glance

​Understanding the code

​File structure

​Customizing the agent

​Productionizing

​Improvements beyond the demo

​Deploying the worker to LiveKit Cloud

​Troubleshooting

Prerequisites

Quickstart

Understanding the flow

Ownership at a glance

Understanding the code

File structure

Customizing the agent

Productionizing

Improvements beyond the demo

Deploying the worker to LiveKit Cloud

Troubleshooting