Events

LiveAvatar processes events over the WebSocket connection returned when the session starts. Each frame is one JSON object sent as a text frame, up to 1 MB. All events follow this JSON structure:

{
  "type": "string",
  "event_id": "string",
  "source_event_id": "string | null"
}

type — the event name.
event_id — on commands, an optional UUID you assign (the server generates one if omitted). On response events, a server-generated UUID.
source_event_id — response events only. The event_id of the command that caused the event, or null when there is none.

Audio sent in any event must be PCM 16-bit 24 kHz mono, Base64-encoded.

Command Events

Send these over the WebSocket. event_id is optional on every command.

Wait for session.state_updated with state "connected" before sending any command events.

Event	Payload	Description
`agent.speak`	`{"audio": "<base64>"}`	Appends audio to the current utterance. The first chunk after `agent.speak_end` or `agent.interrupt` opens a new utterance identified by its `event_id` (see below). Recommended chunk size: ~1 second. Acknowledged with `agent.audio_buffer_appended`.
`agent.speak_end`	`{"audio": "<base64>"}` (optional)	Commits and seals the current utterance. May carry a final trailing audio chunk. Acknowledged with `agent.audio_buffer_committed`.
`agent.interrupt`	—	Barge-in. Clears buffered audio and pending video generation, and seals the current utterance. Emits `agent.speak_interrupted` (if an utterance was open) and `agent.audio_buffer_cleared`.
`agent.start_listening`	—	Switches the avatar to a listening state (from idle).
`agent.stop_listening`	—	Returns the avatar from listening to idle state.
`session.keep_alive`	—	Application-level heartbeat. Refreshes session activity and keeps the session alive beyond the 5-minute idle timeout. No response event.

Unknown type, malformed JSON, missing audio, or invalid Base64 return an error event. There is no separate “clear buffer” command; use agent.interrupt.

Utterances

An utterance is a run of audio the avatar speaks as one unit. It is identified by the event_id of its first agent.speak chunk. Later chunks in the same utterance may carry any event_id; only the first names the utterance. agent.speak_end or agent.interrupt seals it, and the next agent.speak opens a new one. Every response event caused by an utterance carries that first-chunk event_id as source_event_id. Set one UUID per utterance on the first agent.speak (or on every chunk of it) and match response events by source_event_id to know which utterance the avatar is speaking, has finished, or was interrupted.

Response Events

Listen for these on the WebSocket.

Event	Payload	Description
`session.state_updated`	`{"state": "..."}`	WebSocket connection state changed (see below). Emitted only on an actual change. Wait for `"connected"` before sending commands. `source_event_id` is `null`.
`agent.state_updated`	`{"previous_state": "...", "new_state": "..."}`	The avatar changed state (see below). Carries both sides so a late joiner can reconstruct state. Consecutive transitions into the same state are collapsed. `source_event_id` is the utterance id when `new_state` is `"talking"`, else `null`.
`agent.speak_started`	—	The avatar has started speaking an utterance. `source_event_id` is the utterance id.
`agent.speak_ended`	—	The avatar has finished speaking an utterance. `source_event_id` is the utterance id.
`agent.speak_interrupted`	—	The utterance was cut off by `agent.interrupt`. `agent.speak_ended` is not emitted for it. `source_event_id` is the utterance id.
`agent.audio_buffer_appended`	—	Acknowledges `agent.speak`. `source_event_id` is the utterance id.
`agent.audio_buffer_committed`	—	Acknowledges `agent.speak_end`. `source_event_id` is the utterance id.
`agent.audio_buffer_cleared`	—	Acknowledges `agent.interrupt`. `source_event_id` is the interrupt’s `event_id`.
`error`	`{"error": {"type": "...", "message": "...", "event_id": "..."}}`	Invalid request or server error (see below). `source_event_id` is the offending command’s `event_id`, or `null`. `error.event_id` duplicates it.
`warning`	`{"warning": {"type": "...", "message": "..."}}`	Non-fatal diagnostic, such as a `video_starvation` frame gap. Same shape as `error` so both can be parsed uniformly. `source_event_id` is `null`.

`session.state_updated` states

The state field on session.state_updated is one of:

Value	Meaning
`new`	The WebSocket connection is being established.
`connected`	The connection is ready. Emitted once on connect. Commands are accepted.
`disconnected`	The connection has closed.

`agent.state_updated` states

The previous_state and new_state fields on agent.state_updated are one of:

Value	Meaning
`idle`	The avatar is in its idle pose.
`listening`	The avatar is in its listening pose.
`talking`	The avatar is speaking an utterance.

`error` types

The error.type field on error is one of:

Value	Meaning
`invalid_request_error`	The command was malformed: unknown `type`, invalid JSON, missing `audio`, or invalid Base64.
`server_error`	The server failed to process the command.
`video_starvation`	Video generation could not keep up with the audio supplied.

Connection and liveness

One WebSocket per session. Opening a new connection replaces the existing one.
Send session.keep_alive periodically as a heartbeat to keep the session alive beyond the 5-minute idle timeout.

Getting Started

Core Concepts

FULL Mode

Avatar Only (LITE mode)

Socials

FAQ

Command Events

Utterances

Response Events

`session.state_updated` states

`agent.state_updated` states

`error` types

Connection and liveness

​Command Events

​Utterances

​Response Events

​session.state_updated states

​agent.state_updated states

​error types

​Connection and liveness

Command Events

Utterances

Response Events

`session.state_updated` states

`agent.state_updated` states

`error` types

Connection and liveness