ElevenLabs Agent Connector

The ElevenLabs Agent Connector bridges ElevenLabs Agents with LiveAvatar. ElevenLabs handles the conversational AI while LiveAvatar renders the avatar video.

The ElevenLabs Agent Connector is not available with ElevenLabs free-tier API keys. ElevenLabs blocks third-party usage on free plans. You must have a paid ElevenLabs subscription to use this connector.

Requirements

ElevenLabs API key with permissions: convai_read, user_read, voices_read
ElevenLabs Agent ID
Audio output configured as PCM 24K

Setup

1. Register your ElevenLabs API key:

curl -X POST https://api.liveavatar.com/v1/secrets \
  -H "X-API-KEY: <YOUR_API_KEY>" \
  -H "content-type: application/json" \
  -d '{
    "secret_type": "ELEVENLABS_API_KEY",
    "secret_value": "<elevenlabs_api_key>",
    "secret_name": "ElevenLabs Agent Key"
  }'

2. Start a session with the connector:

{
  "mode": "LITE",
  "avatar_id": "<avatar_id>",
  "elevenlabs_agent_config": {
    "secret_id": "<secret_id>",
    "agent_id": "<elevenlabs_agent_id>"
  }
}

How it works

A LiveKit room is automatically created (no WebSocket endpoint returned)
LiveAvatar dispatches a worker that handles interaction with your ElevenLabs Agent
Agent audio drives avatar animation in real time

The ElevenLabs Agent Connector uses the FULL Mode event system, not standard LITE Mode events. The WebSocket command events described in the LITE Mode Events page do not apply here. See the Events section below for details.

Events

The ElevenLabs Agent Connector wraps the ElevenLabs Agent in LiveKit. Events are exchanged over the LiveKit data channel and differentiated by LiveKit room topic:

agent-control — commands you send to the avatar
agent-response — events the server sends to you

Two layers of events are exposed:

LiveAvatar events — a subset of the FULL Mode response events, so existing FULL Mode integrations work unchanged.
ElevenLabs passthrough events — the underlying ElevenLabs Conversational AI protocol, wrapped and forwarded in both directions.

LiveAvatar events

The following FULL Mode response events fire on every ElevenLabs session, on the agent-response topic:

Event	Source
`user.speak_started` / `user.speak_ended`	LiveKit user-state changes
`user.transcription_chunk` / `user.transcription`	Streaming and final user STT (driven by ElevenLabs STT)
`avatar.transcription_chunk` / `avatar.transcription`	Streaming and final agent text
`avatar.speak_started` / `avatar.speak_ended`	Avatar TTS lifecycle
`session_stopped`	Session lifecycle (includes `stop_reason`)

When the ElevenLabs Agent cleanly closes the WebSocket (e.g. agent hangs up), the session ends with a session_stopped event carrying stop_reason: AGENT_HANG_UP. There is no separate conversation_ended event on the wire.

ElevenLabs passthrough events

Raw ElevenLabs Conversational AI events are wrapped and forwarded in both directions, giving you direct access to the underlying agent protocol without per-feature plumbing.

Direction	Wrapper event	Topic	Payload
Server → client	`elevenlabs_agent_event`	`agent-response`	`{"elevenlabs_event_type": "...", "data": {...}}`
Client → server	`elevenlabs_agent_command`	`agent-control`	`{"elevenlabs_event_type": "...", "data": {...}}`

In both directions, elevenlabs_event_type is the original ElevenLabs event type and data is the raw payload. For inbound commands, data must not include a type field — the wrapper’s elevenlabs_event_type is authoritative.

Outbound (server → client)

The following ElevenLabs server events are forwarded as elevenlabs_agent_event:

`elevenlabs_event_type`	Description
`conversation_initiation_metadata`	Initial metadata, includes `conversation_id`.
`user_transcript`	Final user transcription.
`agent_response`	Final agent text.
`agent_response_correction`	Correction to a previous agent response.
`internal_tentative_agent_response`	Streaming partial agent text.
`interruption`	Agent was interrupted.
`client_tool_call`	Tool invocation request from the agent.
`contextual_update`	Echo of context updates.
`vad_score`	Per-frame VAD score from ElevenLabs.

The following are not forwarded:

`elevenlabs_event_type`	Reason
`audio`	High-volume binary; consumed internally and rendered by the avatar.
`ping`	Auto-handled internally with a `pong` reply.

Inbound (client → server)

Only the following inner types are accepted in elevenlabs_agent_command (the developer-facing whitelist):

`elevenlabs_event_type`	Description
`user_message`	Inject text as a user turn.
`user_activity`	Heartbeat / nudge to elicit a reply.
`contextual_update`	Push instructions or context without a user turn.
`client_tool_result`	Reply to an ElevenLabs `client_tool_call`.

conversation_initiation_client_data and pong are blocked because they are managed by LiveAvatar and forwarding them would corrupt the protocol. Any other inner type is dropped.

Listening for ElevenLabs events

If migrating from an existing ElevenLabs agent implementation, note that the ElevenLabs agent WebSocket is not exposed by this connector. Subscribe to the LiveKit data channel and parse the elevenlabs_agent_event event. The original ElevenLabs payload is passed through verbatim under the data field, so your existing parsers can consume data as-is.

import { AgentEventsEnum } from "@heygen/liveavatar-web-sdk";

session.on(AgentEventsEnum.ELEVENLABS_AGENT_EVENT, (event) => {
  console.log("ElevenLabs event:", event.elevenlabs_event_type, event.data);
});

{
  "event_id": "abc-123",
  "event_type": "elevenlabs_agent_event",
  "session_id": "session-456",
  "elevenlabs_event_type": "agent_response",
  "data": { ... }
}

Sending ElevenLabs commands

{
  "event_type": "elevenlabs_agent_command",
  "elevenlabs_event_type": "user_message",
  "data": { "text": "hi there" }
}

For the full ElevenLabs event reference, see the ElevenLabs Conversational AI documentation.

Billing

LiveAvatar: 1 credit per session minute (avatar video only)
ElevenLabs: charges separately for agent usage

Getting Started

Core Concepts

FULL Mode

LITE Mode

Socials

FAQ

Requirements

Setup

How it works

Events

LiveAvatar events

ElevenLabs passthrough events

Outbound (server → client)

Inbound (client → server)

Listening for ElevenLabs events

Sending ElevenLabs commands

Billing

​Requirements

​Setup

​How it works

​Events

​LiveAvatar events

​ElevenLabs passthrough events

​Outbound (server → client)

​Inbound (client → server)

​Listening for ElevenLabs events

​Sending ElevenLabs commands

​Billing

Requirements

Setup

How it works

Events

LiveAvatar events

ElevenLabs passthrough events

Outbound (server → client)

Inbound (client → server)

Listening for ElevenLabs events

Sending ElevenLabs commands

Billing