Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.liveavatar.com/llms.txt

Use this file to discover all available pages before exploring further.

The ElevenLabs Agent Connector bridges ElevenLabs Agents with LiveAvatar. ElevenLabs handles the conversational AI while LiveAvatar renders the avatar video.
The ElevenLabs Agent Connector is not available with ElevenLabs free-tier API keys. ElevenLabs blocks third-party usage on free plans. You must have a paid ElevenLabs subscription to use this connector.

​
Requirements

  • ElevenLabs API key with permissions: convai_read, user_read, voices_read
  • ElevenLabs Agent ID
  • Audio output configured as PCM 24K

​
Setup

1. Register your ElevenLabs API key: 
curl -X POST https://api.liveavatar.com/v1/secrets \
  -H "X-API-KEY: <YOUR_API_KEY>" \
  -H "content-type: application/json" \
  -d '{
    "secret_type": "ELEVENLABS_API_KEY",
    "secret_value": "<elevenlabs_api_key>",
    "secret_name": "ElevenLabs Agent Key"
  }'
2. Start a session with the connector: 
{
  "mode": "LITE",
  "avatar_id": "<avatar_id>",
  "elevenlabs_agent_config": {
    "secret_id": "<secret_id>",
    "agent_id": "<elevenlabs_agent_id>"
  }
}

​
How it works

  • A LiveKit room is automatically created (no WebSocket endpoint returned)
  • LiveAvatar dispatches a worker that handles interaction with your ElevenLabs Agent
  • Agent audio drives avatar animation in real time
The ElevenLabs Agent Connector uses the FULL Mode event system, not standard LITE Mode events. The WebSocket command events described in the LITE Mode Events page do not apply here. See the Events section below for details.

​
Events

The ElevenLabs Agent Connector wraps the ElevenLabs Agent in LiveKit. Events are exchanged over the LiveKit data channel and differentiated by LiveKit room topic:
  • agent-control — commands you send to the avatar
  • agent-response — events the server sends to you
Two layers of events are exposed:
  1. LiveAvatar events — a subset of the FULL Mode response events, so existing FULL Mode integrations work unchanged.
  2. ElevenLabs passthrough events — the underlying ElevenLabs Conversational AI protocol, wrapped and forwarded in both directions.

​
LiveAvatar events

The following FULL Mode response events fire on every ElevenLabs session, on the agent-response topic:
EventSource
user.speak_started / user.speak_endedLiveKit user-state changes
user.transcription_chunk / user.transcriptionStreaming and final user STT (driven by ElevenLabs STT)
avatar.transcription_chunk / avatar.transcriptionStreaming and final agent text
avatar.speak_started / avatar.speak_endedAvatar TTS lifecycle
session_stoppedSession lifecycle (includes stop_reason)
When the ElevenLabs Agent cleanly closes the WebSocket (e.g. agent hangs up), the session ends with a session_stopped event carrying stop_reason: AGENT_HANG_UP. There is no separate conversation_ended event on the wire.

​
ElevenLabs passthrough events

Raw ElevenLabs Conversational AI events are wrapped and forwarded in both directions, giving you direct access to the underlying agent protocol without per-feature plumbing.
DirectionWrapper eventTopicPayload
Server → clientelevenlabs_agent_eventagent-response{"elevenlabs_event_type": "...", "data": {...}}
Client → serverelevenlabs_agent_commandagent-control{"elevenlabs_event_type": "...", "data": {...}}
In both directions, elevenlabs_event_type is the original ElevenLabs event type and data is the raw payload. For inbound commands, data must not include a type field — the wrapper’s elevenlabs_event_type is authoritative.

​
Outbound (server → client)

The following ElevenLabs server events are forwarded as elevenlabs_agent_event:
elevenlabs_event_typeDescription
conversation_initiation_metadataInitial metadata, includes conversation_id.
user_transcriptFinal user transcription.
agent_responseFinal agent text.
agent_response_correctionCorrection to a previous agent response.
internal_tentative_agent_responseStreaming partial agent text.
interruptionAgent was interrupted.
client_tool_callTool invocation request from the agent.
contextual_updateEcho of context updates.
vad_scorePer-frame VAD score from ElevenLabs.
The following are not forwarded:
elevenlabs_event_typeReason
audioHigh-volume binary; consumed internally and rendered by the avatar.
pingAuto-handled internally with a pong reply.

​
Inbound (client → server)

Only the following inner types are accepted in elevenlabs_agent_command (the developer-facing whitelist):
elevenlabs_event_typeDescription
user_messageInject text as a user turn.
user_activityHeartbeat / nudge to elicit a reply.
contextual_updatePush instructions or context without a user turn.
client_tool_resultReply to an ElevenLabs client_tool_call.
conversation_initiation_client_data and pong are blocked because they are managed by LiveAvatar and forwarding them would corrupt the protocol. Any other inner type is dropped.

​
Listening for ElevenLabs events

If migrating from an existing ElevenLabs agent implementation, note that the ElevenLabs agent WebSocket is not exposed by this connector. Subscribe to the LiveKit data channel and parse the elevenlabs_agent_event event. The original ElevenLabs payload is passed through verbatim under the data field, so your existing parsers can consume data as-is. 
import { AgentEventsEnum } from "@heygen/liveavatar-web-sdk";

session.on(AgentEventsEnum.ELEVENLABS_AGENT_EVENT, (event) => {
  console.log("ElevenLabs event:", event.elevenlabs_event_type, event.data);
});

​
Sending ElevenLabs commands

{
  "event_type": "elevenlabs_agent_command",
  "elevenlabs_event_type": "user_message",
  "data": { "text": "hi there" }
}
For the full ElevenLabs event reference, see the ElevenLabs Conversational AI documentation.

​
Billing

  • LiveAvatar: 1 credit per session minute (avatar video only)
  • ElevenLabs: charges separately for agent usage