> ## Documentation Index
> Fetch the complete documentation index at: https://docs.liveavatar.com/llms.txt
> Use this file to discover all available pages before exploring further.

# ElevenLabs Agent Connector

> Bridge ElevenLabs Agents with LiveAvatar for real-time avatar video

The ElevenLabs Agent Connector bridges ElevenLabs Agents with LiveAvatar. ElevenLabs handles the conversational AI while LiveAvatar renders the avatar video.

<Warning>
  The ElevenLabs Agent Connector is not available with ElevenLabs free-tier API keys. ElevenLabs blocks third-party usage on free plans. You must have a paid ElevenLabs subscription to use this connector.
</Warning>

## Requirements

* ElevenLabs API key with permissions: `convai_read`, `user_read`, `voices_read`
* ElevenLabs Agent ID
* Audio output configured as **PCM 24K**

## Setup

**1. Register your ElevenLabs API key:**

```bash theme={null}
curl -X POST https://api.liveavatar.com/v1/secrets \
  -H "X-API-KEY: <YOUR_API_KEY>" \
  -H "content-type: application/json" \
  -d '{
    "secret_type": "ELEVENLABS_API_KEY",
    "secret_value": "<elevenlabs_api_key>",
    "secret_name": "ElevenLabs Agent Key"
  }'
```

**2. Start a session with the connector:**

```json theme={null}
{
  "mode": "LITE",
  "avatar_id": "<avatar_id>",
  "elevenlabs_agent_config": {
    "secret_id": "<secret_id>",
    "agent_id": "<elevenlabs_agent_id>"
  }
}
```

## How it works

* A LiveKit room is automatically created (no WebSocket endpoint returned)
* LiveAvatar dispatches a worker that handles interaction with your ElevenLabs Agent
* Agent audio drives avatar animation in real time

```mermaid theme={null}
sequenceDiagram
    participant User
    participant LiveAvatar
    participant ElevenLabs Agent

    User->>LiveAvatar: Start LITE session with agent config
    LiveAvatar->>ElevenLabs Agent: Connect worker to agent
    User->>ElevenLabs Agent: Audio input (speech)
    ElevenLabs Agent->>ElevenLabs Agent: Audio orchestration + LLM
    ElevenLabs Agent->>LiveAvatar: Audio output
    LiveAvatar->>User: Avatar video stream with lip sync
```

<Warning>
  The ElevenLabs Agent Connector uses the **FULL Mode event system**, not standard LITE Mode events. The WebSocket command events described in the [LITE Mode Events](/docs/lite-mode/events) page do not apply here. See the [Events](#events) section below for details.
</Warning>

## Events

The ElevenLabs Agent Connector wraps the ElevenLabs Agent in LiveKit. Events are exchanged over the LiveKit data channel and differentiated by LiveKit room topic:

* **`agent-control`** — commands you send to the avatar
* **`agent-response`** — events the server sends to you

Two layers of events are exposed:

1. **LiveAvatar events** — a subset of the [FULL Mode response events](/docs/full-mode/events), so existing FULL Mode integrations work unchanged.
2. **ElevenLabs passthrough events** — the underlying ElevenLabs Conversational AI protocol, wrapped and forwarded in both directions.

### LiveAvatar events

The following [FULL Mode response events](/docs/full-mode/events) fire on every ElevenLabs session, on the `agent-response` topic:

| Event                                                 | Source                                                  |
| ----------------------------------------------------- | ------------------------------------------------------- |
| `user.speak_started` / `user.speak_ended`             | LiveKit user-state changes                              |
| `user.transcription_chunk` / `user.transcription`     | Streaming and final user STT (driven by ElevenLabs STT) |
| `avatar.transcription_chunk` / `avatar.transcription` | Streaming and final agent text                          |
| `avatar.speak_started` / `avatar.speak_ended`         | Avatar TTS lifecycle                                    |
| `session_stopped`                                     | Session lifecycle (includes `stop_reason`)              |

When the ElevenLabs Agent cleanly closes the WebSocket (e.g. agent hangs up), the session ends with a `session_stopped` event carrying `stop_reason: AGENT_HANG_UP`. There is no separate `conversation_ended` event on the wire.

### ElevenLabs passthrough events

Raw ElevenLabs Conversational AI events are wrapped and forwarded in both directions, giving you direct access to the underlying agent protocol without per-feature plumbing.

| Direction       | Wrapper event              | Topic            | Payload                                           |
| --------------- | -------------------------- | ---------------- | ------------------------------------------------- |
| Server → client | `elevenlabs_agent_event`   | `agent-response` | `{"elevenlabs_event_type": "...", "data": {...}}` |
| Client → server | `elevenlabs_agent_command` | `agent-control`  | `{"elevenlabs_event_type": "...", "data": {...}}` |

In both directions, `elevenlabs_event_type` is the original ElevenLabs event type and `data` is the raw payload. For inbound commands, `data` must **not** include a `type` field — the wrapper's `elevenlabs_event_type` is authoritative.

#### Outbound (server → client)

The following ElevenLabs server events are forwarded as `elevenlabs_agent_event`:

| `elevenlabs_event_type`             | Description                                   |
| ----------------------------------- | --------------------------------------------- |
| `conversation_initiation_metadata`  | Initial metadata, includes `conversation_id`. |
| `user_transcript`                   | Final user transcription.                     |
| `agent_response`                    | Final agent text.                             |
| `agent_response_correction`         | Correction to a previous agent response.      |
| `internal_tentative_agent_response` | Streaming partial agent text.                 |
| `interruption`                      | Agent was interrupted.                        |
| `client_tool_call`                  | Tool invocation request from the agent.       |
| `contextual_update`                 | Echo of context updates.                      |
| `vad_score`                         | Per-frame VAD score from ElevenLabs.          |

The following are **not** forwarded:

| `elevenlabs_event_type` | Reason                                                              |
| ----------------------- | ------------------------------------------------------------------- |
| `audio`                 | High-volume binary; consumed internally and rendered by the avatar. |
| `ping`                  | Auto-handled internally with a `pong` reply.                        |

#### Inbound (client → server)

Only the following inner types are accepted in `elevenlabs_agent_command` (the developer-facing whitelist):

| `elevenlabs_event_type` | Description                                       |
| ----------------------- | ------------------------------------------------- |
| `user_message`          | Inject text as a user turn.                       |
| `user_activity`         | Heartbeat / nudge to elicit a reply.              |
| `contextual_update`     | Push instructions or context without a user turn. |
| `client_tool_result`    | Reply to an ElevenLabs `client_tool_call`.        |

`conversation_initiation_client_data` and `pong` are blocked because they are managed by LiveAvatar and forwarding them would corrupt the protocol. Any other inner type is dropped.

### Listening for ElevenLabs events

If migrating from an existing ElevenLabs agent implementation, note that the ElevenLabs agent WebSocket is not exposed by this connector. Subscribe to the **LiveKit data channel** and parse the `elevenlabs_agent_event` event. The original ElevenLabs payload is passed through verbatim under the `data` field, so your existing parsers can consume `data` as-is.

<CodeGroup>
  ```typescript Web SDK theme={null}
  import { AgentEventsEnum } from "@heygen/liveavatar-web-sdk";

  session.on(AgentEventsEnum.ELEVENLABS_AGENT_EVENT, (event) => {
    console.log("ElevenLabs event:", event.elevenlabs_event_type, event.data);
  });
  ```

  ```json LiveKit Data Channel theme={null}
  {
    "event_id": "abc-123",
    "event_type": "elevenlabs_agent_event",
    "session_id": "session-456",
    "elevenlabs_event_type": "agent_response",
    "data": { ... }
  }
  ```
</CodeGroup>

### Sending ElevenLabs commands

<CodeGroup>
  ```json LiveKit Data Channel theme={null}
  {
    "event_type": "elevenlabs_agent_command",
    "elevenlabs_event_type": "user_message",
    "data": { "text": "hi there" }
  }
  ```
</CodeGroup>

For the full ElevenLabs event reference, see the [ElevenLabs Conversational AI documentation](https://elevenlabs.io/docs/conversational-ai/customization/events).

## Billing

* **LiveAvatar**: 1 credit per session minute (avatar video only)
* **ElevenLabs**: charges separately for agent usage