[Beta] ElevenLabs Agent Plugin

This guide explains how to integrate your own ElevenLabs Conversational AI Agent into a LiveAvatar session. By using the LITE session mode, you can offload conversation handling to ElevenLabs while LiveAvatar provides the high-fidelity visual streaming layer.

Overview

The LiveAvatar ElevenLabs integration creates a seamless bridge between ElevenLabs' Conversational AI and our real-time streaming avatars.

When a session is initialized in LITE mode with an Agent configuration, LiveAvatar dispatches a worker that handles the interaction with your ElevenLabs Agent. Your agent's audio output is then used to drive our avatar's in real-time. The end result is a a low-latency, interactive experience powered by both ElevenLabs and LiveAvatar.

Prerequisites

Before starting, ensure you have the following:

1. ElevenLabs API Key: Can be found as
2. ElevenLabs Agent ID: Located in your ElevenLabs Conversational AI dashboard.
   1. Make sure to prepare an ElevenLabs Agent before starting.

We will save your ElevenLabs API key via our secrets endpoint. Then, when creating a LITE mode session, you'll have the option to pass the relevant information in to forward the data to our LiveAvatar session.

Adding your API Key

First, register your ElevenLabs API key with LiveAvatar. We encrypt all user secrets with Amazon Encryption library and Amazon KMS. We will return a newly created secret_id in the response that references your stored secret.

See endpoint details here.

curl --request POST \
     --url https://api.liveavatar.com/v1/secrets \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "secret_type": "ELEVENLABS_API_KEY",
  "secret_value": "your_elevenlabs_api_key",
  "secret_name": "<display name of your secret>"
}
'

Starting a session

When starting a LITE mode session, pass in the following data:

{
  "mode": "LITE",
  "elevenlabs_agent_config": {
    "secret_id": "<elevenlabs_secret_id>",
    "agent_id": "<elevenlabs_agent_id>"
  }
}

On our side, this plugin leverages our existing FULL mode infrastructure to spin up a LiveKit room tailored to the ElevenLabs Agent. As a result, there are some subtle differences when calling LITE mode in this manner.

• We do not return a WebSocket endpoint. Instead, we automatically create and manage a LiveKit room on your
  • Standard LITE mode events will not be emitted or available for subscription.
  • Instead, events are delivered through the LiveKit room using the same events as in FULL mode
• This runs in LITE mode, so billing remains 1 credit per session minute.
  • Since we delegate the VAD, STT, LLM and TTS calls to ElevenLabs, you're only charged for the generated avatar video.
  • ElevenLabs Agent usage will be billed directly to your ElevenLabs account.

Session Lifecycle

As mentioned above, you can monitor what is happening inside the session with FULL mode events. Given that we leverage most of our existing FULL mode infrastructure, we recommend setting this session up similar to a FULL mode session.

See the FULL Mode Lifecycle documentation for more details.