Change the avatar background - LiveAvatar Documentation

What you’ll build

Some of LiveAvatar’s avatars are rendered with solid green backgrounds. By detecting and removing those green pixels on the client, you can:

Make the background transparent so the avatar sits on top of your own UI.
Replace the background with an image (branded backdrop, product shot, office scene).
Replace the background with a video (looping environment, ambient footage).

This guide shows how to do all three in real time, with no server-side processing and no additional credits consumed. It works in both FULL Mode and LITE Mode.

Want to skip ahead and try it live? Jump to Run the reference demo.

How it works

The avatar video element exposes a normal MediaStream with a green background. You render that stream to a canvas, and for every frame:

Convert each pixel to HSV and check whether its hue falls inside a green range.
Replace qualifying pixels — either with full transparency, or with a pixel from your custom background.
Draw the result to the visible canvas.

This runs in any modern browser with no external libraries.

If you’re using the default embed iframe, you won’t have direct access to the underlying video element because it’s in a cross-origin frame. Use the Web SDK or a LITE Mode integration (LiveKit, Agora, Pipecat) instead — those give you the raw MediaStream.

Training a custom avatar? Record against a specific solid background color (green is standard, but any saturated, uniform color works) and use the same chroma key technique below to swap it out — adjust minHue / maxHue to match the color you trained on.

Migrating from HeyGen Interactive Avatar? The chroma key approach below is the same one used in the legacy Interactive Avatar green-screen guide — the parameter names (minHue, maxHue, minSaturation, threshold) are identical, so existing code ports over with minimal changes.

Run the reference demo

liveavatar-web-sdk · apps/bg-removal-demo

Complete Next.js reference app — chroma key toggle, solid color, preset images, custom image/video URL, voice chat.

Demonstrates:

Toggling the chroma key on/off
Swapping to a solid color
Preset image backgrounds
A custom image or video URL pasted at runtime
Voice chat with the avatar while the backdrop changes

Initialize

Clone repo, install deps, configure env:

git clone https://github.com/heygen-com/liveavatar-web-sdk.git
cd liveavatar-web-sdk
pnpm install
cp apps/bg-removal-demo/.env.example apps/bg-removal-demo/.env.local
# fill in API_KEY and defaults in apps/bg-removal-demo/.env.local

Start the dev server

Run from monorepo root:

pnpm demo:bg-removal

Open http://localhost:3002, click Start session, grant microphone permission, then use the Background panel on the right to swap backdrops live.

Environment variables

Variable	Description
`API_KEY`	LiveAvatar API key (server-side only)
`API_URL`	LiveAvatar API base URL
`NEXT_PUBLIC_API_URL`	Same as `API_URL` (read by the SDK in the browser)
`DEFAULT_AVATAR_ID`	Default avatar ID for the start form
`DEFAULT_VOICE_ID`	Default voice ID
`DEFAULT_CONTEXT_ID`	Default context ID
`DEFAULT_LANGUAGE`	Default language code (e.g. `en`)

See apps/bg-removal-demo/README.md for full setup details, project structure, and the lib/chromaKey.ts source the production code is derived from.

Understanding the details

Here’s how the reference demo does it. Three steps, each with a clear job:

Mount the DOM surfaces — a <video> element for the raw avatar stream, a <canvas> for the keyed output, and a toggle to switch between them.
Run the chroma key per frame — read pixels off the video, zero out alpha on green ones, write back to the canvas.
Wire the toggle to swap layers — show the canvas (with effect) or the raw video (without), and stop the loop when the session ends.

The steps below walk through each piece. Assumes you already have a <video> element playing the LiveAvatar stream via the Web SDK or a LITE Mode integration.

Step 1: Update your HTML

Add a canvas next to the avatar video, plus a checkbox to toggle the effect.

<!-- Video Section -->
<article style="width: fit-content;">
  <video id="avatarVideo" autoplay playsinline></video>
  <canvas id="avatarCanvas"></canvas>
</article>

<!-- Chroma Key Toggle -->
<div class="chromakey-toggle">
  <input type="checkbox" id="chromaKeyToggle" />
  <label for="chromaKeyToggle">Enable Chroma Keying</label>
</div>

Step 2: Create the chroma key module

Create src/chromaKey.ts with the per-frame keying logic.

/**
 * Apply chroma key effect to a video frame on canvas.
 */
export function applyChromaKey(
  sourceVideo: HTMLVideoElement,
  targetCanvas: HTMLCanvasElement,
  options: {
    minHue: number;        // 60  - minimum hue value (0-360)
    maxHue: number;        // 180 - maximum hue value (0-360)
    minSaturation: number; // 0.10 - minimum saturation (0-1)
    threshold: number;     // 1.00 - threshold for green detection
  }
): void {
  const ctx = targetCanvas.getContext("2d", {
    willReadFrequently: true,
    alpha: true,
  });

  if (!ctx || sourceVideo.readyState < 2) return;

  targetCanvas.width = sourceVideo.videoWidth;
  targetCanvas.height = sourceVideo.videoHeight;

  ctx.clearRect(0, 0, targetCanvas.width, targetCanvas.height);
  ctx.drawImage(sourceVideo, 0, 0, targetCanvas.width, targetCanvas.height);

  const imageData = ctx.getImageData(0, 0, targetCanvas.width, targetCanvas.height);
  const data = imageData.data;

  for (let i = 0; i < data.length; i += 4) {
    const r = data[i];
    const g = data[i + 1];
    const b = data[i + 2];

    // Convert RGB to HSV.
    const max = Math.max(r, g, b);
    const min = Math.min(r, g, b);
    const delta = max - min;

    let h = 0;
    if (delta === 0) {
      h = 0;
    } else if (max === r) {
      h = ((g - b) / delta) % 6;
    } else if (max === g) {
      h = (b - r) / delta + 2;
    } else {
      h = (r - g) / delta + 4;
    }
    h = Math.round(h * 60);
    if (h < 0) h += 360;

    const s = max === 0 ? 0 : delta / max;
    const v = max / 255;

    const isGreen =
      h >= options.minHue &&
      h <= options.maxHue &&
      s > options.minSaturation &&
      v > 0.15 &&
      g > r * options.threshold &&
      g > b * options.threshold;

    if (isGreen) {
      // Soft edges: the "greener" the pixel, the more transparent.
      const greenness = (g - Math.max(r, b)) / (g || 1);
      const alphaValue = Math.max(0, 1 - greenness * 4);
      data[i + 3] = alphaValue < 0.2 ? 0 : Math.round(alphaValue * 255);
    }
  }

  ctx.putImageData(imageData, 0, 0);
}

/**
 * Setup continuous chroma keying on an animation frame loop.
 * Returns a cleanup function that stops the processing.
 */
export function setupChromaKey(
  sourceVideo: HTMLVideoElement,
  targetCanvas: HTMLCanvasElement,
  options: {
    minHue: number;
    maxHue: number;
    minSaturation: number;
    threshold: number;
  }
): () => void {
  let animationFrameId: number | null = null;

  const render = () => {
    applyChromaKey(sourceVideo, targetCanvas, options);
    animationFrameId = requestAnimationFrame(render);
  };

  render();

  return () => {
    if (animationFrameId !== null) {
      cancelAnimationFrame(animationFrameId);
    }
  };
}

Step 3: Wire up the toggle

In the code that owns your avatar video element, import setupChromaKey and switch between the canvas and the raw video based on the checkbox state.

import { setupChromaKey } from "./chromaKey";

const videoElement = document.getElementById("avatarVideo") as HTMLVideoElement;
const canvasElement = document.getElementById("avatarCanvas") as HTMLCanvasElement;
const chromaKeyToggle = document.getElementById("chromaKeyToggle") as HTMLInputElement;

let stopChromaKeyProcessing: (() => void) | null = null;

function updateChromaKeyState() {
  if (!videoElement.srcObject) return;

  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }

  if (chromaKeyToggle.checked) {
    canvasElement.style.display = "block";
    videoElement.style.display = "none";

    stopChromaKeyProcessing = setupChromaKey(videoElement, canvasElement, {
      minHue: 60,
      maxHue: 180,
      minSaturation: 0.1,
      threshold: 1.0,
    });
  } else {
    videoElement.style.display = "block";
    canvasElement.style.display = "none";
  }
}

chromaKeyToggle.addEventListener("click", updateChromaKeyState);

Always stop processing when the stream disconnects or the session ends, to avoid dangling animation frames:

function handleStreamDisconnected() {
  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }
  // ...rest of your disconnect handling
}

async function terminateAvatarSession() {
  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }
  // ...rest of your termination logic
}

Flip the checkbox and the green background becomes transparent.

Add an image or video background

Transparency works when you want the avatar to float over your existing page. For a custom backdrop — a branded scene, an office environment, a product shot — composite a background layer behind the canvas in the DOM. No changes to the chroma key module are required. The canvas output is already transparent where the green used to be, so anything behind it shows through.

Image background

<article style="width: fit-content; position: relative;">
  <img
    id="avatarBackground"
    src="/backgrounds/office.jpg"
    style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover; z-index: 0;"
  />
  <video id="avatarVideo" autoplay playsinline style="position: relative; z-index: 1;"></video>
  <canvas id="avatarCanvas" style="position: relative; z-index: 1;"></canvas>
</article>

Video background

<article style="width: fit-content; position: relative;">
  <video
    src="/backgrounds/office-loop.mp4"
    autoplay
    loop
    muted
    playsinline
    style="position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover; z-index: 0;"
  ></video>
  <video id="avatarVideo" autoplay playsinline style="position: relative; z-index: 1;"></video>
  <canvas id="avatarCanvas" style="position: relative; z-index: 1;"></canvas>
</article>

Match the background asset’s aspect ratio to the avatar canvas (typically 16:9) and encode videos at the same resolution. A 1080p background behind a 720p avatar wastes GPU bandwidth and can cause stutter on lower-end devices.

Swap backgrounds at runtime

Because the background is a sibling DOM element, swapping it is a one-liner. No session restart, no chroma key reconfiguration.

function setImageBackground(url: string) {
  const bg = document.getElementById("avatarBackground") as HTMLImageElement;
  bg.src = url;
}

Make the avatar fully transparent

For overlaying the avatar on your own page — a floating assistant in the corner of a product UI, an in-app tutor, a kiosk overlay — skip the background entirely and let the page behind the canvas show through.

#avatarCanvas {
  background: transparent;
}

article {
  background: transparent;
}

This is the recommended setup for sales assistants, customer support overlays, and in-app tutors where the avatar should feel integrated into your product rather than boxed into a video window.

Tuning the chroma key

The chroma key parameters can be adjusted to fine-tune the effect:

minHue / maxHue — range of green hues to detect. 60–180 covers most greens. Narrow it (e.g., 90–150) if the avatar’s clothing is being partially keyed out.
minSaturation — minimum saturation for detection. Avoids treating unsaturated grays and whites as green. Default 0.1 is usually fine.
threshold — how much “greener” a pixel must be vs. its red/blue components. Higher is stricter. Lower it toward 0.8 if you see a green halo; raise it toward 1.2 if avatar pixels are becoming transparent.

Troubleshooting

Green halo around the avatar’s edges. The soft-edge falloff is controlled by the greenness * 4 multiplier in applyChromaKey. Increase it (try * 6 or * 8) to make the fade-to-transparent more aggressive, which eats further into the halo. You can also tighten maxHue to 160 so only the purest greens are keyed. Parts of the avatar are becoming transparent. The avatar is likely wearing something close to the key color (green tie, green shirt). Raise threshold to 1.2 or 1.3 so only pixels where green strongly dominates red and blue are removed. If the avatar is wearing green by design, pick a different avatar — chroma keying a green-on-green subject isn’t solvable client-side. Edges flicker frame-to-frame. Each frame is keyed independently, so small boundary flicker is expected. To reduce it, render the canvas at the same resolution as the source video (don’t upscale) and avoid CSS filters like filter: blur() on the canvas. Performance is poor on mobile or at 1080p. The getImageData → per-pixel loop → putImageData path is CPU-bound and can struggle above 720p on low-end devices. Two options:

Render at a lower resolution by setting the canvas width/height to 640×360 or 854×480 — the avatar still looks sharp because the source stream scales gracefully.
Use WebGL — a fragment-shader implementation runs on the GPU and handles 1080p at 60 FPS comfortably. Bigger lift, but worth it for high-resolution or mobile-first deployments.

Common pitfalls

The canvas is blank but the video is playing. applyChromaKey guards against this with a readyState < 2 check, but don’t call setupChromaKey before the video has a srcObject. Wait for your stream-ready event (e.g., the Web SDK’s STREAM_READY) before starting processing. The embed iframe doesn’t expose the video element. The default embed is designed for drop-in simplicity and doesn’t expose the raw video stream. For chroma keying, switch to the Web SDK or a LITE Mode integration where you control the rendering layer. Chroma key keeps running after the session ends. setupChromaKey returns a cleanup function — call it whenever the stream disconnects or the session terminates. Otherwise the animation frame loop keeps running against a stale video element and wastes CPU.

Guides

Documentation Index

​What you’ll build

​How it works

​Run the reference demo

liveavatar-web-sdk · apps/bg-removal-demo

​Initialize

​Start the dev server

​Environment variables

​Understanding the details

​Step 1: Update your HTML

​Step 2: Create the chroma key module

​Step 3: Wire up the toggle

​Add an image or video background

​Image background

​Video background

​Swap backgrounds at runtime

​Make the avatar fully transparent

​Tuning the chroma key

​Troubleshooting

​Common pitfalls

What you’ll build

How it works

Run the reference demo

Initialize

Start the dev server

Environment variables

Understanding the details

Step 1: Update your HTML

Step 2: Create the chroma key module

Step 3: Wire up the toggle

Add an image or video background

Image background

Video background

Swap backgrounds at runtime

Make the avatar fully transparent

Tuning the chroma key

Troubleshooting

Common pitfalls