CUSTOM Mode Events

In CUSTOM Mode, we will process events sent through via the websocket we expose. These events are transmitted as JSON formatted strings. The general format will be as follows:

{
  "type": str, 
  ... // event data
}

Expect to process them as JSON strings and send the event data to use as JSON strings via the websocket. If you any events you'd like us to pass in, we'd recommend giving feedback and recommendations.

Command Events

In Custom Mode, you can send us events via websocket endpoint. As of now, we primarily support transitioning the Avatar between speaking and listening, as well as interrupting any existing events that are happening.

Name / `type`	Event Data	Description
`agent.interrupt`	`{"type": "agent.interrupt"}`	Interrupts and stops any scheduled commands. This will clear all scheduled speaking events.
`agent.speak`	`{"type": "agent.speak", "audio": <string>}`	Adds to the avatar buffer for what it will say. The event_id can be set to indicate grouping between sets of interactions. Audio should be PCM 16Bit 24KHz bytes encoded as Base64 Note - we recommend sending many small byte packets. We may close the connection if we detect too large of an audio string input (> 1MB).
`avatar.speak_end`	`{ "type": "agent.speak_end", "event_id": <str> }`	Signal the end of a speaking event.
`agent.start_listening`	`{"type":"agent.start_listening", "event_id": <str>}`	Transition the avatar to a listening state.
`agent.stop_listening`	`{"type":"agent.stop_listening", "event_id": <str>}`	Transition the avatar from a listening state to an idle state.
`session.keep_alive`	`{ "type": "session.keep_alive", "event_id": "<event_id>", }`	Keep the session alive longer. We close the session after a 5 minute idle time window. Periodically send this event if you want to keep the session alive longer.

Server Events

We will emit these events from the websocket that is public when calling custom mode. This is a smaller set of events.

Name	Event Data	Description
`session.state_updated`	`{ "type": "session.state_updated", "event_id": uuid, "state": strEnum}`	We track what the current state of the session is. Prior to sending any events, we require the state to be "connected" Possible states: "connected", "connecting", "closed", "closing".
`agent.speak_started`	`{ "type":"agent.speak_started", "event_id": <str>, "task": {"id": <str> } }`	Avatar started speaking. The event id matches the corresponding, `agent.speak` command event.
`agent.speak_ended`	`{"type":"agent.speak_started", "event_id": <str>, "task": {"id": <str> }}`	Avatar started speaking. The event id matches the corresponding, `agent.speak` command event. Note - should always follow a `agent.speak_started` event.

Name

Event Data

Description

session.state_updated

{ "type": "session.state_updated", "event_id": uuid, "state": strEnum}

We track what the current state of the session is. Prior to sending any events, we require the state to be "connected"

Possible states:
"connected", "connecting", "closed", "closing".

agent.speak_started

{ "type":"agent.speak_started", "event_id": <str>, "task": {"id": <str> } }

Avatar started speaking. The event id matches the corresponding, agent.speak command event.

agent.speak_ended

{"type":"agent.speak_started", "event_id": <str>, "task": {"id": <str> }}

Avatar started speaking. The event id matches the corresponding, agent.speak command event.

Note - should always follow a agent.speak_started event.