How to fix "400006 - Session is not in correct state"?

We use CUSTOM mode, and when we start sending audio through the websocket (ws_url) we get frequent:

Received event: {
"type":"error",
"event_id":"773253df-1a77-44d5-b66a-50ad5a8312d6",
"error":{
"type":"server_error",
"message":"http_code: 400,
api_code: 400006,
message: Session is not in correct state","event_id":"0"
}}

and

Received event: {"type":"agent.audio_buffer_cleared","event_id":"9d308039-2b86-4ab4-b91d-7c3eb6d55156"}

Sometimes (totally random) it works pretty well:

Received event: {
"type":"agent.audio_buffer_appended",
"event_id":"4f935bfc-cb2c-4e2d-aea3-10fce7479aa7",
"task":{"id":"4016979c-b6b8-4107-8e57-8a88bf3e9073"}
}
Received event: {
"type":"agent.audio_buffer_appended",
"event_id":"f3397d25-fe24-4050-ac46-ecd86937a41d",
"task":{"id":"4016979c-b6b8-4107-8e57-8a88bf3e9073"}
}

We tried waiting after calling v1/sessions/start to give the avatar time to join our LiveKit room and ensure everything was ready, but we still got the error.

How can we fix the error so that sending audio packages will always work?