Assistants

Assistants are headless participants that connect to a Whereby room to enhance your user experience.

They can:

• Access all participant audio and video streams in a connected room, including a combined audio stream of all participants

• Perform in room actions like sending chat messages, starting cloud recording, letting participants into the room

• Alongside our Trigger API be triggered automatically via webhooks (e.g. when a first participant joins a room or when two or more participants join and a room session starts).

Assistants run in Node.js environments and are designed for backend integrations and realtime AI use cases. The SDK can not be run in the browser - instead use Core or Browser SDK for creating frontend-based integrations.

Set Up

There are a few steps required to set up Assistants to be used in your Whereby sessions.

Creating an Assistant

Before an Assistant will be granted access to your sessions, they must first be created in the Whereby dashboard. This will assign each individual assistant an assistantKey This key uniquely identifies every assistant and will be later used in the implementation stage when joining a call as a Whereby Assistant.

In this stage, you can assign some key details to your Assistant including:

• Avatar

• Display Name

• Description

• If you are using the Trigger API, you can also set the URL that you have listening for Whereby Webhook events.

Once you've assigned these details, you can save the Assistant and obtain the assistantKey.

Your Assistant will be created in a disabled state initially. Once you're ready to start using your Assistant, you can toggle it to the enabled state via the dashboard.

Once enabled, any Whereby Assistant that now provides the assistantKey copied from this interface will self-identify as this assistant.

That's it for the configuration steps!

When an assistant now joins any room (using its assistantKey) an in-room notification will be shown to all participants.

A Whereby Assistant then sits in the room status bar for the duration that it is connected and until it leaves the room.

Interacting with the assistant icon in the room status displays the title of the assistant.

When the assistant leaves the room it will announce that it is leaving to all participants remaining in the room.

And that's it!

Now, you can view our Quick Start guides or see our example apps to set up and explore the possibilities for your Assistant.

Key Features

Individual Streams

Access the individual audio and media streams of all participants in a session.

Use cases: video analysis, video recording, per-participant live transcription and recording

Combined Audio

All remote participant audio mixed into a single MediaStream

Use cases: live transcription, AI models. sentiment analysis, audio only recording

Trigger API

Start assistants automatically when specific webhooks arrive (eg room.client.joined )

See the Trigger API Reference for more details

In-Room Actions

Assistants can perform a variety of actions inside a room - they are given the same action permissions as host users. These actions include such as:

• Sending and receiving chat messages

• Starting or stopping cloud recordings

• Spotlighting participants

• Request audio / video to be enabled for participants

• Admitting waiting participants

See the Core API Reference the full list of in room actions available.

Usage and Pricing

There are no additional costs for enabling assistants - they are treated as normal participants in a room. Standard Whereby participant minute billing applies.

What next?

• Take a look at our Assistant SDK Reference documentation for more details and to get started building a Whereby Assistant.