# Assistants

{% hint style="warning" %}
⚠️ **Closed Beta**\
The Assistants feature is currently in Closed Beta and available to Enterprise plan customers on annual plans. If your organisation is on a Enterprise plan, you can request access by emailing <embedded@whereby.com>. Assistants will be made generally available after the beta.
{% endhint %}

{% hint style="info" %}
Enabling assistants is not charged as an additional feature — they are treated as normal participants in a room. Your standard Whereby participant minute billing applies to this usage.
{% endhint %}

Assistants are headless participants that connect to a Whereby room to enhance your user experience.&#x20;

They can:&#x20;

* Access all participant audio and video streams in a connected room, including a combined audio stream of all participants
* Perform in-room actions like sending chat messages, starting cloud recording, letting participants into the room
* Alongside our [Trigger API](https://docs.whereby.com/reference/assistant-sdk-reference/api-reference/trigger) be triggered automatically via webhooks (e.g. when a first participant joins a room or when two or more participants join and a room session starts).

## Prerequisites

The Assistant SDK is intended for Node.js environments and depends on some external tooling:&#x20;

* **Node.js ≥ 20**
* [**FFmpeg**](https://ffmpeg.org/) must be installed on the host machine if you wish to make use of the **combined audio stream** functionality.&#x20;

{% hint style="warning" %}
Note: The SDK cannot be run in the browser - use [Core](https://docs.whereby.com/reference/core-sdk-reference) or [Browser](https://docs.whereby.com/reference/react-hooks-reference) SDK for creating user-interface based integrations.&#x20;
{% endhint %}

## Setup

There are a few steps required to set up Assistants to be used in your Whereby sessions.&#x20;

### Creating an Assistant

Before an Assistant will be granted access to your sessions, they must first be created in the Whereby dashboard. This will assign each individual assistant an `assistantKey`. This key uniquely identifies every assistant and will be later used in the implementation stage when joining a call as a Whereby Assistant.&#x20;

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2FY5q7C6RfvhyggbWQ6EuQ%2Fimage.png?alt=media&#x26;token=48231073-57ee-42ce-80c7-4ff324a7b020" alt=""><figcaption></figcaption></figure>

In this stage, you can assign some key details to your Assistant including:&#x20;

* Avatar
* Display Name
* Description
* If you are using the [Trigger API](https://docs.whereby.com/reference/assistant-sdk-reference/api-reference/trigger), you can also set the URL that you have listening for Whereby Webhook events.

Once you've assigned these details, you can save the Assistant and obtain the `assistantKey`.&#x20;

{% hint style="info" %}
You can always edit an Assistant's configuration, view its `assistantKey`, and enable or disable it in your organization after it has been created.&#x20;
{% endhint %}

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2FhA1yKB3Ej1kTWSSXvGwE%2Fimage.png?alt=media&#x26;token=3e3d73ad-b6fa-4ee8-9fb9-cd6acf3d2859" alt=""><figcaption></figcaption></figure>

Your Assistant will be created in a `disabled` state initially. Once you're ready to start using your Assistant, you can toggle it to the `enabled` state via the dashboard.&#x20;

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2FAd4eH0GtztzrHFQGVi0J%2Fimage.png?alt=media&#x26;token=d904c9f4-fc80-4c83-8df4-3e83851219b1" alt=""><figcaption></figcaption></figure>

Once enabled, any Whereby Assistant that now provides the `assistantKey` copied from this interface will self-identify as this assistant.

If you'd like room hosts to be able to request this assistant from within the room, you can enable **Manual** **Invite**. More on that below.

{% hint style="info" %}
You'll need an Endpoint URL configured before Manual Invite can be enabled.
{% endhint %}

That's it for the configuration steps!

## In Room

When an assistant now joins any room (using its `assistantKey`), an in-room notification will be shown to all participants.

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2FaBwVidCIB4u3u2B673Wk%2Fimage.png?alt=media&#x26;token=17d2e904-49b2-409a-89a5-e387bf0f75c4" alt=""><figcaption></figcaption></figure>

A Whereby Assistant then sits in the room status bar for the duration that it is connected and until it leaves the room.&#x20;

Interacting with the assistant icon in the room status displays the assistants menu.&#x20;

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2FkHNnEf59gDC3dLcEqXIz%2Fimage.png?alt=media&#x26;token=8a5c8a03-8030-43b1-9b49-4189e4d3dec5" alt=""><figcaption></figcaption></figure>

### Manual Invite

When enabled, hosts will be able to invite your Whereby Assistant into their room with the **Invite** button in the assistants menu.

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2Fgn3rOhbNydRwHltiyDat%2Fimage.png?alt=media&#x26;token=50693810-b75f-4c93-af27-ae958f78dc93" alt=""><figcaption></figcaption></figure>

The **Invite** button will send a webhook event with the type `assistant.requested` endpoint URL configured for this assistant. You can configure your webhook server to start the requested assistant and have it join the room specified in the webhook event payload.

Once an assistant has joined the room, hosts have the ability to remove it through the assistants menu.

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2FfGnQWEmTfJJ3UUOvtLIP%2Fimage.png?alt=media&#x26;token=e4eed882-3b1a-4bd4-b656-bc07a29d570a" alt=""><figcaption></figcaption></figure>

When the assistant leaves the room it will announce that it is leaving to all participants remaining in the room.

<figure><img src="https://888889433-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLH2UIJDLc2EElQvYUxWX%2Fuploads%2F7uSIBhB9wg7jhO1H8K82%2Fimage.png?alt=media&#x26;token=f77f1a2a-56f5-4d7a-9d69-b766993dbcc1" alt=""><figcaption></figcaption></figure>

And that's it!

Now, you can view our [Quick Start](https://docs.whereby.com/reference/assistant-sdk-reference/quick-start) guide to set up and explore the possibilities for your Assistant.&#x20;

## Key Features

### Individual Streams

Access the individual audio and media streams of all participants in a session.&#x20;

**Use cases**: video analysis, video recording, per-participant live transcription and recording

### Combined Audio&#x20;

All remote participant audio mixed into a single audio stream.

**Use cases**: single track live transcription, AI models, sentiment analysis, audio only recording

### Trigger API&#x20;

Start assistants automatically when specific webhooks arrive (eg `room.client.joined` )

See the [Trigger API Reference](https://docs.whereby.com/reference/assistant-sdk-reference/api-reference/trigger) for more details

### In-Room Actions&#x20;

Assistants can perform a variety of actions inside a room — they are given the same action permissions as host users. These actions include:&#x20;

* Sending and receiving chat messages
* Starting or stopping cloud recordings
* Spotlighting participants&#x20;
* Request audio / video to be enabled for participants
* Admitting waiting participants

See the [Core API Reference](https://docs.whereby.com/reference/core-sdk-reference) for the full list of in-room actions available.&#x20;

## See Also

* [Assistant SDK Reference](https://docs.whereby.com/reference/assistant-sdk-reference): Complete documentation to get started building a Whereby Assistant.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.whereby.com/whereby-product-features/assistants.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.