Assistant SDK Reference

⚠️ Early Access Only This feature is currently available by invite only. If your organization has been granted access, you can follow the documentation below. For others, stay tuned - Assistants will be available in a wider beta soon.

The Whereby Assistant SDK is a Node.js-focused SDK for building assistants that can join Whereby rooms. It provides all the polyfills and plumbing needed to join a Whereby Room in Node, and exposes a simple API to connect assistants, manage their lifecycle and access audio and video streams and other in-room features for all participants. This SDK can be used in conjunction with our Trigger API to listen for webhooks and join the room on certain room conditions being met.

Audience

This package is designed for backend developers and AI/assistant builders who want to:

• Run assistants, bots or agents headlessly (no browser UI)

• Integrate Whereby with real-time transcriptions services

• Stream room audio into LLM-based agents for conversation, summarization or automation

• Trigger assistants to join rooms automatically using webhooks

Use cases

• Capturing a single combined audio stream of all participants for transcription, captioning or analysis

• Feeding live meeting audio to real-time AI agents (eg OpenAI Realtime)

• Running a support bot that automatically joins rooms when triggered, and can perform in room actions

• Building meeting recorders or pipelines that store and process session audio

Requirements

The Assistant SDK is intended for Node.js environments and depends on some external tooling:

• Node.js ≥ 20

• FFmpeg must be installed on the host machine if you wish to make use of the combined audio stream functionality.

Note: The SDK can not be run in the browser - use Core or Browser SDK for creating user-interface based integrations.

Getting started

The Assistant SDK is currently in Early Access and has not yet been published publicly.

Installation

Install the @whereby.com/assistant-sdk package from the public npm registry.

npm

npm install @whereby.com/assistant-sdk

yarn

yarn add @whereby.com/assistant-sdk

pnpm

pnpm add @whereby.com/assistant-sdk

Installing FFmpeg

To use the combined audio stream functionality you will need FFmpeg to be installed on the machine running your assistant. You can check if FFmpeg is already installed on your local machine by running:

ffmpeg -version

If you see the version number, you're good to go! if not, install it using one of the methods below:

macOS (Homebrew)

brew install ffmpeg

Ubutntu/Debian

sudo apt update
sudo apt install ffmpeg

Windows (Chocolatey)

choco install ffmpeg

Or download a prebuilt binary directly from ffmpeg.org/download.

After installation, make sure ffmpeg is available in your system PATH so the Assistant SDK can call it.