Technical Support Request: Rocket.Chat + LiveKit Integration Fails for External Students
1. Project overview
We are developing a self-hosted online classroom and trading education platform.
Our objective is to provide a classroom experience similar to Messenger or TikTok:
-
Students log into Rocket.Chat once.
-
Students communicate through Rocket.Chat channels.
-
Students click Join Live Classroom to enter a video classroom.
-
The instructor controls who can activate microphones and cameras.
-
Students join as viewers by default.
-
Students can request permission to speak.
-
The platform should work reliably on desktop and mobile browsers, including outside our local network.
We previously used Rocket.Chat with Jitsi Meet. Video meetings could be launched directly through Rocket.Chat’s camera button.
However, we experienced a permissions problem: ordinary students were receiving moderator privileges.
We subsequently developed a custom LiveKit classroom to implement stricter instructor controls.
The new LiveKit integration has a critical authentication problem.
2. Server environment
Operating system: Ubuntu 26.04 LTS
Hardware:
-
Intel Core i5-10500T
-
32 GB RAM
-
1 TB NVMe SSD
Infrastructure:
-
nginx reverse proxy
-
Let’s Encrypt HTTPS
-
Rocket.Chat 8.8.1 running in Docker
-
MongoDB 8.2
-
LiveKit media server
-
Custom Next.js 15.5.21 classroom application
-
Node.js and PM2
Application addresses
Rocket.Chat:
LiveKit classroom:
Classroom login:
OAuth callback:
The Next.js application runs locally on port 3202.
PM2 process:
classroom-new-release
Project directory:
/home/birhane/projects/livekit-meet-release-20261023
3. The main problem
The administrator can enter the LiveKit classroom, but an external student cannot.
Administrator:
-
Rocket.Chat username:
LinuxAdmin -
Can authenticate.
-
Can enter the LiveKit classroom.
-
Can see instructor controls.
External student:
-
Rocket.Chat username:
Mieraf -
Rocket.Chat account is active.
-
Can log into Rocket.Chat.
-
Can open the classroom page.
-
Can see the Join Room interface.
-
Encounters a white screen during the joining process.
-
Receives the error:
HTTP 403 — Invalid OAuth state
The student also tried opening the classroom login endpoint directly, without using the Rocket.Chat button.
The same error occurred.
The administrator can authenticate successfully through the normal login flow.
Opening the OAuth callback URL directly also produces Invalid OAuth state, which is expected because a direct request does not contain the required OAuth state.
4. Current authentication architecture
Rocket.Chat uses OAuth to authenticate users entering the Next.js classroom.
The login endpoint generates a random state:
const state = randomBytes(32).toString('hex');
It redirects the browser to Rocket.Chat:
const authorizationUrl = new URL('/oauth/authorize', rocketChatUrl);
authorizationUrl.searchParams.set('response_type', 'code');
authorizationUrl.searchParams.set('client_id', clientId);
authorizationUrl.searchParams.set('redirect_uri', redirectUri);
authorizationUrl.searchParams.set('state', state);
It sets the following cookie:
response.cookies.set('_oauth_state', state, {
httpOnly: true,
secure: true,
sameSite: 'lax',
path: '/api/auth/callback/rocketchat',
maxAge: 900,
});
The callback retrieves the cookie:
const expectedState =
request.cookies.get('_oauth_state')?.value;
const receivedState =
request.nextUrl.searchParams.get('state');
It validates the values:
if (!validOAuthState(expectedState, receivedState)) {
return reply('Invalid OAuth state', 403);
}
The validation function:
import { timingSafeEqual } from 'node:crypto';
export function validOAuthState(
expected: string | undefined,
received: string | null,
): boolean {
if (!expected || !received) {
return false;
}
const a = Buffer.from(expected, 'utf8');
const b = Buffer.from(received, 'utf8');
if (a.length !== 64 || b.length !== 64) {
return false;
}
return timingSafeEqual(a, b);
}
The validation code appears reasonable, but we have not established whether the student is missing the cookie, receiving a different state, or encountering another browser-specific issue.
5. Rocket.Chat integration
Rocket.Chat contains a custom JavaScript button.
Relevant code:
(() => {
const BUTTON_ID = '-live-classroom-link';
const LOGIN_URL =
'https://meet/api/auth/login';
if (window.__ClassroomButtonInitialized) return;
window.__ClassroomButtonInitialized = true;
const button = document.createElement('a');
button.id = BUTTON_ID;
button.href = LOGIN_URL;
button.target = '_blank';
button.rel = 'noopener noreferrer';
button.textContent = '🎥 Join Live Classroom';
})();
The actual script also styles the button and appends it to the document.
The button opens a new browser tab.
The student receives the same OAuth error when bypassing this button and visiting the login endpoint directly.
6. Verified OAuth configuration
The Next.js .env.local contains:
ROCKETCHAT_URL=https://
ROCKETCHAT_REDIRECT_URI=https:///api/auth/callback/rocketchat
Rocket.Chat’s MongoDB OAuth application configuration contains:
name: OSCE Academy Classroom
redirectUri:
https:///api/auth/callback/rocketchat,
https://api/auth/callback/rocketchat
active: true
Both the older classroom domain and the current meet domain are registered.
The current Next.js application uses the meet domain.
7. nginx configuration
The active classroom domain uses:
server {
server_name ;
auth_basic off;
location / {
proxy_pass http://127.0.0.1:3202;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
}
listen 443 ssl;
}
HTTPS is active.
The reverse proxy forwards the host and protocol headers.
There is also an older nginx configuration for:
That configuration routes its OAuth callback to port 3202, but the main application currently uses the meet domain.
8. PM2 logs
The Next.js application starts successfully:
▲ Next.js 15.5.21
Local: http://127.0.0.1:3202
✓ Starting...
✓ Ready in 396ms
The error log repeatedly contains:
Local storage is not available.
No detailed OAuth state failure diagnostics were observed in the supplied PM2 logs.
We have not established whether the local-storage warnings are related to the authentication failure.
9. What has already been tested
-
Administrator login works.
-
Student Rocket.Chat login works.
-
Student can open the classroom page.
-
Student encounters a white screen and
Invalid OAuth state. -
Closing classroom tabs and retrying did not solve the problem.
-
Opening the login endpoint directly did not solve the problem.
-
OAuth state generation and validation code were inspected.
-
Cookie attributes were inspected.
-
nginx reverse-proxy configuration was inspected.
-
Rocket.Chat OAuth application registration was inspected.
Not yet verified:
-
Whether the student’s browser receives the
Set-Cookieheader. -
Whether the browser stores
osce_oauth_state. -
Whether the cookie is included in the callback request.
-
Whether the callback’s returned state matches the originally generated state.
-
Whether multiple login requests overwrite the cookie.
-
Whether browser privacy settings interfere with the redirect.
-
Whether the production environment differs from the inspected source configuration.
10. Required solution
We do not want to disable OAuth state validation or weaken authentication security.
We want a stable, production-ready solution that:
-
Allows Rocket.Chat users to enter the classroom without repeated authentication problems.
-
Supports desktop and mobile browsers.
-
Works for external students.
-
Maintains secure authentication and session management.
-
Keeps the administrator as the only instructor/moderator.
-
Allows students to join as viewers.
-
Allows instructors to grant microphone and camera permissions.
-
Supports automatic reconnection after network interruptions.
-
Does not require users to log into two separate applications unnecessarily.
We prefer an established open-source integration or proven implementation over writing extensive custom authentication code.
11. Questions for experienced developers
A. OAuth diagnosis
What is the most likely explanation for the OAuth state validation succeeding for the administrator but failing for an external student?
What is the fastest reliable method to determine whether the OAuth state cookie is missing, overwritten, or mismatched?
B. Architecture
Should we continue using Rocket.Chat OAuth with a custom Next.js LiveKit application?
Or is there a more reliable existing integration that allows Rocket.Chat to provide authentication while LiveKit provides video and audio?
C. Existing implementations
Are there maintained open-source projects demonstrating Rocket.Chat SSO integration with LiveKit, including instructor/participant roles?
Can we reuse their authentication and session-management approach rather than developing our own?
D. Alternative solution
Would Rocket.Chat’s native Jitsi integration with properly configured moderator roles and audio/video moderation be simpler and more reliable?
If so, what exact Jitsi configuration ensures that only the instructor receives moderator privileges while students remain ordinary participants?
12. Important constraints
-
This is a self-hosted system.
-
We want to avoid unnecessary cloud-service costs.
-
The existing Rocket.Chat installation must remain operational.
-
The current LiveKit application must not be deleted until a replacement has been tested.
-
Security protections must remain enabled.
-
We prefer minimal, tested configuration changes over extensive custom development.
Primary objective:
A student logs into Rocket.Chat, clicks one camera or classroom button, and joins a secure video classroom where only the instructor has moderator privileges.
We need a proven implementation that works consistently for external users.