Troubleshooting Event Propagation/Reactivity Issues in Rocket.Chat 7.7.0+
Summary
With the release of version 7.7.0, Rocket.Chat has changed its default mechanism for propagating events (such as messages, setting changes, etc) between instances in a multi-instance deployment. This change disables database changestreams by default, relying instead on the event transport.
Important Note: Running a multi-instance Rocket.Chat deployment requires a valid Enterprise License.
If your environment is not configured correctly for event transport, you may experience symptoms like:
- Messages or settings changes not appearing for users connected to different instances.
- Ephemeral events like typing indicators and user presence not updating across the cluster.
- Having to refresh in order to see a change made by someone else
- Enabling / Disabling an app not being fully disabled
- A generally “buggy” or inconsistent user experience.
This article outlines the cause of this change and provides the recommended solutions.
Background: The Shift from Changestreams
Historically, Rocket.Chat used two methods for inter-instance communication:
- P2P TCP Connections: Instances would discover each other via a shared instances collection in the database and form a direct peer-to-peer TCP connection. This connection was primarily used for ephemeral events (e.g., typing indicators).
- Database Changestreams: Instances would “watch” the database for persistent changes (e.g., new messages, setting updates). When a change occurred, other instances would see it and relay the update to their connected users.
As of version 7.7.0, database changestreams are disabled by default. This means all events both ephemeral and persistent must now be propagated through the event transport between the instances. While this improves performance and reliability, it requires proper network configuration, especially in containerized environments like Docker or Kubernetes.
Solutions
There are a few paths to ensure your instances can communicate correctly. The recommended approach depends on the scale and complexity of your deployment.
1. Recommended Solution: Use NATS Transport
Using a NATS message broker is the officially recommended method for most deployments. It provides the best balance of performance and ease of setup. In this model, each Rocket.Chat instance makes a single, outbound connection to the NATS server, which then broadcasts events to all other connected instances.
Configuration:
- Deploy a NATS server that is accessible from all your Rocket.Chat instances.
- Set the following environment variable on all Rocket.Chat instances:
TRANSPORTER=monolith+nats://<ip-of-nats-server>:4222
This method is robust and avoids the complexities and overhead of direct peer-to-peer connections.
You can find the nats specific code for adding to your existing deployment here:
https://github.com/RocketChat/rocketchat-compose/blob/main/compose.database.yml#L2-L25
For more details on our full recommended Docker Compose deployment, please refer to our official documentation.
2. Highly Recommended for Scale: Deploy with Helm Chart (Microservices)
For large-scale, high-performance deployments, we strongly recommend using our official Helm Chart to deploy Rocket.Chat in a microservices architecture. This mode splits different functionalities of the application into independent services, making it easier to scale specific parts of your deployment based on load.
You can find out more details about deploying our helm chart in our official documentation.
3. Alternative: Correctly Configure P2P TCP Transport
If you choose to continue using the default P2P TCP transport, you must ensure that every instance can reach every other instance directly. This is often a challenge in containerized environments.
Configuration:
Two environment variables are critical for this setup:
- INSTANCE_IP: By default, Rocket.Chat will register the container’s internal IP, which is not reachable from other hosts. You must set this to the host machine’s IP address that is accessible by other nodes in your cluster.
- TCP_PORT: By default, Rocket.Chat uses a random port. You must set this to a static port number for each instance. If running multiple instances on the same host, each must have a unique TCP_PORT.
Example docker-compose.yml snippet:
services:
rocketchat-1:
image: rocketchat/rocket.chat:latest
environment:
- INSTANCE_IP=192.168.1.10 # Host IP accessible by other nodes
- TCP_PORT=9001
ports:
- "3000:3000"
- "9001:9001" # Expose the TCP port to the host
rocketchat-2:
image: rocketchat/rocket.chat:latest
environment:
- INSTANCE_IP=192.168.1.11 # IP of the second host
- TCP_PORT=9001
ports:
- "3001:3001"
- "9002:9002"
Important Note: This P2P method creates a full mesh network where every instance connects to every other instance. This introduces significant overhead and is not recommended for large-scale deployments.
4. Temporary Fallback: Re-enable ChangeStreams
As a temporary measure, you can revert to the previous behavior by re-enabling database watchers.
Configuration:
Set the following environment variable:
DISABLE_DB_WATCHERS=false
WARNINGS:
- This option is a temporary fallback and will be removed in Rocket.Chat 8.0.
- This method only propagates persistent database changes (messages, settings). Ephemeral events like typing indicators and presence will not work unless a functional transport layer (NATS or correctly configured TCP) is also in place. Relying solely on this fallback will still result in a degraded user experience.
Further Recommendations
To ensure the health and performance of your Rocket.Chat deployment, we strongly recommend that all administrators implement a monitoring solution. Our official documentation now includes detailed instructions for deploying a full monitoring stack. Please revisit our docs to take advantage of these guides.