Oplog tailing in mongo shard cluster

Community Support
1

Description

I have shard enabled in my mongo cluster. Since rocketchat prefers mongo oplog tailing, it won’t be possible in a shard cluster environment.
Is there any alternative for mongo oplog tailing?
How can realtime pushing of data be achieved in a shard cluster environment

Server Setup Information

  • Version of Rocket.Chat Server: 3.1.0-develop
  • Operating System: Linux
  • Deployment Method: Kubernetes (docker)
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 12.14.1 - x64
  • MongoDB Version: 4.2.5
  • Proxy: Ingress
2

This might be my naive deployment but if you are running a sharded cluster with the mongos proxy, you could try using the config-server for the oplog tailing.

As an example, lets say your mongos database is called rocketchat. Your connection strings would be as follows:

MONGO_URL = mongodb://rocketuser:rocketpass@server1:27017,server2:27017,server3:27017/rocketchat?replSet=rs0

MONGO_OPLOG_URL = mongodb://oploguser:oplogpass@server1:27019,server2:27019,server3:27019/local?authSource=rocketchat&replSet=configset0

I have not validated this fully, however it has enabled me to deploy a rocket chat instance, so it might not work fully, but hopefully it’s a starting point

3

The problem with sharded clusters is the oplog is not available. As oplog is actually designed just for mongo replication. A workaround is to tail a specific replicaset in a shard. But then at that point there is no point in using sharing.

3.7.x introduces the utilization of change streams. In the near future the requirement for oplog tailing I think will go away if you have change streams support in your cluster.