Cannot connect to Rocket Cloud when upgrading MongoDB from 4.4.16 to 5.0.11

Description

I’m trying to do a MongoDB upgrade, moving data between two different hosts.

Here’s the OLD host:

Version
4.8.2

Deployment ID
mzvqHRfBFP7fS6s7J

Apps Engine Version
1.32.0

Node Version
v14.18.3

Database Migration
265 (October 19, 2022 4:41 PM)

MongoDB
4.4.16 / wiredTiger (oplog Enabled)

Here’s the NEW host:

Version
4.8.2

Deployment ID
mzvqHRfBFP7fS6s7J

Apps Engine Version
1.32.0

Node Version
v14.18.3

Database Migration
265 (November 20, 2022 10:55 AM)

MongoDB
5.0.11 / wiredTiger (oplog Enabled)

The PROBLEM is that, on the new host, I’m getting this:

speranta-rocketchat-1  | {"level":50,"time":"2022-11-20T08:46:24.102Z","pid":1,"hostname":"58d49a6ac133","name":"System","err":{"type":"FetchError","message":"network timeout at: https://cloud.rocket.chat/api/oauth/token","stack":"FetchError: network timeout at: https://cloud.rocket.chat/api/oauth/token\n    at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/fetch/node_modules/node-fetch/lib/index.js:1437:13)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)","name":"FetchError"},"msg":"network timeout at: https://cloud.rocket.chat/api/oauth/token"}
speranta-rocketchat-1  | {"level":50,"time":"2022-11-20T08:46:24.280Z","pid":1,"hostname":"58d49a6ac133","name":"System","err":{"type":"FetchError","message":"network timeout at: https://cloud.rocket.chat/api/oauth/token","stack":"FetchError: network timeout at: https://cloud.rocket.chat/api/oauth/token\n    at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/fetch/node_modules/node-fetch/lib/index.js:1437:13)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)","name":"FetchError"},"msg":"network timeout at: https://cloud.rocket.chat/api/oauth/token"}
speranta-rocketchat-1  | {"level":50,"time":"2022-11-20T08:47:17.938Z","pid":1,"hostname":"58d49a6ac133","name":"System","err":{"type":"FetchError","message":"network timeout at: https://cloud.rocket.chat/api/oauth/revoke","stack":"FetchError: network timeout at: https://cloud.rocket.chat/api/oauth/revoke\n    at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/fetch/node_modules/node-fetch/lib/index.js:1437:13)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)","name":"FetchError"},"msg":"network timeout at: https://cloud.rocket.chat/api/oauth/revoke"}
speranta-rocketchat-1  | {"level":50,"time":"2022-11-20T08:48:33.716Z","pid":1,"hostname":"58d49a6ac133","name":"System","err":{"type":"FetchError","message":"network timeout at: https://cloud.rocket.chat/api/v2/register/workspace?resend=false","stack":"FetchError: network timeout at: https://cloud.rocket.chat/api/v2/register/workspace?resend=false\n    at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/fetch/node_modules/node-fetch/lib/index.js:1437:13)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)","name":"FetchError"},"msg":"network timeout at: https://cloud.rocket.chat/api/v2/register/workspace?resend=false"}

Please note that I don’t get any other connectivity errors, I can successfully connect via WebUI and Clients.

I tried, without success, to:

  • Click “Sync”: works on old, doesn’t works on new
  • Disconnect the instances from the Rocket Cloud and reconnect: works on old, doesn’t works on new
  • Remove the instance from Rocket Cloud, create a new Token, use the new Token: works on old, doesn’t works on new

Can you please let me know how can I fix/debug this? Thank you!

Server Setup Information

  • Operating System: Debian 11
  • Deployment Method: docker
  • Proxy: nginx
  • Firewalls involved: none

Docker .env

## Rocket.Chat Version https://github.com/RocketChat/Rocket.Chat/releases
RELEASE=4.8.2

## MongoDB
MONGO_URL=mongodb://speranta:PASS@172.17.0.1:27017/speranta?authSource=speranta
MONGO_OPLOG_URL=mongodb://speranta:PASS@172.17.0.1:27017/speranta?authSource=speranta

## Rocket
ROOT_URL=https://chat.sperantatv.ro

## Docker
# Port on the host to bind to
HOST_PORT=3002

Docker docker-compose.yml

version: "3.7"

services:
  rocketchat:
    image: registry.rocket.chat/rocketchat/rocket.chat:${RELEASE:-latest}
    restart: unless-stopped
    environment:
      MONGO_URL: "${MONGO_URL:-\
        mongodb://${MONGODB_ADVERTISED_HOSTNAME:-mongodb}:${MONGODB_INITIAL_PRIMARY_PORT_NUMBER:-27017}/\
          ${MONGODB_DATABASE:-rocketchat}?replicaSet=${MONGODB_REPLICA_SET_NAME:-rs0}}"
      MONGO_OPLOG_URL: "${MONGO_OPLOG_URL:\
        -mongodb://${MONGODB_ADVERTISED_HOSTNAME:-mongodb}:${MONGODB_INITIAL_PRIMARY_PORT_NUMBER:-27017}/\
          local?replicaSet=${MONGODB_REPLICA_SET_NAME:-rs0}}"
      ROOT_URL: ${ROOT_URL:-http://localhost:${HOST_PORT:-3000}}
      PORT: ${PORT:-3000}
    expose:
      - ${PORT:-3000}
    ports:
      - host_ip: ${BIND_IP:-0.0.0.0}
        target: ${PORT:-3000}
        published: ${HOST_PORT:-3000}
        protocol: tcp
        mode: host

NGINX speranta.conf

server {
        listen 80;
        server_name chat.sperantatv.ro;
        if ($host = chat.sperantatv.ro) { return 301 https://chat.sperantatv.ro$request_uri; }
}

server {
        listen 443 http2 ssl;
        server_name chat.sperantatv.ro;

        location / {
                proxy_pass http://127.0.0.1:3002;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Nginx-Proxy true;
                proxy_redirect off;
        }
        ssl_certificate /etc/letsencrypt/live/chat.sperantatv.ro/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/chat.sperantatv.ro/privkey.pem;  # managed by Certbot
}

When you upgraded your mongo did you grant the clusterMonitor role to the user?

We’re working on this being removed… but I’ve seen a lot of issues come about because this role wasn’t added. Causing the change stream watching to not work properly.

This mentions the role needed