Rocket Chat does not start after 6.3.5 update (from 6.3.0)

I updated my docker install from 6.3.0 directly to 6.3.5 (latest), but that broke the whole thing. Trying to go back to 6.3.0 (or 6.3.1) did not resolve it.

As far as I can tell it’s outputing the exact same output as a clean install up until the last part, just not coming to the point of showing (which should be next):

+---------------------------------------------------+
|                   SERVER RUNNING                  |
+---------------------------------------------------+
|                                                   |
|  Rocket.Chat Version: 6.3.5                       |
|       NodeJS Version: 14.21.3 - x64               |
|      MongoDB Version: 5.0.21                      |
|       MongoDB Engine: wiredTiger                  |
|             Platform: linux                       |
|         Process Port: 3000                        |
|             Site URL: https://chat.domain.tld     |
|     ReplicaSet OpLog: Enabled                     |
|          Commit Hash: 5c9c3f7864                  |
|        Commit Branch: HEAD                        |
|                                                   |
+---------------------------------------------------+

(Gotton from a clean install)

Tried restoring the backup dump, but unfortunately that also gives the exact same error. Importing it “over” a default install messes up the install since it does not overwrite existing/duplicate keys (and doing that would probably make it go back to how it was).

I’m really stuck, so any tips is hugely appreciated!

Steps to reproduce:

# docker compose pull (from 6.3.0 to 6.3.5)
# docker compose up -d

Expected behavior:

Should upgrade and work as normal.

Actual behavior:

It halts in the startup process, don’t see the “running-badge” running. But no super obvious huge error?
Usually the “running-badge” shows up right after where it halts.

See logs below

Server Setup Information:

  • Version of Rocket.Chat Server: 6.3.5
  • Operating System: Linux Debian
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: not sure?
  • NodeJS Version: v14.21.3
  • MongoDB Version: v5.0.21

Client Setup Information

  • Desktop App or Browser Version: Tested both browser, app and mobile app.
  • Operating System: macos, linux and ios.

Relevant logs:

Testing with curl @ domain:

➜  rocketchat curl -vvvv chat.domain.tld
*   Trying [ipv6address]:80...
* Connected to chat.domain.tld (ipv6address) port 80 (#0)
> GET / HTTP/1.1
> Host: chat.domain.tld
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://chat.domain.tld/
< Server: Caddy
< Date: Thu, 14 Sep 2023 19:59:21 GMT
< Content-Length: 0
<
* Closing connection 0

Testing with curl @ localhost:

➜  rocketchat curl -vvvv localhost:3000
*   Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.88.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

➜ rocketchat docker logs rocketchat

strict mode: missing type "object" for keyword "additionalProperties" at "#" (strictTypes)
strict mode: missing type "object" for keyword "properties" at "#" (strictTypes)
strict mode: use allowUnionTypes to allow union type keyword at "#/properties/value" (strictTypes)
(node:9) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
MessageType.render is deprecated. Use MessageType.message instead. livechat_webrtc_video_call
{"level":40,"time":"2023-09-14T19:49:51.136Z","pid":9,"hostname":"ecdb202e7feb","name":"VoIPService","msg":"Voip is not enabled. Cant start the service"}
{"level":51,"time":"2023-09-14T19:49:51.789Z","pid":9,"hostname":"ecdb202e7feb","name":"Migrations","msg":"Not migrating, already at version 301"}
[2023-09-14T19:49:51.820Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Moleculer v0.14.29 is starting...
[2023-09-14T19:49:51.820Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Namespace: <not defined>
[2023-09-14T19:49:51.820Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Node ID: feeb6de5-e94a-4a16-aefb-5def8891fd1e
[2023-09-14T19:49:51.821Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/REGISTRY: Strategy: RoundRobinStrategy
[2023-09-14T19:49:51.821Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/REGISTRY: Discoverer: LocalDiscoverer
[2023-09-14T19:49:51.822Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Serializer: JSONSerializer
[2023-09-14T19:49:51.827Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Validator: FastestValidator
[2023-09-14T19:49:51.828Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Registered 13 middleware(s).
[2023-09-14T19:49:51.829Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: Transporter: TcpTransporter
{"level":51,"time":"2023-09-14T19:49:51.846Z","pid":9,"hostname":"ecdb202e7feb","name":"DatabaseWatcher","msg":"Using change streams"}
[2023-09-14T19:49:51.923Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/TRANSIT: Connecting to the transporter...
[2023-09-14T19:49:51.926Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/TRANSPORTER: TCP server is listening on port 43297
[2023-09-14T19:49:51.927Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/TRANSPORTER: UDP Discovery is disabled.
[2023-09-14T19:49:51.927Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/TRANSPORTER: TCP Transporter started.
ufs: temp directory created at "/tmp/ufs"
(node:9) [MONGODB DRIVER] Warning: cursor.count is deprecated and will be removed in the next major version, please use `collection.estimatedDocumentCount` or `collection.countDocuments` instead
[2023-09-14T19:49:52.436Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/REGISTRY: '$node' service is registered.
[2023-09-14T19:49:52.437Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/REGISTRY: 'matrix' service is registered.
[2023-09-14T19:49:52.437Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/$NODE: Service '$node' started.
[2023-09-14T19:49:52.437Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/MATRIX: Service 'matrix' started.
[2023-09-14T19:49:52.438Z] INFO  feeb6de5-e94a-4a16-aefb-5def8891fd1e/BROKER: ✔ ServiceBroker with 2 service(s) started successfully in 597ms.

➜ rocketchat docker logs rocketmongo

{"t":{"$date":"2023-09-14T21:49:42.210+02:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-09-14T21:49:42.211+02:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true
}}}
(… complete log is on github …)
{"t":{"$date":"2023-09-14T21:49:54.863+02:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn18","msg":"client metadata","attr":{"remote":"172.20.0.4:40530","client":"conn18","doc":{"driver":{"name":"nodejs","version":"4.16.0"},"platform":"Node.js v14.21.3, LE","os":{"name":"linux","architecture":"x64","version":"6.1.0-12-amd64","type":"Linux"}}}}
{"t":{"$date":"2023-09-14T21:50:42.664+02:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1694721042:664249][1:0x7f918e76f700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 52, snapshot max: 52 snapshot count: 0, oldest timestamp: (1694720742, 1) , meta che
ckpoint timestamp: (1694721042, 1) base write gen: 14809349"}}

(I also have an issue on github, but posting here for more exposure)

I’m sorry for not being able to offer any further assistance, but I have my eye on this issue before I consider upgrading from 6.3.x.

According to the docs, downgrades are not officially supported: Database Migration - Rocket.Chat Docs

This is not a downgrade, so that should not influence anything. And I could not find any info on this or a similar issue anywhere, and I was stuck with no apparent error.

Instead, I migrated away from RocketChat to Mattermost, which was not super convenient and a bit time-consuming, but I had all the data and succeeded in the end.