Description
Server Setup Information
- Version of Rocket.Chat Server: 2.4.12
- Operating System: Ubuntu 18.04
- Deployment Method: snap
- Number of Running Instances: 2
- DB Replicaset Oplog:
- NodeJS Version:
- MongoDB Version: 3.
- Proxy: nginx
- Firewalls involved:
Any additional Information
Our rocket.chat snaps auto-refreshed from 2.4.11 to 2.4.12 today automatically on two different hosts at different times. When they refreshed, mongodb would not come up - the database complained that it only supported major/minor versions of up to 1/0, but the existing files were 2/0…
May 12 11:02:41 rocket rocketchat-server.rocketchat-mongo[23722]: 2020-05-12T11:02:41.266-0700 E STORAGE [initandlisten] WiredTiger error (-31802) [1589306561:266627][23761:0x7fe1d1c3dd00], txn-recover: unsupported WiredTiger file version: this build only supports major/minor versions up to 1/0, and the file is version 2/0: WT_ERROR: non-specific WiredTiger error
May 12 11:02:41 rocket rocketchat-server.rocketchat-mongo[23722]: 2020-05-12T11:02:41.266-0700 E STORAGE [initandlisten] WiredTiger error (0) [1589306561:266759][23761:0x7fe1d1c3dd00], txn-recover: WiredTiger is unable to read the recovery log.
A snap revert allows the database to restart, but then the the snap.rocketchat-server.rocketchat-server died on restart with no errors in the systemctl logs (logs say it starts successfully, but its status is dead).
It seems like the 2.4.11 snap is running with Mongodb 3.6.14, but the 2.4.12 snap installed MongoDb 3.4.20? To resolve the issue, I was forced to restore an old VM image of the server, disable network lookups of api.snapcraft.io (to prevent it from refreshing when the vm boots), and then restore the database from backup.
I currently have forced snaps to not update to prevent it from breaking our production instances, but I can re-apply the refresh on our test instance if necessary. The problem occurred on both our production and test instances within 30 minutes of each other.