Migrate self hosted server from 3.18 snap to 5.0 docker based installation

Community Support
, ,
1

Description

We want to move the production environment from snap to docker deployment. I have backuped the snap deployment as stated in Snap Backup and Restore - Rocket.Chat Docs
I then followed the guide [HOW TO] Migrate from Snap to Manual installed RocketChat
I did the mongo commands from inside the container. Restore went through without any errors but when I get to the page I am getting a 502 Bad Gateway error. I assume it has to do something with our configured environment:

nginx proxy is doing the ssl termination and rocketchat server is doing plain http on port 3000
The old snap server is having a different domain name than the new docker server. I think the old configuration does not fit to the new server. Is there any way to just backup/restore user data (rooms, channels, chat history) without server configuration?

Server Setup Information

  • Version of Rocket.Chat Server:
    → old snap server: 3.18.7
    → new docker server: 5.0.4
  • Operating System: Ubuntu 20.04
  • Deployment Method: migration from snap to docker deployment
  • Number of Running Instances: 1 snap server, 1 docker server
  • MongoDB Version:
    → on the snap server: 3.6.14
    → on the docker server: 4.4.15
  • Proxy: nginx ssl

Any additional Information

rocketchat log:

{"level":40,"time":"2022-08-25T06:54:00.423Z","pid":1,"hostname":"a472d4be0973","name":"Migrations","msg":"Not migrating, control is locked. Attempt 26/30. Trying again in 10 seconds."}
/app/bundle/programs/server/node_modules/fibers/future.js:280
						throw(ex);
						^
Error: Can't find migration version 232
    at migrateDatabase (server/lib/migrations.ts:249:9)
Some indexes for collection 'rocketchat_integration_history' could not be created:
	Index with name: _updatedAt_1 already exists with different options
Some indexes for collection 'rocketchat_nps_vote' could not be created:
	Index with name: npsId_1_identifier_1 already exists with different options
LocalStore: cannot create store at /var/snap/rocketchat-server/common/uploads (EACCES: permission denied, mkdir '/var/snap')
{"level":40,"time":"2022-08-25T06:55:03.853Z","pid":1,"hostname":"a472d4be0973","name":"Migrations","msg":"Not migrating, control is locked. Attempt 2/30. Trying again in 10 seconds."}
2

Hi!

you will need to run a docker version with 3.18.7.
Then you will restore mongo db from snap to the same version in a docker environment.

While upgrading, don’t jump majors. Go first to 4.8.4, then to latest 5.0.4

Let me know if this helps.

Thanks!

3

Hi,

Thanks for your answer and your time. Following tests failed:

Set up container with rocketchat 3.8.17 and mongo 4.4.15 > Setup wizard page successfull > restoring backup successfull > Error 502 Bad Gateway

Same behaviour with rc 3.8.17 and mongo 3.6.23

Logfile in rocketchat throws error:
LocalStore: cannot create store at /var/snap/rocketchat-server/common/uploads (EACCES: permission denied, mkdir '/var/snap')
Exception in callback of async function: Error: EACCES: permission denied, mkdir '/var/snap'

So, yes, that path is used for file uploads on the snap server. But obviously it is not existing in the docker container.

4

Ok, the problem here is that you probably has the file system for uploads set as System File, pointing to /var/snap/rocketchat-server/common/uploads

Usually it uses GridFS, that stores files inside Mongodb itself.

So, you should mount a folder or volume inside Rocket.Chat on this very same path.

Something like

volumes:
  - /home/rocketchat/uploads:/var/snap/rocketchat-server/common/uploads

/home/rocketchat/uploads will be a folder on the server you should create (or any other path).

You should also copy the files from the snap deployment to there.

Let me know if this helps.

5

That hint was good. I am a little bit further now.

I had to copy the 3 folders “uploads”, “user-sounds” and “user-emojis” and added the volumes entry in the compose.yml:

volumes:
      - /var/lib/docker/volumes/rocketchat_common:/var/snap/rocketchat-server/common

Now, the setup wizard is starting, when calling the site. The problem now is, that registration is disabled (set by the former configuration, because we use ldap as authentication mechanism) and I am not getting past that setup wizard.

server.js:204 System âž” error Exception while invoking method registerUser User registration is disabled [error-user-registration-disabled]

I am wondering as well why there is still the setup wizard…

Further log errors while starting the container:

LocalStore: cannot set store permissions 0744 (EPERM: operation not permitted, chmod '/var/snap/rocketchat-server/common/uploads')
6

if using a docker volume, you should not point it to the path directly.

You can create docker volumes inside your docker-compose.yml and reference it.

and inside the service

volumes:
  rocketcat_uploads: {}

services:
   rocketchat:
      .....
     volumes:
       - volumes:
          rocketcat_uploads:/var/snap/rocketchat-server/common

For the registration, you can mark it as completed adding this env variable:

OVERWRITE_SETTING_Show_Setup_Wizard=completed

Let me know if this helps.

7

Thanks for the hint with the volume. I have changed that and added the variable to .env file as well. But it is still pointing to the setup wizard.
The error in rocketchat container is as well still throwing:

LocalStore: cannot set store permissions 0744 (EPERM: operation not permitted, chmod '/var/snap/rocketchat-server/common/uploads')

A further error is:

=== UnHandledPromiseRejection ===
Error: connect ECONNREFUSED 10.3.8.2:27017
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
    at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:126:14)
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at app/models/server/raw/NotificationQueue.ts:60:15
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at app/notification-queue/server/NotificationQueue.ts:55:21
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
  name: 'MongoNetworkError'
}
---------------------------------
Errors like this can cause oplog processing errors.
8

Strange, after some hours of waiting, it is now working as expected. What is the recommandation for upgrading now?

Should I first upgrade mongodb 3.x to 4.x and after that rocketchat 3.x to 4.x to 5.x?

Edit:
I have upgraded to rocketchat 4.8.4 with mongodb 3.6.23 and this works but warns about deprecated mongo version.
If I now upgrade to mongo 4.4 the mongodb container hangs in a restart loop and throws the error:

{"t":{"$date":"2022-08-29T12:31:47.496+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2022-08-29T12:31:47.497+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
{"t":{"$date":"2022-08-29T12:31:47.497+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
{"t":{"$date":"2022-08-29T12:31:47.497+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
9

And finally I could upgrade mongodb and rocketchat. Here is the order and steps what to do:

Coming from rocketchat 3.8.x and mongodb 3.6.x
Step 1: Upgrade to rocketchat 4.8.x with mongodb 3.6.1
Step 2: Upgrade mongodb to 4.0.x
Step 3: Set the featureCompatibilityVersion in mongodb from 3.6 to 4.0
→ https://www.mongodb.com/docs/manual/release-notes/4.0-upgrade-standalone/
Step 4: Upgrade mongodb to 4.2.x
Step 5: Set the featureCompatibilityVersion in mongodb from 4.0 to 4.2
→ https://www.mongodb.com/docs/manual/release-notes/4.2-upgrade-standalone/
Step 6: Upgrade mongodb to 4.4.x
Step 7: Set the featureCompatibilityVersion in mongodb from 4.2. to 4.4
Step 8: Upgrade rocketchat 4.8.x to 5.0.x

Only one warning is left in the logs:

LocalStore: cannot set store permissions 0744 (EPERM: operation not permitted, chmod '/var/snap/rocketchat-server/common/uploads')

But for the moment I do not see any implications what this may cause.

10

Hi!

Can you check this post? It may be related:

11

Yes, that was the problem. See my post before. I had to upgrade step by step and modify the featureCompatibilityVersion in each step.

It is now working, thanks a lot for your help. One log entry is still complaining about store permissions. It seems that a chmod 0744 is not possible on the new created /var/snap path. I do not see if it has any impact on the migration. For the moment it seems, that everything is working as expected.

1 Like
12

Try using a mounted docker volumes on a new path, and change the upload path setting to it.

Glad everything worked out!

Please, Consider joining our Community Support Channel to help others and attending our Upcoming Community Open Call

Thanks!

13

im using 3.8.17, im already upgrade mongodb to 5.11, everything is fine.
and tried to upgrade to 4.8.4, many errors appear, and database is corrupted

Error creating indexes for integration_history MongoError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { _updatedAt: 1 }, name: "_updatedAt_1", expireAfterSeconds: 2592000 }, existing index: { v: 2, key: { _updatedAt: 1 }, name: "_updatedAt_1" }
    at MessageStream.messageHandler (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:272:20)


Error creating indexes for nps_vote MongoError: An existing index has the same name as the requested index. When index names are not specified, they are auto generated and can cause conflicts. Please refer to our documentation. Requested index: { v: 2, unique: true, key: { npsId: 1, identifier: 1 }, name: "npsId_1_identifier_1" }, existing index: { v: 2, key: { npsId: 1, identifier: 1 }, name: "npsId_1_identifier_1" }
    at MessageStream.messageHandler (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:272:20)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
    at MessageStream._write (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
    at writeOrBuffer (internal/streams/writable.js:358:12)

{"level":51,"time":"2022-09-07T07:27:13.215Z","pid":7298,"hostname":"ica-app-0","name":"Migrations","msg":"Running up() on version 237"}
{"level":51,"time":"2022-09-07T07:27:13.374Z","pid":7298,"hostname":"ica-app-0","name":"Migrations","msg":"Running up() on version 238"}
Error creating index: apps_logs -> { _updatedAt: 1 } { expireAfterSeconds: 2592000 } MongoError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { _updatedAt: 1 }, name: "_updatedAt_1", expireAfterSeconds: 2592000 }, existing index: { v: 2, key: { _updatedAt: 1 }, name: "_updatedAt_1" }
    at MessageStream.messageHandler (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:272:20)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
    at MessageStream._write (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:42:5)


{"level":51,"time":"2022-09-07T07:27:14.110Z","pid":7298,"hostname":"ica-app-0","name":"Migrations","msg":"Running up() on version 246"}
Error occurred:  Error: Timed out while authenticating with server
    at Timeout._onTimeout (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Connection.js:139:17)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  source: 'timeout-auth'
}
meteor://đź’»app/app/app.js:208750
      throw err;
      ^


{"level":51,"time":"2022-09-07T07:31:31.947Z","pid":8796,"hostname":"ica-app-0","name":"Migrations","msg":"Running up() on version 260"}
Error occurred:  Error: Timed out while authenticating with server
    at Timeout._onTimeout (/opt/Rocket.Chat/programs/server/npm/node_modules/imap/lib/Connection.js:139:17)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  source: 'timeout-auth'
}
meteor://đź’»app/app/app.js:208750
      throw err;
14

I came from version 3.18.7 with mongodb 3.6 and had first updated to rocketchat 4.8 and after that upgraded the mongo db in several steps to 4.4. I don’t know what is causing your error beside the one message that states

Error occurred:  Error: Timed out while authenticating with server

But your order of upgrade steps is different from mine which finally worked for me.