wiredTiger migration hasn't started yet. Waiting 30 secs

Community Support
1

Description

Server Setup Information

  • Version of Rocket.Chat Server: 4.0.0
  • Operating System: Debian
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: ?
  • NodeJS Version: ?
  • MongoDB Version: 4.0.25
  • Proxy: traefik
  • Firewalls involved: pfSense

Any additional Information

I am trying to upgrade my MongoDB to wiredTiger (https://docs.rocket.chat/quick-start/installing-and-updating/docker-containers/mongodb-mmap-to-wiredtiger-migration) and followed all the steps through to the end.

But there I seem to be stuck: “wiredTiger migration hasn’t started yet. Waiting 30 secs…” has been repeating for the last 30 minutes or so.

What can I do?
Thanks a lot!

2

Hi!

Sorry for the late response. Were you able to solve this?

Please, do a backups and always do the steps first on a staging environment.

3

Hi!

Thanks for getting back to me.

Unfortunately, I was not able to resolve (in a conventional sense of the word).

I tried a couple of times, repulled the images but to no avail.

So my solution was to reinstall (which was okay because it was a test installation anyway without any meaningful data).

But since I have your attention: I was surprised to see that the new installation (with the latest docker-compose file from your website) did again end up without wiredTiger. So I will probably need to go through the same updating exercise again at some probably not so distant time in the future.

So - what do I have to install in order to get a version running wiredTiger?

And for future reference - which files need to be backed up in order to restore a site to a new installation (i.e. had I had lots of chats etc. in my old installation, how could I have restored them after the new installation)?

Thanks a lot!

4

This docker was supposed to be changed already. indeed. :grimacing:

For using with wiredtiger, on a new installation, you need only to change the mongo services, like so:

  mongo:
    image: mongo:4.4
    restart: unless-stopped
    volumes:
     - ./data/rocketchat_db:/data/db
     - ./data/rocketchat_dump/dump:/dump
    command: mongod --oplogSize 128 --replSet rs0 --storageEngine=wiredTiger
    labels:
      - "traefik.enable=false"

  # this container's job is just run the command to initialize the replica set.
  # it will run the command and remove himself (it will not stay running)
  mongo-init-replica:
    image: mongo:4.4
    command: >
      bash -c
        "for i in `seq 1 30`; do
          mongo mongo/rocketchat --eval \"
            rs.initiate({
              _id: 'rs0',
              members: [ { _id: 0, host: 'localhost:27017' } ]})\" &&
          s=$$? && break || s=$$?;
          echo \"Tried $$i times. Waiting 5 secs...\";
          sleep 5;
        done; (exit $$s)"
    depends_on:
      - mongo

We are discussing about a better docker-compose experience and hopefully will have a specific repo for it. Hang tight.

5

Perfect - thanks! That worked like a charme.

This morning I already got a new notification that another update is available - but I guess that updating is going to be a lot easier again going forward…

It is my understanding that I can simply amend my docker-compose file to specify the image I want to pull. I did read in some other support thread that it is not a good idea to pull “latest” but rather a specific version (I take it that I should not specify the latest version by version number as well, right? In other words: the problem ist not using the “latest” tag but the actual latest version.)

This should leave my data (if I had any) untouched.

But coming back to my other question: What data should I backup in order to restore it later on and to a new version of rocket.chat? (I do a full site backup but restoring that will also restore the old version of rocket.chat as well. What, if I want to restore only the data to a new version of rocket.chat?)

Thanks for your help!

6

That’s the best practice.

You, as a SYS ADMIN, should always be in control of what versions you run. leaving the word “latest”, will just pull the latest version, and may upgrade without your knowledge, on a server reboot, for example.

so please, always change latest (an alias to the most recent version) to the exact latest version release tag number (as of now, 4.3.2)

Whenever you want to migrate, the suggestion is to:

  • do a backup
  • spin up an staging environment with the exact same version
  • restore the backup and files when applicable
  • change the version to the latest tag version (as of now 4.3.2)
  • pull the image (docker-compose pull)
  • update the docker stack (docker-compose up -d)

if everything went well, go ahead, backup again and do the upgrade in production.

according to your second question, it will depend.

If you keep uploads in GridFS, it means your files will be store at DB. On this case, just backup up the mongoDB will be enough.

However, if you keep uploads in System Storage (a path in your server), you should backup and restore both the mongo and this path, and make it available where Rocket.Chat expects it to be, with the proper permissions at the restored server.

Let me know if you have any other doubts, and please, consider showing up at our next community call :slight_smile:

https://app.livestorm.co/rocket-chat/community-open-call?type=detailed&utm_source=COMMUNITY

7

Hi, I would like to piggyback on this thread. We are running rocket chat and I am planning the migration to Mongo4.2.

I am starting the migration as mentioned in the post but it doesn’t go through. Its showing the same message for a few hours:

migrator_1            | 2022-04-02T01:40:06.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:09.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:12.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:15.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:18.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:21.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:24.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:27.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:30.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:33.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:36.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:39.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:42.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:45.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:48.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:51.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:54.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:40:57.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:00.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:03.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:06.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:09.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:12.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:15.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:18.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:21.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:24.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:27.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:30.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:41:33.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)

I am using Mongo 4.0 and RocketChat v3.0.6 on docker with Docker Compose. Any Idea?

EDIT 1 *** It looks like its going forward but very slow.

igrator_1            | 2022-04-02T01:57:21.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:57:24.424+0000    [........................]  rocketchat._raix_push_notifications  101/69016245  (0.0%)
migrator_1            | 2022-04-02T01:57:27.424+0000    [........................]  rocketchat._raix_push_notifications  23728/69016245  (0.0%)
migrator_1            | 2022-04-02T01:57:30.424+0000    [........................]  rocketchat._raix_push_notifications  23728/69016245  (0.0%)
migrator_1            | 2022-04-02T01:57:33.424+0000    [........................]  rocketchat._raix_push_notifications  23728/69016245  (0.0%)

Why are there 67 Million Push notifications in the database?

EDIT 2 *** I further checked the configuration and PUSH is turned off. Any Idea why there are push notifications all with sent: false