Rocketchat quits (or maybe not?) silently after update - Instance not reacheable

Community Support
1

Description

Hi Community!
I encountered an interesting problem while updating my RC 3.10.0 docker (on Synology) instance.

My up and running setup:

  • Version of Rocket.Chat Server: 3.10.0
  • Operating System: DSM7
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: yup
  • NodeJS Version: 12.18.4
  • MongoDB Version: 4.4
  • Proxy: nginx (i guess)
  • Firewalls involved: nope
    This setup is running happily since installed!

Since my RC server version is fairly outdated i decided to upgrade step by step over major versions to :latest.

How i upgrade:

  1. Fetch the image
  2. Create a new container providing the following variables
  • PATH
  • NODE_ENV
  • NOVE_VERSION
  • RC_VERSION
  • DEPLOY_METHOD
  • MONGO_URL
  • HOME
  • PORT
  • ROOT_URL
  • Accounts_AvatarStorePath
  • MONGO_OPLOG_URL
  1. Link the container to a network bridge (assign 4040 internal, and auto external)
  2. Link the container to the container hosting the mongo db
  3. Start the RC container

The issue:
No matter to which version i’d like to upgrade, i start my newly created container and it keeps running. What’s puzzeling me is that is doesn’t log a single line. When i try to reach my server (i do this by accessing the RC instance via browser and ip/port) the response is: The connection was reset

So i can’t access my (maybe) running server (ERR_CONN_RESET) and have no way of tracking down the issue.
Out of desperation i provided wrong DB credentials - those errors are logged, which leaves me with two assumptions:

  1. Server running and totally fine - issue is maybe network/ssl/reverse proxy related. But then again - why i don’t see the “server up and running” log i know from my running instance?
  2. Server hangs up during boot/quits silently

Has someone an idea how to solve this issue?
Thanks in advance!

2

Hi!

My suggestion is:

  • Deploy the very same version you have using our [official docker compose] by changing the mage tag (Docker & Docker Compose - Rocket.Chat Docs)
  • Dump production data, and restore it there.
  • Start upgrading, and do not skip majors. 3.x, 4.x, 5.x
  • Test your integrations, etc.
  • Fix/mitigate eventual issues

Now you plan and execute the migration :slight_smile:

let me know if this helps :slight_smile:

1 Like
3

Hi!
Thanks for your suggestions and your reply.
My solution was to backup my old DB, create a new DB with higher MongoDB version and restore the backup within the new DB. Then i upgraded my RocketChat in small steps.

In detail (for anyone facing similar problems):

  • Shutdown your RocketChat container
  • Backup the MongoDB with mongodump. You can access the terminal of your Docker image inside the Docker app for DSM
  • Copy your backup to another location on your NAS (outside the Mongo container)
  • Shutdown your old MongoDB
  • Fetch a Mongo Docker image below v5.0. Since Mongo 5.0 requires CPU AVX support (this is a hardware requirement and most probably your NAS CPU doesn’t meet it). I chose 4.4.9
  • Configure a new container for your new MongoDB with according variables
  • Follow these instructions for creating a replica set.
  • Give your new MongoDB container access to your backup
  • Restore your backup into your new MongoDB container using mongorestore in your container terminal.
  • Spin up your DB

Now you are ready to upgrade in major RocketChat version steps. Create a new Docker container for every major version you need to upgrade. I got migration errors (you will see them in your Docker log in the DSM Docker app). You can solve them by upgrading in smaller versions. E.g.: you get a migration error upgrading from 4.0 to 5.0, do an upgrade from 4.0 to 4.5. If you still get an migration error, do even a smaller upgrade step.

I was able to get my RocketChat back up and running with the latest version on MongoDB 4.4.9. without data loss.

This solved my problem, but doesn’t provide an explanation why my previous version was running without logging anything and beeing not responsive. All after all this is not imporant any more :grinning:

:warning: Heads up devs :warning:
MongoDB 4.4 EOL is Feb. 24. This is the last version without AVX support needed. In my usecase (self hosted RocketChat and DB on Synology NAS) i can’t install a newer version of MongoDB on my device. I did a quick research of Synology CPU’s and AVX support and i came to the conclusion that no Diskstation supports AVX today. With the EOL of MongoDB 4.4 todays Synology NAS can’t be used to host RocketChat anymore.

1 Like