Failed to start since night update 3.8.1

Community Support
1

Description

Not enough details, since this night update, rocket-chat not starting.
Sorry.

Server Setup Information

  • Version of Rocket.Chat Server: 3.8.1
  • Operating System: Ubuntu 18.04
  • Deployment Method: snap in LXD container
  • Number of Running Instances: 1
  • DB Replicaset Oplog: True
  • NodeJS Version: 3.6.0
  • MongoDB Version: 3.6.14
  • Proxy: haproxy
  • Firewalls involved: NO firewall

Any additional Information

sudo journalctl -f -u snap.rocketchat-server.rocketchat-server

Nov 21 09:36:13 rocketchat systemd[1]: snap.rocketchat-server.rocketchat-server.service: Failed to reset devices.list: Operation not permitted
Nov 21 09:36:13 rocketchat systemd[1]: Started Service for snap application rocketchat-server.rocketchat-server.
Nov 21 09:36:13 rocketchat rocketchat-server.rocketchat-server[1759]: Checking if oplog has been enabled, and enabling if not
Nov 21 09:36:13 rocketchat rocketchat-server.rocketchat-server[1759]: MongoDB shell version v3.6.14
Nov 21 09:36:13 rocketchat rocketchat-server.rocketchat-server[1759]: connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Nov 21 09:36:13 rocketchat rocketchat-server.rocketchat-server[1759]: Implicit session: session { "id" : UUID("db979fde-45ce-440e-9eb9-af69970e0454") }
Nov 21 09:36:13 rocketchat rocketchat-server.rocketchat-server[1759]: MongoDB server version: 3.6.14
Nov 21 09:36:13 rocketchat rocketchat-server.rocketchat-server[1759]: Checking if mongo featureCompatibilityVersion is correct, changing if not
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]: /snap/rocketchat-server/1444/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:218
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:       throw error;
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:       ^
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]: TypeError: Cannot read property 'split' of null
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at app/lib/server/lib/validateEmailDomain.js:17:31
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at app/settings/lib/settings.ts:80:37
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Array.forEach (<anonymous>)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at app/settings/lib/settings.ts:80:15
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Array.forEach (<anonymous>)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Settings.load (app/settings/lib/settings.ts:77:14)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Settings.storeSettingValue (app/settings/server/functions/settings.ts:342:8)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at app/settings/server/functions/settings.ts:365:9
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Array.forEach (<anonymous>)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Settings.init (app/settings/server/functions/settings.ts:364:32)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at module (app/lib/server/startup/settings.js:2908:10)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Module.require (packages/modules-runtime.js:238:14)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at Module.moduleLink [as link] (/snap/rocketchat-server/1444/programs/server/npm/node_modules/meteor/modules/node_modules/reify/lib/runtime/index.js:52:22)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at module (app/lib/server/index.js:1:153)
Nov 21 09:36:22 rocketchat rocketchat-server.rocketchat-server[1759]:     at fileEvaluate (packages/modules-runtime.js:336:7)
Nov 21 09:36:22 rocketchat systemd[1]: snap.rocketchat-server.rocketchat-server.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 09:36:22 rocketchat systemd[1]: snap.rocketchat-server.rocketchat-server.service: Failed with result 'exit-code'.

snap revert not solve problem.

root@rocketchat:~# sudo service snap.rocketchat-server.rocketchat-server status
● snap.rocketchat-server.rocketchat-server.service - Service for snap application rocketchat-server.rocketchat-server
   Loaded: loaded (/etc/systemd/system/snap.rocketchat-server.rocketchat-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sat 2020-11-21 09:49:31 UTC; 1s ago
  Process: 10177 ExecStart=/usr/bin/snap run rocketchat-server (code=exited, status=0/SUCCESS)
 Main PID: 10177 (code=exited, status=0/SUCCESS)

Nov 21 09:49:31 rocketchat systemd[1]: Started Service for snap application rocketchat-server.rocketchat-server.
root@rocketchat:~#

Help!
Thanks

2

I try to create another LXC container, and restore backup mongodb, same error.

And I cannot retore older backups mongodb. Each time it failed to restore.

root@rocketchat:/var/snap/rocketchat-server/common# cat /var/snap/rocketchat-server/common/restore/restore_20201121.1008.log
[*] Extracting backup file with "tar --no-same-owner --overwrite -xzvf /var/snap/rocketchat-server/common/rocketchat_backup_20201119.0200.tar.gz"

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

or
cat /var/snap/rocketchat-server/common/restore/restore_20201121.1007.log

dump/parties/rocketchat_statistics.bson
dump/parties/rocketchat_uploads.chunks.bson

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
3

That I tried to switch in “recovery mode”.

Sorry, it’s LXD solution.

  1. I created backup of mongodb of latest rocketchat release (3.8.1) and pull it on host of LXD
  2. I created a new LXD container with old snapshot of my Rocket-Chat container.
  3. I stopped rocketchat snap, restore DB from backup.
  4. Block auto update. $ sudo snap set system refresh.metered=hold

And It’s work. But I’m stuck in version 3.6.2.

If I try to update, RocketChat failed and back to initial failed problem. With same error.

I’m thinking it’s work to with install the sudo snap install rocketchat-server --candidate

Hope someone had solution to update.
Thanks

4

I did the following to get it running back with 3.6.2

  1. snap run --shell rocketchat-server
  2. snapctl set snap-refreshing=false
  3. sudo service snap.rocketchat-server.rocketchat-server stop
    3.1 Mongo DB needs to be up and running
  4. sudo snap revert rocketchat-server
  5. sudo cp /var/snap/rocketchat-server/common/backup/rocketchat_backup_20201121.0413.tar.gz /var/snap/rocketchat-server/common/rocketchat_backup.tgz
  6. snap run rocketchat-server.restoredb /var/snap/rocketchat-server/common/rocketchat_backup.tgz
  7. sudo systemctl restart snap.rocketchat-server.rocketchat-server
5

Hello Fred! I managed to solve my problem like this - https://github.com/RocketChat/Rocket.Chat/issues/19644 . I hope that the developers will take this issue into account in the next releases.