Rocket chat via snap has error on restart of the application

I installed the server via snap on debian. That went fine and I went through the wizzard without issues. The issue is if you stop the server and then try to restart it you get a error saying you are trying to send a http request to a https server. I never enabled https and it was working fine via http before stopping it and restarting. I had to stop it because I want to take a snap snapshot to see how that went and if it’s running snap gives a error it can’t take a snapshot of the mongo db cause it changed during the process. Stopping it via “snap stop rocketchat-server” and then “snap save rocketchat-server” creates the snap. But, like I said restarting it gives the error above. Even stopping and restarting the service via systemctl does not work even though the status shows it restarted as you can see below. The only way to get it back is to reboot. Also, another issue is after rebooting it says I’m accessing it from localhost instead of the domain name that I setup during the install and that was working to. You can see below the data base seems to think it’s localhost so that does not see to be getting saved despite the UI saying it was. First, I want to have snapshots so what’s the best way and sequence to do this. Second, how stable is rocketchat, how stable are the updates? How stable is it on snap and is the new revisions checked in a lab before pushing it to the snap repo because that auto updates. I just read a thread here from a dev that seems concerned about the quality of code and updates.

systemctl status snap.rocketchat-server.rocketchat-server.service

● snap.rocketchat-server.rocketchat-server.service - Service for snap application rocketchat-server.rocketchat-server
Loaded: loaded (/etc/systemd/system/snap.rocketchat-server.rocketchat-server.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-06-08 19:08:55 CDT; 30s ago
Main PID: 752 (start_rocketcha)
Tasks: 12 (limit: 1111)
Memory: 619.8M
CPU: 27.156s
CGroup: /system.slice/snap.rocketchat-server.rocketchat-server.service
├─752 /bin/bash /snap/rocketchat-server/1516/bin/start_rocketchat.sh
└─800 node /snap/rocketchat-server/1516/main.js

Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | MongoDB Version: 4.2.17 |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | MongoDB Engine: wiredTiger |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | Platform: linux |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | Process Port: 3000 |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | Site URL: http://localhost:3000 |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | ReplicaSet OpLog: Enabled |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | Commit Hash: 596881ec9a |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | Commit Branch: HEAD |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: | |
Jun 08 19:09:19 Web-1 rocketchat-server.rocketchat-server[800]: ±---------------------------------------------+

Version

4.6.3

Deployment ID

Kq9sdeA3J8c5MQaSf

Apps Engine Version

1.31.0

Node Version

v14.18.3

Database Migration

258 (June 8, 2022 7:09 PM)

MongoDB

4.2.17 / wiredTiger (oplog Enabled)

Hi!

Maybe your URL is not set correctly?

try this:
sudo snap set rocketchat-server siteurl=http://XXX.XXX.XXX

being XXX.XXX.XXX your ip and assuming you are not using a domain, only an IP without SSL.

Also, I strongly suggest using docker instead of snap. It will give you way more control over your deployment.

Thanks!

So, I tired the docker install. I’m not trying to be rude here or insulting but what a shit show. While I got it up and working from your own docker-compose file in the docs/install section I was disappointed to find once I set it up I was notified on the dashboard/UI that the mongo db was deprecated. It sure was…several major revisions behind. I thought ok, I’ll just see if there’s a update db in the settings…nope. The error had a link to your rocketchat releases and db versions where I found out you can’t just update mongo to the latest supported…you have to go one by one. Then I found a user here that posted the same issue and a lengthy video on how he corrected it. It’s way too convoluted! He had to move the entire install to /opt and use a migration tool and go through a process of updating shutting down, updating shutting down over and over until he got to the right version. That’s what I mean by a shit show. The docker file ,your own, should have been on point from the start! Using docker is suppose to make things easier to install and keep updated… but in the case due to what seems as a lack of oversight on the dev team it was not. If this had been a deploy in a prod environment it would have caused major issues with downtime and management would have been looking at alternatives. It seems like the mongo database is a sore subject in general and especially with rocketchat updates working or failing. Since rocketchat is in the space of a professional communications tool, which communications is vital, this is a major downfall of what appears to be a superb product. Things should work right out of the box regardless of the install methods, updates should be flawless and easy; rollbacks should be easy which is why I was looking at snap since you can take snapshots. For example, zulip is easy to upgrade and the upgrading tool has a built in db migration tool that handles it all for you. They lack e2e which is why I was looking at rocketchat. Again, this isn’t meant to be rude or insulting but rather me sharing my end user experience. I hope this all improves. Here is a user here that solved this and as you can see it’s fare from ideal of what a end user should experience.

Upgrade Deprecated Mongo 4.0.28 - #2 by vmsman

Neal, Your exact experience is what led me to creating the video that I linked. Open source software can be a dream to configure and maintain or it can be a chore and then some. In particular, my history with MongoDB on other products has been a torrid experience as well. The longer you self-host, the more components you will find deprecated. Databases are the most common and then PHP would be the next most common. In general, Docker containers simplify the installation and upgrade process since they are simply statically replaced. The problem is real software systems have external data and the management of that data relies on tools that change. One bright point in the universe is that when you run open source software, you are in charge of when and if you perform upgrades vs. corporate cloud software where upgrades are “done” to you with little or no warning and the results are often catastrophic. That’s one reason why corporate enterprises use Microsoft Windows Server update Services (WSUS). Hopefully my video helped soften the blow of this particular update.

1 Like

Neal, an additional that I did. I implemented my Dockerized RocketChat inside of a LXD container. That way I can easily snapshot and/or export the container even on an automated schedule if I desire.

Hi @neal !

Thanks for the input, we really appreciate it.

But bear in mind, some of that is a necessary change requested by MongoDB stack, as it will no longer support the mmap file storage, and wiredTiger will be the default one.

According to our compatibility Matrix, all those versions are still supported, and will continue to be supported.

Indeed our docker-compose could be already updated for latest mongo versions, avoiding the deprecation message, and the need of a future migration from mmap do wiredTiger.

Here is the docker-compose.yml contents already adapted to use latest 4.4 version and wiredTiger:

version: '2'

services:
  rocketchat:
    image: registry.rocket.chat/rocketchat/rocket.chat:latest
    command: >
      bash -c
        "for i in `seq 1 30`; do
          node main.js &&
          s=$$? && break || s=$$?;
          echo \"Tried $$i times. Waiting 5 secs...\";
          sleep 5;
        done; (exit $$s)"
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=http://localhost:3000
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
      - REG_TOKEN=${REG_TOKEN}
#       - MAIL_URL=smtp://smtp.email
#       - HTTP_PROXY=http://proxy.domain.com
#       - HTTPS_PROXY=http://proxy.domain.com
    depends_on:
      - mongo
    ports:
      - 3000:3000
    labels:
      - "traefik.backend=rocketchat"
      - "traefik.frontend.rule=Host: your.domain.tld"

  mongo:
    image: mongo:4.4
    restart: unless-stopped
    volumes:
     - ./data/db:/data/db
     #- ./data/dump:/dump
    command: mongod --oplogSize 128 --replSet rs0 --storageEngine=wiredTiger
    labels:
      - "traefik.enable=false"

  # this container's job is just run the command to initialize the replica set.
  # it will run the command and remove himself (it will not stay running)
  mongo-init-replica:
    image: mongo:4.4
    command: >
      bash -c
        "for i in `seq 1 30`; do
          mongo mongo/rocketchat --eval \"
            rs.initiate({
              _id: 'rs0',
              members: [ { _id: 0, host: 'localhost:27017' } ]})\" &&
          s=$$? && break || s=$$?;
          echo \"Tried $$i times. Waiting 5 secs...\";
          sleep 5;
        done; (exit $$s)"
    depends_on:
      - mongo

I will make sure our docker-compose.yml gets updated ASAP.

Also, let me know if you find any other issues along your discovery journey :slight_smile:

You can shout me anytime at our open: duda.nogueira

Right, I’m on proxmox so yes I generally take snapshots to roll back at the system level… BTRFS/ZFS makes life easier. I guess my opinion stems from decades in IT and working with devs. The issues with smooth updates and outdated components with a release, in my experience, come from a few things; Improper communication, lack of cohesive workflow within the team/s and a lack of testing / QA before release. Thankfully these things can be addressed and fixed with proper management and leadership. If this was just some random app on github where a few guys were releasing what they were working on this would be different. But, rocketchat is a successfully professional company that’s in the enterprise space. My experience with installing it multiple ways and having issues with both deteriorated my confidence using it long term if these things don’t improve. It seems like lots of people for sometime now have had concerns and issues with updating. Thanks for your feedback!

Thanks for this, my question is once this docker file you provided is deployed and a new version of mongo is needed will the upgrade path be easy and seamless or will it require steps like the user above? It seems like there should be a script that handles this, perhaps a new container for migration purposes. In docker deploy once and update then on after would be ideal. Thanks for taking the feedback constructively.

Neal, OMG YES. This is like talking to myself. It’s why I created the “ScottiBYTE” YouTube channel. I too have 40 years of IT background and I experienced the great times of properly vetted releases and upgrades with clear and concise documentation. My biggest issue today is the general lack of people to listen and communicate. I believe that is due to the fact that we have fewer and fewer experts. There was a significant time in IT where I was the guy to call that could pass his hand over the system and fix anything wrong. Companies now hire people to do such small “microcosm” jobs and they generally lack the ability or desire to develop advanced skills or communication in employees. More and more of our society is becoming subjective and the development of analytical skills is becoming a lost art.

@neal I was so excited, I forgot to answer the question. RocketChat says that this new docker-compose file is supposed to have the in-built capability to support any future updates or migrations. Once you are at Mongo 5.0 with the wiredTiger database engine, you are set for whenever RocketChat 5.0 is released and I think that the docker configuration should carry you into the future. However, as you well know, anything is possible.

ZFS entered the chat:

lol

The docker-file dudanogueira provided did install rocket chat with mongo 4.4 and the twisted tiger engine. I wanted to upgrade it to 5.0 so I was following your instructions at the url below. Since it’s already at 4.4 I should only have to stop the container, update the compose file for the db and it’s replica image to 5.0 and compose up to pull the new image right…then attach to the container and run the compatibility command? I followed the article…stopped the container that was at 4.4, updated the compose file and restarted it…or tried to it will not start after the pull to 5.0. I can’t obviously attach to it since it’s not running. Just seeing if I missed a step in your article.

dffad90b1d86 mongo:5.0 “docker-entrypoint.s…” 53 seconds ago Restarting (132) 21 seconds ago

I’m following the steps from here in your article since it was already at 4.4.

https://discussion.scottibyte.com/t/upgrading-rocketchat-deployed-in-docker/69

***To go from Mongo 4.0 to Mongo 4.2:
Edit your docker-compose.yml and replace the 4.0 with 4.2 in the mongo section and in the mongo-init-replica section.

Once this is completed, perform the following:

docker-compose stop mongo
docker-compose rm mongo
docker-compose up -d mongo

We now have to change the replication feature set in the database. Connect to the Mongo container. You may need to explicitly specify the name by first doing a “docker ps” to see what it is called:

docker exec -it mongo bash

Now that you are connected to the container:

mongo
db.adminCommand( { setFeatureCompatibilityVersion: "4.2" } )
exit
exit

I have had two others indicate that their system ran mongo 4.4 just fine but could not make the jump to mongo 5.0. Note that when Rocketchat moves to v5.0, its mongo dependency is only 4.4.

Mongo 5 has some restriction regarding hardware CPU: MongoDB 5.0 CPU Intel G4650 compatibility - Installation & Upgrades - MongoDB Developer Community Forums

I have had problems myself running mongo 5 and have helped around other users with hardware incompatible with “old” CPU.

The “problem” here is that Mongo dropped support for the mmap storage, and this required some extra steps do migrate to wiredTiger.

Once you are running with wiredTiger filestorage, you should be fine for future migrations. Unless, of course something else change, as @vmsman said. heheheh

Also, Rocket.Chat 5.X will probably keep mongo 4.x supported as well (I check this information and will push to update our compatibility Matrix)

I realized that MongoDB 4.4 with wiredTiger would be the requirement. Iin my video tutorial I showed folks how to go all the way to MongoDB 5.0 and they have taken 5.0 to be a requirement.