Docker-compose not working properly

Description

Im trying to update my docker based Rocketchat containers following the official Rocketchat instructions

I want to upgrade from 4.2.0 to 4.8.1

The issue is that the command “sudo docker-compose stop rocketchat” goes through without giving me an error, but doesnt actually stop the rocketchat-related containers.

When I do an “sudo docker ps” after this, it tells me that the Rocketchat-Containers are still running.

and with “sudo docker-compose rm rocketchat” it only removes the weird “zmwadmin_rocketchat_1” container.

After that, the container default_rocketchat_1 does still exist and is running:

This makes it kinda impossible for me to update our Rocketchat-Containers.
Ive already tried removing the containers manually with “sudo docker rm default_rocketchat_1”, then building version 4.8.1 with “sudo-docker-compose up -d”, but this results in my Nginx Reverse Proxy giving me the message “502 Bad Gateway”.

Im not sure how I could fix this issue.

The docker-containers are stored directly in the Home-Path of user “zmwadmin”: /home/zmwadmin/
As well as the docker-compose file: /home/zmwadmin/docker-compose.yml

Server Setup Information

  • Version of Rocket.Chat Server:
  • Operating System: Ubuntu 22.04 LTS
  • Deployment Method: Docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog:
  • NodeJS Version: v12.22.1
  • MongoDB Version: 4.0.19 / mmapv1 (oplog activated)
  • Proxy: Local Reverse Proxy is Nginx
  • Firewalls involved: probably not

Any additional Information

My docker-compose.yml:

Blockquote
version: ‘2’

services:
rocketchat:
image: rocketchat/rocket.chat:4.2.0
command: >
bash -c
“for i in seq 1 30; do
node main.js &&
s=$$? && break || s=$$?;
echo "Tried $$i times. Waiting 5 secs…";
sleep 5;
done; (exit $$s)”
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=…
- MONGO_URL=…
- MONGO_OPLOG_URL=…
- MAIL_URL=…

- HTTP_PROXY=…

- HTTPS_PROXY=…

depends_on:
  - mongo
ports:
  - 3000:3000
labels:
  - "traefik.backend=rocketchat"
  - "traefik.frontend.rule=Host: your.domain.tld"

mongo:
image: mongo:4.0
restart: unless-stopped
volumes:
- ./data/db:/data/db
#- ./data/dump:/dump
command: mongod --smallfiles --oplogSize 128 --replSet rs0 --storageEngine=mmapv1
labels:
- “traefik.enable=false”

this container’s job is just run the command to initialize the replica set.

it will run the command and remove himself (it will not stay running)

mongo-init-replica:
image: mongo:4.0
command: >
bash -c
“for i in seq 1 30; do
mongo mongo/rocketchat --eval "
rs.initiate({
_id: ‘rs0’,
members: [ { _id: 0, host: ‘localhost:27017’ } ]})" &&
s=$$? && break || s=$$?;
echo "Tried $$i times. Waiting 5 secs…";
sleep 5;
done; (exit $$s)”
depends_on:
- mongo

hubot, the popular chatbot (add the bot user first and change the password before starting this image)

hubot:
image: rocketchat/hubot-rocketchat:latest
restart: unless-stopped
environment:
- ROCKETCHAT_URL=this.ismydomain
- ROCKETCHAT_ROOM=GENERAL
- ROCKETCHAT_USER=bot
- ROCKETCHAT_PASSWORD=botpassword
- BOT_NAME=bot

you can add more scripts as you’d like here, they need to be installable by npm

  - EXTERNAL_SCRIPTS=hubot-help,hubot-seen,hubot-links,hubot-diagnostics
depends_on:
  - rocketchat
labels:
  - "traefik.enable=false"
volumes:
  - ./scripts:/home/hubot/scripts

this is used to expose the hubot port for notifications on the host on port 3001, e.g. for hubot-jenkins-notifier

ports:
  - 3001:8080

#traefik:

image: traefik:latest

restart: unless-stopped

command: >

traefik

–docker

–acme=true

–acme.domains=‘your.domain.tld’

–acme.email=‘your@email.tld’

–acme.entrypoint=https

–acme.storagefile=acme.json

–defaultentrypoints=http

–defaultentrypoints=https

–entryPoints=‘Name:http Address::80 Redirect.EntryPoint:https’

–entryPoints=‘Name:https Address::443 TLS.Certificates:’

ports:

- 80:80

- 443:443

volumes:

- /var/run/docker.sock:/var/run/docker.sock

zmwadmin@zmwrc:~$ version: ‘2’

services:
rocketchat:
image: rocketchat/rocket.chat:4.2.0
command: >
bash -c
“for i in seq 1 30; do
node main.js &&
s=$$? && break || s=$$?;
echo "Tried $$i times. Waiting 5 secs…";
sleep 5;
done; (exit $$s)”
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=…
- MONGO_URL=…
- MONGO_OPLOG_URL=…
- MAIL_URL=…

- HTTP_PROXY=…

- HTTPS_PROXY=…

depends_on:
  - mongo
ports:
  - 3000:3000
labels:
  - "traefik.backend=rocketchat"
  - "traefik.frontend.rule=Host: your.domain.tld"

mongo:
image: mongo:4.0
restart: unless-stopped
volumes:
- ./data/db:/data/db
#- ./data/dump:/dump
command: mongod --smallfiles --oplogSize 128 --replSet rs0 --storageEngine=mmapv1
labels:
- “traefik.enable=false”

this container’s job is just run the command to initialize the replica set.

it will run the command and remove himself (it will not stay running)

mongo-init-replica:
image: mongo:4.0
command: >
bash -c
“for i in seq 1 30; do
mongo mongo/rocketchat --eval "
rs.initiate({
_id: ‘rs0’,
members: [ { _id: 0, host: ‘localhost:27017’ } ]})" &&
s=$$? && break || s=$$?;
echo "Tried $$i times. Waiting 5 secs…";
sleep 5;
done; (exit $$s)”
depends_on:
^C# - /var/run/docker.sock:/var/run/docker.sock.Certificates:‘https’ on port 3001, e.g. for hubot-jenkins-notifier

bump, must be 20 characters

Hi maybe this is a silly answer. But are you sure you’re running the command in the right folder where your started your rocketchat instance?
Why do you use sudo with docker-compose command? I run it as a regular user always.
If you followed the tutorial for setup you may have passed the permissions to a user named rocketchat. Have you logged in with it?
I hope some of this helps.

Hi!

Upgrading should be something like:

  1. edit docker-compose.yml
  2. change the server image tag to the version you want, something like, in your case:
    from: rocketchat/rocket.chat:4.2.0
    to: rocketchat/rocket.chat:4.8.2

now you need to grab the latest version, so:
3) docker-compose pull

and finally:
4) docker-compose up -d

please, always backup everything before migrating :slight_smile:

Hi, I managed to get rid of the containers including “zmwadmin”. Rocketchat still responded correctly after removing those. Now only the “default” containers are left.

I also managed to update the Rocketchat-Container default_rocketchat_1 to the latest version.
All containers are running but my Nginx-Reverse-Proxy gives me an “502 Bad Gateway”.

What I did was the following:

  1. Stopping all related containers
  2. Removed “zmwadmin”-Containers
  3. Changed directory to /
  4. Edited the tag rocketchat:4.2.0 to rocketchat:latest
  5. docker-compose pull
  6. docker-compose up -d

/var/log/nginx/rocketchat.access.log shows me the follwing errors when trying to reach the Rocketchat-URL:

2022/08/02 06:24:32 [error] 2332#2332: *1 recv() failed (104: Unknown error) while reading response header from upstream, client: 172.22.4.34, server: some.domain.de:57300, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:3000/favicon.ico", host: "some.domain.de:57300", referrer: "https://some.domain.de:57300/"
2022/08/02 06:24:34 [error] 2332#2332: *1 recv() failed (104: Unknown error) while reading response header from upstream, client: 172.22.4.34, server: some.domain.de:57300, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "some.domain.de:57300"
2022/08/02 06:24:34 [error] 2332#2332: *1 recv() failed (104: Unknown error) while reading response header from upstream, client: 172.22.4.34, server: some.domain.de:57300, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:3000/favicon.ico", host: "some.domain.de:57300", referrer: "https://some.domain.de:57300/"

won’t latest tag download Rocket.Chat version 5.0.0?
If it does it does no longer support your mongo version as it looks from your first post you have version 4.0…
The release note from RC 5.0.0 states:
MongoDB: 4.2, 4.4, 5.0 are supported

If you run “docker ps” command, does it show your chat running?

your upstream should point to something like http://rocketchat:3000

I prefer using traefik for reverse proxying. Here is a nice example I use:

@cestev My Rocketchat-Containers including MongoDB are up when using the tag :latest

@dudanogueira Ive tried changing the upstream URL to any combination of http://localhost:3000 and when restarting nginx it always gave me the following error:

Aug 03 05:01:34 zmwrc nginx[2285]: nginx: [emerg] invalid host in upstream "http://127.0.0.1:3000" in /etc/nginx/sites-enabled/default:3

Switching over to traefik as my reverse proxy isnt an option for me. Im working as a sysadmin and im already struggling to maintain our current docker-based rocketchat instance, as I have no deep understanding to how docker works.
When I installed rocketchat via docker it was officially suggested to use nginx as the local reverse proxy instead of directly connecting to the unencrypted rocketchat-container via port tcp/3000. Given Nginx-config was also provided by the rocketchat documentation.

HI!

the address of the upstream will depend on how your servers are setup.

I suspect you added http://127.0.0.1:3000 at the upstream, while you only need the 127.0.0.1:3000 (without http://)

you should add the protocol at the proxy_pass directive inside the location / {} section

Oh, sorry I probably misunderstood what you meant.

My proxy_pass is already configured to point towards 127.0.0.1:3000

I believe it makes sense to post my full Nginx-Config here, so here it is:

cat /etc/nginx/sites-enabled/default

# Upstreams
upstream backend {
    server 127.0.0.1:3000;
}

# HTTPS Server
server {
    listen 443;
    server_name zmwrc.somedomain.de:57300;

    # You can increase the limit if your need to.
    client_max_body_size 200M;

    error_log /var/log/nginx/rocketchat.access.log;

    ssl on;
    ssl_certificate /etc/nginx/wildcard.cer;
    ssl_certificate_key /etc/nginx/wildcard.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don’t use SSLv3 ref: POODLE

    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;

        proxy_redirect off;
    }
}

I also tried to set proxy_pass directly to http://127.0.0.1:3000 but the result was the same when connecting to the Nginx Reverse Proxy from my Browser: 502 Bad Gateway

Hi!

can you state the NGINX version you are running?

Maybe it’s an old version. The problem here is that its saying the nginx confi file is invalid.

Hi, sorry for the late reply, I was on holidays :slight_smile:

nginx -v 

shows me the following output:

nginx version: nginx/1.18.0 (Ubuntu)

According to my 2 seconds of research, this should be the latest stable version for nginx as of now.

If you run nginx -t, what will it bring?

This is certainly some validation error in NGNIX

nginx -t gives me the following output:

nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/sites-enabled/default:16
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Line 16 is where the Config-File “default” says:

ssl on;

Bump

Theres one more question regarding business support.
We contacted the Rocketchat Business Support a month ago as suggested here:

Nobody replied to our form though… :frowning:
What can we do to get Business Support for our on-prem Rocketchat Instance?

Hi! Commercial support is only available for Enterprise Edition licensed workspaces. If you do have EE, you can reach our support at https://support.rocket.chat/

Bear in mind that this is not actually a Rocket.Chat issue, but a deployment one.

please feel free to ping me at our open so I can take a closer look.