RC HA - No messages send after single mongo server goes down

Community Support
1

Description

I have a problem with my RC HA instance.

I have 2 indentical hosts with docker-compose running RC and mongodb and an additional host running mongodb as arbiter and haproxy as the only public faced service of the cluster.

Everything works fine, I can stop a RC instance and with very short delay every user gets put on the running instance. But if I stop a mongodb instance the problem starts.

As soon as I stop it (doesn’t matter if primary or secondary) everything database related in RC goes dead. No new messages can be send (stay gray) and most functions in admin area are greyed out.

Any idea what the problem is? Do I need a third host running the full stack? Is anything in my stack configured wrong?

Server Setup Information

  • Version of Rocket.Chat Server: 3.5.0
  • Operating System: Ubuntu 18.04
  • Deployment Method: docker
  • Number of Running Instances: 2
  • DB Replicaset Oplog: yes
  • NodeJS Version: 12.16.1
  • MongoDB Version: 3.6.9
  • Proxy: haproxy
  • Firewalls involved: no

Any additional Information

docker-compose.yml

version: '2'

services:
  rocketchat:
    image: rocketchat/rocket.chat:latest
    command: >
      bash -c
        "for i in `seq 1 30`; do
          node main.js &&
          s=$$? && break || s=$$?;
          echo \"Tried $$i times. Waiting 5 secs...\";
          sleep 5;
        done; (exit $$s)"
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=http://chat.<domain>.com
      - MONGO_URL=mongodb://rocket:<password>@rocket-1:27017,rocket-2:27017/rocketchat?authSource=admin&replicaSet=rs0&readPreference=primaryPreferred&w=majority
      - MONGO_OPLOG_URL=mongodb://oploguser:<password>@rocket-1:27017,rocket-2:27017/local?authSource=admin&replicaSet=rs0
      - INSTANCE_IP=172.16.0.2
#      - MAIL_URL=smtp://smtp.email
#       - HTTP_PROXY=http://proxy.domain.com
#       - HTTPS_PROXY=http://proxy.domain.com
    depends_on:
      - mongo
    ports:
      - 172.16.0.2:3000:3000
    extra_hosts:
        - "rocket-1:172.16.0.2"
        - "rocket-2:172.16.0.3"
        - "rocket-proxy:172.16.0.4"

  mongo:
    image: mongo:3.6.9
    restart: unless-stopped
    volumes:
     - /root/mongo/arb:/data/db
     - /root/mongo:/opt/keyfile
    command: mongod --smallfiles --keyFile /opt/keyfile/mongodb-keyfile --replSet rs0 --enableMajorityReadConcern false --storageEngine=mmapv1
    ports:
      - 172.16.0.2:27017:27017
    extra_hosts:
        - "rocket-1:172.16.0.2"
        - "rocket-2:172.16.0.3"
        - "rocket-proxy:172.16.0.4"
2

I “solved” this by deploying a whole additional stack and removing the arbiter. Now everything works as expected.