Connection to mongodb cluster failing with RC cluster

Description

I’m currently trying to setup a 2 node RC and mongodb cluster and everything seemed fine until RC decided to just not connect to the mongodb cluster.

I followed this tutorial: https://rocket.chat/docs/installation/docker-containers/high-availability-install/

Using the exact same mongodb URI directly, works without a problem.

I really have no idea, what the problem could be. I have attached the docker-compose.yml, logs from starting and testing the URI and the users present on the mongodb cluster.

Server Setup Information

version: '2'

services:
  rocketchat:
    image: rocketchat/rocket.chat:latest
    hostname: rocket-1
    command: >
      bash -c
        "for i in `seq 1 30`; do
          node main.js &&
          s=$$? && break || s=$$?;
          echo \"Tried $$i times. Waiting 5 secs...\";
          sleep 5;
        done; (exit $$s)"
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=http://chat.mbconnectline.com
      - MONGO_URL=mongodb://rocket:supersecurepassword@rocket-1:27017,rocket-2:27017/rocketchat?authSource=admin&replicaSet=rs0&readPreference=nearest&w=majority
      - MONGO_OPLOG_URL=mongodb://oploguser:supersecurepassword@rocket-1:27017,rocket-2:27017/local?authSource=admin&replicaSet=rs0
      - INSTANCE_IP=172.16.0.2
#      - MAIL_URL=smtp://smtp.email
#       - HTTP_PROXY=http://proxy.domain.com
#       - HTTPS_PROXY=http://proxy.domain.com
    depends_on:
      - mongo
    ports:
      - 172.16.0.2:3000:3000
    extra_hosts:
        - "rocket-1:172.16.0.2"
        - "rocket-2:172.16.0.3"
        - "rocket-proxy:172.16.0.4"

  mongo:
    image: mongo:3.6.9
    hostname: rocket-1
    restart: unless-stopped
    volumes:
     - /root/mongo/arb:/data/db
     - /root/mongo:/opt/keyfile
    command: mongod --smallfiles --keyFile /opt/keyfile/mongodb-keyfile --replSet rs0 --enableMajorityReadConcern false --storageEngine=mmapv1
    ports:
      - 172.16.0.2:27017:27017
    extra_hosts:
        - "rocket-1:172.16.0.2"
        - "rocket-2:172.16.0.3"
        - "rocket-proxy:172.16.0.4"

Any additional Information

Running docker-compose up

mongo_1       | 2020-05-06T12:16:29.373+0000 I ASIO     [NetworkInterfaceASIO-RS-0] Successfully connected to rocket-2:27017, took 5ms (2 connections now open to rocket-2:27017)
mongo_1       | 2020-05-06T12:16:30.203+0000 I NETWORK  [listener] connection accepted from 172.16.0.3:50200 #2 (2 connections now open)
mongo_1       | 2020-05-06T12:16:30.204+0000 I NETWORK  [conn2] received client metadata from 172.16.0.3:50200 conn2: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.6.9" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.15.0-99-generic" } }
mongo_1       | 2020-05-06T12:16:30.227+0000 I ACCESS   [conn2] Successfully authenticated as principal __system on local
rocketchat_1  | /app/bundle/programs/server/node_modules/fibers/future.js:313
rocketchat_1  |                                                 throw(ex);
rocketchat_1  |                                                 ^
rocketchat_1  | 
rocketchat_1  | Error: connect ECONNREFUSED 172.18.0.3:27017
rocketchat_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1137:16) {
rocketchat_1  |   name: 'MongoNetworkError',
rocketchat_1  |   errorLabels: [ 'TransientTransactionError' ],
rocketchat_1  |   [Symbol(mongoErrorContextSymbol)]: {}
rocketchat_1  | }
rocketchat_1  | Tried 1 times. Waiting 5 secs...

The logs on rocket-2 (currently PRIMARY)

2020-05-06T12:16:30.202742818Z 2020-05-06T12:16:30.202+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to rocket-1:27017
2020-05-06T12:16:30.228444987Z 2020-05-06T12:16:30.228+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to rocket-1:27017, took 26ms (1 connections now open to rocket-1:27017)
2020-05-06T12:16:30.229085661Z 2020-05-06T12:16:30.228+0000 I REPL     [replexec-10] Member rocket-1:27017 is now in state SECONDARY
2020-05-06T12:16:30.682224798Z 2020-05-06T12:16:30.681+0000 I NETWORK  [listener] connection accepted from 172.16.0.2:41124 #646 (6 connections now open)
2020-05-06T12:16:30.696569069Z 2020-05-06T12:16:30.696+0000 I NETWORK  [conn646] end connection 172.16.0.2:41124 (5 connections now open)
2020-05-06T12:16:37.043417526Z 2020-05-06T12:16:37.043+0000 I NETWORK  [listener] connection accepted from 172.16.0.2:41128 #647 (6 connections now open)
2020-05-06T12:16:37.058876004Z 2020-05-06T12:16:37.058+0000 I NETWORK  [conn647] end connection 172.16.0.2:41128 (5 connections now open)
2020-05-06T12:16:43.389324847Z 2020-05-06T12:16:43.389+0000 I NETWORK  [listener] connection accepted from 172.16.0.2:41132 #648 (6 connections now open)
2020-05-06T12:16:43.404446929Z 2020-05-06T12:16:43.404+0000 I NETWORK  [conn648] end connection 172.16.0.2:41132 (5 connections now open)
2020-05-06T12:16:44.394504693Z 2020-05-06T12:16:44.394+0000 I NETWORK  [conn644] end connection 172.16.0.2:54640 (4 connections now open)
2020-05-06T12:16:44.428933376Z 2020-05-06T12:16:44.428+0000 I NETWORK  [conn645] end connection 172.16.0.2:54642 (3 connections now open)
2020-05-06T12:16:44.428980073Z 2020-05-06T12:16:44.428+0000 I NETWORK  [conn643] end connection 172.16.0.2:54636 (2 connections now open)

Testing the mongodb URI with docker exec -it rocketchat_mongo_1 /bin/bash

root@rocket-1:/# mongo 'mongodb://rocket:supersecurepassword-1:27017,rocket-2:27017/rocketchat?authSource=admin&replicaSet=rs0&readPreference=nearest&w=majority'
MongoDB shell version v3.6.9
connecting to: mongodb://rocket-1:27017,rocket-2:27017/rocketchat?authSource=admin&replicaSet=rs0&readPreference=nearest&w=majority
2020-05-06T12:22:37.160+0000 I NETWORK  [thread1] Starting new replica set monitor for rs0/rocket-1:27017,rocket-2:27017
2020-05-06T12:22:37.162+0000 I NETWORK  [thread1] Successfully connected to rocket-1:27017 (1 connections now open to rocket-1:27017 with a 5 second timeout)
2020-05-06T12:22:37.163+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to rocket-2:27017 (1 connections now open to rocket-2:27017 with a 5 second timeout)
Implicit session: session { "id" : UUID("bab275f9-7ed8-42a7-91db-eb3d1b4f3c54") }
MongoDB server version: 3.6.9
rs0:PRIMARY> db.isMaster()
{
        "hosts" : [
                "rocket-1:27017",
                "rocket-2:27017"
        ],
        "arbiters" : [
                "rocket-proxy:27017"
        ],
        "setName" : "rs0",
        "setVersion" : 3,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "rocket-2:27017",
        "me" : "rocket-2:27017",
        "electionId" : ObjectId("7fffffff0000000000000002"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1588767779, 1),
                        "t" : NumberLong(2)
                },
                "lastWriteDate" : ISODate("2020-05-06T12:22:59Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1588767779, 1),
                        "t" : NumberLong(2)
                },
                "majorityWriteDate" : ISODate("2020-05-06T12:22:59Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2020-05-06T12:23:06.856Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "minWireVersion" : 0,
        "maxWireVersion" : 6,
        "readOnly" : false,
        "ok" : 1,
        "operationTime" : Timestamp(1588767779, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1588767779, 1),
                "signature" : {
                        "hash" : BinData(0,"iHro1JLnUlJCdOlYRRTWyouRe/4="),
                        "keyId" : NumberLong("6823659583924338689")
                }
        }
}
root@rocket-1:/# mongo 'mongodb://oploguser:supersecurepassword@rocket-1:27017,rocket-2:27017/local?authSource=admin&replicaSet=rs0'
MongoDB shell version v3.6.9
connecting to: mongodb://rocket-1:27017,rocket-2:27017/local?authSource=admin&replicaSet=rs0
2020-05-06T12:41:38.956+0000 I NETWORK  [thread1] Starting new replica set monitor for rs0/rocket-1:27017,rocket-2:27017
2020-05-06T12:41:38.958+0000 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to rocket-1:27017 (1 connections now open to rocket-1:27017 with a 5 second timeout)
2020-05-06T12:41:38.958+0000 I NETWORK  [thread1] Successfully connected to rocket-2:27017 (1 connections now open to rocket-2:27017 with a 5 second timeout)
Implicit session: session { "id" : UUID("e28d1aa3-6603-4ff1-9342-8b49de128738") }
MongoDB server version: 3.6.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
rs0:PRIMARY>

Output of show users

rs0:PRIMARY> show users
{
        "_id" : "admin.oploguser",
        "user" : "oploguser",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "read",
                        "db" : "local"
                }
        ]
}
{
        "_id" : "admin.rocket",
        "user" : "rocket",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "readWrite",
                        "db" : "rocketchat"
                }
        ]
}
{
        "_id" : "admin.siteRootAdmin",
        "user" : "siteRootAdmin",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "root",
                        "db" : "admin"
                }
        ]
}
{
        "_id" : "admin.siteUserAdmin",
        "user" : "siteUserAdmin",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "userAdminAnyDatabase",
                        "db" : "admin"
                }
        ]
}

I solved the problem by removing the hostname from the docker-compose.yml. Docker somehow messed things up with it set.