Replication settings for MongoDB used in Rocket.Chat

Community Support
1

Description

I have been ordered by the boss of Team Rocket to implement redundancy for MongoDB used in Rocket.Chat.

I set up MongoDB with a Primary, Secondary, and Arbiter configuration, but when the Primary goes down, my message inputs get grayed out, and the messages I send are not visible to other users.I have confirmed that setting up two Secondaries along with the Primary solves the issue.I believe this is due to the behavior of MongoDB’s ChangeStream API.

If you have any knowledge on how to configure MongoDB so that Rocket.Chat works with a Primary, Secondary, and Arbiter setup, I would greatly appreciate your advice.

Server Setup Information

  • Version of Rocket.Chat Server: 6.5.2
  • Operating System: Red Hat Enterprise Linux 8.8
  • Deployment Method: tar
  • Number of Running Instances: 2
  • DB Replicaset Oplog: true
  • NodeJS Version: 14.21.3
  • MongoDB Version: 5.0.21
  • Proxy: nginx
  • Firewalls involved:

Any additional Information

This is the configuration information for the replica set.

{
        "_id" : "chat",
        "version" : 9,
        "term" : 61,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.21.203:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 2,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "192.168.21.204:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "192.168.21.248:27017",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 0,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : true,
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("667d056118c880954dc247a5")
        }
}
2

Upgrade. First you are only on 6.5.2 and the latest of that series was 6.5.8 - you really need to keep up to date with updates - there were security fixes you have missed which put you at risk of getting hacked :fearful:

On top of that:

https://docs.rocket.chat/docs/support-prerequisites

Each release of Rocket.Chat comes with a six-month support period from its release date.

Once this period expires, hosted services such as the Push Gateway and Marketplace will cease to function for that particular version.

https://docs.rocket.chat/docs/version-durability

Your version was EOL in June 2024. You need to be on a higher version - I’d suggest 6.11.2

Can’t help you otherwise.

3

Thank you for your response. I understand about the support period.
As expected, upgrading is necessary for security reasons.
I will report to our boss, Giovanni. I will check the operation again after the upgrade.

4

I tried setting up the latest version in a new environment and checked it.
The same graying out issue occurs in this environment as well.
Should I be scrutinizing the source code or studying MongoDB…?

  • Version of Rocket.Chat Server: 6.12.0
  • Operating System: Rocky Linux 9.2
  • Deployment Method: tar
  • Number of Running Instances: 1
  • DB Replicaset Oplog: true
  • NodeJS Version: 14.21.3
  • MongoDB Version: 6.0.14
  • Proxy: nginx
  • Firewalls involved:
5

I have no idea about arbiter and whether it would or should work.

You have a replica set?

As normal debug procedure I’d suggest you look at logs first.

So Rocket.Chat logs ands the Mongo logs.

See where that leads.

6

Thank you for your response.
By using an arbiter, we can set up replication with a low-spec server.
Team Rocket doesn’t have much money, so we are focused on saving costs.

For now, I will patiently check the logs.
I’ll consult you again if there is any progress.
Thank you.

7

I suspect that is an unsupported option. Sorry.

How many users are you expecting to support?

8

I can’t say it too loudly, but Team Rocket is an organization made up of 50 to 100 people.

9

Kk.

In which case you don’t need anything fancy.

I run similar on a VM with 2x cpu, 6gb Ram, docker rocket & single mongo.

Think Rocket is limited to 4Gb.

It’s not massively busy but it just idles.

There is no such thing as “free”, even with open source. You need to budget a little somewhere.

You can get a VM with 4gb at say vultr or digital ocean for around $20 per month. Between 50 of you? Couple of dollars each per year.

10

Thank you for your response.

The purpose of setting up replication was not for load balancing but for redundancy.
The goal is to ensure that if one MongoDB server fails, another MongoDB server can take over, allowing Rocket.Chat services to continue.
It seems that MongoDB has a separate mechanism for load balancing.

Indeed, it is difficult to achieve everything without spending money.
For now, having one primary server and two secondary servers seems sufficient, so I will proceed with that.
I hope the information about the issues with having a primary, secondary, and arbiter setup is helpful.

Thank you.

11

I think if you are operating on a limited budget then this is a) overkill and b) not cheap. Hey ho.

we are focused on saving costs

Redundancy costs more…

I can honestly say that in the several years that I have run my servers the only downtime has been when I have updated. That’s it. It has been reliable and robust.

I’d spend my money on decent reliable hardware and have some good backups - I think you are chasing some ghosts here.

Not really I’m afraid - you have not got a cure and I don’t believe arbiter/sharding is supported.

Ah yes - note this has been covered many times before. You need to search thoroughly.

So sharding isn’t, and won’t ever be, supported.

If you need redundancy that badly then invest the money and run a couple of Mongo servers which is supported.

12

Thank you for your response. I learned a lot from it. Thank you very much :grinning:

13

Theory v practice :slight_smile:

For 50-100 one instance will be fine. Run a separate test box for testing upgrades, and do upgrades slowly and incrementally with a good eye on github issues.

A few notes for you!

You only need to think about increasing backend resources above 200 users due to constraints with node etc.

1 Like