Rocket.Chat's Community Open Call 🎤 Dec 15, 2021 Join us!

MongoDB - v4.2 wiredTiger - Slower?

Description

I updated our production server to 4.1.2, and switched from MMAP Engine to wiredTiger.

The result has caused some performance issues. RocketChat feels slower and hangs.

I thought it was server resource issue so I upgraded our instance to T3.Medium, but the result still the same.

Thoughts?

Server Setup Information

  • Version of Rocket.Chat Server: 4.1.2
  • Operating System: Ubuntu 20 LTS
  • Deployment Method: Docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: v12.22.1
  • MongoDB Version: 4.2.17
  • Proxy: nginx
  • Firewalls involved:

Any additional Information

Example1: Reloading the RC using the app from time to time takes a good minute or two. Reloading the web version seems to be just fine.

Expample2: Sending messages seems to lag from time.

Are you using XFS file system?

I think we’re on ext4 system. Would that make a huge difference?

I read that wiredTiger supposedly utilize CPU usage.

Additional Details:



------------ Notes ------------
I upgraded to 5.0.3 MongoDB. The performance have improved a little bit, but nothing compared to when we were using MMAP. Anyone else experiencing the same issue?

We have about 100 users in here now. The one that seems to come up the most in the logs is the “too many request” error. Is there a way to increase this or by pass?

I dont exactly know the impact of XFS vs ext4, but there are dire warnings in the startup log of mongo, if you start it on ext4.

Hi! I have asked our deployment team about this. I’ll keep you posted here.

1 Like

Hi!

According to here:

and to what our deployment specialists have answered, wiredTiger has indeed some different memory strategy from mmap:

" a server running both wired tiger Mongo and RC on the same VPS or machine with limited memory will see drastic deterioration of performance because RC will now be severely constrained."

2 Likes

Thanks, @dudanogueira!

I’ve certainly tried to keep up with the resources and compensated with 2vCPU and 4GB RAM (AWS T3.Medium); however, the performance issue is still there despite of only using just about 15% of the resources.

Out of curiosity, is there a safe way to migrate back from MongoDB 5.0 wiredTiger to MongoDB 4.0 MMAP? I assume its a sequential downgrade for mongo, then migrate to MMAP once I downgrade to 4.0.

That is certainly something to look into.

One thing I would try first is to limit the resources for the container. Downgrading MongoDB is a solution for now, but it will certainly bite you at the newer future.

here is how you can specify docker resources for containers:

Downgrading MongoDB didn’t end well. I got as far as 4.2 wiredTriger. The performance is much worse at that version. The 5.0.3 seems to be the sweet spot at the moment.

I tried limiting resources but it looks like it have some compatibility issue with “deploy” tag. That or I have to build a “swarm”, of which at the moment beyond my current knowledge.

if not on swarm, you can try limiting using docker-compose v2 syntax:

That didn’t work out well for me. I mean I reviewed this as well. Compose file version 2 reference | Docker Documentation


I got this response when posted on Github - RocketChat slow performance on MongoDB wiredTiger (4.2-5.0) · Issue #23721 · RocketChat/Rocket.Chat · GitHub

Any clues on how to better optimize MongoDB?

Update: The instance is still a little bit sluggish and some messages are being delayed.

On docker compose for mongo container, I added memory reservation, limit, and swap.
Ref: Compose file version 2 reference | Docker Documentation

"
mem_limit: 3000M
mem_reservation: 2500M
memswap_limit: 3200M
privileged: true

"

On RocketChat app setting, I disabled DDP Rate Limiter.

Admin → Rate Limiter → DDP Rate Limiter

1 Like