RocketChat 7 slow API responses with default docker deployment

michal.mocnak · February 13, 2025, 5:01pm

Description

Default docker deployment with rocker.server v7 (7.2.2 and 7.3.1 tested) is super slow on each API request = approx 7-12 seconds to response.
Checked same configuration on v6 and it does work properly without any issue.

Server Setup Information

Version of Rocket.Chat Server: 7.2.2
Operating System: Docker 27.01 / Ubuntu 22.04 / Swarm
Deployment Method: docker
Number of Running Instances: 1
DB Replicaset Oplog: 1
NodeJS Version: 20.18.3 - x64
MongoDB Version: 6.0.13
Proxy: traefik
Firewalls involved:

Any additional Information

x-default-opts:
  &default-opts
  logging:
    options:
      max-size: "10m"

services:
  mongodb:
    <<: *default-opts
    image: bitnami/mongodb:6.0
    volumes:
     - mongodb:/bitnami/mongodb
    environment:
      MONGODB_REPLICA_SET_MODE: primary
      MONGODB_REPLICA_SET_NAME: rs0
      MONGODB_PORT_NUMBER: 27017
      MONGODB_INITIAL_PRIMARY_HOST: mongodb
      MONGODB_INITIAL_PRIMARY_PORT_NUMBER: 27017
      MONGODB_ADVERTISED_HOSTNAME: mongodb
      MONGODB_ENABLE_JOURNAL: 'true'
      ALLOW_EMPTY_PASSWORD: 'yes'
    networks:
      - traefik_proxy

  rocketchat:
    <<: *default-opts
    image: rocket.chat:7.2.2
    volumes:
      - uploads:/app/uploads
    environment:
      PORT: 3000
      ROOT_URL: https://chat.my.domain
      MONGO_URL: mongodb://mongodb:27017/rocketchat?replicaSet=rs0
      MONGO_OPLOG_URL: mongodb://mongodb:27017/local?replicaSet=rs0
      DEPLOY_METHOD: docker
      Accounts_UseDNSDomainCheck: 'false'
    networks:
      - chat_internal
      - traefik_proxy
    deploy:
      mode: replicated
      replicas: 1
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik_proxy
        - traefik.http.routers.rocketchat.rule=Host(`chat.my.domain`)
        - traefik.http.routers.rocketchat.entrypoints=websecure
        - traefik.http.routers.rocketchat.tls.certresolver=resolver
        - traefik.http.services.rocketchat.loadbalancer.server.port=3000  
    depends_on:
      - mongodb
 
networks:
  traefik_proxy:
    external: true
  chat_internal:
    external: true

volumes:
  uploads:
  mongodb:

reetp · February 13, 2025, 9:41pm

You’ll have to tell us what you are actually trying to do.

What API calls?

michal.mocnak · February 13, 2025, 9:53pm

I am sorry. Literally every single API call is that slow. Checking via Chrome debug tools.

michal.mocnak · February 14, 2025, 9:09am

For example endpoint /api/v1/settings.public took 5.76s in v7.3.1 but in v6.13 it took 10ms

reetp · February 14, 2025, 6:22pm

What do your server logs tell you is going on?

What file storage are you using?

michal.mocnak · February 15, 2025, 7:51am

Nothing wrong in logs seems to me and storage is now local SSD.

LocalStore: store created at  
LocalStore: store created at  
LocalStore: store created at  
{"level":40,"time":"2025-02-15T07:38:16.227Z","pid":1,"hostname":"5bf00cf20b15","name":"VoIPAsteriskService","msg":"Voip is not enabled. Cant start the service"} 
{"level":51,"time":"2025-02-15T07:38:16.252Z","pid":1,"hostname":"5bf00cf20b15","name":"Migrations","msg":"Not migrating, already at version 318"} 
[2025-02-15T07:38:16.261Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Moleculer v0.14.35 is starting... 
[2025-02-15T07:38:16.262Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Namespace: <not defined> 
[2025-02-15T07:38:16.262Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Node ID: 0f673441-ee1f-4d80-b55a-d94dcc4235eb 
[2025-02-15T07:38:16.263Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/REGISTRY: Strategy: RoundRobinStrategy 
[2025-02-15T07:38:16.263Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/REGISTRY: Discoverer: LocalDiscoverer 
[2025-02-15T07:38:16.265Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Serializer: EJSONSerializer 
[2025-02-15T07:38:16.271Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Validator: FastestValidator 
[2025-02-15T07:38:16.273Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Registered 13 middleware(s). 
[2025-02-15T07:38:16.274Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: Transporter: TcpTransporter 
{"level":51,"time":"2025-02-15T07:38:16.293Z","pid":1,"hostname":"5bf00cf20b15","name":"DatabaseWatcher","msg":"Using change streams"} 
[2025-02-15T07:38:16.299Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/TRANSIT: Connecting to the transporter... 
[2025-02-15T07:38:16.319Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/TRANSPORTER: TCP server is listening on port 38679 
[2025-02-15T07:38:16.319Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/TRANSPORTER: UDP Discovery is disabled. 
[2025-02-15T07:38:16.319Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/TRANSPORTER: TCP Transporter started. 
{"level":51,"time":"2025-02-15T07:38:16.671Z","pid":1,"hostname":"5bf00cf20b15","name":"License","msg":"License installed","version":"3.0","hash":"O3XdD1N0"} 
ufs: temp directory created at "/tmp/ufs" 
[2025-02-15T07:38:16.828Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/REGISTRY: '$node' service is registered. 
[2025-02-15T07:38:16.829Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/REGISTRY: 'matrix' service is registered. 
[2025-02-15T07:38:16.830Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/$NODE: Service '$node' started. 
[2025-02-15T07:38:16.830Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/MATRIX: Service 'matrix' started. 
[2025-02-15T07:38:16.830Z] INFO  0f673441-ee1f-4d80-b55a-d94dcc4235eb/BROKER: ✔ ServiceBroker with 2 service(s) started successfully in 541ms. 
+--------------------------------------------+ 
|               SERVER RUNNING               | 
+--------------------------------------------+ 
|                                            | 
|  Rocket.Chat Version: 7.3.1                | 
|       NodeJS Version: 22.14.0 - x64        | 
|      MongoDB Version: 6.0.13               | 
|       MongoDB Engine: wiredTiger           | 
|             Platform: linux                | 
|         Process Port: 3000                 | 
|             Site URL: https://chat.my.dom  | 
|     ReplicaSet OpLog: Enabled              | 
|          Commit Hash: 6a2724740a           | 
|        Commit Branch: HEAD                 | 
|                                            | 
+--------------------------------------------+ 
{"level":51,"time":"2025-02-15T07:38:28.412Z","pid":1,"hostname":"5bf00cf20b15","name":"License","msg":"License installed","version":"3.0","hash":"x7MO8UVQ"}

reetp · February 15, 2025, 10:03am

Are you using GridFS or “local storage”?

Check your collection sizes too.

michal.mocnak · February 15, 2025, 1:30pm

It seems i figured it out. I am using just local storage. Later i’ll switch into MinIO. I realized mongodb was in traefik_proxy network instead of chat_internal. As rocketchat service is in both but chat_internal has as the main. Probably in v7 there is different dns resolve or something like that and it timeouts resolving mongo in different network.

However in v6 it worked well. And what is strange too is when i have set networks correctly and switched to mongo 7 it started to be slow again ;o)

Anyway now i am running mongo v6 and rocketchat v7 and it works well!!

reetp · February 15, 2025, 6:34pm

This I guess?

Glad you got it sorted.

michal.mocnak · February 15, 2025, 7:10pm

Actually this was ok but mongodb was in traefik_proxy instead of chat_internal. That was the problem. Why it has problem with mongo v7 i still don’t know. But it works with v6 so i’ll leave it ;o)

reetp · February 15, 2025, 7:23pm

Kk.

I’d start figuring the upgrade. You’ll have to eventually!!

michal.mocnak · February 16, 2025, 5:03pm

And finally i’ve figured it out with mongo version. All the problems were connected to the os version. Initially i’ve tested it on Ubuntu 22.04 LTS and it seems after upgrading to 24.04 LTS it starts to work correctly. Probably it was connected to mongo and it’s dependencies (even 22.04 is offically supported for version 7).

reetp · February 16, 2025, 6:21pm

That’s a ballache but excellent feedback thanks.

Typical with this fangled rapid release, beta at best, who cares if we didn’t test throroughly nonsense…

Not good enough at Enterprise level IMHO.

It will get mentioned

Topic		Replies	Views
Rocker chat version updating issue 4.8 to 6.4 Community Support rocketchat-apps , docker	2	522	November 21, 2023
Is there a way to speed up docker compose up -d Community Support docker	1	22	November 28, 2024
Fail to upgrade from 6.13.1 to 7.0.9 Upgrade Help	6	67	April 18, 2025
Migrating from a manual install to a snap install causes a slow problem on RocketChat Community Support	0	1356	January 8, 2020
Server became slow after update Community Support	3	702	November 24, 2021

[Secure CommsOS™ Launch] Join our next Roadmap Reveal webinar to learn more Register now ->

RocketChat 7 slow API responses with default docker deployment

Description

Server Setup Information

Any additional Information

Related topics