Slow running Queries to rocketchat_messages collection

Community Support
1

Description

We are seeing some long running queries to rocketchat.rocketchat_message. Both command and query. Just a few random picks from them, we have:

  • 18 second command
  • 32 second query
  • 23 second query
  • 12 second query

These happen fairly often in the morning. Usually this presents itself as messages not sending (staying gray).

Is there any Tuning we can do to stop slow running queries? One thing we are looking at changing is the MongoURL with readPreference=nearest since currently we are reading and writing to the master in the Replicaset.

Server Setup Information

  • Version of Rocket.Chat Server: 3.0.3
  • Operating System: CentOS 7
  • Deployment Method: tar / extraction
  • Number of Running Instances: 11
  • DB Replicaset Oplog:
  • NodeJS Version: v12.16.1
  • MongoDB Version: 3.6.11
  • Proxy: Haproxy

Any additional Information

We have 1400 Users.

7.5GB rocketchat_message collection
12.3 GB rocketchat_app_logs collection

{
    "host": "mongodb03:27017",
    "desc": "conn3079",
    "connectionId": 3079,
    "client": "10.45.9.22:41910",
    "clientMetadata": {
        "driver": {
            "name": "nodejs",
            "version": "3.2.7"
        },
        "os": {
            "type": "Linux",
            "name": "linux",
            "architecture": "x64",
            "version": "3.10.0-862.14.4.el7.x86_64"
        },
        "platform": "Node.js v12.16.1, LE, mongodb-core: 3.2.7"
    },
    "active": "true",
    "currentOpTime": "2020-03-12T08:42:45.903-0400",
    "opid": 134946789,
    "secs_running": 32,
    "microsecs_running": 32368731,
    "op": "query",
    "ns": "rocketchat.rocketchat_message",
    "command": {
        "find": "rocketchat_message",
        "filter": {
            "_hidden": {
                "$ne": true
            },
            "rid": "RWXWWNtwXzzCLxgfg",
            "_updatedAt": {
                "$gt": "2020-02-20T12:33:48.249Z"
            }
        },
        "sort": {
            "ts": -1
        },
        "returnKey": false,
        "showRecordId": false,
        "$db": "rocketchat"
    },
    "numYields": 2047,
    "locks": {
        "Global": "r",
        "MMAPV1Journal": "r",
        "Database": "r",
        "Collection": "R"
    },
    "waitingForLock": "false",
    "lockStats": {
        "Global": {
            "acquireCount": {
                "r": 4096
            }
        },
        "MMAPV1Journal": {
            "acquireCount": {
                "r": 2110
            },
            "acquireWaitCount": {
                "r": 19
            },
            "timeAcquiringMicros": {
                "r": 2293895
            }
        },
        "Database": {
            "acquireCount": {
                "r": 2048
            }
        },
        "Collection": {
            "acquireCount": {
                "R": 2048
            },
            "acquireWaitCount": {
                "R": 62
            },
            "timeAcquiringMicros": {
                "R": 2709138
            }
        },
        "Mutex": {
            "acquireCount": {
                "r": 814
            }
        }
    }
}
2

Hello, any update on this one?
Same problem here

3

I have a similar problem. Any news?