Long response after migrate

Community Support
1

Description

After the migration, the new server responds for a very long time. Users enter for 15 minutes, message delay for 5-10 minutes.

Server Setup Information

  • server has 8 cores, 32gb ram, 250ssd
  • Version of Rocket.Chat Server: 2.1.1 rev.1414
  • Operating System: Ubuntu Server 18.04
  • Deployment Method: snap
  • Number of Running Instances:
  • DB Replicaset Oplog:
  • NodeJS Version:
  • MongoDB Version: 3.6.14 wiredtiger engine
  • About 100 users

Any additional Information

Did backup and restore with this instructions https://rocket.chat/docs/installation/manual-installation/ubuntu/snaps/
without any errors
My database takes 29gb size, most of all it is uploads (Gridfs)
now it 10 hours after starting new server but nothing changes , users still enter for 15 minutes(login page said “Please wait”), message delay for 5-10 minutes

2

Hi, could you execute this and confirm the output is 3.6?

rocketchat-server.mongo --eval "printjson(db.adminCommand ({getParameter: 1, featureCompatibilityVersion: 1}))"|grep '"version" :'

3

Output:
“version” : “3.6”

4

and i run htop and seem that mongo processes get Red 15.0G VIRT and 13.5G RES and at this moment message delay is 1-2 minutes. this is faster that 2 hour ago))))))))

5

Your system seems to have enough memory, but it could be a limitation of the use of gridfs. Also these latest versions changed the default configuration for notifications, you think it could be related to that? if you disable most the default notify all messages do you see any changes?

6

No, all my notifications settings like in old server. Yesterday after two hour from my last post lags are almost gone. But still there are periodically present. And yesterday at the moment when the server stopped lagging, a bunch of processes appeared in htop.

htop
htop1845×1006 380 KB

And output of journalctl -f -u snap.rocketchat-server.rocketchat-mongo

-- Logs begin at Thu 2019-12-12 16:04:35 CET. --

Dec 20 15:45:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:45:57.361+0100 I COMMAND [conn18] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “SDGeQJtS84iB9q9wr” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853099, 3), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:3025 nreturned:1 reslen:32594 locks:{ Global: { acquireCount: { r: 6052 } }, Database: { acquireCount: { r: 3026 } }, Collection: { acquireCount: { r: 3026 } } } protocol:op_query 56452ms
Dec 20 15:45:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:45:57.362+0100 I COMMAND [conn20] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “cgEg7S7EK8ntsj8EK” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853101, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:2997 nreturned:1 reslen:198905 locks:{ Global: { acquireCount: { r: 5996 } }, Database: { acquireCount: { r: 2998 } }, Collection: { acquireCount: { r: 2998 } } } protocol:op_query 55449ms
Dec 20 15:45:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:45:57.362+0100 I COMMAND [conn17] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “2X5ZDuQuN4QCE7mFE” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853087, 5), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:3234 nreturned:1 reslen:179962 locks:{ Global: { acquireCount: { r: 6470 } }, Database: { acquireCount: { r: 3235 } }, Collection: { acquireCount: { r: 3235 } } } protocol:op_query 61960ms
Dec 20 15:45:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:45:57.363+0100 I COMMAND [conn14] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “tfGfndsiwGCALkNCB” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853101, 2), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:2972 nreturned:13 reslen:3271927 locks:{ Global: { acquireCount: { r: 5946 } }, Database: { acquireCount: { r: 2973 } }, Collection: { acquireCount: { r: 2973 } } } protocol:op_query 54273ms
Dec 20 15:46:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:46:57.523+0100 I COMMAND [conn17] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “waS7FYMdvykG3Gg7x” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853160, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:2897 nreturned:1 reslen:63855 locks:{ Global: { acquireCount: { r: 5796 } }, Database: { acquireCount: { r: 2898 } }, Collection: { acquireCount: { r: 2898 } } } protocol:op_query 51555ms
Dec 20 15:46:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:46:57.524+0100 I COMMAND [conn18] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “SDGeQJtS84iB9q9wr” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853157, 26), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:3128 nreturned:1 reslen:32594 locks:{ Global: { acquireCount: { r: 6258 } }, Database: { acquireCount: { r: 3129 } }, Collection: { acquireCount: { r: 3129 } } } protocol:op_query 59269ms
Dec 20 15:46:57 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:46:57.524+0100 I COMMAND [conn20] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “yQTBA6hNckNwfHZXQ” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853167, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:2707 nreturned:5 reslen:1168817 locks:{ Global: { acquireCount: { r: 5416 } }, Database: { acquireCount: { r: 2708 } }, Collection: { acquireCount: { r: 2708 } } } protocol:op_query 46175ms
Dec 20 15:48:42 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:48:42.520+0100 I COMMAND [conn17] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “SDGeQJtS84iB9q9wr” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853257, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:3193 nreturned:1 reslen:32594 locks:{ Global: { acquireCount: { r: 6388 } }, Database: { acquireCount: { r: 3194 } }, Collection: { acquireCount: { r: 3194 } } } protocol:op_query 60758ms
Dec 20 15:48:42 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:48:42.520+0100 I COMMAND [conn14] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “udWW9fBMNQQdQ9rFn” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853267, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:2921 nreturned:1 reslen:57938 locks:{ Global: { acquireCount: { r: 5844 } }, Database: { acquireCount: { r: 2922 } }, Collection: { acquireCount: { r: 2922 } } } protocol:op_query 52307ms
Dec 20 15:48:42 Ubuntu-1804-bionic-64-minimal rocketchat-server.rocketchat-mongo[7106]: 2019-12-20T15:48:42.520+0100 I COMMAND [conn20] command parties.rocketchat_uploads.chunks command: find { find: “rocketchat_uploads.chunks”, filter: { files_id: “SDGeQJtS84iB9q9wr” }, sort: { n: 1 }, returnKey: false, showRecordId: false, $clusterTime: { clusterTime: Timestamp(1576853267, 6), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: “parties” } planSummary: COLLSCAN keysExamined:0 docsExamined:141888 hasSortStage:1 cursorExhausted:1 numYields:2969 nreturned:1 reslen:32594 locks:{ Global: { acquireCount: { r: 5940 } }, Database: { acquireCount: { r: 2970 } }, Collection: { acquireCount: { r: 2970 } } } protocol:op_query 53792ms

and these connections just swap places…