Chat works perfectly fine most of the time. This happens only when the chat server was inactive for some days as per my observation. I’ll get you the logs once I face the issue again.
I was wondering if I have to enable any specific settings when mongoDB is external as this never occurred when running mongoDB along with the chat in the same machine.
I20210707-08:52:51.724(0) Exception in callback of async function: TypeError: Cannot read property 'name' of null at ReplSetState.remove (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset_state.js:162:27) at Object.handler [as cb] (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset.js:1218:30) at Connection.flush (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:209:18) at connectionFailureHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:260:10) at Connection.Pool._connectionCloseHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:179:5) at Connection.emit (events.js:314:20) at Connection.EventEmitter.emit (domain.js:483:12) at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:438:12) at Object.onceWrapper (events.js:421:26) at TLSSocket.emit (events.js:314:20) at TLSSocket.EventEmitter.emit (domain.js:483:12) at net.js:675:12 at TCP.done (_tls_wrap.js:568:7) at TCP.callbackTrampoline (internal/async_hooks.js:126:14)
I20210707-08:52:51.750(0) Exception in setInterval callback: MongoNetworkError: connection 6 to test1234-shard-00-01.yphjv.mongodb.net:27017 closed at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:440:9) at Object.onceWrapper (events.js:421:26) at TLSSocket.emit (events.js:314:20) at TLSSocket.EventEmitter.emit (domain.js:483:12) at net.js:675:12 at TCP.done (_tls_wrap.js:568:7) at TCP.callbackTrampoline (internal/async_hooks.js:126:14) { [Symbol(errorLabels)]: Set { 'RetryableWriteError' } }
I20210707-12:21:18.649(0) Exception in setInterval callback: MongoError: not master at Connection.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61) at Connection.emit (events.js:314:20) at Connection.EventEmitter.emit (domain.js:483:12) at processMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10) at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15) at TLSSocket.emit (events.js:314:20) at TLSSocket.EventEmitter.emit (domain.js:483:12) at addChunk (_stream_readable.js:297:12) at readableAddChunk (_stream_readable.js:272:9) at TLSSocket.Readable.push (_stream_readable.js:213:10) at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) at TLSWrap.callbackTrampoline (internal/async_hooks.js:126:14) { topologyVersion: { processId: 60e59c3282d880a648425e17, counter: 4 }, operationTime: Timestamp { _bsontype: 'Timestamp', low_: 4, high_: 1625647890 }, ok: 0, code: 10107, codeName: 'NotWritablePrimary', '$clusterTime': { clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 7, high_: 1625660477 }, signature: { hash: [Binary], keyId: [Long] } } }
I20210707-12:21:18.654(0) Exception in setInterval callback: MongoError: not master at Connection.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61) at Connection.emit (events.js:314:20) at Connection.EventEmitter.emit (domain.js:483:12) at processMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10) at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15) at TLSSocket.emit (events.js:314:20) at TLSSocket.EventEmitter.emit (domain.js:483:12) at addChunk (_stream_readable.js:297:12) at readableAddChunk (_stream_readable.js:272:9) at TLSSocket.Readable.push (_stream_readable.js:213:10) at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) at TLSWrap.callbackTrampoline (internal/async_hooks.js:126:14) { topologyVersion: { processId: 60e59c3282d880a648425e17, counter: 4 }, operationTime: Timestamp { _bsontype: 'Timestamp', low_: 4, high_: 1625647890 }, ok: 0, code: 10107, codeName: 'NotWritablePrimary', '$clusterTime': { clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 7, high_: 1625660477 }, signature: { hash: [Binary], keyId: [Long] } } }
I20210712-13:25:06.814(0) Exception while invoking method saveSettings Error: TOTP Required [totp-required] at checkCodeForUser (app/2fa/server/code/index.ts:186:9) at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:31:4) at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12) at packages/ddp-server/livedata_server.js:1689:15 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at packages/ddp-server/livedata_server.js:1687:36 at new Promise (<anonymous>) at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12) at Server.apply (packages/ddp-server/livedata_server.js:1625:26) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9 => awaited here: at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12) at Server.apply (packages/ddp-server/livedata_server.js:1638:22) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9
I20210712-13:31:39.231(0) Exception while invoking method saveSettings Error: TOTP Required [totp-required] at checkCodeForUser (app/2fa/server/code/index.ts:186:9) at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:31:4) at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12) at packages/ddp-server/livedata_server.js:1689:15 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at packages/ddp-server/livedata_server.js:1687:36 at new Promise (<anonymous>) at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12) at Server.apply (packages/ddp-server/livedata_server.js:1625:26) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9 => awaited here: at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12) at Server.apply (packages/ddp-server/livedata_server.js:1638:22) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9
I20210712-18:33:43.686(0) Exception while invoking method saveSettings Error: TOTP Required [totp-required] at checkCodeForUser (app/2fa/server/code/index.ts:186:9) at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:31:4) at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12) at packages/ddp-server/livedata_server.js:1689:15 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at packages/ddp-server/livedata_server.js:1687:36 at new Promise (<anonymous>) at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12) at Server.apply (packages/ddp-server/livedata_server.js:1625:26) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9 => awaited here: at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12) at Server.apply (packages/ddp-server/livedata_server.js:1638:22) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9
I20210714-07:19:08.540(0) Exception while invoking method login Error: User not found [403] at AccountsServer.Accounts._runLoginHandlers (app/lib/server/lib/loginErrorMessageOverride.js:10:18) at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:557:31) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12) at packages/ddp-server/livedata_server.js:1689:15 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at packages/ddp-server/livedata_server.js:1687:36 at new Promise (<anonymous>) at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12) at Server.apply (packages/ddp-server/livedata_server.js:1625:26) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9 => awaited here: at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12) at Server.apply (packages/ddp-server/livedata_server.js:1638:22) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:263:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9
Since the beginning, that has been my suspicion too. As of now Mongo is sitting in an external network.
Next plan is to associate a Mongo DB instance within the same network to see if that resolves the issue.
The default users can only send messages inside some channel or group, but direct/private messages are not being sent. The app was build with Snap in a Digital Ocean droplet on Ubuntu 18.04.
I applied the steps present here Snaps - Rocket.Chat Docs (I nano a config.env file and pasted USE_NATIVE_OPLOG = true inside it).
Hi,
I had the same issue for a while. searched a lot but no useful response found. I went through bellow:
either with mongo shell or compass find lard collections or the collections that has the most documents in it. for me it was the rocketchat_message_read_reciept thas has 24.4 GB of space and about 109M records in it!.
for alternate check, the gray-out messages while sending them, usually occur at the start of the day, when a bunch of employees start to work and precisely, it occurs when groups/channels which including the most of the messages in them or the groups or channels with most members in it.
you can disable this feature in Administration --> message --> Show Read Receipts and switch it off.
or also, check the DB cpu utilizaion with htop or sar command or also, run atop 1 to find if database disk face IOwait.