Roadmap Reveal 🗺 What's next for Rocket.Chat, Sep 22

Messages are greyed out

Description

Chat messages appear greyed out when trying to send new messages.

The only workaround I have found is to restart the server.

Do note that the rocket chat server runs in Azure as an app service while Mongo DB instance is in Atlas externally.

Server Setup Information

  • Version of Rocket.Chat Server: 3.15.0
  • Operating System: Ubuntu
  • Deployment Method: docker (Azure App service)
  • Number of Running Instances: 1
  • DB Replicaset Oplog: (oplog Enabled)
  • NodeJS Version: v12.22.1
  • MongoDB Version: External MongoDB instance in Atlas (4.4.6)
  • Proxy:
  • Firewalls involved:

I would imagine that you have connection issues somewhere.

What do your logs tell you?

Hey John
Thanks for the reply.

Chat works perfectly fine most of the time. This happens only when the chat server was inactive for some days as per my observation. I’ll get you the logs once I face the issue again.

I was wondering if I have to enable any specific settings when mongoDB is external as this never occurred when running mongoDB along with the chat in the same machine.

it shouldn’t matter - localhost vs someotherhost - it can’t tell where they are!

However, the further away the more likely something gets in the way.

Greyed out messages are usually indicators of some sort of connectivity issue. You might want to try and monitor that.

Error logs as follows


I20210707-08:52:51.724(0) Exception in callback of async function: TypeError: Cannot read property 'name' of null     at ReplSetState.remove (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset_state.js:162:27)     at Object.handler [as cb] (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset.js:1218:30)     at Connection.flush (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:209:18)     at connectionFailureHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:260:10)     at Connection.Pool._connectionCloseHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:179:5)     at Connection.emit (events.js:314:20)     at Connection.EventEmitter.emit (domain.js:483:12)     at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:438:12)     at Object.onceWrapper (events.js:421:26)     at TLSSocket.emit (events.js:314:20)     at TLSSocket.EventEmitter.emit (domain.js:483:12)     at net.js:675:12     at TCP.done (_tls_wrap.js:568:7)     at TCP.callbackTrampoline (internal/async_hooks.js:126:14) 
I20210707-08:52:51.750(0) Exception in setInterval callback: MongoNetworkError: connection 6 to test1234-shard-00-01.yphjv.mongodb.net:27017 closed     at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:440:9)     at Object.onceWrapper (events.js:421:26)     at TLSSocket.emit (events.js:314:20)     at TLSSocket.EventEmitter.emit (domain.js:483:12)     at net.js:675:12     at TCP.done (_tls_wrap.js:568:7)     at TCP.callbackTrampoline (internal/async_hooks.js:126:14) {   [Symbol(errorLabels)]: Set { 'RetryableWriteError' } } 
I20210707-12:21:18.649(0) Exception in setInterval callback: MongoError: not master     at Connection.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61)     at Connection.emit (events.js:314:20)     at Connection.EventEmitter.emit (domain.js:483:12)     at processMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10)     at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15)     at TLSSocket.emit (events.js:314:20)     at TLSSocket.EventEmitter.emit (domain.js:483:12)     at addChunk (_stream_readable.js:297:12)     at readableAddChunk (_stream_readable.js:272:9)     at TLSSocket.Readable.push (_stream_readable.js:213:10)     at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)     at TLSWrap.callbackTrampoline (internal/async_hooks.js:126:14) {   topologyVersion: { processId: 60e59c3282d880a648425e17, counter: 4 },   operationTime: Timestamp { _bsontype: 'Timestamp', low_: 4, high_: 1625647890 },   ok: 0,   code: 10107,   codeName: 'NotWritablePrimary',   '$clusterTime': {     clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 7, high_: 1625660477 },     signature: { hash: [Binary], keyId: [Long] }   } } 
I20210707-12:21:18.654(0) Exception in setInterval callback: MongoError: not master     at Connection.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61)     at Connection.emit (events.js:314:20)     at Connection.EventEmitter.emit (domain.js:483:12)     at processMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10)     at TLSSocket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15)     at TLSSocket.emit (events.js:314:20)     at TLSSocket.EventEmitter.emit (domain.js:483:12)     at addChunk (_stream_readable.js:297:12)     at readableAddChunk (_stream_readable.js:272:9)     at TLSSocket.Readable.push (_stream_readable.js:213:10)     at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)     at TLSWrap.callbackTrampoline (internal/async_hooks.js:126:14) {   topologyVersion: { processId: 60e59c3282d880a648425e17, counter: 4 },   operationTime: Timestamp { _bsontype: 'Timestamp', low_: 4, high_: 1625647890 },   ok: 0,   code: 10107,   codeName: 'NotWritablePrimary',   '$clusterTime': {     clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 7, high_: 1625660477 },     signature: { hash: [Binary], keyId: [Long] }   } } 
I20210712-13:25:06.814(0) Exception while invoking method saveSettings Error: TOTP Required [totp-required]     at checkCodeForUser (app/2fa/server/code/index.ts:186:9)     at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:31:4)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210712-13:31:39.231(0) Exception while invoking method saveSettings Error: TOTP Required [totp-required]     at checkCodeForUser (app/2fa/server/code/index.ts:186:9)     at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:31:4)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210712-18:33:43.686(0) Exception while invoking method saveSettings Error: TOTP Required [totp-required]     at checkCodeForUser (app/2fa/server/code/index.ts:186:9)     at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:31:4)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210714-07:19:08.540(0) Exception while invoking method login Error: User not found [403]     at AccountsServer.Accounts._runLoginHandlers (app/lib/server/lib/loginErrorMessageOverride.js:10:18)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:557:31)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9

Hmmm…

Cannot read property ‘name’ of null

Exception in setInterval callback: MongoNetworkError: connection 6 to test1234-shard-00-01.yphjv.mongodb.net:27017 closed at TLSSocket
Set { ‘RetryableWriteError’ }

MongoError: not master at Connection
codeName: ‘NotWritablePrimary’

I would suggest you have a look at your network and Mongo conections.

Error: TOTP Required [totp-required]
Exception while invoking method login Error: User not found

You might also want to look at your users!

Thanks for reply @john.crisp

Since the beginning, that has been my suspicion too. As of now Mongo is sitting in an external network.
Next plan is to associate a Mongo DB instance within the same network to see if that resolves the issue.

Certainly worth trying for starters. let us know how you get along.

Hello guys, I’m having the same issue here…

The default users can only send messages inside some channel or group, but direct/private messages are not being sent. The app was build with Snap in a Digital Ocean droplet on Ubuntu 18.04.

I applied the steps present here Snaps - Rocket.Chat Docs (I nano a config.env file and pasted USE_NATIVE_OPLOG = true inside it).

Please don’t crash in on threads.

Unless you are running exactly the same system then, despite appearances, you probably have a different issue.

Start a new thread with your server setup information, hardware, and proxy/firewall info too.

Hi,
I had the same issue for a while. searched a lot but no useful response found. I went through bellow:

either with mongo shell or compass find lard collections or the collections that has the most documents in it. for me it was the rocketchat_message_read_reciept thas has 24.4 GB of space and about 109M records in it!.

for alternate check, the gray-out messages while sending them, usually occur at the start of the day, when a bunch of employees start to work and precisely, it occurs when groups/channels which including the most of the messages in them or the groups or channels with most members in it.

you can disable this feature in Administration --> message --> Show Read Receipts and switch it off.

or also, check the DB cpu utilizaion with htop or sar command or also, run atop 1 to find if database disk face IOwait.

here is my problem and more info for you. Hope it help.