Some Rocket.Chat channels not getting ELK Stack alerts

Community Support
1

Description

I have set-up ElastAlert2 (an alerting service for ELK Stack logs) to send alerts to several Rocket.Chat channels. Recently however some channels don’t get alerts when they should. For example zimbra last got a alert at 3:58 AM but it is now 9:52AM (The zimbra channel should get alerts every few minutes or so). Checking docker logs reveals some errors which I will post below.

Server Setup Information

  • Version of Rocket.Chat Server: 6.9.0
  • Operating System: CentOS 7 Linux VM
  • Deployment Method: docker compose
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Not sure
  • NodeJS Version: Not sure
  • MongoDB Version: Not sure but this is my configuration in the docker-compose.yml
    image: docker.io/bitnami/mongodb:${MONGODB_VERSION:-5.0}
  • Proxy:
  • Firewalls involved: SonicWall

Any additional Information

These seem to be the relevant logs from my Rocket.Chat container:

Errors like this can cause oplog processing errors.

Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process

Future node.js versions will automatically exit the process

=================================

=== UnHandledPromiseRejection ===

errorClass [Error]: Error, too many requests. Please slow down. You must wait 10 seconds before trying this endpoint again. [error-too-many-requests]

at app/api/server/api.ts:347:10

at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {

isClientSafe: true,

error: 'error-too-many-requests',

reason: 'Error, too many requests. Please slow down. You must wait 10 seconds before trying this endpoint again.',

details: { timeToReset: 9020, seconds: 10 },

errorType: 'Meteor.Error'

}

---------------------------------

Errors like this can cause oplog processing errors.

Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process

Future node.js versions will automatically exit the process

=================================

{"level":50,"time":"2024-07-04T12:00:20.646Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"Error during fallback workspace sync","err":{"type":"AbortError","message":"The user aborted a request.","stack":"AbortError: The user aborted a request.\n at abort (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1401:16)\n at AbortSignal.abortAndFinalize (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1416:4)\n at AbortSignal.dispatchEvent (/app/bundle/programs/server/npm/node_modules/event-target-shim/dist/event-target-shim.js:818:35)\n at abortSignal (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:52:12)\n at AbortController.abort (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:91:9)\n at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/dist/index.js:37:51)\n at listOnTimeout (internal/timers.js:557:17)\n at processTimers (internal/timers.js:500:7)\n => awaited here:\n at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n at app/cloud/server/functions/syncWorkspace/legacySyncWorkspace.ts:95:16\n at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40\n => awaited here:\n at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n at app/cloud/server/functions/syncWorkspace/legacySyncWorkspace.ts:160:15\n at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40\n => awaited here:\n at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n at app/cloud/server/functions/syncWorkspace/index.ts:35:6\n at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","name":"AbortError"}}

{"level":50,"time":"2024-07-04T12:00:25.853Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"Failed to communicate with Rocket.Chat Cloud","url":"https://releases.rocket.chat/v2/server/supportedVersions","err":{"type":"AbortError","message":"The user aborted a request.","stack":"AbortError: The user aborted a request.\n at abort (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1401:16)\n at AbortSignal.abortAndFinalize (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1416:4)\n at AbortSignal.dispatchEvent (/app/bundle/programs/server/npm/node_modules/event-target-shim/dist/event-target-shim.js:818:35)\n at abortSignal (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:52:12)\n at AbortController.abort (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:91:9)\n at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/dist/index.js:37:51)\n at listOnTimeout (internal/timers.js:557:17)\n at processTimers (internal/timers.js:500:7)\n => awaited here:\n at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n at app/cloud/server/functions/supportedVersionsToken/supportedVersionsToken.ts:45:17\n at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","name":"AbortError"}}

{"level":50,"time":"2024-07-04T12:00:25.869Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"An error occurred syncing workspace. 'The user aborted a request.'"}

{"level":50,"time":"2024-07-05T00:00:25.803Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"Failed to communicate with Rocket.Chat Cloud","url":"https://releases.rocket.chat/v2/server/supportedVersions","err":{"type":"AbortError","message":"The user aborted a request.","stack":"AbortError: The user aborted a request.\n at abort (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1401:16)\n at AbortSignal.abortAndFinalize (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1416:4)\n at AbortSignal.dispatchEvent (/app/bundle/programs/server/npm/node_modules/event-target-shim/dist/event-target-shim.js:818:35)\n at abortSignal (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:52:12)\n at AbortController.abort (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:91:9)\n at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/dist/index.js:37:51)\n at listOnTimeout (internal/timers.js:557:17)\n at processTimers (internal/timers.js:500:7)\n => awaited here:\n at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n at app/cloud/server/functions/supportedVersionsToken/supportedVersionsToken.ts:45:17\n at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","name":"AbortError"}}

{"level":40,"time":"2024-07-05T03:12:20.915Z","pid":1,"hostname":"a831cb6f09f0","name":"SyncedCron","msg":"Failed to send usage report"}

{"level":50,"time":"2024-07-05T10:30:23.263Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"No real time data received recently"}

{"level":50,"time":"2024-07-05T10:30:45.869Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"No real time data received recently"}

{"level":50,"time":"2024-07-05T11:45:17.157Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"No real time data received recently"}

{"level":50,"time":"2024-07-05T12:00:22.686Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"Failed to communicate with Rocket.Chat Cloud","url":"https://releases.rocket.chat/v2/server/supportedVersions","err":{"type":"AbortError","message":"The user aborted a request.","stack":"AbortError: The user aborted a request.\n at abort (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1401:16)\n at AbortSignal.abortAndFinalize (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/node_modules/node-fetch/lib/index.js:1416:4)\n at AbortSignal.dispatchEvent (/app/bundle/programs/server/npm/node_modules/event-target-shim/dist/event-target-shim.js:818:35)\n at abortSignal (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:52:12)\n at AbortController.abort (/app/bundle/programs/server/npm/node_modules/abort-controller/dist/abort-controller.js:91:9)\n at Timeout.<anonymous> (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/dist/index.js:37:51)\n at listOnTimeout (internal/timers.js:557:17)\n at processTimers (internal/timers.js:500:7)\n => awaited here:\n at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n at app/cloud/server/functions/supportedVersionsToken/supportedVersionsToken.ts:45:17\n at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","name":"AbortError"}}

{"level":50,"time":"2024-07-05T19:08:13.182Z","pid":1,"hostname":"a831cb6f09f0","name":"Rocket.Chat Apps","msg":"Error getting the app requests stats from marketplace 'request to https://marketplace.rocket.chat/v1/app-request/stats failed, reason: getaddrinfo ENOTFOUND marketplace.rocket.chat'"}

{"level":50,"time":"2024-07-05T19:13:38.233Z","pid":1,"hostname":"a831cb6f09f0","name":"Rocket.Chat Apps","msg":"Error getting the app requests stats from marketplace 'request to https://marketplace.rocket.chat/v1/app-request/stats failed, reason: getaddrinfo ENOTFOUND marketplace.rocket.chat'"}

[root@ELK-Stack rocketchat]#

2

reason: ‘Error, too many requests. Please slow down. You must wait 10 seconds before trying this endpoint again.’,

Check your rate limiting.

:"Error getting the app requests stats from marketplace 'request to https://marketplace.rocket.chat/v1/app-request/stats failed, reason: getaddrinfo ENOTFOUND marketplace.rocket.chat

Etc

I’d suggest checking your networking.

1 Like
3

I also seem to now be getting this error which may be related:

{"level":50,"time":"2024-07-09T15:15:41.726Z","pid":1,"hostname":"a831cb6f09f0","name":"System","msg":"Error sending notification","err":{"type":"PoolClearedOnNetworkError","message":"Connection to mongodb:27017 interrupted due to server monitor timeout","stack":"PoolClearedOnNetworkError: Connection to mongodb:27017 interrupted due to server monitor timeout\n    at ConnectionPool.interruptInUseConnections (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:264:36)\n    at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:251:41\n    at processTicksAndRejections (internal/process/task_queues.js:77:11)\n    at runNextTicks (internal/process/task_queues.js:64:3)\n    at processTimers (internal/timers.js:497:9)\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at server/models/raw/NotificationQueue.ts:74:15\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at app/notification-queue/server/NotificationQueue.ts:55:21\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at app/notification-queue/server/NotificationQueue.ts:46:5\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","address":"mongodb:27017"}}
=== UnHandledPromiseRejection ===
PoolClearedOnNetworkError: Connection to mongodb:27017 interrupted due to server monitor timeout
    at ConnectionPool.interruptInUseConnections (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:264:36)
    at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:251:41
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
    at runNextTicks (internal/process/task_queues.js:64:3)
    at processTimers (internal/timers.js:497:9) {
  address: 'mongodb:27017',
  [Symbol(errorLabels)]: Set(1) { 'RetryableWriteError' }
}
---------------------------------
Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================
=== UnHandledPromiseRejection ===
PoolClearedOnNetworkError: Connection to mongodb:27017 interrupted due to server monitor timeout
    at ConnectionPool.interruptInUseConnections (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:264:36)
    at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:251:41
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
    at runNextTicks (internal/process/task_queues.js:64:3)
    at processTimers (internal/timers.js:497:9) {
  address: 'mongodb:27017',
  [Symbol(errorLabels)]: Set(1) { 'RetryableWriteError' }
}
---------------------------------
Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================

Any advice?

4

It is too vague to really know.

Show us your compose file.

Got to Workspace/Administration and show us the deployment details.

I suspect there is a lot more going on here than you are telling us.

5

Here’s my compose file:

volumes:
  mongodb_data: { driver: local }

services:
  rocketchat:
    image: registry.rocket.chat/rocketchat/rocket.chat:${RELEASE:-latest}
    restart: always
    labels:
      traefik.enable: "true"
      traefik.http.routers.rocketchat.rule: Host(`${DOMAIN:-}`)
      traefik.http.routers.rocketchat.tls: "true"
      traefik.http.routers.rocketchat.entrypoints: https
      traefik.http.routers.rocketchat.tls.certresolver: le
    environment:
      MONGO_URL: "${MONGO_URL:-\
        mongodb://${MONGODB_ADVERTISED_HOSTNAME:-mongodb}:${MONGODB_INITIAL_PRIMARY_PORT_NUMBER:-27017}/\
        ${MONGODB_DATABASE:-rocketchat}?replicaSet=${MONGODB_REPLICA_SET_NAME:-rs0}}"
      MONGO_OPLOG_URL: "${MONGO_OPLOG_URL:\
        -mongodb://${MONGODB_ADVERTISED_HOSTNAME:-mongodb}:${MONGODB_INITIAL_PRIMARY_PORT_NUMBER:-27017}/\
        local?replicaSet=${MONGODB_REPLICA_SET_NAME:-rs0}}"
      ROOT_URL: ${ROOT_URL:-http://localhost:${HOST_PORT:-3000}}
      # ROOT_URL: http://terrarium.uhtasi.local:3000
      PORT: ${PORT:-3000}
      DEPLOY_METHOD: docker
      DEPLOY_PLATFORM: ${DEPLOY_PLATFORM:-}
      REG_TOKEN: ${REG_TOKEN:-}
    depends_on:
      - mongodb
    expose:
      - ${PORT:-3000}
    ports:
      - "${BIND_IP:-0.0.0.0}:${HOST_PORT:-3000}:${PORT:-3000}"
    logging:
      driver: "json-file"
      options:
        max-size: "50m"

  mongodb:
    image: docker.io/bitnami/mongodb:${MONGODB_VERSION:-5.0}
    restart: always
    volumes:
      - mongodb_data:/bitnami/mongodb
    environment:
      MONGODB_REPLICA_SET_MODE: primary
      MONGODB_REPLICA_SET_NAME: ${MONGODB_REPLICA_SET_NAME:-rs0}
      MONGODB_PORT_NUMBER: ${MONGODB_PORT_NUMBER:-27017}
      MONGODB_INITIAL_PRIMARY_HOST: ${MONGODB_INITIAL_PRIMARY_HOST:-mongodb}
      MONGODB_INITIAL_PRIMARY_PORT_NUMBER: ${MONGODB_INITIAL_PRIMARY_PORT_NUMBER:-27017}
      MONGODB_ADVERTISED_HOSTNAME: ${MONGODB_ADVERTISED_HOSTNAME:-mongodb}
      MONGODB_ENABLE_JOURNAL: ${MONGODB_ENABLE_JOURNAL:-true}
      ALLOW_EMPTY_PASSWORD: ${ALLOW_EMPTY_PASSWORD:-yes}
    logging:
      driver: "json-file"
      options:
        max-size: "50m"

Actually I noticed this error come up after I updated. So maybe the issue has something to do with the update? I updated through my .env file. Then followed the instructions on the Rocket.Chat website.