Push Notifications Stop Working After a Few Hours

Community Support
1

Description

I have read threw the forums here on how to get push notifications to work. They do work every time I do a reboot of the docker (In Unraid) but only for a few hours. After the 2 hours or so and I use the test push notification it says it sent to 3 devices but I dont get anything.

Server Setup Information

  • Version of Rocket.Chat Server: 4.5.0
  • Operating System: Unraid docker app
  • Deployment Method: Docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog:
  • NodeJS Version: v14.18.3
  • MongoDB Version: 5.0.6
  • Proxy: Nginx Proxy Manager (Also in Docker)
  • Firewalls involved: Pfsense

Any additional Information

Here is the error log I get when trying to use the test push notification

I20220320-17:51:29.062(-4) push_test {   id: 'PK75tr8zu8zQhPKng',   clientAddress: '192.168.50.1',   httpHeaders: {     host: 'chat.fusionit.net',     'x-forwarded-scheme': 'https',     'x-forwarded-proto': 'https',     'x-forwarded-for': '192.168.50.1',     'x-real-ip': '192.168.50.1',     'content-length': '87',     accept: '*/*',     'x-auth-token': 'xb7c46rkJ17Vl6PYarzY8PzgmuwrG_wNuahOIgeL8uf',     'x-requested-with': 'XMLHttpRequest',     'user-agent': 'Mozilla/5.0 (Linux; Android 12; SM-S906U1 Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/99.0.4844.73 Mobile Safari/537.36',     'x-user-id': 'XjtyvAQmaicrmpj52',     'content-type': 'application/json',     origin: 'https://chat.fusionit.net',     'sec-fetch-site': 'same-origin',     'sec-fetch-mode': 'cors',     'sec-fetch-dest': 'empty',     referer: 'https://chat.fusionit.net/admin/Push',     'accept-encoding': 'gzip, deflate',     'accept-language': 'en-US,en;q=0.9',     cookie: 'rc_uid=XjtyvAQmaicrmpj52; rc_token=xb7c46rkJ17Vl6PYarzY8PzgmuwrG_wNuahOIgeL8uf'   },   userId: 'XjtyvAQmaicrmpj52' } 
{"level":35,"time":"2022-03-20T21:51:29.063Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Meteor","method":"push_test","userId":"XjtyvAQmaicrmpj52","userAgent":"Mozilla/5.0 (Linux; Android 12; SM-S906U1 Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/99.0.4844.73 Mobile Safari/537.36","referer":"https://chat.fusionit.net/admin/Push","remoteIP":"192.168.50.1","instanceId":"2Sk7rGfMvPnyERSXi"} 
{"level":20,"time":"2022-03-20T21:51:29.070Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"Sending notification {\"createdAt\":\"2022-03-20T21:51:29.069Z\",\"createdBy\":\"<SERVER>\",\"sent\":false,\"sending\":0,\"from\":\"push\",\"title\":\"@Ryan-Stocks\",\"text\":\"This is a push test message\",\"userId\":\"XjtyvAQmaicrmpj52\",\"sound\":\"default\",\"apn\":{\"text\":\"@Ryan-Stocks:\<br>This is a push test message\"}}"} 
{"level":20,"time":"2022-03-20T21:51:29.070Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send message \"@Ryan-Stocks\" to userId 'XjtyvAQmaicrmpj52'"} 
{"level":20,"time":"2022-03-20T21:51:29.071Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send to token {\"gcm\":\"eXIPUGyXQhO2kmT3HmFKNo:APA91bHqQcLtqndz1r_WjPlSGsNemM6gqBv0v_kZxcj0CISYJB19rgRY4iLdc8NOFuGMxmzHFtHhLTjUBVFxQuZDuApn2AHd6zM6NuA0wJ9DdZJjgVf372NVLFxG4_C7Fy30c7xzjwJu\"}"} 
{"level":20,"time":"2022-03-20T21:51:29.072Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send to token {\"gcm\":\"eXIPUGyXQhO2kmT3HmFKNo:APA91bHqQcLtqndz1r_WjPlSGsNemM6gqBv0v_kZxcj0CISYJB19rgRY4iLdc8NOFuGMxmzHFtHhLTjUBVFxQuZDuApn2AHd6zM6NuA0wJ9DdZJjgVf372NVLFxG4_C7Fy30c7xzjwJu\"}"} 
{"level":20,"time":"2022-03-20T21:51:29.079Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send to token {\"gcm\":\"eG7xSFaZSK-WmUIrTFLoiz:APA91bGdpCQ87CKbCgnbR-64Q0eAnUYn1B2TBuaEMIfZmquFy2-ExPFFJdJ-cZbodsuHIkt9GiG4gKIjywqEvyaUP7Ljdv8EUvNUG7KwFOFKGJnLc9aI0R_lsHtG8Y-wEM2IeS9jD1EA\"}"} 
{"level":20,"time":"2022-03-20T21:51:29.080Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send to token {\"gcm\":\"eG7xSFaZSK-WmUIrTFLoiz:APA91bGdpCQ87CKbCgnbR-64Q0eAnUYn1B2TBuaEMIfZmquFy2-ExPFFJdJ-cZbodsuHIkt9GiG4gKIjywqEvyaUP7Ljdv8EUvNUG7KwFOFKGJnLc9aI0R_lsHtG8Y-wEM2IeS9jD1EA\"}"} 
{"level":20,"time":"2022-03-20T21:51:29.085Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send to token {\"gcm\":\"euSMre_8R0qwYyJJx9YdGf:APA91bFLnPgn8wPKYTcgRts_6tjWd6OMIuIeGhlfOulv9U3hsb-RCRfcyzda_OA5ArSOJxcO7-glKnSmx36fDxMbATPMg8U48OujdTRghXvvOkNU10cxJsAYa_4ToRpPKfGnUC0XXstL\"}"} 
{"level":20,"time":"2022-03-20T21:51:29.085Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"send to token {\"gcm\":\"euSMre_8R0qwYyJJx9YdGf:APA91bFLnPgn8wPKYTcgRts_6tjWd6OMIuIeGhlfOulv9U3hsb-RCRfcyzda_OA5ArSOJxcO7-glKnSmx36fDxMbATPMg8U48OujdTRghXvvOkNU10cxJsAYa_4ToRpPKfGnUC0XXstL\"}"} 
{"level":20,"time":"2022-03-20T21:51:29.090Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"Sent message \"@Ryan-Stocks\" to 0 ios apps 3 android apps"} 
{"level":20,"time":"2022-03-20T21:51:29.091Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"GUIDE: The \"appTokensCollection\" - No APN clients have registered on the server yet..."} 
{"level":35,"time":"2022-03-20T21:51:29.092Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"API","method":"POST","url":"/api/v1/method.call/push_test","userId":"XjtyvAQmaicrmpj52","userAgent":"Mozilla/5.0 (Linux; Android 12; SM-S906U1 Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/99.0.4844.73 Mobile Safari/537.36","length":"87","host":"chat.fusionit.net","referer":"https://chat.fusionit.net/admin/Push","remoteIP":"192.168.50.1","status":200,"responseTime":31} 
{"level":40,"time":"2022-03-20T21:51:29.190Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"Error sending push to gateway (not authorized) {\"statusCode\":401,\"content\":\"{\\\"code\\\":121,\\\"error\\\":\\\"rocket.chat cloud registration required\\\",\\\"requestId\\\":\\\"a1797b11-6c9e-49ae-a0db-15b630a90f20\\\",\\\"status\\\":401}\",\"headers\":{\"access-control-allow-headers\":\"Content-Type, Authorization, Content-Length, X-Requested-With\",\"access-control-allow-methods\":\"GET, PUT, POST, DELETE, OPTIONS\",\"access-control-allow-origin\":\"*\",\"access-control-expose-headers\":\"Content-Type, Authorization, Cache-Control, Expires, Pragma, X-powered-by\",\"cache-control\":\"private, no-cache, no-store, must-revalidate\",\"connection\":\"close\",\"content-length\":\"126\",\"content-type\":\"application/json; charset=utf-8\",\"date\":\"Sun, 20 Mar 2022 21:51:29 GMT\",\"expires\":\"-1\",\"pragma\":\"no-cache\",\"vary\":\"Accept-Encoding\",\"x-powered-by\":\"Rocket Fuel and Rocketeers\"},\"ok\":false,\"data\":{\"code\":121,\"error\":\"rocket.chat cloud registration required\",\"requestId\":\"a1797b11-6c9e-49ae-a0db-15b630a90f20\",\"status\":401}}"} 
{"level":40,"time":"2022-03-20T21:51:29.221Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"Error sending push to gateway (not authorized) {\"statusCode\":401,\"content\":\"{\\\"code\\\":121,\\\"error\\\":\\\"rocket.chat cloud registration required\\\",\\\"requestId\\\":\\\"a7fcab69-2b30-4571-b12e-266852e70978\\\",\\\"status\\\":401}\",\"headers\":{\"access-control-allow-headers\":\"Content-Type, Authorization, Content-Length, X-Requested-With\",\"access-control-allow-methods\":\"GET, PUT, POST, DELETE, OPTIONS\",\"access-control-allow-origin\":\"*\",\"access-control-expose-headers\":\"Content-Type, Authorization, Cache-Control, Expires, Pragma, X-powered-by\",\"cache-control\":\"private, no-cache, no-store, must-revalidate\",\"connection\":\"close\",\"content-length\":\"126\",\"content-type\":\"application/json; charset=utf-8\",\"date\":\"Sun, 20 Mar 2022 21:51:29 GMT\",\"expires\":\"-1\",\"pragma\":\"no-cache\",\"vary\":\"Accept-Encoding\",\"x-powered-by\":\"Rocket Fuel and Rocketeers\"},\"ok\":false,\"data\":{\"code\":121,\"error\":\"rocket.chat cloud registration required\",\"requestId\":\"a7fcab69-2b30-4571-b12e-266852e70978\",\"status\":401}}"} 
{"level":40,"time":"2022-03-20T21:51:29.226Z","pid":1,"hostname":"Stocks-Unraid","name":"","name":"Push","msg":"Error sending push to gateway (not authorized) {\"statusCode\":401,\"content\":\"{\\\"code\\\":121,\\\"error\\\":\\\"rocket.chat cloud registration required\\\",\\\"requestId\\\":\\\"7109db6a-b973-40c0-985b-68a14dc3c98c\\\",\\\"status\\\":401}\",\"headers\":{\"access-control-allow-headers\":\"Content-Type, Authorization, Content-Length, X-Requested-With\",\"access-control-allow-methods\":\"GET, PUT, POST, DELETE, OPTIONS\",\"access-control-allow-origin\":\"*\",\"access-control-expose-headers\":\"Content-Type, Authorization, Cache-Control, Expires, Pragma, X-powered-by\",\"cache-control\":\"private, no-cache, no-store, must-revalidate\",\"connection\":\"close\",\"content-length\":\"126\",\"content-type\":\"application/json; charset=utf-8\",\"date\":\"Sun, 20 Mar 2022 21:51:29 GMT\",\"expires\":\"-1\",\"pragma\":\"no-cache\",\"vary\":\"Accept-Encoding\",\"x-powered-by\":\"Rocket Fuel and Rocketeers\"},\"ok\":false,\"data\":{\"code\":121,\"error\":\"rocket.chat cloud registration required\",\"requestId\":\"7109db6a-b973-40c0-985b-68a14dc3c98c\",\"status\":401}}"}
2

Im still having this problem. Not getting notifications is quite anoying and I dont want to restart the docker every few hours just to get them to work.

Anyone…?

3

Hi!

I asked internally about this. Can you please:

  • Try syncing again and checking for any outstanding logs?
  • How is the authentication with you mongo setup? We have reports on some scenarios that it can break oplog/changestream tailing/monitoring.
1 Like
4

Hi! I have almost the same issue.

  • For me it does not generate any error logs when I sync again (also shows as successful in the dashboard).
  • for me I authenticate via URI mongodb://rocketchat:PASSWORD@HOST:27017/rocketchat and for Oplog mongodb://root:ROOTPASSWORD@HOST:27017/local?replicaSet=rs0&authSource=admin

The only difference for me is that I get a slightly different error message (as in I never get the Send to token thing). The error appears almost exactly two hours after restarting the server.

{"level":40,"time":"2022-09-09T12:27:06.539Z","pid":1,"hostname":"rocketchat-prod-rocketchat-6445dbfd8b-2bvq7","name":"Push","msg":"Error sending push to gateway (not authorized) {\"statusCode\":401,\"content\":\"{\\\"code\\\":121,\\\"error\\\":\\\"rocket.chat cloud registration required\\\",\\\"requestId\\\":\\\"6cac0fbe-9028-42de-ba0d-c7c6ae7db235\\\",\\\"status\\\":401}\",\"headers\":{\"access-control-allow-headers\":\"Content-Type, Authorization, Content-Length, X-Requested-With\",\"access-control-allow-methods\":\"GET, PUT, POST, DELETE, OPTIONS\",\"access-control-allow-origin\":\"*\",\"access-control-expose-headers\":\"Content-Type, Authorization, Cache-Control, Expires, Pragma, X-powered-by\",\"cache-control\":\"private, no-cache, no-store, must-revalidate\",\"connection\":\"close\",\"content-length\":\"126\",\"content-type\":\"application/json; charset=utf-8\",\"date\":\"Fri, 09 Sep 2022 12:27:06 GMT\",\"expires\":\"-1\",\"pragma\":\"no-cache\",\"vary\":\"Accept-Encoding\",\"x-powered-by\":\"Rocket Fuel and Rocketeers\"},\"ok\":false,\"data\":{\"code\":121,\"error\":\"rocket.chat cloud registration required\",\"requestId\":\"6cac0fbe-9028-42de-ba0d-c7c6ae7db235\",\"status\":401}}"}
{"level":50,"time":"2022-09-09T12:28:26.636Z","pid":1,"hostname":"rocketchat-prod-rocketchat-6445dbfd8b-2bvq7","name":"System","err":{"type":"Error","message":"failed [401]{\"status\":401}","stack":"Error: failed [401]{\"status\":401}\n    at makeErrorByStatus (server/lib/http/call.ts:59:9)\n    at server/lib/http/call.ts:168:19\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","response":{"statusCode":401,"content":"{\"status\":401}","headers":{"access-control-allow-headers":"Content-Type, Authorization, Content-Length, Last-Event-ID, X-Requested-With","access-control-allow-methods":"GET, PUT, POST, DELETE, OPTIONS","access-control-allow-origin":"*","access-control-expose-headers":"Content-Type, Authorization, Cache-Control, Expires, Pragma, X-powered-by","cache-control":"private, no-cache, no-store, must-revalidate","connection":"close","content-length":"14","content-type":"application/json; charset=utf-8","date":"Fri, 09 Sep 2022 12:28:26 GMT","expires":"-1","pragma":"no-cache","vary":"Accept-Encoding","x-fleet-version":"-77b34ba","x-powered-by":"Rocket Fuel and Rocketeers"},"ok":false,"data":{"status":401}}},"msg":"failed [401]{\"status\":401}"}
5

Hmm maybe the connection is renewed by cronjobs… I have the feeling that cronjobs may (no longer) work on my system. I will check that out tonight (UTC) and report back.

1 Like
6

Hm I couldn’t really get it resolved. But I still can’t really say if it has to do with cronjobs or not (will have to do some more digging, any pointers are greatly apreciated of course :slight_smile: )

7

I could never get this working properly so I ended up going with Mattermost. Even that though had a little finagling to get the notifications to work…but in the end it did work with Matermost.

1 Like
8

I am 90% sure I just managed to get it to work again.
The reason was, that changeStream was not working (I never noticed the error in the logs due to some other spam).

 Change Stream is available for your installation, give admin permissions to your database user to use this improved version

It happened because I transfered to another Mongo Instance and only copied the rocketchat DB over (and not the admin) and then just created new users in the new db, forgetting to give the rocket user special rights on the admin DB.

So in short this is what fixed it: Adds clusterMonitor role to doc. by pierreozoux · Pull Request #1853 · RocketChat/docs · GitHub

9

Now I am sure, this fixed my issue

1 Like
10

Glad to hear that, @stanner !

Thanks for sharing!