Failure to reach cloud.rocket.chat servers from OVH instance

Community Support
1

Description

I have been using my registered starter instance which is deployed on OVH servers for a few months. Since April, I lost access to the marketplace, push notifications and I started getting a warning that my server’s workspace version is outdated (I am using latest 6.7.0). I am aware that this latter problem will be solved in the upcomming release. However, this is a solution for air-gapped systems, while mine is connected to the internet.

I tried unregistering my instance and since then I have not been able to re-register it except via the offline method. I have not touched my default iptables firewall config in months and I do not suspect my outgoing https traffic would be filtered by my own server.

It seems to me that cloud.rocket.chat API started blocking my server via its firewall or some blacklist. I wonder if anyone else has experienced this phenomenon recently, as I have not been able to find similar experiences on the forums.

Server Setup Information

  • Version of Rocket.Chat Server: 6.7.0
  • Operating System: Debian 10
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 14.21.3 - x64
  • MongoDB Version: 5.0.18
  • Proxy: apache
  • Firewalls involved: iptables

Any additional Information

Latest logs when visiting the marketplace or attempting to register the instance via token or email:

{"level":50,"time":"2024-04-21T09:04:16.415Z","pid":1,"hostname":"a66f73449a59","name":"Rocket.Chat Apps","msg":"Error getting the app requests stats from marketplace 'The user aborted a request.'"}
{"level":40,"time":"2024-04-21T09:06:08.180Z","pid":1,"hostname":"a66f73449a59","name":"Rocket.Chat Apps","msg":"Unable to access Marketplace. Does the server has access to the internet? 'The user aborted a request.'"}

Other logs relating to connection issues:

{"level":40,"time":"2024-04-21T08:53:53.326Z","pid":1,"hostname":"a66f73449a59","name":"SyncedCron","msg":"Failed to send usage report"}
2

What version were you running before?

How many users?

What licence type is shown in your cloud panel?

3

I was running 6.5.0 then updated to 6.6.6 then 6.7.0 on its release day.
My instance contains 20 users and the licence type shown on the panel is Starter Seats with an Active status.

4

Not seen anyone blocked.

(Note - there is a very good reason to not use ‘latest’ in your compose file. ALWAYS set a specific version and upgrade when you are ready and have tested - I personally would never, ever, upgrade on day 1 of a new release)

What actually changed when you got blocked/lost connection? Was it an upgrade or did something else happen? Did you change anything on your server? Exactly when did it occur?

5

Okay, I get it that being an early adopter of new releases poses a risk on production systems.

Looking back at the backups, the update to v6.6.5 was on 22/03/2024. I haven’t noticed when push notifications stopped working but according to the push notification usage stats (below), they have not worked on April. Only this week we started getting the “unsupported version” warning relating to 6.5.0 which we moved away from way earlier this year.

Other than the version update, there were no other changes on the server. Is there a way for me to check if the API is accessible to me from the server? An http request sent via curl for example? Just to locate the problem (RocketChat version issue vs networking issue).

rc notifications
rc notifications1062×321 12.2 KB

6

IIRC there is a bug open on Unsupported version.

However, not this bug first and check your server connectivity.

If that fails then open a thread in open.rocket.chat #support and I can ping the appropriate person to take a look. (You can’t DM me but I will be there)

7

Hello, @youcef!

We are sorry that you are having this problem.
Please run the command below on your database so the message can go away:
db.rocketchat_settings.remove( {"_id": "Cloud_Workspace_Supported_Versions_Token"} )

Let us know if it doesn’t work.
Best regards, Bárbara Zanella.

1 Like
8

@reetp @barbara.zanella after testing, both fixes, DB command or Update to 6.7.1, solve the issue of the unsupported version as I stated in the original post. However, my online system is still considered offline or air-gapped due to connectivity problems to RocketChat Cloud.

I will proceed by opening a thread on open.rocket.chat #support as recommended

Best regards

9

Someone will take a look over there in due course - please be patient.

10

Hey there! I know this thread is already a year old, but unfortunately I’m going through pretty much the same thing right now on 7.0.0. After hours of trial and error and searching the web for solutions I’m giving up and hoping for your help!

I was running our nonprofit community rocket.chat server in community version happily for many years and now like two weeks after updating to 7.0.0 we suddenly went into read-only mode and got the message, that this is an air-gapped workspace and we should connect it to the internet or update to a premium plan to restore full functionality. Both wouldn’t work, because the workspace doesn’t connect to the Rocket.Chat Cloud anymore. After some research I found out, that I should delete the existing workspace registration in the Rocket Chat Cloud first and then re-register it. That didn’t work either.

At first I thought it doesn’t work because it showed that it is still registered (not showing a registration button). Since I couldn’t get rid of the registration, I deleted the entry in the mongodb with db.rocketchat_settings.remove( {"id": /Cloud/} ). After that I got the “Register workspace” button back, but unfortunately that didn’t help either because the registration would stuck - with “use token” saying “Token not recognized”, by cloud account mail being stuck in step 1. It feels like Rocket.Chat Cloud blocked our server.

Please, how can we get rid of air-gapped read-only mode and get our community version re-registered? Thanks a lot in advance!

Server Setup Information

  • Version of Rocket.Chat Server: 7.0.4
  • Operating System: Ubuntu 22.04.5 LTS
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 20.17.0
  • MongoDB Version: 6.0.20
  • Proxy: nginx
  • Firewalls involved: iptables

Rocket.Chat Logs Excerpts

“Failed to Connect with Rocket.Chat Cloud”,“url”:“/api/oauth/clients”,“err”:{“type”:“AbortError”,“message”:“The user aborted a request.”,“stack”:“AbortError: The user aborted a request.\n at abort (/app/bundle/programs/server/npm/node_modules/node-fetch/lib/index.js:1458:16)\n at AbortSignal.abortAndFinalize (/app/bundle/programs/server/npm/node_modules/node-fetch/lib/index.js:1473:4)\n at AbortSignal.dispatchEvent (/app/bundle/programs/server/npm/node_modules/event-target-shim/src/event-target.mjs:337:35)\n at abortSignal (/app/bundle/programs/server/npm/node_modules/abort-controller/src/abort-signal.ts:68:12)\n at AbortController.abort (/app/bundle/programs/server/npm/node_modules/abort-controller/src/abort-controller.ts:26:9)\n at Timeout. (/app/bundle/programs/server/npm/node_modules/@rocket.chat/server-fetch/src/index.ts:42:48)\n at listOnTimeout (node:internal/timers:581:17)\n at processTimers (node:internal/timers:519:7)”,“name”:“AbortError”}}

“Failed to communicate with Rocket.Chat Cloud”,“url”:"https://releases.rocket.chat/v2/server/supportedVersions",“err”:{“type”:“FetchError”,“message”:"request to https://releases.rocket.chat/v2/server/supportedVersions failed, reason: ",“stack”:“FetchError: request to https://releases.rocket.chat/v2/server/supportedVersions failed, reason: \n at ClientRequest. (/app/bundle/programs/server/npm/node_modules/node-fetch/lib/index.js:1501:11)\n at ClientRequest.emit (node:events:519:28)\n at ClientRequest.emit (node:domain:488:12)\n at emitErrorEvent (node:_http_client:108:11)\n at TLSSocket.socketErrorListener (node:_http_client:511:5)\n at TLSSocket.emit (node:events:519:28)\n at TLSSocket.emit (node:domain:488:12)\n at emitErrorNT (node:internal/streams/destroy:169:8)\n at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n at processTicksAndRejections (node:internal/process/task_queues:82:21)”,“errno”:“ETIMEDOUT”,“code”:“ETIMEDOUT”,“name”:“FetchError”}}

“Failed to get Checkout URL with Rocket.Chat Billing Service”,“url”:“/api/v2/checkout”,“err”:{“type”:“CloudWorkspaceAccessTokenEmptyError”,“message”:“Workspace access token is empty”,“stack”:“Error: Workspace access token is empty\n at getWorkspaceAccessTokenOrThrow (app/cloud/server/functions/getWorkspaceAccessToken.ts:82:9)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at getCheckoutUrl (app/cloud/server/functions/getCheckoutUrl.ts:17:17)\n at ee/server/requestSeatsRoute.ts:28:71”}}

11

Chasing old posts is never great.

Scatter gun comments everywhere trying to get attention just makes work for me.

One place please.

And:

https://docs.rocket.chat/docs/rocketchat-release-notes#release-700

  • Rocket.Chat Community Edition no longer supports air-gapped environments (environments without internet access). To ensure compliance with licensing agreements, only users on premium plans—including the free Starter plan—can run Rocket.Chat in these air-gapped setups. Our standard builds include commercial code, so restricting air-gapped environments to premium users helps maintain security and proper licensing compliance. Community users who need a fully open-source setup can use the fossify script to remove all premium code, allowing them to build their own offline solutions.
12

Hello there. My problem was never solved since I am still using my OVH instance, upgraded to v 7.0.0. For 10 months, my Starter Plan instance cannot reach Rocket.Chat Cloud servers in any way. I never changed anything in my configs, and it happened suddenly on April 2024.
My instance was read-only for a few hours due to the air-gapped limitation, until I copy pasted the licence from my RocketChat Cloud account. But the main issue, of it not being air-gapped in the first place, still remains unsolved.

13

Can you please update your server information including your licence and number of users. Too much time has passed.

Can you also shell into your container and check you can access this URL

docker exec -it <mycontainer> bash
wget https://releases.rocket.chat/v2/server/supportedVersions

That is the URL the server checks. If you can’t get that you need to fix your networking.

If you do get a page of info then let us know - we will then need:

Deployment ID
Cloud Workspace ID

Thanks.

14

Here is the new server setup info.

Version of Rocket.Chat Server: 7.0.0
Operating System: Debian 10
Deployment Method: docker
Number of Running Instances: 1
DB Replicaset Oplog: Enabled
NodeJS Version: v20.17.0 - x64
MongoDB Version: 5.0.24
Proxy: apache
Firewalls involved: iptables

This is the output of the wget command inside docker container:

Connecting to releases.rocket.chat (172.67.42.51:443)
wget: can't connect to remote host (172.67.42.51): Operation timed out

The command works outside docker though. This means I got to check my docker setup?

15

Note that I can wget other stuff from the internet from inside the container. So it seems, the problem is only when connecting to rocket.chat servers from the container.

16

I’d start by fnding the answer to that then.

DNS seems to be answering as you get an IP address.

If you can’t get that URL your server can’t check itself.

Firewalling/reverse proxy (you are running a reverse proxy aren’t you??) somewhere at a guess.

Check iptables logs?

17

Okay, I found out that docker was messing with the firewall with all its DOCKER-USER, DOCKER, DOCKER-ISOLATION-STAGE-* iptables chains. I confirmed this by running the rocketchat container in host mode, then by flushing those chains using iptables -F <CHAIN> and switching back to bridge mode.

I added "iptables": false to my /etc/docker/daemon.json config file and restarted docker service in order to prevent it from blocking anything behind my back.

I am glad this problem is resolved, and that the problem is not related to my cloud provider nor rocketchat servers.

Best Regards

18

Yup - I hate docker for doing this.

Yup - first thing I do when I install docker. It is utterly useless.

I also tell it which DNS server to use - in my case my server handles DNS itself so I point docker at localhost to stop it deciding to go off piste and say use Google or other.

Me too :slight_smile:

Well done for persevering. So often ‘issues’ are just configuration.

It’s just a case of finding out exactly what bit needs a tweak, and I suspect there are others with similar issues.

eg

1 Like