Switching to air-gapped mode after Pro Trial ends despite having access to cloud.rocket.chat

Description

Good day RC forum
Today I encountered a problem, the solution to which I did not find in this topic

and not in the others

The point is that on my workspace it gave out about the limitation of push notifications (because the limit was exceeded by 10 thousand, but there was no such volume in the previous month and years of use, and I have been using RC in the community version for about 1.5-2 years)
Okay, let’s say this has nothing to do with it
But after that it gave out “This air-gapped workspace is read-only mode”
I restarted the docker, restarted the system - nothing helped
I climbed through the forums - tried to do wget releases.rocket.chat/v2/server/supportedVersions - and this command was successfully executed
I also noticed that
If I delete my space from cloud.rocket.chat - the license will fly off and on the workspace
Okay, I do the license synchronization - drop down in the space registration menu - successfully pass it, get the Pro Trial - and the error with air-gapped goes away, but if I end the subscription - after 1 minute air-gapped returns.

That is, the server sees cloud.rocket.chat, downloads releases.rocket.chat/v2/server/supportedVersions, activates the new workspace, but for some reason switching to the community version - it does air-gapped
What information do I need to provide to get help?

Server Setup Information

  • Version of Rocket.Chat Server: 7.5.1
  • Operating System: Oracle Linux Server 8.6
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog:
  • NodeJS Version: 22.13.1 - x64
  • MongoDB Version: 5.0.10
  • Proxy: nginx 1.23.1
  • Firewalls involved:

Any additional Information

1 Like

Number of users.
Logs.

Hello!

Total 628
Active 394
avg online ~60
I know about the 200 online users limit for community subscriptions

Today I tried to reproduce this situation

Because after activating the workspace, I am on pro trial, I cancelled it in my personal account on the cloud
Everything went ok
I went to my rocket.chat, synchronized the license - my workplace became with a community subscription
restarted the docker - it became air-grapped again and in read-only mode

Here are the logs with the chronology

I am on the community license and restart the docker and after that the system is in read-only mode

“level”:50,“time”:“2025-04-29T16:18:49.291Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“SyncedCron”,“msg”:“Failed to send usage report”,“err”:{“type”:“FetchError”,“message”:“invalid json response body at /collector.rocket.chat/ reason: Unexpected end of JSON input”,“stack”:“FetchError: invalid json response body at collector.rocket.chat/ reason: Unexpected end of JSON input
at /app/bundle/programs/server/npm/node_modules/node-fetch/lib/index.js:273:32
at processTicksAndRejections (node:internal/process/task_queues:105:5)
at sendStats (app/statistics/server/functions/sendUsageReport.ts:25:26)
at usageReportCron (server/cron/usageReport.ts:9:21)
at async Promise.all (index 2)
at startCronJobs (server/startup/cron.ts:15:2)”,“name”:“FetchError”}}
{“level”:20,“time”:“2025-04-29T16:18:49.986Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“Callbacks”,“msg”:“Executing callback with id Qm2QT3xDRmCiGf7BT for hook workspaceLicenseRemoved”}
{“level”:30,“time”:“2025-04-29T16:18:49.987Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“System”,“msg”:“Synced with Rocket.Chat Cloud”,“function”:“syncCloudData”}
{“level”:20,“time”:“2025-04-29T16:18:49.988Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“System”,“msg”:“Resetting cached value Cloud_Workspace_Supported_Versions_Token in settings”}
{“level”:20,“time”:“2025-04-29T16:18:49.990Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“System”,“msg”:“Workspace credentials cache hit using scope: workspace:license:read workspace:client:write workspace:stats:write workspace:push:send marketplace:read marketplace:download fedhub:register. Avoiding generating a new access token from cloud services.”}

Now I reactivate my workspace and receive a trial subscription

{“level”:30,“time”:“2025-04-29T16:18:59.460Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“License”,“msg”:“Limit validation failed”,“kind”:“privateApps”,“limit”:{“behavior”:“prevent_action”,“max”:0}}
{“level”:20,“time”:“2025-04-29T16:19:03.183Z”,“pid”:1,“hostname”:“7373a426fccd”,“name”:“System”,“msg”:“Workspace credentials cache miss using scope: workspace:license:read workspace:client:write workspace:stats:write workspace:push:send marketplace:read marketplace:download fedhub:register, fetching new access token from cloud services.”}

since there are a lot of logs, it won’t fit here, but I highlighted what I think is appropriate

i.e. in fact, after restarting the docker - rocket.chat seems to not see the Internet and forced synchronization of the license does not help
but after I delete my workspace from the cloud - rocket.chat will tell me in a friendly manner “Register me”
Synchronization is happening, there is Internet

And this is the text that the neural network issued

Start of server operation (16:18:48)
Invalid host (host validation error)

The server detects that it is already at migration version 318 (Not migrating, already at version 318)

  1. Attempting to synchronize with the cloud (16:18:48–16:18:49)
    16:18:48.830 – License cache found (Workspace credentials cache hit…), but server still tries to request a new token.

16:18:48.875 – Setting up queues and workers.

16:18:49.080 – First license failure:

json
{
“level”:20,
“msg”:“Workspace credentials cache miss using scope: workspace:license:read… fetching new access token from cloud services.”
}
The server cannot obtain an access token for cloud services.

16:18:49.291 – Error sending report to Rocket.Chat Cloud:

json
{
“level”:50,
“msg”:“Failed to send usage report”,
“err”: {
“type”:“FetchError”,
“message”:“invalid json response body at collector.rocket.chat/ reason: Unexpected end of JSON input”
}
}
The server cannot send statistics, but continues to work.

  1. Successful synchronization after activating the Pro trial (16:18:49–16:18:50)
    16:18:49.986 – The workspaceLicenseRemoved callback is triggered (possibly due to subscription cancellation).

16:18:49.987 – Successful sync with cloud:

json
{
“level”:30,
“msg”:“Synced with Rocket.Chat Cloud”,
“function”:“syncCloudData”
}
16:18:49.988 – Resetting cached versions (Resetting cached value Cloud_Workspace_Supported_Versions_Token).

16:18:49.990 – New successful license cache (Workspace credentials cache hit…).

  1. Problems after restart (16:18:50–16:19:03)
    16:18:50.767 – room_is_blocked error when sending messages (possibly due to read-only mode).

16:18:59.953–16:19:03.183 – Repeated licensing errors:

The server is losing access to the cloud again (Workspace credentials cache miss…).

Returning to a state where it cannot verify the license.

What are the options for diagnosing/solving the problem boss?

Hello,

  • Community plan HAS a limit od 10k push/month
  • Check firewall configuration if our domains are blocked/whitelisted: Firewall Configuration
  • If our portal can’t be reached Community edition will be read-only as if it were air-gapped.
  • Behavior when deleting and creating a new register on our portal is expected, it will go to Pro trial after push limit is reached
  • Community plan HAS a limit od 10k push/month

but after this limit is exceeded, they simply become unavailable or the entire service switches to a read-only system?

did it, but still, what is the license sync time? 2 min? I do a forced sync after disabling the sublist in cloud.rocket.chat - and everything is fine
but when I reboot the docker - read-only service
and my mini note
When I turn off the service - the favicon of my service becomes gray, but when it is launched - it becomes the one I have

I ended the subscription and restarted docker (you can see the icon is grey)
photo_2025-04-29_22-30-20

and the system loaded, but here it is active, and in mine it is in read-only mode

NB for reference there is no user limit.

There is a “presence” limit of 200 user after which you can’t see online status.

Yes, yes, here it only talks about 200 simultaneous online users

Today the push notifications banner went away, but after rebooting it went back to read-only mode, but a notification also came
But I didn’t quite understand, what do you mean for 7 days?)))

How does the registration/activation of a new workplace work?
Using the Internet and a connection to the cloud, right?

I see this in the logs
That is, the license was applied, it is not offline and this means that the connection to the cloud was successful

{“level”:20,“time”:“2025-04-30T21:09:40.420Z”,“pid”:1,“hostname”:“HOSTNAME”,“name”:“System”,“msg”:“Workspace credentials cache miss using scope: … fetching new access token from cloud services.”}
{“level”:20,“time”:“2025-04-30T21:09:41.029Z”,“pid”:1,“hostname”:“HOSTNAME”,“name”:“Callbacks”,“msg”:“Executing callback with id ABC123 for hook workspaceLicenseChanged”}
{“level”:30,“time”:“2025-04-30T21:09:41.036Z”,“pid”:1,“hostname”:“HOSTNAME”,“name”:“License”,“msg”:“New Enterprise License”}
{“level”:20,“time”:“2025-04-30T21:09:41.037Z”,“pid”:1,“hostname”:“HOSTNAME”,“name”:“License”,“msg”:“license”,“decrypted”:{
“version”:“3.0”,
“information”:{
“id”:“LICENSE_ID”,
“autoRenew”:true,
“createdAt”:“2025-04-30T21:09:40.674Z”,
“visualExpiration”:“2025-05-30T21:09:40.669Z”,
“notifyAdminsAt”:“2025-05-30T21:09:40.669Z”,
“notifyUsersAt”:“2025-05-30T21:09:40.669Z”,
“trial”:true,
“cancellable”:true,
“offline”:false,
“grantedBy”:{},
“grantedTo”:{},
“tags”:[{“name”:“Pro trial”,“color”:“#F3BE08”}]
},
“validation”:{
“serverUrls”:[{“value”:“example.com”,“type”:“url”}],
“cloudWorkspaceId”:“CLOUD_WS_ID”,
“serverUniqueId”:“SERVER_UNIQUE_ID”,
“validPeriods”:[{“validUntil”:“2025-06-13T21:09:40.669Z”,“invalidBehavior”:“invalidate_license”}],
“legalTextAgreement”:{“type”:“required”,“acceptedVia”:“cloud”},
“statisticsReport”:{“required”:true,“allowedStaleInDays”:2}
},
“grantedModules”:[
{“module”:“canned-responses”},
{“module”:“ldap-enterprise”},
… (остальные модули без изменений)
],
“limits”:{
“activeUsers”:[{“max”:500,“behavior”:“start_fair_policy”},{“max”:501,“behavior”:“prevent_action”}],
… (остальные лимиты без изменений)
},
“cloudMeta”:{}
}}

But when I cancel the subscription, the GUI also crashes and says that in 7 days the system will go into read-only mode (although this has never happened before)
In relation to Russian users, were any blocks made?))

Successful license cancellation:

json
{“level”:35,“method”:“POST”,“url”:“/v1/cloud.removeLicense”,“status”:200,“responseTime”:1233}
Status 200 confirms that the license removal request has been completed.

Checking the license status after cancellation:

json
{“level”:35,“method”:“GET”,“url”:“/v1/licenses.info”,“status”:200,“responseTime”:6}
The system is requesting up-to-date license information.

Notification about read-only mode:

json
{“msg”:“Your air-gapped workspace will enter read-only mode in 7 days…”}
The message is sent via push services to users.

Cloud dependency:

json
{“msg”:“Workspace credentials cache hit using scope: … Avoiding generating a new access token from cloud services.”}

The log confirms that the system continues to interact with the cloud to verify the license (offline: false).

is the problem really that the service doesn’t see the internet?

and it turns out that the services are available!