Connectivity Services has been unreachable

Community Support
1

Description

Without any reasons stopped working push, market app. In logs I see errors.
My instance works behind http-proxy. Suddenly it stopped using proxy-server and try send requests directly.

Server Setup Information

  • Version of Rocket.Chat Server: 4.4.0
  • Operating System:
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: on
  • NodeJS Version: 14.18.1
  • MongoDB Version: 4.0.28
  • Proxy: haproxy

Any additional Information

{"level":50,"time":"2022-02-02T15:09:03.978Z","pid":9,"hostname":"1fcd4251f4cf","name":"","name":"System","err":{"type":"FetchError","message":"request to https://cloud.rocket.chat/api/oauth/token failed, reason: getaddrinfo ENOTFOUND cloud.rocket.chat","stack":"FetchError: request to https://cloud.rocket.chat/api/oauth/token failed, reason: getaddrinfo ENOTFOUND cloud.rocket.chat<br>    at ClientRequest.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/fetch/node_modules/node-fetch/lib/index.js:1444:11)<br>    at ClientRequest.emit (events.js:400:28)<br>    at ClientRequest.emit (domain.js:475:12)<br>    at TLSSocket.socketErrorListener (_http_client.js:475:9)<br>    at TLSSocket.emit (events.js:400:28)<br>    at TLSSocket.emit (domain.js:475:12)<br>    at emitErrorNT (internal/streams/destroy.js:106:8)<br>    at emitErrorCloseNT (internal/streams/destroy.js:74:3)<br>    at processTicksAndRejections (internal/process/task_queues.js:82:21)","errno":"ENOTFOUND","code":"ENOTFOUND","name":"FetchError"},"msg":"request to https://cloud.rocket.chat/api/oauth/token failed, reason: getaddrinfo ENOTFOUND cloud.rocket.chat"} 
{"level":40,"time":"2022-02-02T15:09:14.068Z","pid":9,"hostname":"1fcd4251f4cf","name":"","name":"Rocket.Chat Apps","msg":"Unable to access Marketplace. Does the server has access to the internet? 'request to https://marketplace.rocket.chat/v1/apps failed, reason: getaddrinfo ENOTFOUND marketplace.rocket.chat'"}

Rocket.chat’s docker-compose part:

version: '3.1'

services:

  rocketchat:
      image: rocketchat/rocket.chat:4.4.0
      command: >
         bash -c
            "for i in `seq 1 30`; do
              node main.js &&
              s=$$? && break || s=$$?;
              echo \"Tried $$i times. Waiting 5 secs...\";
              sleep 5;
             done; (exit $$s)"
      volumes:
            - /var/data/rchat/uploads:/app/uploads
      environment:
         - PORT=3000
         #- INSTANCE_IP=rchat.example.loc
         - ROOT_URL=https://rchat.example.loc
         - MONGO_URL=mongodb://mongo_2:27017,mongo_3:27017,mongo_1:27017/rocketchat?replicaSet=rs0&readPreference=nearest&w=majority
         - MONGO_OPLOG_URL=mongodb://mongo_2:27017,mongo_3:27017,mongo_1:27017/local?replicaSet=rs0
         - HTTP_PROXY=http://a.b.c.d:3128
         - HTTPS_PROXY=http://a.b.c.d:3128
      ports:
         - 3000:3000
2

Hi!

We have seen some unexpected behaviors in 4.4.0 version due to an underlying meteor upgrade.

Did it started after the upgrade?

We have a milestone to deal with this issues:

3

That behavior I found out at 4.4.0 after update. According to log investigation problem has been appeared several hour earlier when instance worked at 4.3.2
Extra data:
inside rocket.chat HTTP(S)_PROXY environment was transfered from docker-compose

HTTPS_PROXY=http://a.b.c.d:3128
NODE_VERSION=14.18.2
HOSTNAME=1fcd4251f4cf
MONGO_URL=mongodb://mongo_2:27017,mongo_3:27017,mongo_1:27017/rocketchat?replicaSet=rs0&readPreference=nearest&w=majority
YARN_VERSION=1.22.15
PORT=3000
HOME=/tmp
MONGO_OPLOG_URL=mongodb://mongo_2:27017,mongo_3:27017,mongo_1:27017/local?replicaSet=rs0
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ROOT_URL=https://rchat.example.loc
Accounts_AvatarStorePath=/app/uploads
DEPLOY_METHOD=docker
PWD=/app/bundle
HTTP_PROXY=http://a.b.c.d:3128
NODE_ENV=production
4

Hi!

This is a known issue. 4.4.0 upgraded meteor which caused some unexpected behavior.

here you can keep track of what issues we have discovered so far: