Cannot connect to Rocket.Chat from external

Description:

I am having issues getting connected to my new Rocket.Chat server from external sources. It works internally without any issue. It should be working with SSL Reverse Proxy and I have tried both nginx SSL Reverse Proxyas well as Apache SSL Reverse Proxy. Everything appears to be set properly within my firewall configuration.

If I run tcpdump -i eth0 port 443 and then try to hit it from outside, it does show the server being hit. This SHOULD indicate the firewall is accepting the connection through. My firewall is a pfSense firewall.

I have also disabled the local firewall on the server as well as disabled SELinux.

I have gone through countless forums, and suggestions and have not been able to resolve this particular issue. Does anyone out there have any suggestions for me?

Server Setup Information:

  • Version of Rocket.Chat Server: 0.74.3
  • Operating System: Centos7
  • Deployment Method: Followed instructions here
  • Number of Running Instances: 1
  • NodeJS Version: v8.16.0
  • MongoDB Version: v3.6.12

Additional context

Here are my relevant config files:

nginx.conf

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  ************.com;
#        root         /usr/share/nginx/html;

#       ssl_certificate "/etc/nginx/certificate.crt";
#       ssl_certificate_key "/etc/nginx/certificate.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;

#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;

#        location / {
#        }

#        error_page 404 /404.html;
#            location = /40x.html {
#        }

#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

default.conf

# Upstreams
upstream backend {
    server 127.0.0.1:3000;
}

# HTTPS Server
server {
    listen 443 ssl;
    server_name ***********.com

    # You can increase the limit if your need to.
    client_max_body_size 200M;

    error_log /var/log/nginx/rocketchat.access.log;

    ssl on;
    ssl_certificate /etc/nginx/certificate.crt;
    ssl_certificate_key /etc/nginx/certificate.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don’t use SSLv3 ref: POODLE

    location / {
        proxy_pass http://backend/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forward-Proto http;
        proxy_set_header X-Nginx-Proxy true;

        proxy_redirect off;
    }
}

rocketchat.service

[Unit]
Description=The Rocket.Chat server
After=network.target remote-fs.target nss-lookup.target nginx.target mongod.target mongod.service
[Service]
ExecStart=/usr/local/bin/node /opt/Rocket.Chat/main.js
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=rocketchat
User=rocketchat
Environment=LD_PRELOAD=/opt/Rocket.Chat/programs/server/npm/node_modules/sharp/vendor/lib/libz.so NODE_ENV=production MONGO_URL=mongodb://localhost:27017/rocketchat ROOT_URL=https://***********.com PORT=3000
[Install]
WantedBy=multi-user.target

Relevant logs:

View Logs

e[34mI20190424-09:04:58.331(-7) LocalStore: store created at  
e[34mI20190424-09:04:58.389(-7) LocalStore: store created at  
e[34mI20190424-09:04:58.391(-7) LocalStore: store created at  
e[34mI20190424-09:04:58.648(-7) Updating process.env.MAIL_URL 
e[34mI20190424-09:04:58.892(-7) Setting default file store to GridFS 
e[34mI20190424-09:04:59.134(-7) Starting Email Intercepter... 
e[34mI20190424-09:05:29.890(-7) (migrations.js:121) Migrations: Not migrating, already at version 137
e[34mI20190424-09:05:31.024(-7) Updating process.env.MAIL_URL 
e[34mI20190424-09:05:36.694(-7) Using GridFS for custom sounds storage 
e[34mI20190424-09:05:36.757(-7) Using GridFS for custom emoji storage 
e[34mI20190424-09:06:01.124(-7) ➔ System ➔ startup 

e[34mI20190424-09:06:01.125(-7) ➔ +----------------------------------------------------------+ 
e[34mI20190424-09:06:01.126(-7) ➔ |                      SERVER RUNNING                      | 
e[34mI20190424-09:06:01.127(-7) ➔ +----------------------------------------------------------+ 
e[34mI20190424-09:06:01.128(-7) ➔ |                                                          | 
e[34mI20190424-09:06:01.128(-7) ➔ |  Rocket.Chat Version: 0.74.3                             | 
e[34mI20190424-09:06:01.129(-7) ➔ |       NodeJS Version: 8.11.3 - x64                       | 
e[34mI20190424-09:06:01.130(-7) ➔ |             Platform: linux                              | 
e[34mI20190424-09:06:01.131(-7) ➔ |         Process Port: 3000                               | 
e[34mI20190424-09:06:01.131(-7) ➔ |             Site URL: https://************.com  | 
e[34mI20190424-09:06:01.132(-7) ➔ |     ReplicaSet OpLog: Disabled                           | 
e[34mI20190424-09:06:01.133(-7) ➔ |          Commit Hash: 202a465f1c                         | 
e[34mI20190424-09:06:01.134(-7) ➔ |        Commit Branch: HEAD                               | 
e[34mI20190424-09:06:01.134(-7) ➔ |                                                          | 
e[34mI20190424-09:06:01.135(-7) ➔ +----------------------------------------------------------+

So Rocket seems up fine then.

I’d suggest you start with a MUCH simpler conf file and keep your firewalls off or the ports temporarily open until you have it running.

Check the docs for an example eg.

https://rocket.chat/docs/installation/manual-installation/configuring-ssl-reverse-proxy/#running-behind-a-nginx-ssl-reverse-proxy

K.I.S.S. You can always add complexity later :slight_smile:

Thanks so much for the reply. The nginx.conf file is the default setup after nginx is installed. The default.conf (also listed above) is set up as instructed in the link you provided.

Note the comments on the CentOS setup page you ignored

“Rocket.Chat recommends …and NodeJS version 8.11.3”

That probably should be 8.11.4 now, but you do need to follow things exactly.

Nginx. Move ALL the existing stuff out the way for testing and keep it as simple as possible.

(On my mobile so hard to see with the clutter)

Just use a REALLY basic setup. On my test nginx rig it doesn’t even run on 80.

Can you connect locally to Rocket via 443 behind pfsense ?? Or only with 3000?

In the server running config information (bottom of my initial post) it shows the server running configuration and lists NodeJS Version: 8.11.3 - x64

Yes i can connect locally via 443 behind the pfsense, only external does not seem to make it through.

I believe it is running just as basic as can be right now, i will double check the config files to see if I missed something but its basically just the default.conf file that Rocketchat specifies and the nginx.conf file is left as is when first installed via ‘yum install nginx -y’

Thanks!

Sorry but your original posted Server info said:

NodeJS Version: v8.16.0

:slight_smile:

So it looks like your issue is pfsense (or whatever is coming OUT of your nginx proxy to pfense) and not Rocket?

Can you see what pfsense is up too? Must be some logging in there (I know it is fiercesomely tricky)

Ya pfSense was my original thought but it seems to be set up correctly. Other services are able to pass through (such as an offsite VoIP device) and thus it should all be ok. The system firewall logs do not show anything of note being blocked.

As mentioned in my original post, the tcpdump does also show a connection hitting the rocketchat server, it just doesn’t serve anything up, (ERR_CONNECTION_TIMED_OUT). So I am not quite sure where to go from here. I’ve also rebuilt the RocketChat server from scratch using Ubuntu (twice) and have had the same issue come up. Works internally but not externally. I must just be missing something incredibly small and simple but for the life of me, I just can’t find it.

Hmmm.

Remember when testing, first Rocket wants to use websockets. So not sure on exact responses. Test from inside and outside.

Next you have also tried Ubuntu with the same issues (I presume exact same)

Logically there is a common thread here…

Yes, other services may be perfect. But this is a DIFFERENT service.

If you can connect on port 3000 and/or 443 locally there is only one place to look.

So what does pfsense do with 80/443 ?

Maybe try turning Rocket.Chat off. And in its place use something like netcat to see if its even getting to the host.

For example something like this:

echo "booooo" | nc -l 3000

Should get that text in browser as well as see something like this from netcat

GET / HTTP/1.1
Host: localhost:3000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: <removed>
Upgrade-Insecure-Requests: 1
1 Like