Mobile app use instantly crashes server. Everything "just works" in a web browser

Community Support
,
1

Description

Mobile app use instantly crashes server. Everything “just works” in a web browser.

Server Setup Information

  • Version of Rocket.Chat Server: 5.4.4
  • Operating System: Debian 11.6
  • Deployment Method: snap
  • Proxy: nginx
  • Firewalls involved: Ubiquiti edgerouter

Any additional Information

I’m setting up a second installation of Rocket.Chat for home use because I’ve enjoyed using it at work, and it seems like it would be a good “backup” to SMS as my wife and I are often in places where we have wi-fi but not cellular data.

Setup process: Installed Debian, apt install snapd, snap install rocketchat-server. Also tried with Ubuntu, selecting the rocketchat-server option during the OS install, but had the same result.

Host is a VM on Proxmox.

I’m having an issue with my Rocket.Chat server crashing immediately, and 100% reproducibly by logging in to it with the Android app, I haven’t tested the iOS app as I don’t have an iOS device. If I log in with a web browser, everything works fine.

I’ve used RC at work for about a year, and my work environment is mostly the same as my “at home” environment (a Proxmox host running an RC snap, reverse proxied by nginx, but the nginx proxy is behind a Cisco ASA instead of an edgerouter), and I’m trying to set up a RC server for family use.

My nginx proxy config file is identical on both installations, with the obvious exceptions of the hostname / IP I’m proxying. (eg. rc.work.tld vs rc.home.tld, being proxied to 10.0.0.20:3000 vs 192.168.1.20:3000)

My at work setup process was the same, I created a VM, installed the snap, and that was it, and it’s been working well for quite a while.

I’m not sure what logs I could look at to figure out why the server crashes when logging in with a mobile app, but doing anything on the mobile app “crashes” the Rocket.Chat snap, that might be the wrong word, but 100% of the time if I open the mobile app, the RC server can’t be reached on port 3000 without using the proxy or on 443 with the proxy.

If I’m logged in on a browser on one device (even as a different account) and open the mobile app, as soon as the app is open (even on a different network, for example over cellular data) the browser will show “Waiting for server connection, trying again in X seconds…” and stay like that forever until I close the mobile app. Once the mobile app is closed, the browser based interface comes back up in about 5-10 seconds. If I wasn’t logged in in a browser when opening the mobile app, I get the nginx 502 Bad Gateway error page that I would get if the server being proxied was “turned off”.

If I leave the mobile app open, I can connect to my server, log in, see the channel / user list (for about 1 second), but the top of the screen will then rapidly swap between “Waiting for network… / Updating… / Waiting for network… / Updating…”

I can happily post whatever logs might be useful to see why this is happening, but I don’t know how or where to find them.

2

Hi!! Welcome to our Community :hugs:

One thing you could do, to make sure they have the same SSL configurations:

run a ssl check, with a tool like SSL Server Test (Powered by Qualys SSL Labs) on both domains and compare their reports.

Rocket.Chat logs might also give us some information. For that, you go to Workspace > Settings > Logs, increase the log verbosity, and check them while you reproduce this issue.

You can both check the logs from within Rocket.Chat (Workspace > Logs) or thru terminal. here you have the doc for that:

Let me know if this helps.

Thanks!

3

I took a look at the logs when connecting from the mobile app, and I’m not sure if it looks like something that I can / know how to fix.

After restarting the snap, and then logging in from a mobile device, I get this: (slightly edited because I’m not sure what’s private and what’s just randomly generated strings)

2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: +--------------------------------------------------------+
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |                     SERVER RUNNING                     |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: +--------------------------------------------------------+
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |                                                        |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |  Rocket.Chat Version: 5.4.4                            |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |       NodeJS Version: 14.19.3 - x64                    |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |      MongoDB Version: 4.4.15                           |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |       MongoDB Engine: wiredTiger                       |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |             Platform: linux                            |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |         Process Port: 3000                             |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |             Site URL: https://MYHOSTNAME!!!!!!!!!!!!!  |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |     ReplicaSet OpLog: Enabled                          |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |          Commit Hash: abcde54321                       |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |        Commit Branch: HEAD                             |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: |                                                        |
2023-03-11T06:21:48Z rocketchat-server.rocketchat-server[3465]: +--------------------------------------------------------+
2023-03-11T06:22:00Z rocketchat-server.rocketchat-server[3465]: {"level":30,"time":"2023-03-11T06:22:00.082Z","pid":3465,"hostname":"chat","name":"SyncedCron","msg":"Starting \"Generate download files for user data\"."}
2023-03-11T06:22:00Z rocketchat-server.rocketchat-server[3465]: {"level":30,"time":"2023-03-11T06:22:00.083Z","pid":3465,"hostname":"chat","name":"SyncedCron","msg":"Finished \"Generate download files for user data\"."}
2023-03-11T06:22:12Z rocketchat-server.rocketchat-server[3465]: {"level":35,"time":"2023-03-11T06:22:12.526Z","pid":3465,"hostname":"chat","name":"Meteor","method":"UserPresence:away","userId":null,"userAgent":"RC Mobile; android 13; v4.36.0 (38053)","remoteIP":"192.168.1.1","instanceId":"qwerty12345"}
2023-03-11T06:22:12Z rocketchat-server.rocketchat-server[3465]: {"level":35,"time":"2023-03-11T06:22:12.529Z","pid":3465,"hostname":"chat","name":"Meteor","method":"UserPresence:online","userId":null,"userAgent":"RC Mobile; android 13; v4.36.0 (38053)","remoteIP":"192.168.1.1","instanceId":"qwerty12345"}
2023-03-11T06:22:13Z rocketchat-server.rocketchat-server[3397]: /snap/rocketchat-server/1540/helpers/rocketchat.sh: line 23:  3465 Illegal instruction     (core dumped) node $SNAP/main.js
2023-03-11T06:22:13Z rocketchat-server.rocketchat-server[3526]: pkill: pidfile not valid
2023-03-11T06:22:13Z rocketchat-server.rocketchat-server[3526]: Try `pkill --help' for more information.
2023-03-11T06:22:13Z systemd[1]: snap.rocketchat-server.rocketchat-server.service: Control process exited, code=exited, status=1/FAILURE
2023-03-11T06:22:13Z systemd[1]: snap.rocketchat-server.rocketchat-server.service: Failed with result 'exit-code'.
2023-03-11T06:22:13Z systemd[1]: snap.rocketchat-server.rocketchat-server.service: Consumed 52.940s CPU time.
2023-03-11T06:22:13Z systemd[1]: snap.rocketchat-server.rocketchat-server.service: Scheduled restart job, restart counter is at 1.
2023-03-11T06:22:13Z systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-server.
2023-03-11T06:22:13Z systemd[1]: snap.rocketchat-server.rocketchat-server.service: Consumed 52.940s CPU time.
2023-03-11T06:22:13Z systemd[1]: Started Service for snap application rocketchat-server.rocketchat-server.
2023-03-11T06:22:15Z rocketchat-server.rocketchat-server[3528]: node $SNAP/main.js
2023-03-11T06:22:34Z rocketchat-server.rocketchat-server[3596]: strict mode: use allowUnionTypes to allow union type keyword at "#/properties/value" (strictTypes)
2023-03-11T06:22:36Z rocketchat-server.rocketchat-server[3596]: LocalStore: store created at
2023-03-11T06:22:36Z rocketchat-server.rocketchat-server[3596]: LocalStore: store created at
2023-03-11T06:22:36Z rocketchat-server.rocketchat-server[3596]: LocalStore: store created at
2023-03-11T06:22:44Z rocketchat-server.rocketchat-server[3596]: MessageType.render is deprecated. Use MessageType.message instead. livechat_webrtc_video_call
2023-03-11T06:22:45Z rocketchat-server.rocketchat-server[3596]: {"level":40,"time":"2023-03-11T06:22:45.686Z","pid":3596,"hostname":"chat","name":"VoIPService","msg":"Voip is not enabled. Cant start the service"}

With…
rocketchat-server.rocketchat-server[3397]: /snap/rocketchat-server/1540/helpers/rocketchat.sh: line 23: 3465 Illegal instruction (core dumped) node $SNAP/main.js
…and the couple lines right above it being where I open the mobile app.

I get essentially the same log but with a different IP I’m connecting from if I try over cellular data to be off of my network.

If you want, I can send you the unedited log, in case it’s something you’d be able to use to point me in the right direction.

4

I had posted a log that Akismet blocked as spam, but it looks like my issue is probably related to issue 23938 on github, the hypervisor I’m running it on at home is old, and doesn’t support AVX, but I guess only the mobile app is making it crash.

I switched channels to 3.x/stable, and then held the package and that seems to work fine on my hypervisor when also using the mobile app.

5

Oh… this issue: 4.2.1 gives 'Illegal instruction' - SIGILL in node on some CPU types · Issue #23938 · RocketChat/Rocket.Chat · GitHub

Unfortunately this is out of our hands. :frowning:

With this new requirement for mongo 5 and the end of life of the 4.X version hitting the end of their lives we are going to see a lot of issues like this.

Thanks for sharing this. :hugs: I am sure it will help other users around!

6

Thanks for pointing me in the right direction of how to find the logs.

I ended up (planning to) switch to the Docker version to run 4.2.0, I’m aware it’s old / EOL / unsupported / unsafe / etc., but it’s going to be used for two to three users, behind a VPN, in a non-production environment, for only unimportant conversations that won’t impact anything if they were suddenly lost, so I’m fine with staying on the old version for home and “latest” at work.

Docker in a Proxmox LXC container is going to be a lot “lighter” than a full VM running a snap anyway.

7

I would wish to appleal for the account abc. Kindly any link with the admin