# Rocket.Chat crashed. Restarting the services doesn't help **URL:** \ **Category:** Community Support\ **Created:** [August 13, 2020, 8:32am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123 "2020-08-13T08:32:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 13, 2020, 8:32am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/1 "2020-08-13T08:32:32Z")
### Description Rocket.Chat unexpectedly crashed out of blue and throws this kind of errors: 2020/08/13 08:56:07 [error] 337681#337681: \*99747 connect() failed (111: Connection refused) while connecting to upstream, client: 9x.8x.2xx.2xx, server name, request: “GET /favicon HTTP/1.1”, upstream: “127.0.0.1:3000/favicon.ico”, host: “[sub.domain.com](http://sub.domain.com)”, referrer: “domain name” Link to the error log dump: [https://s3.amazonaws.com/uploads.use1.cloud.rocket.chat/eoRXMCHBbQCdDnrke/uploads/eFwou2EY5S7bvYWPY/24bJYAkCKQe8QPmgw/btevSNFLgLnd2sjx2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAZQA2H3B2ERCWVEMW%2F20200813%2Fus-east-1%2Fs3%2Faws4\_request&X-Amz-Date=20200813T083053Z&X-Amz-Expires=120&X-Amz-Signature=15d028e8fc6bb97e60fec3cc73eded55e1daaaefb11ac14ce3003521696bab1f&X-Amz-SignedHeaders=host&response-content-disposition=inline%3B%20filename%3D"larkin%20-%20Thu%20Aug%2013"](https://s3.amazonaws.com/uploads.use1.cloud.rocket.chat/eoRXMCHBbQCdDnrke/uploads/eFwou2EY5S7bvYWPY/24bJYAkCKQe8QPmgw/btevSNFLgLnd2sjx2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAZQA2H3B2ERCWVEMW%2F20200813%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200813T083053Z&X-Amz-Expires=120&X-Amz-Signature=15d028e8fc6bb97e60fec3cc73eded55e1daaaefb11ac14ce3003521696bab1f&X-Amz-SignedHeaders=host&response-content-disposition=inline%3B%20filename%3D%22larkin%2520-%2520Thu%2520Aug%252013%22) ### Server Setup Information - Version of Rocket.Chat Server: latest - Operating System: Ubuntu 20.04 - Deployment Method: snap - Number of Running Instances: 1 - Proxy: nginx - Firewalls involved: disabled ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 13, 2020, 11:51am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/2 "2020-08-13T11:51:50Z")
Service status showed this: `at /snap/rocketchat-server/1439/programs/server/boot.js:415:15 Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: at /snap/rocketchat-server/1439/programs/server/boot.js:465:7 Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: at Function.run (/snap/rocketchat-server/1439/programs/server/profile.js:280:14) Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: at /snap/rocketchat-server/1439/programs/server/boot.js:463:13 { Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: errno: -30, Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: syscall: 'mkdir', Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: code: 'EROFS', Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: path: '/snap/rocketchat-server/1439/programs/server/https:' Aug 13 10:54:46 v2201 rocketchat-server.rocketchat-server[2067]: }` ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 13, 2020, 3:27pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/3 "2020-08-13T15:27:42Z")
Seems to be a related issue that affected many people: [Rocket chat on Ubuntu 20.04 no longer works](https://forums.rocket.chat/t/rocket-chat-on-ubuntu-20-04-no-longer-works/8126/12) ---
**Author:** ![binarydad](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/binarydad/32/2035_2.png) [@binarydad](https://forums.rocket.chat/u/binarydad)\ **Post date:** [August 13, 2020, 3:29pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/4 "2020-08-13T15:29:58Z")
Seems to be related to Snap? I see they just pushed 3.5.1 recently. ---
**Author:** ![bradley.hilton](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/bradley.hilton/32/30_2.png) [@bradley.hilton](https://forums.rocket.chat/u/bradley.hilton)\ **Post date:** [August 13, 2020, 4:42pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/5 "2020-08-13T16:42:52Z")
> [@Rocket chat server can't start (problem with cloud.rocket.chat )](https://forums.rocket.chat/t/rocket-chat-server-cant-start-problem-with-cloud-rocket-chat/8131/3): > > Changes have been made and we are investigating why it suddenly stopped even though other areas of the same infrastructure were working just fine. Everything should be working fine as of right now. Sorry to be so vague, but we really don’t know what caused the issue and are looking into it. ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 13, 2020, 5:59pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/6 "2020-08-13T17:59:43Z")
I’m not sure if the “everything should be working fine as of right now” applies to my situation, but I’ve tried restarting rocketchat-server and I’m still getting ‘502 Bad Gateway’. ---
**Author:** ![chaosclarity](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@chaosclarity](https://forums.rocket.chat/u/chaosclarity)\ **Post date:** [August 13, 2020, 6:04pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/7 "2020-08-13T18:04:41Z")
Same here. I’ve restored a fresh copy of the VM from 2 days ago because I’ve mucked with the current one trying to get it working. Still the same ol “502 Bad Gateway” for me. I’ve noticed that if I try to start the snap.rocketchat-server service, it appears it starts OK but for some reason stops immediately - shows it’s Inactive instead of Active and running. Port 3000 also never opens up. ---
**Author:** ![binarydad](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/binarydad/32/2035_2.png) [@binarydad](https://forums.rocket.chat/u/binarydad)\ **Post date:** [August 13, 2020, 7:28pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/8 "2020-08-13T19:28:34Z")
Samsies. Though, I got fed up and did a fresh install using the native installation method (i.e., no more snap). ---
**Author:** ![chaosclarity](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@chaosclarity](https://forums.rocket.chat/u/chaosclarity)\ **Post date:** [August 13, 2020, 8:55pm UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/9 "2020-08-13T20:55:07Z")
I managed to just deploy Mattermost and I think I actually like it better. ---
**Author:** ![aaron.ogle](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/aaron.ogle/32/15_2.png) [@aaron.ogle](https://forums.rocket.chat/u/aaron.ogle)\ **Post date:** [August 14, 2020, 4:42am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/10 "2020-08-14T04:42:35Z")
3.5.2 should fully solve this. Basically two parts of same issue. 1. HTTP calls weren’t deferred causing them to block until they got a response. 2. On our end we had an incident that caused a service that is never down to stop responding and our load balancer kept the connection open too long instead of more aggressively timing out. 3.5.2 protects rocket.chat from that now regardless. The service is fixed and we are making the changes to the loadbalancer ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 14, 2020, 7:19am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/11 "2020-08-14T07:19:46Z")
Unfortunately, it did not. I’ve tried ‘snap refresh rocketchat-server’ but it says there are no updates available, so I guess it’s already updated to 3.5.2. Restarting rc, caddy, mongo, and ubuntu server didn’t help. ---
**Author:** ![aaron.ogle](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/aaron.ogle/32/15_2.png) [@aaron.ogle](https://forums.rocket.chat/u/aaron.ogle)\ **Post date:** [August 14, 2020, 7:36am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/12 "2020-08-14T07:36:06Z")
What version does it say it’s on? snap info rocketchat-server I believe will tell you what version is installed. Look at logs also. [https://docs.rocket.chat/installation/snaps#how-do-i-tell-if-rocket-chat-is-actually-running](https://docs.rocket.chat/installation/snaps#how-do-i-tell-if-rocket-chat-is-actually-running) The FAQ should be of use here to see what’s happening ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 14, 2020, 7:37am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/13 "2020-08-14T07:37:53Z")
3.5.2 In the logs I see this: ```auto -- Logs begin at Sun 2020-08-02 04:52:27 CEST. -- Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: at /snap/rocketchat-server/1440/programs/server/boot.js:415:15 Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: at /snap/rocketchat-server/1440/programs/server/boot.js:465:7 Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: at Function.run (/snap/rocketchat-server/1440/programs/server/profile.js:280:14) Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: at /snap/rocketchat-server/1440/programs/server/boot.js:463:13 { Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: errno: -30, Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: syscall: 'mkdir', Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: code: 'EROFS', Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: path: '/snap/rocketchat-server/1440/programs/server/https:' Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: } Aug 14 08:51:45 v220190810296596149 rocketchat-server.rocketchat-server[1654]: Updating process.env.MAIL_URL ``` ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 14, 2020, 7:42am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/14 "2020-08-14T07:42:27Z")
Found the culprit (apparently): ```auto -- Logs begin at Sun 2020-08-02 04:52:27 CEST. -- Aug 14 08:51:07 v220190810296596149 systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy. Aug 14 08:51:07 v220190810296596149 systemd[1]: Started Service for snap application rocketchat-server.rocketchat-caddy. Aug 14 08:51:07 v220190810296596149 rocketchat-server.rocketchat-caddy[1220]: 2020/08/14 08:51:07 loading Caddyfile via flag: open /var/snap/rocketchat-server/1440/Caddyfile: no such file or directory Aug 14 08:51:07 v220190810296596149 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Main process exited, code=exited, status=1/FAILURE Aug 14 08:51:07 v220190810296596149 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'. Aug 14 08:51:07 v220190810296596149 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Scheduled restart job, restart counter is at 5. Aug 14 08:51:07 v220190810296596149 systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy. Aug 14 08:51:07 v220190810296596149 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Start request repeated too quickly. Aug 14 08:51:07 v220190810296596149 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'. Aug 14 08:51:07 v220190810296596149 systemd[1]: Failed to start Service for snap application rocketchat-server.rocketchat-caddy. ``` How to fix that? I’m in /var/snap/rocketchat-server/1440/ right now and it is empty. But I haven’t touched that folder before. ---
**Author:** ![aaron.ogle](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/aaron.ogle/32/15_2.png) [@aaron.ogle](https://forums.rocket.chat/u/aaron.ogle)\ **Post date:** [August 14, 2020, 7:46am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/15 "2020-08-14T07:46:58Z")
Had you used caddy before for your https and reverse proxy? If so you’ll need to initialize it. The FAQ has instructions for that. If not that error is fine. Just fails to start and sits there unused. For rocket.chat operation caddy isn’t needed. It’s just there available in case you want it. Seems like you should be able to access on port 3000 right now. ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 14, 2020, 7:53am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/16 "2020-08-14T07:53:20Z")
I haven’t. I’ve installed rocket chat with snap and haven’t touched anything that wasn’t in the instructions. I’m using nginx to be able to access my rocket chat instance via an actual domain name url and that’s it. I’m still getting “502 Bad Gateway”. ---
**Author:** ![aaron.ogle](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/aaron.ogle/32/15_2.png) [@aaron.ogle](https://forums.rocket.chat/u/aaron.ogle)\ **Post date:** [August 14, 2020, 7:56am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/17 "2020-08-14T07:56:36Z")
Can you access at port 3000 directly at all? It directly won’t reply with bad gateway. It’ll either reply or not at all ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 14, 2020, 8:14am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/18 "2020-08-14T08:14:07Z")
I’ve tried 1xx.8x.1xx.8x:3000 and it returns “ERR\_CONNECTION\_REFUSED”. ---
**Author:** ![aaron.ogle](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/aaron.ogle/32/15_2.png) [@aaron.ogle](https://forums.rocket.chat/u/aaron.ogle)\ **Post date:** [August 14, 2020, 8:15am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/19 "2020-08-14T08:15:54Z")
On the actual host it’s self can you curl [http://localhost:3000/api/info](http://localhost:3000/api/info) and get a result? If so sounds like firewall. The logs indicate it’s probably up and running ---
**Author:** ![larkin](https://avatars.discourse-cdn.com/v4/letter/l/b2d939/32.png) [@larkin](https://forums.rocket.chat/u/larkin)\ **Post date:** [August 14, 2020, 8:20am UTC](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123/20 "2020-08-14T08:20:05Z")
Firewall was disabled. curl says “(7) Failed to connect to localhost port 3000: Connection refused”. [Next page](https://forums.rocket.chat/t/rocket-chat-crashed-restarting-the-services-doesnt-help/8123.md?page=2)