Suddenly 502 Bad Gateway

Community Support
1

Hi.

Suddenly, at the end of last week, RocketChat stopped working.
I have no idea why. The status is active (running).

Jul 09 11:38:10 rocketchat systemd[1]: Started Service for snap application rocketchat-server.rocketchat-server.
Jul 09 11:38:12 rocketchat rocketchat-server.rocketchat-server[1305]: Mongo is not available, can’t start. Waiting 10 seconds and trying again
Jul 09 11:38:22 rocketchat rocketchat-server.rocketchat-server[1305]: Checking if oplog has been enabled, and enabling if not
Jul 09 11:38:22 rocketchat rocketchat-server.rocketchat-server[1305]: MongoDB shell version: 3.2.7
Jul 09 11:38:22 rocketchat rocketchat-server.rocketchat-server[1305]: connecting to: test
Jul 09 11:38:31 rocketchat rocketchat-server.rocketchat-server[1305]: Will load cache for users
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: 102 records load from users
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: Will load cache for rocketchat_room
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: 13 records load from rocketchat_room
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: Will load cache for rocketchat_subscription
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: 31 records load from rocketchat_subscription
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: Will load cache for rocketchat_settings
Jul 09 11:38:32 rocketchat rocketchat-server.rocketchat-server[1305]: 706 records load from rocketchat_settings
Jul 09 11:38:34 rocketchat rocketchat-server.rocketchat-server[1305]: Updating process.env.MAIL_URL
Jul 09 11:38:34 rocketchat rocketchat-server.rocketchat-server[1305]: Starting Email Intercepter…
Jul 09 11:38:35 rocketchat rocketchat-server.rocketchat-server[1305]: Will load cache for rocketchat_permissions
Jul 09 11:38:35 rocketchat rocketchat-server.rocketchat-server[1305]: 80 records load from rocketchat_permissions
Jul 09 11:38:35 rocketchat rocketchat-server.rocketchat-server[1305]: Will load cache for rocketchat_roles
Jul 09 11:38:35 rocketchat rocketchat-server.rocketchat-server[1305]: 11 records load from rocketchat_roles
Jul 09 11:38:37 rocketchat rocketchat-server.rocketchat-server[1305]: LocalStore: store created at
Jul 09 11:38:37 rocketchat rocketchat-server.rocketchat-server[1305]: LocalStore: store created at
Jul 09 11:38:37 rocketchat rocketchat-server.rocketchat-server[1305]: LocalStore: store created at
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: Warning: connect.session() MemoryStore is not
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: designed for a production environment, as it will leak
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: memory, and will not scale past a single process.
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: Setting default file store to GridFS
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: LocalStore: store created at
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: LocalStore: store created at
Jul 09 11:38:39 rocketchat rocketchat-server.rocketchat-server[1305]: LocalStore: store created at
Jul 09 11:38:45 rocketchat rocketchat-server.rocketchat-server[1305]: Mon, 09 Jul 2018 09:38:45 GMT connect deprecated multipart: use parser (multiparty, busboy, formidable) npm module instead at npm/node_modules/connect/lib/middleware/bodyParser.js:56:20
Jul 09 11:38:45 rocketchat rocketchat-server.rocketchat-server[1305]: Mon, 09 Jul 2018 09:38:45 GMT connect deprecated limit: Restrict request size at location of read at npm/node_modules/connect/lib/middleware/multipart.js:86:15
Jul 09 11:38:45 rocketchat rocketchat-server.rocketchat-server[1305]: {“line”:“160”,“file”:“rocketchat_migrations.js”,“message”:“Migrations: Not migrating, already at version 129”,“time”:{"$date":1531129125500},“level”:“info”}
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: Using GridFS for custom sounds storage
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: Using FileSystem for custom emoji storage
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: Exception in callback of async function: Error: EROFS: read-only file system, mkdir ‘/data’
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at Object.fs.mkdirSync (fs.js:885:18)
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at sync (/snap/rocketchat-server/1307/programs/server/npm/node_modules/mkdirp/index.js:71:13)
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at Function.sync (/snap/rocketchat-server/1307/programs/server/npm/node_modules/mkdirp/index.js:77:24)
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at new RocketChatFile.FileSystem (/snap/rocketchat-server/1307/programs/server/packages/rocketchat_file.js:223:12)
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at /snap/rocketchat-server/1307/programs/server/packages/rocketchat_emoji-custom.js:97:44
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at Function.time (/snap/rocketchat-server/1307/programs/server/profile.js:309:28)
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at /snap/rocketchat-server/1307/programs/server/boot.js:427:13
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at /snap/rocketchat-server/1307/programs/server/boot.js:472:5
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at Function.run (/snap/rocketchat-server/1307/programs/server/profile.js:510:12)
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: at /snap/rocketchat-server/1307/programs/server/boot.js:470:11
Jul 09 11:38:46 rocketchat rocketchat-server.rocketchat-server[1305]: Updating process.env.MAIL_URL

I tried to revert with snap, but still 502 Bad Gateway.
Maybe someone can help me.

2

It appears you set your file upload to /data which is a directory not writable by snaps.

Snaps are only allowed to write to certain directories.

Directories writable would be something like:

  • /var/snap/rocketchat-server/common (this is the default)
  • /var/snap/rocketchat-server/current (this folder is versioned. Will be copied every single upgrade.)
1 Like
3

Thanks for looking into it. Now that I’m not able to connect, how can I change the directory back?

4

I found a solution by myself. If someone lands here an don’t know what to do:
db.rocketchat_settings.update ({_id: “EmojiUpload_Storage_Type”}, {$set: {“value”:“GridFS”}}, false, true);

Again thanks for investigation. Keep up.

5

Thanks for following up with the solution. Yes it really depended on which was using file upload. Looks like you found it :slight_smile:

6

@osmg Can you please tell me how to apply this solution?

Note: I did not install rocket chat using snap

7

you might want to try just rebooting the service
ex. systemctl restart rocketchat

worked for me, not using snap