Description Rocket.Chat unexpectedly crashed out of blue and throws this kind of errors: 2020/08/13 08:56:07 [error] 337681#337681: *99747 connect() failed (111: Connection refused) while connecting to upstream, client: 9x.8x.2xx.2xx, server name, request: “GET /favicon HTTP/1.1”, upstream: “127.0.0.1:3000/favicon.ico”, host: “sub.domain.com”, referrer: “domain name” Link to the error log dump: https://s3.amazonaws.com/uploads.use1.cloud.rocket.chat/eoRXMCHBbQCdDnrke/uploads/eFwou2EY5S7bvYWPY/24bJYAkCKQe8QPmgw/btevSNFLgLnd2sjx2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAZQA2H3B2ERCWVEMW%2F20200813%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200813T083053Z&X-Amz-Expires=120&X-Amz-Signature=15d028e8fc6bb97e60fec3cc73eded55e1daaaefb11ac14ce3003521696bab1f&X-Amz-SignedHeaders=host&response-content-disposition=inline%3B%20filename%3D"larkin%20-%20Thu%20Aug%2013" Server Setup Information Version of Rocket.Chat Server: latest Operating System: Ubuntu 20.04 Deployment Method: snap Number of Running Instances: 1 Proxy: nginx Firewalls involved: disabled

Alright this is what I suspected. [image] larkin: EmojiUpload_FileSystemPath - https://domain.com/emotes We’ll need to drop into a mongo shell again. Also switch to parties db just like above. Then choose which path forward If you want custom emoji on file system you can update this value: db.rocketchat_settings.update({_id:”EmojiUpload_FileSystemPath”},{$set:{value:”/var/snap/rocketchat-server/common/emoji”}}) Or replace with folder that is snap writable. See more details here: Deploy with Snaps - Rocket.Chat Docs Alternatively switch it to use gridfs db.rocketchat_settings.update({_id:”EmojiUpload_Storage_Type”},{$set:{value:”GridFS”}}) Once you do one or the other you should be able to restart Rocket.chat. Looks like this value was changed while it was running and must have never needed to restart since set. Then the update caused it to and failed to start up because it had web address instead of file system address and it couldn’t find the folder

Firewall was disabled. curl says “(7) Failed to connect to localhost port 3000: Connection refused”.

Hm… sure enough looks like it didn’t get booted. Then we’re going to need more output for Rocket.Chat startup logs. journalctl -u snap.rocketchat-server.rocketchat-server

journalctl -u snap.rocketchat-server.rocketchat-server output: Aug 14 10:17:48 v2201 systemd[1]: Started Service for snap application rocketchat-server.rocketchat-server. Aug 14 10:17:51 v2201 rocketchat-server.rocketchat-server[714]: Mongo is not available, can't start. Waiting 10 seconds and trying again Aug 14 10:18:01 v2201 rocketchat-server.rocketchat-server[714]: Checking if oplog has been enabled, and enabling if not Aug 14 10:18:02 v2201 rocketchat-server.rocketchat-server[1473]: MongoDB shell version v3.6.14 Aug 14 10:18:02 v2201 rocketchat-server.rocketchat-server[1473]: connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb Aug 14 10:18:02 v2201 rocketchat-server.rocketchat-server[1473]: Implicit session: session { "id" : UUID("26f6e030-794a-4680-8cf2-b01f3afa3821") } Aug 14 10:18:02 v2201 rocketchat-server.rocketchat-server[1473]: MongoDB server version: 3.6.14 Aug 14 10:18:02 v2201 rocketchat-server.rocketchat-server[714]: Checking if mongo featureCompatibilityVersion is correct, changing if not Aug 14 10:18:22 v2201 rocketchat-server.rocketchat-server[1614]: LocalStore: store created at Aug 14 10:18:22 v2201 rocketchat-server.rocketchat-server[1614]: LocalStore: store created at Aug 14 10:18:22 v2201 rocketchat-server.rocketchat-server[1614]: LocalStore: store created at Aug 14 10:18:23 v2201 rocketchat-server.rocketchat-server[1614]: Setting default file store to GridFS Aug 14 10:18:28 v2201 rocketchat-server.rocketchat-server[1614]: {"line":"120","file":"migrations.js","message":"Migrations: Not migrating, already at version 202","time":{"$date":1597393108974},"level":"info"} Aug 14 10:18:29 v2201 rocketchat-server.rocketchat-server[1614]: ufs: temp directory created at "/tmp/ufs" Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: Loaded the Apps Framework and loaded a total of 0 Apps! Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: Using GridFS for custom sounds storage Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: Using FileSystem for custom emoji storage Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: Exception in callback of async function: Error: EROFS: read-only file system, mkdir '/snap/rocketchat-server/1440/programs/server/https:' Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at Object.mkdirSync (fs.js:840:3) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at sync (/snap/rocketchat-server/1440/programs/server/npm/node_modules/mkdirp/index.js:71:13) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at sync (/snap/rocketchat-server/1440/programs/server/npm/node_modules/mkdirp/index.js:77:24) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at Function.sync (/snap/rocketchat-server/1440/programs/server/npm/node_modules/mkdirp/index.js:77:24) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at new RocketChatFile.FileSystem (app/file/server/file.server.js:154:10) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at app/emoji-custom/server/startup/emoji-custom.js:32:38 Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at Function.time (/snap/rocketchat-server/1440/programs/server/profile.js:273:30) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at /snap/rocketchat-server/1440/programs/server/boot.js:415:15 Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at /snap/rocketchat-server/1440/programs/server/boot.js:465:7 Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at Function.run (/snap/rocketchat-server/1440/programs/server/profile.js:280:14) Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: at /snap/rocketchat-server/1440/programs/server/boot.js:463:13 { Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: errno: -30, Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: syscall: 'mkdir', Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: code: 'EROFS', Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: path: '/snap/rocketchat-server/1440/programs/server/https:' Aug 14 10:18:31 v2201 rocketchat-server.rocketchat-server[1614]: } Aug 14 10:18:32 v2201 rocketchat-server.rocketchat-server[1614]: Updating process.env.MAIL_URL __updated to insert the entire log.

It looks like something is trying to use the file system at a path that doesn’t exist and it can’t create at since read only. Can you do: rocketchat-server.mongo Then execute: db.rocketchat_settings.find().forEach(function(s) { if (!s._id.match(/^Assets/) && !s._id.match(/^Layout/) && s._id !== 'css' && s.value !== s.packageValue) { print(s._id + ' - ' + s.value); } }); It will spit out any setting changed. So will need to censor out any secrets or domains. The file paths and file related settings we are most interested in.

I’ve inserted that in the rocketchat-server.mongo terminal and it returned nothing. That piece of code is immediately followed by “rs0: PRIMARY>”

You might need to type use parties first to select the database (this is default db name in snaps)

Should I post the custom .css output here as well? There’s a lot.

Nah that shouldn’t have any effect on boot up. Can leave that out

Accounts_TwoFactorAuthentication_Enabled - false Accounts_TwoFactorAuthentication_By_Email_Enabled - false Accounts_TwoFactorAuthentication_By_Email_Auto_Opt_In - false uniqueID - zn************* Accounts_SearchFields - username, name, bio Accounts_ManuallyApproveNewUsers - true Accounts_RegistrationForm_SecretURL - PPcZDimKCuAP2Jc9W Accounts_Registration_AuthenticationServices_Enabled - false Accounts_Default_User_Preferences_hideRoles - true Show_Setup_Wizard - completed Site_Url - https://sub.domain.com Site_Name - Polises Messaging DeepLink_Url - https://sub.domain.com Statistics_reporting - false Message_Attachments_GroupAttach - true Message_Audio_bitRate - 128 Message_MaxAllowedSize - 15000 Hide_System_Messages - Push_enable_gateway - false Custom_Script_Logged_In - UI_Use_Real_Name - true Organization_Type - community Organization_Name - name Industry - *** Size - 2 Country - worldwide Website - https://domain.com Server_Type - privateTeam Allow_Marketing_Emails - false Register_Server - false Cloud_Workspace_Access_Token_Expires_At - Thu Jan 01 1970 01:00:00 GMT+0100 (CET) EmojiUpload_Storage_Type - FileSystem EmojiUpload_FileSystemPath - https://domain.com/emotes Update_LatestAvailableVersion - 3.5.1 Message_Read_Receipt_Enabled - true Message_Read_Receipt_Store_Users - true FEDERATION_Status - Enabled RetentionPolicy_DoNotPruneDiscussion - true RetentionPolicy_DoNotPruneThreads - true

I get this: db.rocketchat_settings.update({_id:" EmojiUpload_Storage_Type"},{$set:{value:"GridFS"}}) WriteResult({ "nMatched" : 0, "nUpserted" : 0, "nModified" : 0 }) After exit, I restart rocketchat and mongo, but still can’t access my rocketchat instance. I’ve tried running the following code again: db.rocketchat_settings.find().forEach(function(s) { if (!s._id.match(/^Assets/) && !s._id.match(/^Layout/) && s._id !== 'css' && s.value !== s.packageValue) { print(s._id + ' - ' + s.value); } }); And still get EmojiUpload_FileSystemPath - https://domain.com/emotes I’m not an avid mongo user, I thought maybe I exited without saving but I can’t find a command that saves the changes to a database so I assume it saves the changes automatically. What could be a problem?

Rocket.Chat crashed. Restarting the services doesn't help

Community Support

surendrababu.s August 14, 2020, 9:53am 34

dear aaron colud you please help me in this

Topic		Replies	Views
RC stopped working after reboot Community Support	11	2337	August 24, 2021
Sudden 502 error Community Support	21	6818	September 23, 2020
RocketChat suddenly unavailable (MongoDB connection refused) Community Support	2	4448	May 14, 2020
Suddenly, mongod is not starting Community Support	6	8044	November 7, 2019
Rocket chat via snap has error on restart of the application Community Support	14	3179	June 10, 2022

[Secure CommsOS™ Launch] Join our next Roadmap Reveal webinar to learn more Register now ->

Rocket.Chat crashed. Restarting the services doesn't help

Related topics