Getting stuck at 6.2.11 to 6.3.0

Upgrade Help
1

What version are you coming from? 6.2.11

What version are you going to? 6.3.0 (and then later releases)

What deployment method did you use to deploy? droplet from digital ocean, but with some manual changes to apply the updates. (it worked fine from 2.4.9 all the way up to 6.2.11…)

Did you follow a particular doc? Which one? not anything directly, just pieced together since we are coming from a very old version,

What issues are you running into upgrading?

All versions seem to be working OK for right now. However as soon as we stop the 6.2.11 instance which started and worked, and try to go to 6.3.0, the system will start the pid, but just stop.

Our log shows:

Nov 14 22:59:51 chat-noc rocketchat[1932]: strict mode: missing type "object" for keyword "additionalProperties" at "#" (strictTypes)
Nov 14 22:59:51 chat-noc rocketchat[1932]: strict mode: missing type "object" for keyword "properties" at "#" (strictTypes)
Nov 14 22:59:51 chat-noc rocketchat[1932]: strict mode: use allowUnionTypes to allow union type keyword at "#/properties/value" (strictTypes)
Nov 14 22:59:54 chat-noc rocketchat[1932]: (node:1932) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
Nov 14 22:59:54 chat-noc rocketchat[1932]: Please migrate your code to use AWS SDK for JavaScript (v3).
Nov 14 22:59:54 chat-noc rocketchat[1932]: For more information, check the migration guide at https://a.co/7PzMCcy
Nov 14 22:59:54 chat-noc rocketchat[1932]: (Use `node --trace-warnings ...` to show where the warning was created)
Nov 14 22:59:54 chat-noc rocketchat[1932]: LocalStore: store created at
Nov 14 22:59:54 chat-noc rocketchat[1932]: LocalStore: store created at
Nov 14 22:59:54 chat-noc rocketchat[1932]: LocalStore: store created at
Nov 14 22:59:54 chat-noc rocketchat[1932]: MessageType.render is deprecated. Use MessageType.message instead. livechat_webrtc_video_call
Nov 14 22:59:55 chat-noc rocketchat[1932]: {"level":40,"time":"2023-11-14T22:59:55.534Z","pid":1932,"hostname":"chat-noc","name":"VoIPService","msg":"Voip is not enabled. Cant start the service"}
Nov 14 22:59:57 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T22:59:57.060Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Migrating from version 294 -> 301"}
Nov 14 22:59:57 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T22:59:57.061Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 295(Change old \"LDAP_Background_Sync_Interval\" and \"CROWD_Sync_Interval\" to a pre-defined values instead of accept any input from the user)"}
Nov 14 22:59:57 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T22:59:57.071Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 296(Reset the default value of Login Terms setting and replace by empty string)"}
Nov 14 22:59:57 chat-noc rocketchat[1932]: {"level":40,"time":"2023-11-14T22:59:57.083Z","pid":1932,"hostname":"chat-noc","name":"System","msg":"The default value of the setting 'Login Terms' has changed to an empty string. Please review your settings."}
Nov 14 22:59:57 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T22:59:57.084Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 297"}
Nov 14 23:00:00 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T23:00:00.197Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 298(Set overrideDestinationChannelEnabled for all incoming webhook integrations)"}
Nov 14 23:00:00 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T23:00:00.213Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 299(Add public field to existing custom OAuth settings)"}
Nov 14 23:00:00 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T23:00:00.226Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 300"}
Nov 14 23:00:00 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T23:00:00.240Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Running up() on version 301"}
Nov 14 23:00:00 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T23:00:00.249Z","pid":1932,"hostname":"chat-noc","name":"Migrations","msg":"Finished migrating."}
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.319Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Moleculer v0.14.29 is starting...
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.320Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Namespace: <not defined>
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.320Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Node ID: <blanking this out>
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.322Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/REGISTRY: Strategy: RoundRobinStrategy
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.322Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/REGISTRY: Discoverer: LocalDiscoverer
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.324Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Serializer: JSONSerializer
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.342Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Validator: FastestValidator
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.345Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Registered 13 middleware(s).
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.347Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: Transporter: TcpTransporter
Nov 14 23:00:00 chat-noc rocketchat[1932]: {"level":51,"time":"2023-11-14T23:00:00.378Z","pid":1932,"hostname":"chat-noc","name":"DatabaseWatcher","msg":"Using change streams"}
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.466Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/TRANSIT: Connecting to the transporter...
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.476Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/TRANSPORTER: TCP server is listening on port 42229
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.477Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/TRANSPORTER: UDP Discovery is disabled.
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.478Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/TRANSPORTER: TCP Transporter started.
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.993Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/REGISTRY: '$node' service is registered.
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.994Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/REGISTRY: 'matrix' service is registered.
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.995Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/$NODE: Service '$node' started.
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.996Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/MATRIX: Service 'matrix' started.
Nov 14 23:00:00 chat-noc rocketchat[1932]: [2023-11-14T23:00:00.996Z] INFO  ad8444e7-4bf8-4956-9d54-231215cb4152/BROKER: ✔ ServiceBroker with 2 service(s) started successfully in 631ms.
Nov 14 23:00:01 chat-noc rocketchat[1932]: (node:1932) [MONGODB DRIVER] Warning: cursor.count is deprecated and will be removed in the next major version, please use `collection.estimatedDocumentCount` or `collection.countDocuments` instead

I let this sit for 15+ minutes, and nothing seems to happen. No new logs. No errors.
When I run top, I will see node and mongod taking up RAM and anywhere from 1-2% CPU.

In the previous startups, I also had another error, but that has not shown up since 6.3.0 and I am not sure it was related.

That error was:

Nov 14 20:55:41 chat-noc rocketchat[5699]: === UnHandledPromiseRejection ===
Nov 14 20:55:41 chat-noc rocketchat[5699]: MongoServerError: E11000 duplicate key error collection: rocketchat.users index: username_1 dup key: { username: "rocket.cat" }
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/insert.js:53:33
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection_pool.js:333:25
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:210:17
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at handleOperationResult (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/server.js:326:20)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at Connection.onMessage (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:242:9)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at MessageStream.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at MessageStream.emit (events.js:400:28)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at MessageStream.emit (domain.js:475:12)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at processIncomingData (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at MessageStream._write (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at writeOrBuffer (internal/streams/writable.js:358:12)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at MessageStream.Writable.write (internal/streams/writable.js:303:10)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at Socket.ondata (internal/streams/readable.js:731:22)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at Socket.emit (events.js:400:28)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at Socket.emit (domain.js:475:12)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at addChunk (internal/streams/readable.js:293:12)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at readableAddChunk (internal/streams/readable.js:267:9)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at Socket.Readable.push (internal/streams/readable.js:206:10)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at TCP.callbackTrampoline (internal/async_hooks.js:130:17)
Nov 14 20:55:41 chat-noc rocketchat[5699]:  => awaited here:
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at Function.Promise.await (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at server/startup/initialData.js:28:3
Nov 14 20:55:41 chat-noc rocketchat[5699]:     at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
Nov 14 20:55:41 chat-noc rocketchat[5699]:   index: 0,
Nov 14 20:55:41 chat-noc rocketchat[5699]:   code: 11000,
Nov 14 20:55:41 chat-noc rocketchat[5699]:   keyPattern: { username: 1 },
Nov 14 20:55:41 chat-noc rocketchat[5699]:   keyValue: { username: 'rocket.cat' },
Nov 14 20:55:41 chat-noc rocketchat[5699]:   [Symbol(errorLabels)]: Set(0) {}
Nov 14 20:55:41 chat-noc rocketchat[5699]: }
Nov 14 20:55:41 chat-noc rocketchat[5699]: ---------------------------------
Nov 14 20:55:41 chat-noc rocketchat[5699]: Errors like this can cause oplog processing errors.
Nov 14 20:55:41 chat-noc rocketchat[5699]: Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Nov 14 20:55:41 chat-noc rocketchat[5699]: Future node.js versions will automatically exit the process
Nov 14 20:55:41 chat-noc rocketchat[5699]: =================================

I tried to drop indexes on db.users, with “db.users.dropIndexes()” but that did not fix this issue.
I am unsure if this issue is related to the current issue or not.

2

Since it took RC 3 months to approve this, I since found the solution to my problem.
The issue ended up being because our Rocket.Cat user was manually created by one of our scripts, it was added in as a user, and not bot.

MongoServerError: E11000 duplicate key error collection: rocketchat.users index: username_1 dup key: { username: "rocket.cat" }

So our soluition was to go into the db, and dump the user settings, delete the user, and re-add the user, updating settings again.
YMMV, Use at your own risk

monogo  
use rocketchat

# Save the data output from this:
db.users.find({username: new RegExp("rocket.cat", "i")}).pretty()

# delete the user
db.users.deleteOne({username: new RegExp("rocket.cat", "i")})

# restart the service, which adds the user back in
# verify the user
db.users.find({username: new RegExp("rocket.cat", "i")}).pretty()

# update user settings:
db.users.updateOne({_id: "rocket.cat"}, {$set: {  
"emails" : [  
{  
"address" : "not-real@email.com",  
"verified" : true  
}  
],  
"roles" : [  
"bot",  
"app"  
],  
"__rooms" : [  
"GENERAL",  
"blah1",  
"blah2",  
"blah3"  
]  
}})