I’m running an old version of Rocketchat 0.74.3 on Centos 7 and I’m trying to migrate to a new server running Ubuntu 22 and moving Rocketchat to a docker container according to the RC Guides. I figured it would be easier to upgrade to the latest build once in Docker. I got the Rocketchat server up running the same old RC version in the container on the new server with docker compose. I was able to verify by going to the rocketchat install website and see the start up wizard to configure the new install. Then I use mongodump and mongorestore to migrate the data to the new server.
Mongorestore successfully restored all documents. However, there was an error after “restoring users from archive on stdin”…
The error message reads “Failed: restore error: error running merge command: (NoSuchKey) Missing expected field “db””
Then when I load the new RC server web page I get the login page, but after I enter a username and password it tries to login but just hangs getting alerts about reconnecting to the database.
When I look at log files I see a bunch of “slow query” events on command \update \users.
The authentication process never completes. I should also mention that in the old server I’m using ldap to authenticate to Active Directory. Maybe that add another wrinkle to my restore process that isn’t documented?
Any help would be very welcome because I’m currently at a loss on where to go next.
Thanks.
Server Setup Information
- Version of Rocket.Chat Server: 0.74.3
- Operating System: Centos 7 and Ubuntu 22
- Deployment Method: Centos 7 manual install ----> Ubuntu 22 Docker Container
- Number of Running Instances: 1
Logs
{“log”:“{"t":{"$date":"2023-11-08T23:10:23.587+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1699485023:587362][1:0x7f029cf86700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 580012, snapshot max: 580012 snapshot count: 0, oldest timestamp: (1699484723, 4) , meta checkpoint timestamp: (1699485022, 11) base write gen: 279345"}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:10:23.587682902Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:10:32.604+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn234","msg":"Slow query","attr":{"type":"command","ns":"rocketchat.$cmd","command":{"update":"users","ordered":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1699485032,"i":10}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"lsid":{"id":{"$uuid":"198683b7-efc9-4a0e-a98d-3e130ce88dca"}},"$db":"rocketchat"},"numYields":0,"reslen":260,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":3},"readConcern":{"level":"local","provenance":"implicitDefault"},"writeConcern":{"w":"majority","wtimeout":0,"provenance":"implicitDefault"},"storage":{},"remote":"172.18.0.3:47576","protocol":"op_query","durationMillis":121}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:10:32.605511134Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:10:42.771+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn237","msg":"Slow query","attr":{"type":"command","ns":"rocketchat.$cmd","command":{"update":"users","ordered":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1699485042,"i":3}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"lsid":{"id":{"$uuid":"489df412-742c-477b-9189-f4f4fdad9c12"}},"$db":"rocketchat"},"numYields":0,"reslen":260,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":1},"readConcern":{"level":"local","provenance":"implicitDefault"},"writeConcern":{"w":"majority","wtimeout":0,"provenance":"implicitDefault"},"storage":{},"remote":"172.18.0.3:47588","protocol":"op_query","durationMillis":420}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:10:42.77243385Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:10:42.775+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn65","msg":"Slow query","attr":{"type":"command","ns":"rocketchat.$cmd","command":{"update":"users","ordered":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1699485042,"i":5}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"lsid":{"id":{"$uuid":"3ae79da3-6cb5-4983-9cf0-b55f4cf8dd56"}},"$db":"rocketchat"},"numYields":0,"reslen":260,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":2},"readConcern":{"level":"local","provenance":"implicitDefault"},"writeConcern":{"w":"majority","wtimeout":0,"provenance":"implicitDefault"},"storage":{},"remote":"172.18.0.3:59872","protocol":"op_query","durationMillis":406}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:10:42.776029545Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:10:42.775+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn233","msg":"Slow query","attr":{"type":"command","ns":"rocketchat.$cmd","command":{"update":"rocketchat_sessions","ordered":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1699485042,"i":6}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"lsid":{"id":{"$uuid":"8b5dca36-a606-4adb-b181-ffbd54858244"}},"$db":"rocketchat"},"numYields":0,"reslen":260,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":1},"readConcern":{"level":"local","provenance":"implicitDefault"},"writeConcern":{"w":"majority","wtimeout":0,"provenance":"implicitDefault"},"storage":{},"remote":"172.18.0.3:47572","protocol":"op_query","durationMillis":396}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:10:42.776655316Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:11:13.070+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn84","msg":"Slow query","attr":{"type":"command","ns":"rocketchat.$cmd","command":{"update":"users","ordered":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1699485072,"i":10}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"lsid":{"id":{"$uuid":"489df412-742c-477b-9189-f4f4fdad9c12"}},"$db":"rocketchat"},"numYields":0,"reslen":260,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":3},"readConcern":{"level":"local","provenance":"implicitDefault"},"writeConcern":{"w":"majority","wtimeout":0,"provenance":"implicitDefault"},"storage":{},"remote":"172.18.0.3:59894","protocol":"op_query","durationMillis":480}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:11:13.071253984Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:11:13.074+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn234","msg":"Slow query","attr":{"type":"command","ns":"rocketchat.$cmd","command":{"update":"users","ordered":true,"$clusterTime":{"clusterTime":{"$timestamp":{"t":1699485072,"i":11}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"lsid":{"id":{"$uuid":"198683b7-efc9-4a0e-a98d-3e130ce88dca"}},"$db":"rocketchat"},"numYields":0,"reslen":260,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":1}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":1}},"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":3},"readConcern":{"level":"local","provenance":"implicitDefault"},"writeConcern":{"w":"majority","wtimeout":0,"provenance":"implicitDefault"},"storage":{},"remote":"172.18.0.3:47576","protocol":"op_query","durationMillis":383}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:11:13.074673094Z”}
{“log”:“{"t":{"$date":"2023-11-08T23:11:24.131+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1699485084:131879][1:0x7f029cf86700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 582800, snapshot max: 582800 snapshot count: 0, oldest timestamp: (1699484784, 5) , meta checkpoint timestamp: (1699485083, 22) base write gen: 279345"}}\n”,“stream”:“stdout”,“time”:“2023-11-08T23:11:24.132233297Z”}