Howto migrate the files from gridfs to filesystem

[aaron.ogle]

Conversation ported from issue:

Question: Howto Migrate the files from Gridfs to Filesystem?

Conversation

@himpich

We used (the default) gridfs for upload / avatar storage. As this doesn’t scale very good, our Rocketchat installation frequently crashes (after > 100% cpu load on the node instances or mongodb). Now I switched to “filesystem”.

I there any way to convert / migrate / move the files from gridfs to the filesystem storage or am I stuck with the old uploaded files / attachments?

Thanks in advance,
Stefan

@localguru

Would like to know that too!

@aaron.ogle

This would definitely require some decent scripting to pull off.

If you take a look in rocketchat_uploads collection you’ll see that each record has something like this:

{ "_id" : "KkYS33J4wY2Y2a5iY", "name" : "Clipboard - January 2, 2018 2:21 PM", "size" : 81127, "type" : "image/png", "rid" : "{roomid}", "description" : "", "store" : "GridFS:Uploads", "complete" : true, "uploading" : false, "extension" : "", "progress" : 1, "userId" : "{userid}", "_updatedAt" : ISODate("2018-01-02T20:21:15.885Z"), "instanceId" : "{instance_id}", "etag" : "{e_tag}", "path" : "/ufs/GridFS:Uploads/KkYS33J4wY2Y2a5iY/Clipboard%20-%20January%202,%202018%202:21%20PM", "token" : "{token}", "uploadedAt" : ISODate("2018-01-02T20:21:17.147Z"), "url" : "/ufs/GridFS:Uploads/KkYS33J4wY2Y2a5iY/Clipboard%20-%20January%202,%202018%202:21%20PM" }

As well then the actual files are in rocketchat_uploads.files with their binary data in rocketchat_uploads.chunks

Not tested… purely psuedo

1. Set your file store to filesystem and set your folder. In this case lets assume /uploads
2. Then export all of the files from gridfs and put them in a folder somewhere
3. You would need to then loop over this collection and for each record find the file in the folder matching that id and move it in to the appropriate subfolder. (I think rocketchat_uploads will only have GridFS:Upload)
4. Then after the file was there you would need to modify the store to Filesystem:Uploads (or what ever the store is). You’d also have to modify the url too.

Also note… if you move to s3/googlestorage or other block stores I believe the url is prefixed by your server id which can be found in administration->info

The same would also need done for rocketchat_avatars collection and its gridfs collection

[uniquename]

I have instaled ubuntu 20, RC instaled as snap and cant switch to filesystems. All the time when upload files i have "Error: EACCES: permission denied, open ‘/home/master/RCuploads/xxx’ [ufs: cannot upload file]
I dont have any idea how to setup folder permissions to make it to work. Please help someone