GridFS migrator

Community Creations
1

I created a primitive python script allowing to migrate uplods stored in GridFS to FileSystem records.

please feel free to contribute improvements

2 Likes
2

Armin, just a short question: why should you use “Filesystem” instead of “GridFS” for storing files?

3

I can see some sort of nfs being useful in certain installations

4

In my case the mongo DB was simply becoming to big, as well as having the files on a NFS or GridFS storage makes it easier to maintain

1 Like
5

Hello

Need Help

root@ip:~# ./migrate.py -c dump -d /app/uploads -r rocketchat -t AmazonS3 -d bucket_name

  1. Dumping iMWFJMgrwKXT9Bm7w Audio record.mp3
    Traceback (most recent call last):
    File “./migrate.py”, line 232, in
    obj.dumpfiles(“rocketchat_uploads”, store)
    File “./migrate.py”, line 117, in dumpfiles
    key = store.put(filename, data, upload)
    File “./migrate.py”, line 68, in put
    entry[‘name’]) + ‘"’)
    File “/usr/local/lib/python3.6/dist-packages/boto3/resources/factory.py”, line 520, in do_action
    response = action(self, *args, **kwargs)
    File “/usr/local/lib/python3.6/dist-packages/boto3/resources/action.py”, line 83, in call
    response = getattr(parent.meta.client, operation_name)(**params)
    File “/usr/local/lib/python3.6/dist-packages/botocore/client.py”, line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
    File “/usr/local/lib/python3.6/dist-packages/botocore/client.py”, line 648, in _make_api_call
    operation_model, request_dict, request_context)
    File “/usr/local/lib/python3.6/dist-packages/botocore/client.py”, line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
    File “/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py”, line 102, in make_request
    return self._send_request(request_dict, operation_model)
    File “/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py”, line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
    File “/usr/local/lib/python3.6/dist-packages/botocore/endpoint.py”, line 116, in create_request
    operation_name=operation_model.name)
    File “/usr/local/lib/python3.6/dist-packages/botocore/hooks.py”, line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
    File “/usr/local/lib/python3.6/dist-packages/botocore/hooks.py”, line 228, in emit
    return self._emit(event_name, kwargs)
    File “/usr/local/lib/python3.6/dist-packages/botocore/hooks.py”, line 211, in _emit
    response = handler(**kwargs)
    File “/usr/local/lib/python3.6/dist-packages/botocore/signers.py”, line 90, in handler
    return self.sign(operation_name, request)
    File “/usr/local/lib/python3.6/dist-packages/botocore/signers.py”, line 157, in sign
    auth.add_auth(request)
    File “/usr/local/lib/python3.6/dist-packages/botocore/auth.py”, line 425, in add_auth
    super(S3SigV4Auth, self).add_auth(request)
    File “/usr/local/lib/python3.6/dist-packages/botocore/auth.py”, line 357, in add_auth
    raise NoCredentialsError
    botocore.exceptions.NoCredentialsError: Unable to locate credentials

My Amazon S3 Settings
Amamzon S3 ( added the following fields Bucket Name, acl, Access Key , Secret Key, Region, Bucket url , Force Path ( set to True) ,Signature Version (v4) URLs Expiration Timespan ( 120), Proxy Avatars ( false), Proxy Uploads (false) )

Am i missing anything.

6

Can this be used for migrating Emojis as well?

7

@Beezus in principle it should also work for avatars and emojis, but you would need to run it on e.g. “rocketchat_avatars” instead of the rocketchat_uploads collection, please have a look at ln:233,
and in case please make a pull request, to share your enhancement

8

I had actually just make my own scripts that download all emosjis to a folder, deletes the emojis, then once we change to filesystem storage i run another script that reuploads all of the emojis Kind of a roundabout way but it got the job done.