Have problem with exporting file from GridFS

Description

Hello guys,

RocketChat become slowly when uploading/reading picture.

And @john.crisp (Community Manager) suggested me to export the file from GridFS with this script.

arminfelder/gridfsmigrate: RocketChat GridFS to filesytem migration script

To make the plan for doing it,I mongodump the rocketchat database from MongoDB which is working now.

Then do mongorestore to another VM which installed MongoDB.x86_64 (NOT in docker container. ver 4.4).

Executing the python script like below


python3 ./migrate.py -c dump -r rocketchat -t FileSystem -d /root/gridfs_dump

And here is the return error.


Traceback (most recent call last):

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 755, in next

    chunk = self._next_with_retry()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 747, in _next_with_retry

    return self._cursor.next()

  File "/usr/local/lib64/python3.6/site-packages/pymongo/cursor.py", line 1215, in next

    raise StopIteration

StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "./migrate.py", line 229, in <module>

    obj.dumpfiles("rocketchat_uploads", store)

  File "./migrate.py", line 104, in dumpfiles

    data = res.read()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 565, in read

    chunk_data = self.readchunk()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 528, in readchunk

    chunk = self.__chunk_iter.next()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 759, in next

    raise CorruptGridFile("no chunk #%d" % self._next_chunk)

gridfs.errors.CorruptGridFile: no chunk #0

Server Setup Information

  • Version of Rocket.Chat Server: 3.6.1

  • Version of MongoDB server: 4.4

  • Operating System: CentOS 7

  • Deployment Method: both RocketChat and MongoDB in Docker container

  • Number of Running Instances: 2

  • Firewalls involved: no

Thanks for your help.

As per conversations on open…

Have a search for this:
gridfs.errors.CorruptGridFile: no chunk

eg
https://jira.mongodb.org/browse/PYTHON-1552

There are a number of potential reasons - out of space etc etc

Personally I would get a good backup and then update your whole Rocket install first - if you want to be cautious do it a version at a time with backups between just to make sure the DB modifications work.

When moving DB make sure you move between the SAME versions of Rocket or you will hit problems.

Do NOT export from 3.6.1 and import to say 3.14.0

Make a sure you have plenty of space to export from GridFS

Something else I have just seen is you are running 3.6.1 on Mongo 4.4 ?

Did you import to Mongo 4.4 first and THEN try to export from GridFS? I’m not sure how good 4.4 support is.

This may need to be updated but…

https://docs.rocket.chat/installation/manual-installation/mongo-versions

1 Like

I’m very sure that the space of disk is quite enough.

Let me make sure I have it right.

  1. Make the good BACKUP BEFORE DONING ANYTHING. Yes,I got it.

  2. Update the RocketChat version will modify the MongoDB schema. So hold the MongoDB version and update RocketChat.

After update RocketChat make sure it is working well.

  1. Do NOT export DB from RocketChat 3.6.1 and import to 3.14.0, and so on.

Yes,I’m running RocketChat 3.6.1 on MongoDB 4.4.

Yes,I import to MongoDB 4.4 first and try to export from GridFS, maybe this is the answer why I have problem when exporting GridFS ?

I’ll try to use MongoDB ver 3.6.9 as official recommend.

Thank you again.

Have a nice day

Update

  1. arminfelder/gridfsmigrate still not working, but it’s OK.

  2. I follow John’s advise, upgrade RocketChat first (hold the MongoDB version).

Upgrade RocketChat 3.6.1 to 3.14.0 and the “migration” doing fine (version 205 → 222).

After that, I update the MongoDB/RocketChat node OS (yum upgrade -y) and it sloved the problem, picture upload/read won’t slowly as before.

  1. But as John said, it is not a good way to store file in GridFS. So I change the file upload type to AWS S3 (actually MinIO) and working fine.

Both old file in GridFS and the new upload store in MinIO could be showed.

Thanks again.

Glad that worked.

For the GridFS export you need lots of space to do it.

I would try and resolve that and then get all the old data out of GridFS to MinIO

If you have issues with that then please ask.

Thank you so much!

If the admin page shows “Total Uploads Size 3.40 GB”, how much storage space shoud I give?

(Actually there is about less than 3 GB in GridFS, 400 MB already in MinIO.)

Maybe I should wait for RocketChat compatibility MongoDB version 4.4 to get all the old data out of GridFS to MinIO. I prefer hold all old data in GridFS for now.

Yes, I have new issues about MinIO. If I need to post this in new thread, please that me know.


After I change the storage to MinIO (AWS S3 in RocketChat setting) the picture will got “retry” after 15 mins.

rocketchat_picture_retry


Used mc admin trace to show require status and do some search, I found that in the require url, there is a parameter X-Amz- Expires = 900 defined “AWS S3 presigned url” expired time. But I setted URLs Expiration Timespan: 0.

This “picture retry” situation will recover it self the next day.


02:30:09.936 [403 Forbidden] s3.GetObject minio-cvt.domain.com/rocket-cdn/kTKesiQB9hQ5NtsTg/uploads/2YwwxaS8hJqDDBPXF/gzHk8qHmbZaXCSCb7/apiuZWRLvcHeuDfNA?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=rocket%2F20210608%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210608T011121Z&X-Amz-Expires=900&X-Amz-Signature=c82d9b267c8b954b7da26a3ee2cdf62b56e8ab4952af69fe206d5319df45dbf4&X-Amz-SignedHeaders=host&response-content-disposition=inline%3B%20filename%3D%22IMG_8705.png.JPG%22  <Client_IP_Here>    1.146ms      ↑ 253 B ↓ 674 B


Hits “retry” for many times will show the XML like below.

<Error>
<Code>AccessDenied</Code>
<Message>Request has expired</Message>
<Key>
kTKesiQB9hQxxxxTg/uploads/2YwwxaS8hJqDDxxxx/vf3uP3j4rxnuyCyeb/MM7LuDWbxxxxx
</Key>
<BucketName>rocket-cdn</BucketName>
<Resource>
/rocket-cdn/kTKesiQB9hQ5NtsTg/uploads/2YwwxaS8hJqDDBPXF/vf3uP3j4rxnuyCyeb/MM7LuDWbWp65uFspZ
</Resource>
<RequestId>16822D87FE27xxxx</RequestId>
<HostId>60e46de2-deb8-4c87-8f51-xxxxx</HostId>
</Error>

Server Setup Infomation:

  • Version of Rocket.Chat Server: 3.14.0

  • Version of MongoDB server: 4.4

  • Operating System: CentOS 7 (all I used)

  • Deployment Method: Docker (except Nginx reverse proxy)

  • Number of Running Instances: 2

  • Proxy: all server behind Nginx reverse proxy

  • Firewalls involved: no

  • MinIO version: 2021-05-27T22:06:31Z

Amazon S3 settings in RocketChat admin page.

  • Bucket name: rocket-cdn

  • Acl: <null>

  • Access Key: accesskey

  • Secret Key: secretkey

  • CDN Domain for Downloads: https://minio-cvt.domain.com/

  • Region: <null>

  • Bucket URL: https://minio-cvt.domain.com/ (If I use https://minio-cvt.domain.com/rocket-cdn/ it could’t worked.)

  • Signature Version: v4

  • Force Path Style: true

  • URLs Expiration Timespan: 0

Hi,

First thing to do is upgrade your Rocket to at least 3.14.4 - please read the release notes for why.

More than that :wink:

I think you need enough to store a whole copy. So several Gb to be sure.

If you are already on Mongo 4.4, which as far as I am aware is not officially supported yet, then the exporter may not work- it is not in our control! However, I don’t suggest you try and go back to an earlier Mongo DB version now.

There is another exporter in the Rocket.Chat repo you could try that may possibly work - I’ll try and find it.

As far as CDN goes, I’d disable it first and test without it. That can cause all sorts of issues so you need to figure out where the problem really lies - either with Rocket or CDN.

Just read the release note both v3.14.4 and v3.14.5 (latest) but still don’t know why. (Nothing change about AWS S3)

I have enough disk space when used the GridFS expoter script which you suggested.

However it is not my priority for now.

If I can’t fix the “picture retry” issue, there is no meaning to export all old file to MinIO.

During these days, I tried bypass CDN to access MinIO/RocketChat, the “retry” problem still happen.

Also tried those settings…