# Have problem with exporting file from GridFS

**URL:** <https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071>\
**Category:** Community Support\
**Created:** [April 29, 2021, 11:30am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071 "2021-04-29T11:30:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![xxzk](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xxzk](https://forums.rocket.chat/u/xxzk)\
**Post date:** [April 29, 2021, 11:30am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/1 "2021-04-29T11:30:51Z")

</div>

### Description

Hello guys,

RocketChat become slowly when uploading/reading picture.

And @john.crisp (Community Manager) suggested me to export the file from GridFS with this script.

arminfelder/gridfsmigrate: RocketChat GridFS to filesytem migration script

> **[GitHub - arminfelder/gridfsmigrate: RocketChat GridFS to filesytem migration script](https://github.com/arminfelder/gridfsmigrate)**
>
> RocketChat GridFS to filesytem migration script

To make the plan for doing it,I `mongodump` the `rocketchat` database from MongoDB which is working now.

Then do `mongorestore` to another VM which installed MongoDB.x86\_64 (NOT in docker container. ver 4.4).

Executing the python script like below

```auto

python3 ./migrate.py -c dump -r rocketchat -t FileSystem -d /root/gridfs_dump

```

And here is the return error.

```auto

Traceback (most recent call last):

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 755, in next

    chunk = self._next_with_retry()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 747, in _next_with_retry

    return self._cursor.next()

  File "/usr/local/lib64/python3.6/site-packages/pymongo/cursor.py", line 1215, in next

    raise StopIteration

StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "./migrate.py", line 229, in <module>

    obj.dumpfiles("rocketchat_uploads", store)

  File "./migrate.py", line 104, in dumpfiles

    data = res.read()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 565, in read

    chunk_data = self.readchunk()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 528, in readchunk

    chunk = self.__chunk_iter.next()

  File "/usr/local/lib64/python3.6/site-packages/gridfs/grid_file.py", line 759, in next

    raise CorruptGridFile("no chunk #%d" % self._next_chunk)

gridfs.errors.CorruptGridFile: no chunk #0

```

### Server Setup Information

- Version of Rocket.Chat Server: 3.6.1

- Version of MongoDB server: 4.4

- Operating System: CentOS 7

- Deployment Method: both RocketChat and MongoDB in Docker container

- Number of Running Instances: 2

- Firewalls involved: no

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![john.crisp](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/john.crisp/32/2711_2.png) [@john.crisp](https://forums.rocket.chat/u/john.crisp)\
**Post date:** [April 29, 2021, 11:48am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/2 "2021-04-29T11:48:41Z")

</div>

As per conversations on open…

Have a search for this:  
gridfs.errors.CorruptGridFile: no chunk

eg  
[https://jira.mongodb.org/browse/PYTHON-1552](https://jira.mongodb.org/browse/PYTHON-1552)

There are a number of potential reasons - out of space etc etc

Personally I would get a good backup and then update your whole Rocket install first - if you want to be cautious do it a version at a time with backups between just to make sure the DB modifications work.

When moving DB make sure you move between the SAME versions of Rocket or you will hit problems.

Do NOT export from 3.6.1 and import to say 3.14.0

Make a sure you have **plenty** of space to export from GridFS

Something else I have just seen is you are running 3.6.1 on Mongo 4.4 ?

Did you import to Mongo 4.4 first and THEN try to export from GridFS? I’m not sure how good 4.4 support is.

This may need to be updated but…

[https://docs.rocket.chat/installation/manual-installation/mongo-versions](https://docs.rocket.chat/installation/manual-installation/mongo-versions)

---

<div class="post-metadata">

**Author:** ![xxzk](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xxzk](https://forums.rocket.chat/u/xxzk)\
**Post date:** [May 3, 2021, 2:10am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/3 "2021-05-03T02:10:16Z")

</div>

I’m very sure that the space of disk is quite enough.

Let me make sure I have it right.

1. Make the good BACKUP BEFORE DONING ANYTHING. Yes,I got it.

2. Update the RocketChat version will modify the MongoDB schema. So **hold the MongoDB version and update RocketChat.**

After update RocketChat make sure it is working well.

1. Do NOT export DB from RocketChat 3.6.1 and import to 3.14.0, and so on.

Yes,I’m running RocketChat 3.6.1 on MongoDB 4.4.

Yes,I import to MongoDB 4.4 first and try to export from GridFS, maybe this is the answer why I have problem when exporting GridFS ?

I’ll try to use MongoDB ver 3.6.9 as official recommend.

Thank you again.

Have a nice day

---

<div class="post-metadata">

**Author:** ![xxzk](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xxzk](https://forums.rocket.chat/u/xxzk)\
**Post date:** [May 21, 2021, 2:44am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/4 "2021-05-21T02:44:20Z")

</div>

Update

1. arminfelder/gridfsmigrate still not working, but it’s OK.

2. I follow John’s advise, upgrade RocketChat first (hold the MongoDB version).

Upgrade RocketChat 3.6.1 to 3.14.0 and the “migration” doing fine (version 205 → 222).

 ![docker_logs_after_upgrade_to_3_14_0_Node2_0](https://us1.discourse-cdn.com/flex021/uploads/rocketchat/original/2X/6/6b80bfcf474af29ed89aad6fde96d3a34c0a9f15.jpeg)

After that, I update the MongoDB/RocketChat node OS (`yum upgrade -y`) and it sloved the problem, picture upload/read won’t slowly as before.

1. But as John said, it is not a good way to store file in GridFS. So I change the file upload type to AWS S3 (actually MinIO) and working fine.

**Both old file in GridFS and the new upload store in MinIO could be showed.**

Thanks again.

---

<div class="post-metadata">

**Author:** ![john.crisp](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/john.crisp/32/2711_2.png) [@john.crisp](https://forums.rocket.chat/u/john.crisp)\
**Post date:** [May 21, 2021, 10:14am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/5 "2021-05-21T10:14:43Z")

</div>

Glad that worked.

For the GridFS export you need lots of space to do it.

I would try and resolve that and then get all the old data out of GridFS to MinIO

If you have issues with that then please ask.

---

<div class="post-metadata">

**Author:** ![xxzk](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xxzk](https://forums.rocket.chat/u/xxzk)\
**Post date:** [June 8, 2021, 2:37am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/6 "2021-06-08T02:37:27Z")

</div>

Thank you so much!

If the admin page shows “Total Uploads Size 3.40 GB”, how much storage space shoud I give?

(Actually there is about less than 3 GB in GridFS, 400 MB already in MinIO.)

Maybe I should wait for RocketChat compatibility MongoDB version 4.4 to get all the old data out of GridFS to MinIO. I prefer hold all old data in GridFS for now.

Yes, I have new issues about MinIO. **If I need to post this in new thread, please that me know.**

* * *

After I change the storage to MinIO (AWS S3 in RocketChat setting) **the picture will got “retry” after 15 mins.**

 ![rocketchat_picture_retry](https://us1.discourse-cdn.com/flex021/uploads/rocketchat/original/2X/e/e05ab450984faacf7ba723d1d47322c8dc49cad6.jpeg)

  

Used `mc admin trace` to show require status and do some search, I found that in the require url, there is a parameter `X-Amz- Expires = 900` defined “AWS S3 presigned url” expired time. But I setted **URLs Expiration Timespan: 0**.

This “picture retry” situation will **recover it self the next day**.

```auto

02:30:09.936 [403 Forbidden] s3.GetObject minio-cvt.domain.com/rocket-cdn/kTKesiQB9hQ5NtsTg/uploads/2YwwxaS8hJqDDBPXF/gzHk8qHmbZaXCSCb7/apiuZWRLvcHeuDfNA?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=rocket%2F20210608%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210608T011121Z&X-Amz-Expires=900&X-Amz-Signature=c82d9b267c8b954b7da26a3ee2cdf62b56e8ab4952af69fe206d5319df45dbf4&X-Amz-SignedHeaders=host&response-content-disposition=inline%3B%20filename%3D%22IMG_8705.png.JPG%22 <Client_IP_Here> 1.146ms ↑ 253 B ↓ 674 B

```

  

Hits “retry” for many times will show the XML like below.

```auto
<Error>
<Code>AccessDenied</Code>
<Message>Request has expired</Message>
<Key>
kTKesiQB9hQxxxxTg/uploads/2YwwxaS8hJqDDxxxx/vf3uP3j4rxnuyCyeb/MM7LuDWbxxxxx
</Key>
<BucketName>rocket-cdn</BucketName>
<Resource>
/rocket-cdn/kTKesiQB9hQ5NtsTg/uploads/2YwwxaS8hJqDDBPXF/vf3uP3j4rxnuyCyeb/MM7LuDWbWp65uFspZ
</Resource>
<RequestId>16822D87FE27xxxx</RequestId>
<HostId>60e46de2-deb8-4c87-8f51-xxxxx</HostId>
</Error>

```

### Server Setup Infomation:

- Version of Rocket.Chat Server: 3.14.0

- Version of MongoDB server: 4.4

- Operating System: CentOS 7 (all I used)

- Deployment Method: Docker (except Nginx reverse proxy)

- Number of Running Instances: 2

- Proxy: all server behind Nginx reverse proxy

- Firewalls involved: no

- MinIO version: 2021-05-27T22:06:31Z

### Amazon S3 settings in RocketChat admin page.

- Bucket name: rocket-cdn

- Acl: `<null>`

- Access Key: `accesskey`

- Secret Key: `secretkey`

- CDN Domain for Downloads: `https://minio-cvt.domain.com/`

- Region: `<null>`

- Bucket URL: `https://minio-cvt.domain.com/` (If I use `https://minio-cvt.domain.com/rocket-cdn/` it could’t worked.)

- **Signature Version: v4**

- Force Path Style: `true`

- **URLs Expiration Timespan: 0**

---

<div class="post-metadata">

**Author:** ![john.crisp](https://sea1.discourse-cdn.com/flex021/user_avatar/forums.rocket.chat/john.crisp/32/2711_2.png) [@john.crisp](https://forums.rocket.chat/u/john.crisp)\
**Post date:** [June 8, 2021, 9:21am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/7 "2021-06-08T09:21:41Z")

</div>

Hi,

First thing to do is upgrade your Rocket to at least 3.14.4 - please read the release notes for why.

> [@xxzk](#):
>
> If the admin page shows “Total Uploads Size 3.40 GB”, how much storage space shoud I give?

More than that 😉

I think you need enough to store a whole copy. So several Gb to be sure.

If you are already on Mongo 4.4, which as far as I am aware is not officially supported yet, then the exporter may not work- it is not in our control! However, I don’t suggest you try and go back to an earlier Mongo DB version now.

There is another exporter in the Rocket.Chat repo you could try that may possibly work - I’ll try and find it.

As far as CDN goes, I’d disable it first and test without it. That can cause all sorts of issues so you need to figure out where the problem really lies - either with Rocket or CDN.

---

<div class="post-metadata">

**Author:** ![xxzk](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xxzk](https://forums.rocket.chat/u/xxzk)\
**Post date:** [June 8, 2021, 9:55am UTC](https://forums.rocket.chat/t/have-problem-with-exporting-file-from-gridfs/11071/8 "2021-06-08T09:55:44Z")

</div>

Just read the release note both v3.14.4 and v3.14.5 (latest) but still don’t know why. (Nothing change about AWS S3)

> **[Release 3.14.5 · RocketChat/Rocket.Chat](https://github.com/RocketChat/Rocket.Chat/releases/tag/3.14.5)**
>
> 2021-06-06 · 1 🚀 · 1 🐛 · 1 👩‍💻👨‍💻
> Engine versions
> 
> Node: 12.22.1
> NPM: 6.14.1
> MongoDB: 3.4, 3.6, 4.0
> Apps-Engine: 1.25.0
> 
> 🚀 Improvements
> 
> 
> Send only relevant data via WebSocket (#22258)
> Previo...

I have enough disk space when used the GridFS expoter script which you suggested.

However it is not my priority for now.

If I can’t fix the “picture retry” issue, there is no meaning to export all old file to MinIO.

During these days, I tried bypass CDN to access MinIO/RocketChat, the “retry” problem still happen.

Also tried those settings…

- Signature Version: `v2` or `<null>`

- Region: `us-east-1` ([Minio - Rocket.Chat Docs](https://docs.rocket.chat/guides/administrator-guides/settings/file-upload/minio))