Images broken in chat after migration to Docker

Community Support
1

Description

We recently migrated from SNAP to Docker on Rocky 9.7.

Server Setup Information

  • Version of Rocket.Chat Server: 8.2.1
  • Operating System: Rocky 9.7
  • Deployment Method: Docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog:
  • NodeJS Version:
  • MongoDB Version: 8.0.20
  • Proxy: Traefik (Docker)
  • Firewalls involved: Sonicwall 3650

Any additional Information

We are running into a problem where image thumbnails appear randomly as broken. On iPhone app, the images almost always load. On the Windows Rocketchat app, they sometimes load. On a web browser they appear broken almost always. When you add a new image the thumbnail works, for a while. The images are there, if you click really fast on a broken link it will show the whole image. If you click the download button they will download fine. But the process of showing the thumbnail is broken, but not universally broken. The web browser is generally showing a 404 in devtools for the image. We are using the FileSystem Storage Type. The folder is bound mounted in Docker to a folder on the parent OS. Permissions are fine. The app can access the files, it’s just sometimes the thumbnail does not render. But it works nearly all the time on iPhone app. It’s doubtful it is the reverse proxy as it was a problem before we made the site live, when I was just accessing it directly by IP internally in our data center.

If you click the retry button, a quick flash of the image will show and then it will go back to the retry screen. But if you double click really fast it will actually load the image, like you click Retry and then super fast click the blurred out image, it will bring up the image.

There is one other problem where the OpLog is showing disabled but if I manually check in mongodb the replica set looks fine and has activity.

image
image622×382 12.8 KB

2

I’d read this and sort out your DB before you go any further. Make sure your backups are ok.

Your server logs will give you clues eg permissions/paths.

Note user perms for docker may not be the same as snap, and paths in the DB may now be wrong.

3

Just want to be clear you’re saying that moving from Snap v6 to Docker v8 there might be changes in how the database is organized? But are there tools for remedying this? Or only for paid support? I’m not finding glaring issues in the DB and users can send/receive messages and images, just old ones are somewhat broken (can still download them and see them in gallery mode). Replica set is primary. I did a test on a separate server where I migrated MongoDB 6 to 7 and 8 before even starting RC, but it’s still showing oplog disabled, and like I said the only thing is these images not working, and they work fine on iPhone app. If iPhone app is working seamlessly how can there be problems with paths in the DB?

4

You haven’t really documented your exact sequence to get from working snap to broken docker. So it’s hard to be specific.

See the “xy problem”.

From experience your issues most likely stem from not reading docs properly before you moved and not properly testing.

For a start the db is not the same name in snaps as docker. Snaps have potentially differently stored filepath names in the db (there’s a thread on similar somewhere), and the files have different permissions & owners.

There will be a lot of internal db changes between 6, 7, 8, and even 8.2 Hence the changelogs & warnings. That’s a few years of big changes.

You should have had copious backups and checked Rocket was running correctly after each db upgrade. There are docs to tell you how to do it safely.

There is no simple answer to any of it beyond start from “last good backup”. Again, from the symptoms and educated guess I suspect file paths/perms. Your app may have cached files etc - insufficient information to tell. Best test is a clean browser.

No idea on oplogs - normally set enabled for years though not sure it is required now - check the docs/docker compose files.

Your logs will have the answers. Have you read them?

You don’t mention licence type or number of users.

If you have paid support then you can open a ticket. If not it’s open source….

5

All good friend. This is a migration, so we have the old server as is and backups are not a huge concern. In response to your questions, I did a mongodump from the snap server, and migrated it to the docker server. The mongodump was v6 as per Snap 7, what we are on on our old machine. We can roll back, but we are concerned about disruptions to our service with snap, and I do understand that there are limitations to migrating from snap to docker. We’re trying to get to a stable place.

I migrated the dump to new server, copied to docker, imported using mongosh on V6.0 in my docker-compose.yml. I upgraded stepwise from 6 to 7 and then to 8. I did this before even turning on Rocketchat to ensure that the upgrade was effective. Replicaset showing primary and no issues with the DB. I do understand that this is a nine year old instance and switched from gridfs to filesystem, however, all of the images are properly shared to a bind mount directory on host OS, and they work in the iphone app, no problem. We’ve tried multiple browsers and all of them have issues with older images. We’ve done cache resets.

Like I said, everything is operational except for the thumbnail display on browsers. We’ve already resigned that this might be the best we can get. Not trying to battle.

We were seeing sharing issues in the logs, and corrected for those, so the images can be downloaded and are visible in the gallery viewer, so the app can see the images, all of them, but when it does the thumbnail load it fails for some reason, only on browsers.

I was concerned about oplog disabled but I’ve read that V8 of RocketChat is moving away from that, so I wasn’t as concerned.

I do understand I didn’t give full detail, trying to flesh that out here. Also, I understand it’s free software, so I get it. No worries.

I apologize, I didn’t install this server, I inherited it, so I’m not super versed in the license. We only have 40 active users.

6

Assume you setup mingo with a replicaset before importing?

That was probably yoir worsr mistake.

There are db tweaks almost every update, and big ones in a version change.

It probably tried to run them all at once so effectively upgrading directly from 6.x to 8.x

Every chance that that created issues at that will not have been tested.

You didn’t read the docs properly (some of this I wrote myself years ago!)

When? Was everything migrated?

Were all the filepaths properly updated (check the collections themselves)?

There are two migration scripts around if you search. I’d check that.

What happens if you check the url in the dev console?

Normally the fussiest of all eg https etc.

Have you tried on an iphone that is clean?

Sounds exactly like a cache issue apart from the iphone.

Can’t remember if Rocket creates a thumb and caches it somewhere.

Need to check the urls as above.

Sorry way late here & need zzzzs. Let us know.

7

Just so you know we’re mainly on Teams and this is a small team project, so we’re not in meltdown about this. And all new images work fine.

Yes, MongoDB 6.0 with replicaset verified as primary before starting

I will look at this more. We have been looking at docs to try to remedy this. We have tried numerous times. We were on RC V7 on snap, so moving to 8 is not a huge jump in RC version.

So there was in the past a move to filesystem, however, the problem is with new images. If there is a good way to reconstruct thumbnails that would be helpful to try. I also switched the setting to turn off thumbnail max resolution to see if it would recreate thumbnails.

In the Dev console, when you are scrolling up and it tries to create the thumbnail you get a 404. However, the images are available with the cloud download button and show up in the gallery viewer. Here is an example:

This link shows 404 in Devtools

https://chat.digitalreefinc.net/file-upload/69987d52243d7d12484fad27/Clipboard%20-%20February%2020,%202026%2010:27%20AM.png

This link works

https://chat.digitalreefinc.net/file-upload/69987d52243d7d12484fad26/Clipboard%20-%20February%2020,%202026%2010:27%20AM.png?download

So I’m assuming there is a DB problem associated with that file. We’ve compared ones that work and ones that don’t and tried to do surgical corrections in the DB and it didn’t work.

8

IIRC image thumbs are done on the fly/lazy loaded. Don’t think they’re stored in the db at all.

Almost certainly this db between your swapped storage type & db upgrades.

7-8 IS a big jump, as was 6-7. Don’t underestimate it, or the lack of testing…

Migrations of that bug a leap are not usually tested (don’t blame me….!). At best the last few supported versions.

You MUST restore your snap db to the exact same rocket version you backed up.

Again, you really must run rocket after each incremental update to ensure db changes are applied.

It is extremely fussy & temperamental.

I’d go back,

I’d go back, check for the file store migrator scripts and see if they help, and then upgrade more incrementally with tests between to check it is working.

Check the required mongo versions against the Rocket versions too.