Recently the mobile app has stopped loading images for most of my users. My own app (my user is admin, theirs is not), seemed to work ok, so I didn’t realize the problem at all in the beginning. I think it’s being going on for about a week. With any user, the images work fine in a browser (even android browser on the same phone) just not in the app. Cache resets, reinstalls or relogins do not seem to help.
I see nothing at all in the rocket server logs about the image loading attempt, but my proxy access logs show that a successful attempt to a file looks pretty much like a failed one, though the access logs doesn’t show the headers.
Here’s a successful request:
“GET /file-upload/67dfc9520e580db2ce33c3a8/1000000526.jpg HTTP/2.0” 200 7483209 “-” “okhttp/4.9.2”
And here’s a failed one on the same file
“GET /file-upload/67dfc9520e580db2ce33c3a8/1000000526.jpg HTTP/2.0” 403 0 “-” “okhttp/4.9.2”
If I disable the option in Settings->File Upload->Protect uploaded files, the apps start showing the images fine again. Obviously I don’t want to do that as a long-term solution. The 403 (forbidden) and that setting seem to both indicate that the mobile client isn’t authorized to download the images for some reason. Since the same user is authorized to do that in a browser, perhaps the mobile client isn’t sending the cookies as it should or something?
Any tips on where to go from here debugging this? Anyone else having the same issue?
Server Setup Information
Version of Rocket.Chat Server: 7.4.1
Operating System: Debian
Deployment Method: docker (official image)
Number of Running Instances: 1
DB Replicaset Oplog: oplog Enabled
NodeJS Version: v22.13.1
MongoDB Version: 7.0.17 / wiredTiger
Proxy: nginx
Firewalls involved: Server firewalld + router NAT. As the request comes through and gets 403, that’s not likely the issue.
Any additional Information
I found some other topics with similar behavior caused by image downloads being opened in system browser (which is not logged in), so I also tried enabling the mobile option for setting default browser as “in-app”, but that changed nothing. I also tried the rocket.chat experimental app from play store, but that behaved exactly the same.
The issue still persists for me, the only difference is that now it’s happening on all my users Android clients. The previous randomness now seems gone and user permissions don’t appear to affect anything.
Searching for similar issues:
I have tried searching both here and in the github, but haven’t found quite an accurate match. Sorry if I missed some issue, but to me, they all seemed a bit different.
Just to clarify what I’m seeing:
Uploading (POST) images works, both in browser and the Android app.
Viewing images work in a browser, but not the Android app. They work even when using a browser (chrome or firefox) on the same Android phone with the same user. Only the app has this issue.
All avatar images, custom emoji, etc. work fine both in the browser and the Android app.
It’s even possible to successfully upload a new image from the mobile client but it still won’t manage to GET it back and display it. It does show for browser clients, so uploading works.
Now, I’ve done some more debugging in the meantime. Here’s what I found:
On the interim when some mobiles were working and others were not, I added some debug logging in the nginx config. The issue seemed fairly clear on server end:
The failed requests get a 403 from the server. There’s no other errors in the log and the response is immediate - not a hang or a performance thing.
Other images (like avatar images) worked on both clients. These requests appear to send rc_uid and rc_token as query parameters (visible in access log)
The GET requests on the /file-uploads path (where images posted in chat go) don’t use the same query params, but authenticate differently:
The working session sends rc_uid and rc_token as cookies (visible by logging $http_cookie on the nginx).
The non-working session sends no cookies at all. I guess this is the heart of the issue, but I don’t know why the cookies are not included.
I don’t see any particular errors in the logs. The GET /file-uploads requests are the only ones that fail with 403.
If I disable the upload file protection from the server, the mobile clients work fine. As such, it seems obvious that the server can return the images, it just refuses to do so, since the client doesn’t send the authentication tokens on those requests.
My next step will be to debug this in android studio. I’ve got the app running in a emulator (and the issue does reproduce there as well), but I haven’t had the time to read into the code enough to know what to look at.
As for my setup and GridFS
I migrated away from GridFS after reading the docs when this issue appeared. That did not affect the issue. Even on gridfs my perf was decent and the issue was mobile-only… but anyhow, now I’m using FileSystem and the issue still persists.
The FileSystem config is working and permissions are fine, as everything is fine when opening Rocket.Chat in a browser: I can see all old images, I can upload new ones, all users can view them.
My certs (letsencrypt) are valid - no complaints about them in a browser
Websockets do work when checked in browser network tab
I have had this installation a long time and migrated through various versions, mongo updates, migrating to wiredtiger etc. along the way. As such, something could be a little funky in my setup. The mobile-only nature of the issue seems to indicate it’s not about that, though.
My nginx setup is a little more complex than the docs version, as I run multiple services behind a SWAG nginx. I guess there could be something off in my config that would just affect the android client.
My setup has worked fine for years, including with the mobile app. This has become a problem during this year, and I’m not aware of any particular change that would have caused it, except for regular updates on the software itself.
Sorry for the long response, just trying to be throughout.
OK, so I spent a while debugging the android app to see exactly what goes wrong and I managed to fix it for myself. It was indeed an nginx configuration issue.
tl;dr: I needed to add this to my nginx config:
large_client_header_buffers 4 16k;
After that, each user needs to re-login so that the settings refresh.
The longer version (in case your issue differs and you need a point to start)
Why the fetching fails: When fetching images, the mobile app uses formatAttachmentUrl (in formatAttachmentUrl.ts file) to conditionally add the authentication tokens to the request, depending on whether the setting FileUpload_ProtectFiles is true or false.
In my case, the entire setting was missing in redux store, along with most other settings. Missing setting falls throug the conditional so no auth-info is present in the request and the service correctly replies with 403. So why were they missing?
After login, the settings are fetched via the function getSettings (in getSettings.ts).
This was not getting anything in my case, but neither was it logging anything. So I logged the request and realized it was extremely long as it lists all possible settings as queryparams. This request failed with a 414 on my nginx as my config was limiting the header size too much.
I probably missed this in going through my logs earlier because I was just looking at the image requests. The settings fetch happens at login and even if it fails, the app continues (mostly) working.
It’s a little surprising that the app works as well as it does, since the issue prevented almost all server settings from reaching the client.