Some users' avatars are disappearing

Community Support
1

Description

Some users’ avatars are gone and there is no URL etag. I’m using the filesystem approach and I don’t use a retention policy.

Also, some messages are disappearing and “retry” is happening.

Please help

Server Setup Information

  • Version of Rocket.Chat Server: 3.16.0
  • Operating System: CentOS 7
  • Deployment Method: Hosted
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enable
  • NodeJS Version: 12.18.4 - x64
  • MongoDB Version: 4.0.23
  • Proxy: Nginx
  • Firewalls involved: OS Firewall

Any additional Information

2021/07/26 17:56:32 [error] 13976#0: *169 upstream prematurely closed connection while reading upstream, client: 192.168.1.1, server: test.kro.kr, request: “GET /avatar/Zabbix HTTP/1.1”, upstream: “http://127.0.0.1:3000/avatar/Zabbix”, host: “test.kro.kr”, referrer: “https://test.kro.kr/group/monitoring”

2021/07/26 17:56:42 [error] 13975#0: *214 upstream prematurely closed connection while reading upstream, client: 192.168.1.1, server: test.kro.kr, request: “GET /avatar/room/Wic9vFQbNGKZnwWsy HTTP/1.1”, upstream: “http://127.0.0.1:3000/avatar/room/Wic9vFQbNGKZnwWsy”, host: “test.kro.kr”, referrer: “https://test.kro.kr/direct/YnffmZCTw997b72qKrkDayorodpeHt6MRw”

2

First thing to do is upgrade. Don’t use ‘latest’ tag, but do check releases all the time.

Which filesystem? GridFS? Something else?

3

The storage type is file system. I’m using it for the path “/opt/Rocket.Chat/data/”

4

OK.

Please. upgrade.

Let us know your hardware specs, and rough number of seats.

Then let us know what you have in your nginx file - I suspect that this is the issue somehow.

Also a generic read for your error “upstream prematurely closed connection while reading upstream”

5

using 2 CPU cores and 8 GB of memory, and the number of concurrent users is 60-70. I don’t think running out of resources.

I added proxy_read_time and proxy_connect_timeout options to solve “upstream prematurely closed connection while reading upstream”, but it is the same.

Migrated Rocket Chat to another server. There are no errors after migration, but the avatar and some images will disappear the same.

I suspect that old avatar images and uploaded files are being automatically deleted. Can you determine the cause?

Also Where are the avatar images stored? It doesn’t seem to be stored in FileSystem.

# Upstreams
upstream backend {
    server 127.0.0.1:3000;
}

# HTTPS Server
server {
    listen               443 ssl;
    #ssl                  on;
    ssl_certificate      /etc/nginx/ssl/certificate.crt;
    ssl_certificate_key  /etc/nginx/ssl/private.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    server_name  test.kro.kr;
    access_log   /var/log/nginx/nginx.vhost.access.log;
    error_log    /var/log/nginx/nginx.vhost.error.log;

    location     / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;

        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;

        proxy_redirect off;
    }
}
6

HMmm…

OK, here’s my recent working nginx config with letsencrypt for comparison.

So you are on 3.16.3 correct?

Not without more information. What are your logs telling you? Both your Rocket.Chat logs plus your nginx and system logs?

There will be some action, some where.

What is happening in this directory?

/opt/Rocket.Chat/data/

Are you deleting anything or modifying it at all?

Can you see the files being saved in there? (I have a feeling Avatars get stored in Mongo but can’t remember)

It might be better if it is outside of your actual Rocket directory??

7

I was facing the same issue
I checked the avatar collection on mongodb
There were some avatars with different path or storage type, so the old ones were not showing

1 Like