File upload stuck or doesn't work

Description

Server Setup Information

  • Version of Rocket.Chat Server: 1.3.2
  • Operating System: Centos 7
  • Deployment Method: tar
  • Number of Running Instances: 2
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 8.11.4
  • MongoDB Version: 4.0.9
  • Proxy: haproxy
  • Firewalls involved:
  • File Storage: GridFS

Any additional Information

In RC logs after file uploading -
Sep 10 12:30:00 10.21.161.135: { Error: ENOENT: no such file or directory, stat ‘/tmp/zipFiles/QydhZTK2YJpX8NwFf.zip’
Sep 10 12:30:00 10.21.161.135: errno: -2,
Sep 10 12:30:00 10.21.161.135: code: ‘ENOENT’,
Sep 10 12:30:00 10.21.161.135: syscall: ‘stat’,
Sep 10 12:30:00 10.21.161.135: path: ‘/tmp/zipFiles/QydhZTK2YJpX8NwFf.zip’ }
Sep 10 12:34:28 10.21.161.135: Exception while invoking method ‘ufsComplete’ { MongoNetworkError: connection 10 to 10.21.161.126:27017 timed out
Sep 10 12:34:28 10.21.161.135: at Socket. (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:259:7)
Sep 10 12:34:28 : at Object.onceWrapper (events.js:313:30)
Sep 10 12:34:28 10.21.161.135: at emitNone (events.js:106:13)
Sep 10 12:34:28 10.21.161.135: at Socket.emit (events.js:208:7)
Sep 10 12:34:28 10.21.161.135: at Socket._onTimeout (net.js:410:8)
Sep 10 12:34:28 10.21.161.135: at ontimeout (timers.js:498:11)
Sep 10 12:34:28 10.21.161.135: at tryOnTimeout (timers.js:323:5)
Sep 10 12:34:28 10.21.161.135: at Timer.listOnTimeout (timers.js:290:5)
Sep 10 12:34:28 10.21.161.135: name: ‘MongoNetworkError’,
Sep 10 12:34:28 10.21.161.135: errorLabels: [ ‘TransientTransactionError’ ],
Sep 10 12:34:28 10.21.161.135: [Symbol(mongoErrorContextSymbol)]: {} }

I have my prod RC infrastructure (RC 4-nodes v 0.74.2, MongoDB v3.2.10 3 - nodes ReplsSet/Oplog - enabled, File storage - GridFS) - file upload works fine. I would like to upgrade RC version to newest.

Server Setup Information

  • Version of Rocket.Chat Server: 1.3.2 and 1.0.3
  • Operating System: Centos 7
  • Deployment Method: RPM
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 8.11.4
  • MongoDB Version: 3.6.8-2
  • Proxy: AWS ALB
  • Firewalls involved:
  • File Storage: GridFS

This is happening to me as well on versions 1.0.3 and 1.3.2. This is on a Stage environment with Ops data dumped and migrated to Stage. In addition, I ran iostat on the Mongo replica set primary host during the upload. The device for the Mongo data directory goes from almost nothing to 100 percent on %util. I do not see any writes on the device. It seems as though this is all coming from reads.