This is how I understand this. I have yet to test it, but I will update as soon I can.
MongoDB 4.0.x uses an MMAP storage engine. In order to update to a higher 4.x or 5.x, you will need to migrate from MMAP to wiredTiger storage engine first.
Ref: https://docs.rocket.chat/quick-start/installing-and-updating/docker-containers/mongodb-mmap-to-wiredtiger-migration
MongoDB needs to be updated sequentially from my understanding. To upgrade to 5.x, you will have to upgrade from 4.0 to 4.2; then from 4.2 to 4.4, and so on. I don’t think you can just jump from 4.0 to 5.0.
Ref: https://docs.mongodb.com/manual/tutorial/upgrade-revision/
After you migrated from MMAP to wiredTiger engine, (Step 1)stop all docker instances. (Step 2)update your Docker Compose file and change mongo and mongo-init-replica image version to the next sequential version. (Step 3)Then, run the Docker Compose file. Repeat from step 1 to step 3 if needed.
Ref: https://raw.githubusercontent.com/RocketChat/Rocket.Chat/develop/docker-compose.yml
Backup the database and even the entire server if possible before trying anything, and create a fail-safe solution.
Hope this helps in some way!