Mongodb crashed: write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection

Description

My mongodb crashed and after googling for a few days I have been unable to come up with any solution. I would greatly appreciate any pointers. I am using docker-compose file for setup. File below.

[rsSync-0] Assertion: Location17322: write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:134217728 src/mongo/db/repl/oplog.cpp 142
2020-09-01T14:41:47.219+0000 F -        [rsSync-0] terminate() called. An exception is active; attempting to gather more information
2020-09-01T14:41:47.236+0000 F -        [rsSync-0] DBException::toString(): Location17322: write to oplog failed: DocTooLargeForCapped: document doesn't fit in capped collection. size: 124 storageSize:134217728
Actual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException
 0x5574f1ed7611 0x5574f1ed6ff5 0x5574f1fcb896 0x5574f205fad9 0x5574f1fcb235 0x7fa2e346bf83 0x7fa2e346c487 0x5574f07c3838 0x5574f084dd4b 0x5574f084e57f 0x5574f083d216 0x5574f16e6b85 0x5574f16e70fb 0x5574f166dadc 0x5574f166f0ac 0x5574f166fae6 0x5574f1fe6af0 0x7fa2e32466ba 0x7fa2e2f7c4dd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5574EFA58000","o":"247F611","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5574EFA58000","o":"247EFF5"},{"b":"5574EFA58000","o":"2573896","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"5574EFA58000","o":"2607AD9"},{"b":"5574EFA58000","o":"2573235","s":"__gxx_personality_v0"},{"b":"7FA2E345C000","o":"FF83"},{"b":"7FA2E345C000","o":"10487","s":"_Unwind_Resume"},{"b":"5574EFA58000","o":"D6B838","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx"},{"b":"5574EFA58000","o":"DF5D4B","s":"_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_11OplogBufferEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE"},{"b":"5574EFA58000","o":"DF657F","s":"_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_11OplogBufferEPNS0_22ReplicationCoordinatorE"},{"b":"5574EFA58000","o":"DE5216"},{"b":"5574EFA58000","o":"1C8EB85","s":"_ZN5mongo8executor22ThreadPoolTaskExecutor11runCallbackESt10shared_ptrINS1_13CallbackStateEE"},{"b":"5574EFA58000","o":"1C8F0FB"},{"b":"5574EFA58000","o":"1C15ADC","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"5574EFA58000","o":"1C170AC","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"5574EFA58000","o":"1C17AE6","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"5574EFA58000","o":"258EAF0"},{"b":"7FA2E323F000","o":"76BA"},{"b":"7FA2E2E75000","o":"1074DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.20", "gitVersion" : "e2416422da84a0b63cde2397d60b521758b56d1b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-112-generic", "version" : "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "5574EFA58000", "elfType" : 3, "buildId" : "C632903AED38FD4DE4A0A6D7626A215B3C1095E0" }, { "b" : "7FFCE45DA000", "elfType" : 3, "buildId" : "570546F38D51397FA9F52EEA73DDE45AE5E7C9C0" }, { "b" : "7FA2E4650000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "D2A698AA64D6F578A96EA05653BDCF0FF7CA8206" }, { "b" : "7FA2E4435000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "683BB174D849C94BACD676385382A96336F2807F" }, { "b" : "7FA2E3FF0000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "5E0DDD272AEA6EA77600A74887246761A82F263F" }, { "b" : "7FA2E3D87000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "922334DBDD3F86D331FF18633AA16D73FBA91D03" }, { "b" : "7FA2E3B83000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "B72852F16AEE2A6EE4DAA94C4057623427A8E17F" }, { "b" : "7FA2E397B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "3828BFB25DE953DF34817895B3A065E9D57C32A3" }, { "b" : "7FA2E3672000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "6E218A55DB841085AD021AFE2562D1557A5A0D85" }, { "b" : "7FA2E345C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FA2E323F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "3DB0B0EE6244F5B89AB1D535F91B17D162CC1701" }, { "b" : "7FA2E2E75000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "C4FD86EC1EED57A09C79CE601F6C6E3796F574DF" }, { "b" : "7FA2E48BF000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "929894755069C71880927EEE9DC0D5C501616531" }, { "b" : "7FA2E2C42000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7FA2E2A26000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7FA2E27DC000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, { "b" : "7FA2E25CD000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "91C006FC62D3432E121A006D2424D2CA910FE3B2" }, { "b" : "7FA2E237C000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "31269F12DD5DFAD74819B33202BE70DCB257B139" }, { "b" : "7FA2E2162000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "E2357F88F4E53F0DE76266532DDDCB7FEB6F87A7" }, { "b" : "7FA2E1E32000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "B9A7B6BA989F754F13E22FA11CAA07A254488F76" }, { "b" : "7FA2E1BFF000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FA2E19C9000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FA2E1749000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FA2E1477000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, { "b" : "7FA2E1248000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, { "b" : "7FA2E1044000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "E6F8CA010177433A1298DAB8912203E5A5EBB9BA" }, { "b" : "7FA2E0E39000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, { "b" : "7FA2E0C1E000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "4FE2C629045AB0AF26EC5D7F8A3D64B9F8CB21B2" }, { "b" : "7FA2E09DD000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7FA2E0779000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7FA2E0566000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7FA2E0362000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FA2E0159000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7FA2DFECF000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7FA2DFC2D000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7FA2DF9FA000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7FA2DF7E4000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7FA2DF5DC000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7FA2DF3B3000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7FA2DF1A4000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7FA2DEF59000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7FA2DEC84000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "238FAA096B5ACEBE12D74B433E1E6A9313D1C1DA" }, { "b" : "7FA2DEA4C000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "0AD93CE1869E37EBBDAB9C29525FDACF92643289" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5574f1ed7611]
 mongod(+0x247EFF5) [0x5574f1ed6ff5]
 mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x5574f1fcb896]
 mongod(+0x2607AD9) [0x5574f205fad9]
 mongod(__gxx_personality_v0+0x2B5) [0x5574f1fcb235]
 libgcc_s.so.1(+0xFF83) [0x7fa2e346bf83]
 libgcc_s.so.1(_Unwind_Resume+0x57) [0x7fa2e346c487]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl19signalDrainCompleteEPNS_16OperationContextEx+0x4F8) [0x5574f07c3838]
 mongod(_ZN5mongo4repl8SyncTail17_oplogApplicationEPNS0_11OplogBufferEPNS0_22ReplicationCoordinatorEPNS1_14OpQueueBatcherE+0xF1B) [0x5574f084dd4b]
 mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEPNS0_11OplogBufferEPNS0_22ReplicationCoordinatorE+0x17F) [0x5574f084e57f]
 mongod(+0xDE5216) [0x5574f083d216]
 mongod(_ZN5mongo8executor22ThreadPoolTaskExecutor11runCallbackESt10shared_ptrINS1_13CallbackStateEE+0x1A5) [0x5574f16e6b85]
 mongod(+0x1C8F0FB) [0x5574f16e70fb]
 mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x14C) [0x5574f166dadc]
 mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xBC) [0x5574f166f0ac]
 mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x96) [0x5574f166fae6]
 mongod(+0x258EAF0) [0x5574f1fe6af0]
 libpthread.so.0(+0x76BA) [0x7fa2e32466ba]
 libc.so.6(clone+0x6D) [0x7fa2e2f7c4dd]
-----  END BACKTRACE  -----

dokcer-compose.yml

version: '2'

services:
  rocketchat:
    image: rocket.chat:latest
    container_name: rocket-chat
    command: bash -c 'for i in `seq 1 30`; do node main.js && s=$$? && break || s=$$?; echo "Tried $$i times. Waiting 5 secs..."; sleep 5; done; (exit $$s)'
    restart: always
    volumes:
      - ./uploads:/app/uploads
      - ./server:/app/server2
    environment:
      - PORT=3000
      - ROOT_URL=https://chat.domain.com
      - MONGO_URL=mongodb://mongo:27017/rocketchat
      - MONGO_OPLOG_URL=mongodb://mongo:27017/local
      #- MONGO_URL=mongodb://10.100.61.68:27017/rocketchat
      #- MONGO_OPLOG_URL=mongodb://10.100.61.68:27017/local
      - Accounts_UseDNSDomainCheck=True
      - CREATE_TOKENS_FOR_USERS=true
    depends_on:
      - mongo
    ports:
      - 3000:3000
    labels:
      - "traefik.http.routers.chat.rule=Host(`chat.domain.com`)"
      - "traefik.http.routers.chat.entrypoints=websecure"
      - "traefik.http.routers.chat.tls.certresolver=le"
      - "traefik.http.services.chat.loadbalancer.server.port=3000"
      - "traefik.enable=true"
      - "traefik.http.middlewares.https-redirect.redirectscheme.scheme=https"
      - "traefik.docker.network=traefik"
    links:
      - mongo:mongo
    networks:
      - traefik

  mongo:
    image: mongo:4.0
    container_name: mongodb
    restart: always
    volumes:
     - ./data/db:/data/db
     - ./data/dump:/dump
    ports:
     - 27018:27017
    command: mongod --smallfiles --oplogSize 512 --replSet rs0 --storageEngine=mmapv1
    #command: mongod --smallfiles --oplogSize 128 --storageEngine=mmapv1
    networks:
      - traefik
  # this container's job is just run the command to initialize the replica set.
  # it will run the command and remove himself (it will not stay running)
  mongo-init-replica:
    image: mongo
    command: 'bash -c "for i in `seq 1 30`; do mongo mongo/rocketchat --eval \"rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''localhost:27017'' } ]})\" && s=$$? && break || s=$$?; echo \"Tried $$i times. Waiting 5 secs...\"; sleep 5; done; (exit $$s)"'
    depends_on:
      - mongo
    networks:
      - traefik
    links:
      - mongo:mongo

What version of Rocket.Chat was running? You have latest tag in your docker compose but that doesn’t really tell us what version was running

I am not sure because Rocket Chat will no longer load. I did a pull, so it should actually be the latest version.

I just went ahead and specific set it to 3.5.4. Still the same error.

are you able to do docker-compose exec -it mongo mongo and once it brings up mongo shell do:

use local
db.stats()

In order to get mongo to start and stay up. I actually have to do

docker exec -it mongodb bash

then I start mongo with

mongod --smallfiles --oplogSize 128 --storageEngine=mmapv1 

leaving the replicaset off

Now I can run those commands:

use local;
switched to db local
> db.stats();
{
        "db" : "local",
        "collections" : 9,
        "views" : 0,
        "objects" : 567536,
        "avgObjSize" : 238.97275238927574,
        "dataSize" : 135625640,
        "storageSize" : 144756736,
        "numExtents" : 9,
        "indexes" : 6,
        "indexSize" : 384272,
        "fileSize" : 2431647744,
        "nsSizeMB" : 16,
        "extentFreeList" : {
                "num" : 0,
                "totalSize" : 0
        },
        "dataFileVersion" : {
                "major" : 4,
                "minor" : 22
        },
        "fsUsedSize" : 108970721280,
        "fsTotalSize" : 133685121024,
        "ok" : 1
}

Updated the stats. I forgot I had made some other changes that did not work and restored it the time of this post.