Scavenge allocation failure

Community Support
1

Description

Today we had a break down in the server, it seems that there is a process of collecting garbage that consumes a lot of memory, but in our stats we couldn’t find the correct moment because every graph was with enough span to work, I mean disk, memory, processors

In the log I found something about Scavenge and Mark-sweep

[3344:0x542df00] 762289367 ms: Scavenge 1906.3 (2056.7) -> 1905.5 (2071.7) MB, 30.9 / 4.5 ms  (average mu = 0.762, current mu = 0.761) allocation failure
[3344:0x542df00] 762290140 ms: Mark-sweep 1920.4 (2071.7) -> 1919.3 (2081.4) MB, 707.2 / 4.9 ms  (average mu = 0.739, current mu = 0.705) allocation failure scavenge might not succeed

Is there something that could be done to improve this performance and avoid over consumption of server resources? Below I let you the complete log of this event

Thank you and regards

Server Setup Information

  • Version of Rocket.Chat Server: 6.4
  • Number of Running Instances: 1
  • NodeJS Version: v14.21.3
  • MongoDB Version: 5.0.21 / wiredTiger (oplog Enabled)
  • Proxy: nginx

Any additional Information

<--- Last few GCs --->

[3344:0x542df00] 762289332 ms: Scavenge 1905.4 (2055.7) -> 1905.3 (2056.7) MB, 19.6 / 4.4 ms  (average mu = 0.762, current mu = 0.761) allocation failure
[3344:0x542df00] 762289367 ms: Scavenge 1906.3 (2056.7) -> 1905.5 (2071.7) MB, 30.9 / 4.5 ms  (average mu = 0.762, current mu = 0.761) allocation failure
[3344:0x542df00] 762290140 ms: Mark-sweep 1920.4 (2071.7) -> 1919.3 (2081.4) MB, 707.2 / 4.9 ms  (average mu = 0.739, current mu = 0.705) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0xa3ad50 node::Abort() [node]
 2: 0x970199 node::FatalError(char const*, char const*) [node]
 3: 0xbba90e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbbac87 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd76ea5  [node]
 6: 0xda783e v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
 7: 0xdb3876 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
 8: 0xd9fa0f v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
 9: 0xd9fc88 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xd92569 v8::internal::ItemParallelJob::Run() [node]
11: 0xdb57d0 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [node]
12: 0xdb606c v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [node]
13: 0xdb6235 v8::internal::MarkCompactCollector::Evacuate() [node]
14: 0xdc8231 v8::internal::MarkCompactCollector::CollectGarbage() [node]
15: 0xd844f8 v8::internal::Heap::MarkCompact() [node]
16: 0xd85fe8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
17: 0xd8942c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
18: 0xd4ebfd v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
19: 0xd48a84 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
20: 0xd4aa90 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
21: 0x1137425 v8::internal::IncrementalStringBuilder::Extend() [node]
22: 0xe75490 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node]
23: 0xe7754e v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
24: 0xe7b29b v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
25: 0xe78f00 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
26: 0xe78333 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
27: 0xe7b29b v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
28: 0xe7cc5f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
29: 0xc66e1f v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node]
30: 0x1449459  [node]