Scavenge allocation failure

Community Support
1

Description

Today we had a break down in the server, it seems that there is a process of collecting garbage that consumes a lot of memory, but in our stats we couldn’t find the correct moment because every graph was with enough span to work, I mean disk, memory, processors

In the log I found something about Scavenge and Mark-sweep

[3344:0x542df00] 762289367 ms: Scavenge 1906.3 (2056.7) -> 1905.5 (2071.7) MB, 30.9 / 4.5 ms  (average mu = 0.762, current mu = 0.761) allocation failure
[3344:0x542df00] 762290140 ms: Mark-sweep 1920.4 (2071.7) -> 1919.3 (2081.4) MB, 707.2 / 4.9 ms  (average mu = 0.739, current mu = 0.705) allocation failure scavenge might not succeed

Is there something that could be done to improve this performance and avoid over consumption of server resources? Below I let you the complete log of this event

Thank you and regards

Server Setup Information

  • Version of Rocket.Chat Server: 6.4
  • Number of Running Instances: 1
  • NodeJS Version: v14.21.3
  • MongoDB Version: 5.0.21 / wiredTiger (oplog Enabled)
  • Proxy: nginx

Any additional Information

<--- Last few GCs --->

[3344:0x542df00] 762289332 ms: Scavenge 1905.4 (2055.7) -> 1905.3 (2056.7) MB, 19.6 / 4.4 ms  (average mu = 0.762, current mu = 0.761) allocation failure
[3344:0x542df00] 762289367 ms: Scavenge 1906.3 (2056.7) -> 1905.5 (2071.7) MB, 30.9 / 4.5 ms  (average mu = 0.762, current mu = 0.761) allocation failure
[3344:0x542df00] 762290140 ms: Mark-sweep 1920.4 (2071.7) -> 1919.3 (2081.4) MB, 707.2 / 4.9 ms  (average mu = 0.739, current mu = 0.705) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0xa3ad50 node::Abort() [node]
 2: 0x970199 node::FatalError(char const*, char const*) [node]
 3: 0xbba90e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbbac87 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd76ea5  [node]
 6: 0xda783e v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
 7: 0xdb3876 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
 8: 0xd9fa0f v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
 9: 0xd9fc88 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
10: 0xd92569 v8::internal::ItemParallelJob::Run() [node]
11: 0xdb57d0 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [node]
12: 0xdb606c v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [node]
13: 0xdb6235 v8::internal::MarkCompactCollector::Evacuate() [node]
14: 0xdc8231 v8::internal::MarkCompactCollector::CollectGarbage() [node]
15: 0xd844f8 v8::internal::Heap::MarkCompact() [node]
16: 0xd85fe8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
17: 0xd8942c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
18: 0xd4ebfd v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
19: 0xd48a84 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
20: 0xd4aa90 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
21: 0x1137425 v8::internal::IncrementalStringBuilder::Extend() [node]
22: 0xe75490 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node]
23: 0xe7754e v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
24: 0xe7b29b v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
25: 0xe78f00 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
26: 0xe78333 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
27: 0xe7b29b v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
28: 0xe7cc5f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
29: 0xc66e1f v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node]
30: 0x1449459  [node]
3

Hello, I think there is a memory issue causing the server to consume excessive resources. The log shows a “Scavenge” process followed by a “Mark-sweep” process, both of which are garbage collection mechanisms used by V8, the JavaScript engine used by Node.js.

The log also indicates an “allocation failure” and a “JavaScript heap out of memory” error. This suggests that the server’s memory is being exhausted during the garbage collection process, potentially due to high memory usage by your application.

To improve performance and avoid overconsumption of server resources, you can try increasing the memory limit. If the server has available memory, you can allocate more memory to the Node.js process. This can be done by adjusting the --max-old-space-size flag when starting the Node.js process. For example, node --max-old-space-size=4096 server.js sets the maximum heap size to 4GB.