Mongod spam "checkpointer" to syslog - how to prevent?

Community Support
1

Description

Hello community, I am facing the issue that mongod from my snap installation 6.7.0 is spamming the syslog file with their checkpointer nonsense.
I do not want mongod to spam my syslog file with that, how can I change this?
Please no “migrate to docker”, this is not a solution for me.

Server Setup Information

  • Version of Rocket.Chat Server: 6.7.0
  • Operating System: Ubuntu 22
  • Deployment Method: snap
  • Number of Running Instances: 1
  • DB Replicaset Oplog: wat??
  • NodeJS Version: 14.21.2
  • MongoDB Version: 5.0.15 wt oplog
  • Proxy: nginx
  • Firewalls involved: nej

Any additional Information

Apr 23 07:39:46 h2915188 mongod: {“t”:{“$date”:“2024-04-23T07:39:46.479+02:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn27”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:56272”,“uuid”:“XXXXXXXXXXXXXXXXXXXXXXXX”,“connectionId”:27,“connectionCount”:19}}
Apr 23 07:40:20 h2915188 mongod: {“t”:{“$date”:“2024-04-23T07:40:20.977+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22430, “ctx”:“Checkpointer”,“msg”:“WiredTiger message”,“attr”:{“message”:“[1713850820:977551][15074:XXXXXXXXXXXXXXXXXXXXXXXX], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 327, snapshot max: 327 snapshot count: 0, oldest timestamp: (1713850519, 1) , meta checkpoint timestamp: (1713850819, 1) base write gen: 23098198”}}
Apr 23 07:41:21 h2915188 mongod: {“t”:{“$date”:“2024-04-23T07:41:21.036+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22430, “ctx”:“Checkpointer”,“msg”:“WiredTiger message”,“attr”:{“message”:“[1713850881:36002][15074:XXXXXXXXXXXXXXXXXXXXXXXX], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 343, snapshot max: 343 snapshot count: 0, oldest timestamp: (1713850579, 1) , meta checkpoint timestamp: (1713850879, 1) base write gen: 23098198”}}

As I already wrote, I do not want those messages to appear in syslog, they have to go into their own logfile

thank you to anyone in advance who can help me with this issue!

2

I have tried editing the file

/var/snap/rocketchat-server/current/mongod.conf

and commented out:

systemLog:
destination: syslog

and added:

systemLog:
destination: file
path: “/var/log/mongod.log”
logAppend: true

But mongod freaks out and doesnt start for some strange reason, even mongo itself gives this as an example.

I can not change the contents of the file mongo.sh in

/snap/rocketchat-server/current/helpers

or can I somehow? It is in read only snap FS…

3

Snaps are a very closed environment. Personally hate them, but for many they are simple.

They are not really designed for you to be able to alter much.

I am not sure they can write outside their own environment.

My own native mongo install has this in the conf file - be careful as I think it is JSON or XML formatted and must be indented correctly

systemLog:
  verbosity: 0
  destination: file
  logAppend: true
  logRotate: reopen
  path: /var/log/mongodb/mongod.log

You would have to make sure that the snap has permission to write to the destination.

The alternative is using docker and having more control over your install. YMMV.

4

as I already wrote, docker is not an option for me, it totally messes up my iptables.
mongod from that snap install should have permission to write to /var/log since syslog is owned by root and 744 and it can write to syslog.
I mean it would be totally fine for now to just write to /dev/null, I dont care for mongo log spam.
I just dont want to get my syslog filled with nonsense information
just FYI your systemLog config aint working for me, I have tried and mongo doesn’t start, journalctl doesn’t even tell me the exact issue why. (I mean it is obvious)

5

You know you can disable that don’t you? I do it myself on all my docker machines…

Put this in daemon.json

"iptables": false

Unfortunately, as I have said, snaps are a law unto themselves and are tightly contained. It’s why I personally rip the lot out on any install :rofl:

I think they are restricted to their own directory/sub directories. At a guess (and I really don’t know but…) logging is probably done via a socket or similar and not actually writing to a file.

Your only real option is to try and have a hunt for generic snap settings to allow it to write to other directories.

eg this old one “snap can’t write to /var/log”

There may be other issues that prevent it running this as an option and for reasons stated above.

Someone else may have an idea but I know no more.

6

thanks for the infos and attempt of help, but no matter what I do,
snap still spamming syslog, altering mongod.conf to log to a file that is even 777 only makes the thing not start:

rocketchat-server.rocketchat-mongo[31528]: {"t":{"$date":"2024-05-09T08:02:43.445Z"},"s":"F",  "c":"CONTROL",  "id":20574,   "ctx":"-","msg":"Error during global initialization","attr":{"error":{"code":2,"codeName":"BadValue","errmsg":"syslog is not allowed when systemLog.path is specified"}}}

and whoever mainained/wrote the file mongo.sh deep inside /snap folder should be asked for a good reason to have the --syslog parameter specified there, rather than mongod.conf
because I cant find a good reason, it is plainly stupid to force all logs to syslog.

   [[ -n $(pgrep -xf "mongod --config=$SNAP_DATA/mongod.conf --fork --syslog") ]]

(that file is inside /snap mountpoint and read only.

7

I’m not sure even if you modified that mongo.sh file is it can write OUTSIDE the app directory. The person who wrote it knows a fair amount about snaps…

Argue that one with Ubuntu!

As I previously mentioned, if you took time to read about snaps you will see that snaps are VERY tightly controlled and NOT great if you want to do anything other than supplied.

This is not the fault of Rocket.Chat.

I have snaps disabled for this very reason.

So I think your only route is to use docker if this is what you want.