Rocketchat_apps_logs collection is massive

Community Support
1

Our rocket chat database is growing a lot. After some digging into mongodb we noticed the collection rocketchat_apps_logs is about 90GB. Is this just used for app logs?

How can I safely clean this up or limit the amount logged here?

2

hi!

The same question. We have >20gb apps logs. Can we remove it via deleteMany in mongoshell?

Any help?

7

I can top that, our rocketchat_apps_logs.bson file in our Rocket.Chat container is 266 Gb. Has anyone figured out whether or not this data is important, or whether it can be reduced or removed?

8

Have you had a look and see what it contains?

You can use something like Studio3T to have a look at your collections.

9

Thanks for the suggestion. I haven’t been able to look at the data (yet), but I suppose it’s collecting some kind of error logs for the apps integration component of Rocket.Chat. The mongo database isn’t easy to get to with Studio3T in our case, since it’s running inside a docker container on a Cloudron platform. The CLI for mongodb is very clunky IMO, so there’s no easy way to review the data here either.

On a separate note though, I don’t really care about the data. :slight_smile: The problem appears to be that since a recent Rocket.Chat update (I think it was the 7.2.0 server update), these logs are growing at an insane rate. It is filling a 500Gb disk within hours. This wasn’t happening before.

We don’t even have any apps installed. We did have a couple of free ones (polls and reminders) installed before the update, but we have removed them.

Actually since it’s running on Cloudron, the same reason it’s hard to get to the data, seems to be why our server is still operational. A side effect of the container’s virtual file system appears to be that the data lives inside the app container’s virtual file system. Restarting the app starts a new instance, and rebooting the server releases the “virtually” filled disk. I don’t claim to understand this, just that the effect is helping us clean up and stay operational. Otherwise we’d be stuck with a full disk.

The underlying problem is the rocket_chat_apps_logs collection growing at an unmanageable rate. Does anyone have some insight into how this can be fixed?

10

Not without knowing what it contains or causes it …

Need more debugging info. You were replying to a year old thread. We need to know about your setup.

Pretty sure Rocket logs tell part of the story but we need more info.

History of server, deployment details etc etc

Eg

11

Thank you @reetp - I will do some extra work as you requested.

I replied to a January 2024 comment which appeared on a web search that appeared to report the same conditions as our problem. The description of the problem was very simple, so I was hoping someone else had seen the effect and might have a quick solution or tips.

Happy to provide additional information if it’s expected. I was not prompted or invited to do so when logging in to the forum to participate in this thread. I must say, the document you’ve linked me to appears to outline an onerous process for general community support, particularly when asking a question as simple as the topic in this thread. I participate in many support forums and haven’t had to read an essay quite that long before. E.g. I’ve got quite a lot on this topic from the Cloudron community forums even though it’s obviously a Rocket.Chat issue.

I don’t quite understand how much info you need when you say “etc etc”, and the other info you mentioned (history of server, deployment details) does not appear to be listed in the in document you linked. I’ll do my best.

While I’m collecting the additional information that you say you can’t provide any insight without, I respectfully suggest you don’t need more debugging info to provide some helpful related info, so I’ll try to ask some more specific questions:

  • Has the very simple effect I described been seen in testing or by other users, particularly after recent updates? (Or is my problem rare or unique, in which case I will investigate further on our side?)
  • What kind of data is in rocketchat_apps_logs.bson, and what kinds of things could cause it to grow if we don’t have any apps installed? This might help us understand what has happened in our environment.
  • Assuming I do not have a degree in systems administration, how would I retrieve these Rocket logs that you referred to?

I appreciate this is free community support, and I only expect what I pay for. For free, I will try to be polite and helpful to others and hope for the same in return. I appreciate your time, thank you.

12

Excellent.

Things move fast here (regrettably) so you are 12 releases and a year out of date.

By all means refer to an old post but better to start a new thread if it is more than a month or two old.

It is expected. You have to remember we are are blind. You have to describe your own specific circumstances in detail - just because someone else had the issue does not mean it is the same as yours.

A lot of that info is in the documentation, forums, open.rocket and bugs if you read around a lot. I just tried to put it in one doc for my benefit.

It was born out of learning how to get my own questions answered decades ago, and then answering the same questions a thousand times over many years, and not just here. Most people are too lazy to RTFM - a consequence of dumbing down in modern IT.

It saves me hours and hours own my own personal time - I don’t work here (though I did for a while) and do this as a volunteer. Every single time I have to search or look up docs or bugs that a user could do themselves is a waste of my valuable drinking time :beers:

Rocket attracts a lot of very inexperienced admins who have no idea how to do a decent bug report. I have tried to do a fairly comprehensive FAQ. It doesn’t apply to all, but it applies to many and there are some generally good links in there - How to Report Bugs Effectively and How to ask a smart question that will attract attention prevent a lot of XY Info problems

I split it into sections and try to refer people to the relevant ones - in your case “Information that support or devs will require” because it is needed for every single user when they report an issue. Check the github issue templates and the docs in Rocket. Same info required.

Remember that when you take your car to a garage you don’t just say ‘it’s broken’.
You describe the issue as best you can. And they have the car…

I actually referred you to the relevant section. Did you read it?

Information that support or devs will require

At a minimum you will need this:

It does say it quite clearly? if not I’ll amend it.

I respectfully suggest you read the XY Problem above and note this line:

Remember that if your diagnostic theories were accurate, you wouldn’t be asking for help right?

We have zero knowledge of YOUR setup. We do not have access to your brain or your keyboard and monitor and server. Without the info we are just guessing. Hence we ask for it. Also see below.

  • Has the very simple effect I described been seen in testing or by other users, particularly after recent updates? (Or is my problem rare or unique, in which case I will investigate further on our side?)

Not that I have seen.

  • What kind of data is in rocketchat_apps_logs.bson, and what kinds of things could cause it to grow if we don’t have any apps installed? This might help us understand what has happened in our environment.
  • Assuming I do not have a degree in systems administration, how would I retrieve these Rocket logs that you referred to?

I have no qualifications in IT. Just 40+ years of experience.

If you run a server then you are administrator and it is on you to learn how to administer it properly. See ‘dumbing down’ above.

I can’t tell you how to get to Mongo on Cloudron - I’d imagine you can ssh to the instance so you can use that ssh connection with Studio 3T? It might need a bit of jiggery pokery on the server - but again its Cloudron and I have no experience of that.

Then you will see what is being logged which is a good start at diagnosing the issue. At a guess (and only a guess) it is a lot of debugging info for apps. But if you ave none installed I have idea what or why… hence we’ll need details.

Also I presume you are not using GridFS for storage?? Just checking (again it is these sort of details we need to see so we don’t have to ask)

No problems. I am sorry the reply is lengthy but you asked a of questions that have no simple answer.

Bearing all the above in mind I can tell you absolutely that if I don’t have the answer and need to speak to a dev then I know what they will ask for before they’ll even think of looking at it.

So follow the docs and do a proper report with as much relevant info as possible.

Thanks.

13

The original problem:

After updating Rocket.Chat server to the latest version (probably 7.2.0 but I can’t pinpoint the moment), our server began to fill its disk rapidly. Upon investigation, we found the rocketchat_apps_logs.bson file within the Rocket.Chat docker container was growing at a crazy rate, e.g. gigabytes per minute. More details can be found here.

After a further update (probably to the 7.2.1 server) the file and folder structure within the docker app appeared to change and we can no longer verify it is the rocketchat_apps_logs collection growing, but we are seeing the same disk filling effect. Sometimes it appears to begin immediately when the app starts, but other times it doesn’t start until later. It does not appear to be related to any users using the system; sometimes users are online and it doesn’t start, other times users are not online and it does start.

  • Server Setup Information:

    • Licence type eg CE/Starter/Pro :
      – My administration interface says “Starter”, although I don’t think we installed this, we thought we installed the open source version. What is CE?

    • Number of users:
      – 13 active, 24 in total

    • Server hardware: VPS/hypervisor/bare metal
      – VMWare hypervisor

    • Version of Rocket.Chat Server:
      – Currently 7.2.1, problem appeared to start around when 7.2.0 was installed or slightly earlier

    • Operating System: Ubuntu/Redhat etc
      – Ubuntu 20.04 LTS

    • Deployment Method: snap/docker/tar/etc
      – Docker, deployed point-and-click as a Cloudron app (cloudron.io)

    • Number of Running Instances:
      – 1

    • DB Replicaset Oplog:
      – What is that and how would I find it?

    • NodeJS Version:
      – 20.18.0 according to the Cloudron specs

    • MongoDB Version:
      – 6.0.13

  • Client information:

    • Client type: Electron app/React-Native app/Browser and version
      – What do those options mean?
      – We are using a combination of browser access (mostly current Chrome) and mobile apps (mostly IOS) and desktop apps (Windows 10 and 11)
      – The problem seems to be happening on the server, not in clients, and appears to happen whether or not any clients are active

  • A list of the steps required to replicate the issue.

    • If we can’t replicate it we can’t debug it.
      – I don’t know, this problem appears to be a server side problem that began at an unknown moment, perhaps after a Rocket.Chat update.
      – Install Rocket.Chat (Ideally on Cloudron I suppose? Although other non-Cloudron people seem to have the problem?)
      – The problem did not appear to happen < 7.2.0. (However from a previous report, perhaps it happened to someone in 2020.) So perhaps attempt running an update?
      – Start Rocket.Chat and observe the disk filling up. I think it is the rocketchat_apps_logs.bson file filling up, but after the latest Rocket.Chat server update the filesystem inside the docker container appeared to change and I can no longer see this easily.
      – Note we did have a small number of free apps installed (e.g. polls and reminders), but we have now deleted all these. No apps are installed.
      – If it’s helpful, Cloudron people have been trying to be supportive on this forum thread and other recent Rocket.Chat problems in that forum might also be relevant - E.g. could it be related to Deno?

  • Other info from the support guide requirements:
    – I have searched for similar issues to this online extensively
    – I have searched documentation, but I have no idea what area of the system to consider, so I’m stumbling in the dark
    – I did not find anything in open and closed bugs
    – We were not trying to solve any problems, we just ran an app update
    – The original problem began after an automated Cloudron app update. We have since run manually triggered updates and it still happens.
    – This app has been running since mid 2020. There are about 40 channels and the stored chat data seems to be about 12 Gb before the disk filling problem starts. No problems like this until recently. We have previously used a tiny amount of omnichannel chats, but none recently.
    – Other than the app upgrade, no other recent configuration changes were made.
    – I don’t know how to retrieve relevant logs and there is no info on this in the linked support guide.
    – There have been no changes to the network, reverse proxies, or other related components. The app itself and access to it is working fine.

  • Upgrades have happened incrementally, usually between every minor version. We typically haven’t experienced
14

Yes I read it, all of it, before responding. My point that I hoped you would interpret is that your instructions were not clear: “etc etc” is not a helpful guideline, and nothing in your long document matched the other language you used.

16

Attempting to extract and address your direct queries and concerns…

Yes, I read that. The analogies in your document are somewhat ambiguous and difficult for me to interpret, but to me that section of the document doesn’t appear to be related to my specific query about a specific problem. I don’t have any diagnostic theories. I didn’t submit a bug report. I observed an issue and asked for more info about it.

Sorry, let me clarify my question: I was not asking for support for my server environment. I meant to ask, which logs were you referring to, and how do I get these out of Rocket.Chat in a standard scenario? (If there is documentation about this, I haven’t found it, but would be happy to review a link.)

If that’s the sort of thing you need to know, I suggest listing it in the list of information that the devs will require, along with how to check.

I don’t know the answer to this question or how to find out, but I will follow up when I can.

Please don’t waste your drinking time on my behalf.

17

Your deleted quote.

Yes, because the original post was 2020 and the last follow ups were a year old and there is a long way between the original version and this. You have made an assumption that it is the same issue, but you have zero proof.

Also the original OP may not want to be bombarded by irrelevant replies to their original post.

Better to create new post with a link to the old when the original is this old.

Unless we know what your server environment is we can’t help. From long and bitter experience having a good understanding of the server setup and background makes diagnosing issues much easier. It is general good practice when reporting issues. Also, if you need to create a bug report you will have all the relevant details at hand. Note what I said would happen if I have to speak to a dev.

If you want help, then be helpful and supply what we ask for. You are the one that needs help. Not me.

I suggest you spend more time reading the documentation provided by Rocket.Chat and understanding your environment. A bit of self help will go a long way.

Nope I am not going to anymore.

Have a nice day.

18

I deleted it because I noticed you had already answered my question so I was avoiding thread spam. I’m not sure why you would bring it back, other than to man-splain to me further. But since you resurrected it…

I wasn’t trying to prove anything, I don’t know where that thought came from. I asked a question. The perfectly reasonable assumption I made is that in this case, both the 2020 and the 2024 problem descriptions appeared to exactly match my problem, so the ensuing guess I made was that there’s a chance whatever’s happening is a long standing issue. Perhaps over the years someone else had seen it and had a useful thought to share.

Maybe, or maybe they appreciate the unsolved issue being raised again, or even solved the problem themselves and might have dropped in to help. I’ve certainly experienced and offered that myself in the past. Bumping issues for new attention is a very common thing to do in support forums. It cuts both ways.

My humble apologies to OP if it wasn’t useful in this case.

Sure, good suggestion. And if it avoids me being berated for having a different opinion to yours next time, to the point of resurrecting deleted posts to make the point, I will do so.

When asked and given some guidelines, I believe I supplied everything you asked for. I spent a lot of time on it. I’m not sure if you read any of it. No need now I suppose, since you’ve tapped out.

I’m genuinely confused about what self help I could do to understand which logs you were referring to in your comment. There are a lot of potentially relevant ones, and your documented guidelines encouraged me not to dump too much log data on you.

I have in fact read most of the (unspecified) docs and done most of the (unspecified) investigation you keep accusing me of not doing, hours of it, and I have an excellent understanding of my own environment outside the Rocket.Chat proprietary scope (hence why I didn’t ask for help on that front), and the documentation you referred me to does not give an answer to the question of which logs you were referring to in your comment either. (E.g. I doubt you want the 400 Gb of apps_logs data, but given the repetition of advice to supply every piece of information for my original very simple question, perhaps you do.)

I very simply asked for clarification. I accept now, you’re unwilling to give it.

You appear to have missed my (admittedly implied) point there. Your original comment putting me down and telling me getting drunk is more important was actively contemptuous, and unnecessarily so. To state it a little more directly: please go back to your drinking if showing me contempt is the alternative. I hoped you would take that on board and in future when dealing with me or others, speak more respectfully. Though I didn’t expect it, an apology for the insult would have been humble, polite, and appreciated, too.

I won’t. I feel horrible. I’m sorry if you do too, rather I hope something positive might ensue, but acquiescing to bullies in public forums is not okay with me, because it makes it okay for the next person to receive the same treatment.

If it was your intention as Community Liaison Officer to represent Rocket.Chat by speaking down to a community member, expanding the scope of my query and then telling me what I’m doing wrong inside your expanded scope, making me feel that your product group doesn’t want genuine people like me here, and costing me a lot of effort for what I think was originally an extremely simple question, you have succeeded. Perhaps I’ll try to get (and provide) help somewhere else that doesn’t carry the Rocket.Chat brand. I’ll consider paying for an answer to my simple question as suggested in your support docs, but now I’m a very trepidatious about the kind of treatment I’ll get. By contrast, the folks on the Cloudron forum have been extremely friendly and helpful on the same topic.

I believe I did everything you asked me to. I spent a lot of time on it. So it’s disappointing you’re not willing to see it through, but I hope someone else may benefit.