How to debug authentication issues?

[sintertrack]

Description

A user on our instance has problems logging in. He’s getting password errors and rate limit errors.

However, I can’t find anything about this in the logs. Nothing. Same thing in the Gui, I can’t find anything about any of the issues mentioned there.

So, how do I debug user authentication or rate limit issues in general?

Server Setup Information

• Version of Rocket.Chat Server: 8.5.1
• Operating System: Linux
• Deployment Method: Helm chart k8s
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version:
• MongoDB Version: 8.0.20
• Proxy: none
• Firewalls involved: none

[sintertrack]

I have set loglevel to debug, but still no logs about the failing authentication.

[reetp]

Can’t remember but you may need to stop/restart to change logging.

If you have others logging in ok, and one user not you probably ought to check the password is actually correct.

You make no mention of what is doing the authentication.

I trust you are running https with a reverse proxy.?

[sintertrack]

@reetp

Can’t remember but you may need to stop/restart to change logging.

Yes, I already did that.

@reetp

If you have others logging in ok, and one user not you probably ought to check the password is actually correct.

That is not the point, the question is how to troubleshoot auth problems in general, this specific user problem served just as an example.

@reetp

You make no mention of what is doing the authentication.

Rocket.Chat local users with username,password and email 2FA - so RC is doing it itself.

@reetp

I trust you are running https with a reverse proxy.?

There’s an ingress-nginx in front of it, of course.

[reetp]

@sintertrack

That is not the point

It is - I was responding to what you asked, not what you think you asked:

A user on our instance has problems logging in

One user.

@sintertrack

There’s an ingress-nginx in front of it, of course.

There’s no ‘of course’ because here was your info:

• Proxy: none
• Firewalls involved: none

So you should document and ask accurately which will save people wasting their time answering the wrong questions.

@sintertrack

• 8.5.1

You should also upgrade immediately - your version is insecure. See the changelogs:

c80b0c0 - Releases · RocketChat/Rocket.Chat

As far as logging goes I suggest you spend some time reading the documentation which will no doubt answer many of your questions:

Rocket Chat Workspace Log Configuration

Rate limiting:

Rocket Chat Rate Limiter for Secure Communications

[sintertrack]

Hi,

thanks for your response. You’re right, next time I’ll be more accurate.

I’ve done the upgrade today to 8.6.1.

I also read the linked docs already, but audit logs seems to be a premium feature, we’re using community.

[reetp]

KK.

I don’t think you need auditing logs.

I’d suggest reading this for general workspace logs with Promotheus:

Monitor Workspace Logs and Metrics

[sintertrack]

@reetp

I don’t think you need auditing logs.

No I just need regular logs about auth errors.

@reetp

I’d suggest reading this for general workspace logs with Promotheus

I did.

But as I have already established, there are no logs about authentication errors. In fact, there are no logs about authentication at all. Forwarding those logs doesn’t change anything.

I am not missing any logs as well, since I am looking at it directly via k8s:

$ kgp | sd fits "" | t -cname,status
NAME                                                         STATUS
cm-acme-http-solver-t4lbt                                    Running
mongodb-kubernetes-operator-65d8f4b69c-pkg5d                 Running
mongosh-6d65576fc4-9xb4h                                     Running
rocketchat-prod-mgob-0                                       Running
rocketchat-prod-mongodb-instance-0                           Running
rocketchat-prod-mongodb-instance-1                           Running
rocketchat-prod-mongodb-instance-2                           Running
rocketchat-prod-mongodbexporter-prometheus-mongodb-exp9q4gz  Running
rocketchat-prod-nginx-d4478db5b-7mjt5                        Running
rocketchat-prod-postfix-mail-0                               Running
rocketchat-prod-rocketchat-7fb9ff4bfd-bj6w5                  Running

$ k logs -l app.kubernetes.io/name=rocketchat --tail=100000|grep -i "error|fail|fatal"
# (no output!)

I am seeing logs, but these are just access logs, nothing more.

[reetp]

I don’t see Prometheus there (or grafana)?

That’s where I believe it now ends up though I don’t have a setup to confirm.

helm-charts/monitoring at master · RocketChat/helm-charts

[sintertrack]

Hi,

@reetp

I don’t see Prometheus there (or grafana)?

It is, but runs in another namespace. So, yes, we do have Rocketchat metrics in a Grafana dashboard etc, but that’s not the point of this thread.

@reetp

That’s where I believe it now ends up though I don’t have a setup to confirm.

Nope. Metrics and Logs are different things(*). I am talking about logs here.

The only logs Rocketchat produces in a k8s pod goes to STDOUT, which I can look at using the kubectl logs command (as posted above) or using stern. That’s it. There’s no additional logfile inside the pod somewhere. One could ship these logs to Loki or Elasticsearch (what we do), but that doesn’t change anything. You can only ship what’s there.

So, I am pretty sure, that I am looking at 100% of the logs Rocketchat is producing. And it doesn’t contain any authentication logs. No successful authentications, no errors and no rate limit exceedings.

And my question is: is it possible to enable such kind of logs and if so - how? If it’s not possible, I’d like to know how others are debugging authentication problems.

best,
Tom

(*) just to be sure, I also looked at all the metrics Rocketchat is producing, because your insistence of metrics were suspicious to me. So, yes, there are certain pointers to events in the metrics, such as these:

rocketchat_rest_api{quantile="0.9",status="200",method="post",version="v1",entrypoint="method.call/rooms:get",user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Rocket.Chat/4.14.1 Chrome/144.0.7559.236 Electron/40.8.5 Safari/537.36"} 0.010015375600000001

but these are just statistics, just what metrics are for. No pointers towards authentication or even particular errors.

[reetp]

@sintertrack

Nope. Metrics and Logs are different things(*). I am talking about logs here.

I understand that. You are getting shirty with the wrong person here. I no longer work for Rocket. Just volunteer because no one else does.

Please read what it says here again:

Configure an observability stack

Rocket.Chat’s official Docker Compose(opens in new tab) and Helm chart(opens in new tab) includes a recommended observability stack for monitoring your workspace that can be easily enabled during deployment. This stack uses:

Prometheus to collect workspace metrics

Loki to collect and index workspace logs

Grafana to easily visualize logs and metrics through dashboards

Monitor Workspace Logs and Metrics

So take a look at Loki (I don’t see Loki, Promotheus, Grafana in your setup)

As I said - I do not run 8.x at the minute because I prefer LTS and hate the new UI so don’t have this setup.

Others may know more, if you can find anyone to help.

[sintertrack]

Hi,

@reetp

You are getting shirty with the wrong person here.

Sorry if what I wrote sounded like that, it wasn’t meant to upset you. In fact I appreciate your help.

@reetp

So take a look at Loki (I don’t see Loki, Promotheus, Grafana in your setup)

We’re shipping Rocketchat logs to elastic:

So, I don’t have a problem accessing the logs, there’s just no auth stuff in there.

PS: I tried to trigger some error myself with trying to login with a non-existent user. This produces a log entry:

Exception while invoking method login errorClass [Error]: User not found [401]

This error appears when the user doesn’t exist or the password is invalid. That’s all, but since the user is not being mentioned, it’s more or less worthless. I think I’m going to open an issue on github about this.

[reetp]

They are extremely touchy about anything to do with personal information and/or security. Hence not logging a name I’d guess.

It is logging an authentication failure, just not logging all the info you want, and every chance they may not do anything about it.

I presume any user who can’t login would raise it with you anyway?

What sort of authent do you use? Rocket or SAML/LDAP?

Can’t a user reset a password?

[sintertrack]

@reetp

They are extremely touchy about anything to do with personal information and/or security. Hence not logging a name I’d guess.

makes sense.

@reetp

I presume any user who can’t login would raise it with you anyway?

They do and I can’t tell them anything besides “try this”, “try that”

@reetp

What sort of authent do you use? Rocket or SAML/LDAP?

Rocket native.

@reetp

Can’t a user reset a password?

Yes, I have one specific user who has weird problems. They login every day and everything works just fine. Then, out of nowhere, they can’t login anymore, according to Rocketchat their password is invalid. However, they are logged in with the very same credentials from another machine inside the corporate net via a proxy. Only logging in from the internet doesn’t work.

Then they try to reset the password, just in case. It doesn’t work. They get no email and the ui just stays on the same page. That’s it.

Here comes the interesting part: when I restart the mongodb pods, then their problems disappear. I haven’t tried it, but I’m pretty sure it would permanently vanish if I run mongodb with just one pod.

So, that was the reason why I am looking for a way to troubleshoot that users problems somehow.

[reetp]

Ok, as I started to suspect.

How many users do you have approximately, and what licence are you using?

[sintertrack]

@reetp

How many users do you have approximately, and what licence are you using?

600 users and community license.

[reetp]

@sintertrack

600 users and community license.

As I suspected, hence the 3 mongos.

That’ll likely be your issue then. It isn’t logging per se, but the ability of Rocket to handle that number of users.

On CE beyond about 200 users you will struggle with all sorts of odd issues - it’s a limitation of node itself.

If you use a paid version the problem is solved with micro services which can easily scale to millions of users. CE can’t, and won’t.

They’ve done that for a reason… And they aren’t going to fix it.

By all means open a bug on the logging, but bear in mind my comments on security above.

I don’t expect you’ll get much response.

[reetp]

As a follow up I’ve been talking to colleagues at Rocket about this.

Bad news - no, your 600 user CE issues - the likely real problem - will not get fixed. To fix any issues your solution is to pay for a supported version.

Good news - they may be open to a logging bug.

You could open one in the main repo.

[reetp]

Odd how conversation stops when licencing is raised…

Remember folks.

Rocketchat is open source.

But that is not the same as “free”.

If you want pre built binaries the hardware & hosting & development to produce them is not free. It has to be paid for somehow.

If you are using and profiting from Rocket consider paying somehow - even if you just do some support here, or open, or github.

Give a bit back. The team helps people like that.

If you are an open source organisation or other non profit, talk to sales about a licence. They may choose to help you.

[sintertrack]

@reetp

Odd how conversation stops when licencing is raised…

Oh no, I am just switching departments and was too busy to look after the issue.

I have checked again, we have 376 users total, not 600, and I’m not the one here to decide wether to pay for rocketchat or not. If you’d ask me, I’d pay for it.

As for the underlying issue itself: the user in question found out that he can fix the issue by deleting all website data in his browser (not just cookies, but everything), or using an incognito instance.