Matrix Federation can't receive incoming messages

Description

Hello, I try to setup Matrix Bridge. But something not work as should.
Currently I can send messages to outside (e.g. <user.name>:matrix.org) - this part works fine.
But I can’t receive any messages from outside.
I’ve only small record in rocketchat logs:
rocketchat1 | - - [dd/MM/YYYY:hh:mm:ss +0000] “PUT /_matrix/app/v1/transactions/1 HTTP/1.1” 403 55 “-” “Synapse/1.114.0”

Doing “Verify configuration” i’ve see message:
All configuration looks good, homeserver configuration looks good, appservice configuration looks good, total round trip time to homeserver 5ms

And log entry:
rocketchat1 | - - [dd/MM/YYYY:hh:mm:ss +0000] “POST /_matrix/app/v1/ping HTTP/1.1” 200 2 “-” “Synapse/1.114.0”

Any idea what is the root of the issue?

Server Setup Information

  • Version of Rocket.Chat Server: 6.12.0
  • Operating System: Ubuntu 24.04
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog:
  • NodeJS Version: v14.21.3
  • MongoDB Version: 6.0.13
  • Proxy: nginx
  • Firewalls involved:

BTW: Here is a similar topic with no replay
Rocket.Chat / Matrix federation: connected but messaging only works one way - Community Support - Rocket.Chat

I think there are a number of licencing issues caused by the upgraded to 6.12 that are affecting various services.

eg:

Also here and in open.rocket.chat

The team is aware but no idea when they’ll get fix out.

Hi, I’m the one who opened activation a Starter license fails with “Period validation failed”
that problem is solved for me and I can see the Starter license activated.
But I still have the issue to send messages any direction.
From inside the synapse container I can do pings to the rocket chat server using the hs_token from the registration.yml
curl http://rocketchat:3300/_matrix/app/v1/ping -XPOST -H "Authorization: Bearer {hs_token}"
but any other API requests returns {"errcode":"M_FORBIDDEN","error":"Bad token supplied,"}

If you are trying to debug this you need to go back to the start and tell us how you deployed it and a lot of your settings.

Otherwise it is all just guesswork.

It also isn’t clear here if we have two separate issues or the same one.

I’ve managed to setup the federation on a virgin server using the automated installation script and Rocket.Chat 6.6.3
This works, but fails when updating to 6.12.0
see:

Hello,
Here you go my setup. Hope this will be enough to reproduce issue

docker-compose.yml

volumes:
  mongodb:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: ./mongodb

networks:
  app:
    name: app
  db:
    name: db

services:
  rocketchat1:
    container_name: rocketchat1
    image: registry.rocket.chat/rocketchat/rocket.chat:6.12.0
    restart: always
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - ./rocketchat/uploads:/app/uploads
    environment:
      MONGO_URL: "mongodb://mongodb:27017/rocketchat?replicaSet=rs0"
      MONGO_OPLOG_URL: "mongodb://mongodb:27017/local?replicaSet=rs0"
      ROOT_URL: http://localhost:3000
      DEPLOY_METHOD: docker
    depends_on:
      - mongodb
    networks:
      - app
      - db
    ports:
      - 3000:3000
      - 3300:3300
    expose:
      - 3300

  synapse1:
    image: matrixdotorg/synapse:latest
    container_name: synapse1
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - ./synapse_data1:/data:rw
    environment:
      - SYNAPSE_SERVER_NAME=<SERVERNAME>
      - SYNAPSE_REPORT_STATS=no
    restart: always
    depends_on:
      - postgres
    networks:
      - app
      - db
    ports:
      - 8008:8008
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "2"

  mongodb:
    image: docker.io/bitnami/mongodb:6.0.13
    container_name: mongodb
    restart: always
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - mongodb:/bitnami/mongodb:rw
    environment:
      MONGODB_REPLICA_SET_MODE: primary
      MONGODB_REPLICA_SET_NAME: rs0
      MONGODB_PORT_NUMBER: 27017
      MONGODB_INITIAL_PRIMARY_HOST: mongodb
      MONGODB_INITIAL_PRIMARY_PORT_NUMBER: 27017
      MONGODB_ADVERTISED_HOSTNAME: mongodb
      MONGODB_ENABLE_JOURNAL: true
      ALLOW_EMPTY_PASSWORD: yes
    networks:
      - db


  postgres:
    image: docker.io/postgres:16-alpine
    container_name: postgres
    restart: always
    environment:
      - POSTGRES_USER=<PG_USERNAME>
      - POSTGRES_PASSWORD=<PG_PASSWORD>
      - POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
      - ./postgres:/var/lib/postgresql/data
    networks:
      - db
server_name: "<SERVERNAME>"
pid_file: /data/homeserver.pid
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    resources:
      - names: [client, federation]
        compress: false
database:
  name: psycopg2
  txn_limit: 10000
  args:
    user: <PG_USERNAME>
    password: <PG_PASSWORD>
    database: synapse
    host: postgres
    port: 5432
    cp_min: 5
    cp_max: 10
log_config: "/data/<SERVERNAME>.log.config"
media_store_path: /data/media_store
registration_shared_secret: "<registration_shared_secret>"
report_stats: false
macaroon_secret_key: "<macaroon_secret_key>"
signing_key_path: "/data/<SERVERNAME>.signing.key"
trusted_key_servers:
  - server_name: "matrix.org"

app_service_config_files:
  - /data/rocketchat.yaml

rocketchat.yaml - copied from RocketChat UI

NGINX

server {
    listen      80 default_server;
    location / {
        rewrite ^ https://$host$request_uri? permanent;
    }
    include include/certbot.location;
}
server {
    listen      443 ssl;
    server_name  <rocketchat>;
    client_max_body_size 10m;
    location / {
        proxy_pass         http://<docker>:3000/;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
    location /.well-known/matrix/server {
            default_type application/json;
            add_header Access-Control-Allow-Origin *;
            return 200 '{"m.server": "<matrix>"}';
    }
    location /.well-known/matrix/client {
            default_type application/json;
            add_header Access-Control-Allow-Origin *;
            return 200 '{"m.homeserver": {"base_url": "https://<matrix>"}}';
    }

    access_log  /var/log/nginx/<rocketchat>.log ;
    error_log  /var/log/nginx/<rocketchat>.log ;

    ssl_certificate     /etc/nginx/ssl/live/<rocketchat>/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/live/<rocketchat>/privkey.pem;

    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
	server {		
    listen      443 ssl;		
    listen      8448 ssl;		
    server_name  <matrix>;		
    add_header X-Frame-Options DENY;		
    add_header X-Content-Type-Options nosniff;		
    add_header X-XSS-Protection "1; mode=block";		
    location ~ ^(/_matrix|/_synapse/client) {		
        proxy_pass http://<docker>:8008;		
        proxy_set_header X-Forwarded-For $remote_addr;		
        proxy_set_header X-Forwarded-Proto $scheme;		
        proxy_set_header Host $host;		
        client_max_body_size 50M;		
        proxy_http_version 1.1;		
    }		

    access_log  /var/log/nginx/<matrix>.log ;		
    error_log  /var/log/nginx/<matrix>.log ;		

    ssl_certificate     /etc/nginx/ssl/live/<matrix>/fullchain.pem;		
    ssl_certificate_key /etc/nginx/ssl/live/<matrix>/privkey.pem;		
}

NS Records
<rocketchat> IN A <Public IP>
<matrix> IN A <Public IP>

Port Forwarding
ANY to <Public IP> port 80,443,8448 DST-NAT DST-PORT Original

Test Results:
https://federationtester.matrix.org/api/report?server_name= <rocketchat>

{
    "WellKnownResult": {
        "m.server": "<matrix>",
        "CacheExpiresAt": 0
    },
    "DNSResult": {
        "SRVSkipped": false,
        "SRVCName": "",
        "SRVRecords": null,
        "SRVError": {
            "Message": "lookup _matrix._tcp.<matrix> on 8.8.8.8:53: no such host"
        },
        "Hosts": {
            "<matrix>": {
                "CName": "<matrix>.",
                "Addrs": [
                    "<Public IP>"
                ],
                "Error": null
            }
        },
        "Addrs": [
            "<Public IP>:8448"
        ]
    },
    "ConnectionReports": {
        "<Public IP>:8448": {
            "Certificates": [
                {
                    "SubjectCommonName": "<matrix>",
                    "IssuerCommonName": "E5",
                    "SHA256Fingerprint": "<Fingerprin>",
                    "DNSNames": [
                        "<matrix>"
                    ]
                },
                {
                    "SubjectCommonName": "E5",
                    "IssuerCommonName": "ISRG Root X1",
                    "SHA256Fingerprint": "Xf2zzzGybyPYfAnzoM72QvZAaan7fP4pJwu13A8eFrs",
                    "DNSNames": null
                }
            ],
            "Cipher": {
                "Version": "TLS 1.3",
                "CipherSuite": "TLS_AES_128_GCM_SHA256"
            },
            "Checks": {
                "AllChecksOK": true,
                "MatchingServerName": true,
                "FutureValidUntilTS": true,
                "HasEd25519Key": true,
                "AllEd25519ChecksOK": true,
                "Ed25519Checks": {
                    "ed25519:a_Adoq": {
                        "ValidEd25519": true,
                        "MatchingSignature": true
                    }
                },
                "ValidCertificates": true
            },
            "Errors": [],
            "Ed25519VerifyKeys": {
                "ed25519:a_Adoq": "<ED_Key1>"
            },
            "Info": {},
            "Keys": {
                "old_verify_keys": {},
                "server_name": "<rocketchat>",
                "signatures": {
                    "<rocketchat>": {
                        "ed25519:a_Adoq": "<ED_Key2>"
                    }
                },
                "valid_until_ts": 1726051527513,
                "verify_keys": {
                    "ed25519:a_Adoq": {
                        "key": "<ED_Key1>"
                    }
                }
            }
        }
    },
    "ConnectionErrors": {},
    "Version": {
        "name": "Synapse",
        "version": "1.114.0"
    },
    "FederationOK": true
}

It would have been helpful if you had worked through the issue instead of racing off and filing a bug.

This just confuses things and make sit difficult to track. And this is open source. It won’t get fixed any faster unless you have paid support.

This is you.

What makes it more difficult is you say you followed the automated install which says use ‘latest’ (see below) but you used 6.6.3 and then upgraded?

Where does it mention that?

You need to be more methodical when testing these things.

Please test on 6.11.2 so we can make sure it is 6.12 and not other factors - unless you did upgrade to 6.11.2 but never mentioned it.


Note - never use ‘latest’ as per this - I need to get the docs changed:

@reetp I’m very sorry if I caused confusion or made things difficult.
I don’t try to push someone and I’m very aware of the limited support I can expect without paying for it.
You are totally right with the latest tag, I’ve used it for test setup because it’s mentioned in the documentation Deploy a Federated Rocket.Chat Workspace with Docker

In the meantime I’ve spend a couple of hours looking through the code and doing some experiments. I’ve documented my findings and guesses in the GitHub issue.
Again do not want to be pushy or annoying, actually the opposite I hope that my findings will help those who really know their way around Rocket.Chat to find the bug easier.
If you think the GitHub issue is the wrong place to put the findings, feel free to close it and I will dump them here or in an other thread.

@vlivankin short version for you: if you managed to setup the connection in one direction, try doing the same thing using Rocket.Chat 6.11.2 and synapse 1.71

Your work in the bug now is very good thanks.

I also wrote this some while ago.

The links on How to Report Bugs Effectively and How to ask a smart question that will attract attention will save a lot of XYInfo issues and really help devs.

They are much more likely to leap into a well documented, well researched bug than “it’s broke I need help”.

Try and separate issues - they may seem related but that is an ‘assumption’ (makes an ‘ass of u and me’ :slight_smile: ) you have got to prove - as per your registration issue vs Matrix issue. Be careful of rabbit holes.

Thanks.