How to Synapse with Workers and NGINX

13. Juli 2021

How to Synapse with Workers and NginX

This document shows how to run Synapse with a

  • federation worker
  • a client worker
  • a federation sender

The example runs matrix.netfg.net on a VPS with 20G of RAM and 6 vCores on Archlinux. As of writing this document the current Python version is 3.9.6 and I installed synapse using PIP and Virtualenv - and I will not describe how to do that because it is straightforward.

Prerequisits

Install the following four packages using your distribution package manager (you may skip Python as it is most probably already installed). Make sure Nginx, Postgres as well as Redis are running with its default configuration. We will not touch Redis' config at all.

  1. Nginx
  2. Redis
  3. Python
  4. PostgreSQL

To create the Postgres DB run these short steps

su - postgres
createuser --pwprompt synapse
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse synapsedb

Then use synapse as db-user and synapsedb as database name below

Prefix #1

Create a new account on your server. I called mine synapse and gave him /srv/matrix as home directory. That directory is the place where I put all my synapse configuration files, logfiles and that directory is also the base of my python virtual environment, i.e. from where I call source venv/bin/activate. As synapse-user create a subdirectory logs there as we will not use systemd's journal for now. If all runs smoothely you can of course switch to the console log target and let systemd do the logging.

Prefix #2

DO NOT LET SYNAPSE's MAIN PROCESS LOG ON INFO LEVEL - OR YOUR VPS GETS UNUSEABLE!

Is this still (currently at v1.38.0 and tested around 10 weeks ago) the case? I don't know. But you have been warned!

Prefix #3

Using workers fixed not a single performance problem I encountered when using Matrix / Synapse. On my homeserver I am the only user and the server is idling 99% of the time with next-to-nothing-HDD-IO. (I ordered a HDD based VPS because I needed space! SSD based VPS's are still too expensive). There is some amount of web, email, syncthing and xmpp stuff going on but Synapse requires by far more resources then all those combined thanks to an interpreting language like Python and http as protocol. Choose the right tool for the job and not the tool you where teached at college learning IT Business!

Joining Matrix HD (~21000 users) took one hour and the client (Element on Android) timed out after 30 seconds. It returned with an error but the server still synced / federated, eating up CPU and IO resources! After 30 minutes Element did - all of a sudden - join the room. But this didn't mean that the join was complete. No! It took at least another 30 minutes until the server was back to normal. During that time the workers and the main process had fun penetrating the CPUs and the database as hard as can be ;)

There is this excuse where it is said "Well, it only takes a long time for the first user to join the room". Let me answer this: There is always a first user - and that person will be pissed and will piss off all others in that persons social environment. So how can we convince those WhatsApp and other island users (Signal, Threema, etc. pp.) to escape from their islands? Well, if this is the best we can, we can't!

Moving on (Thanks John...)

The shared configuration

Create the homeserver.yaml file. In the worker documentation this file is also referred to as the shared configuration file. Yes, it is shared amongst your workers but it is your homeservers configuration file. Why it is not referred to as homeserver.yaml or - at least - say your shared config file is your homeserver.yaml file?!

What you see below is what I use right now - and yes, this is a working and sufficient homeserver.yaml file. If you change the basic settings and Postgres settings you should be good to go. Make sure your workers a ready too of course. ;)

# basic settings
server_name: "YOUR INSTANCE NAME"
signing_key_path: "The full path to your signing key"
registration_shared_secret: "some random string"
media_store_path: "full path to your media store directory"
public_baseurl: "Most probably YOUR INSTANCE NAME (server_name) prefixed with https://"
admin_contact: "your email for this instance"
log_config: "the full path to the file which configures logging"

# those are my preferences, change as needed
use_presence: false
enable_metrics: false
report_stats: false
suppress_key_server_warning: true
max_upload_size: 50M

# the main process listener
# change the port as needed and leave it as is
listeners:
  - port: 8008
    tls: false
    type: http
    x_forward: true
    bind_addresses: ['::1', '127.0.0.1']
    resources:
      - names: [client, federation]
        compress: false

# The replication listener. Change port number as needed
# Every single worker will talk to this port using its worker_replication_http_port setting
  - port: 9093
    bind_addresses: ['::1', '127.0.0.1']
    type: http
    resources:
    - names: [replication]

# we use Postgres on Localhost
database:
  name: psycopg2
  args:
    user: synapse
    password: xxx
    database: xxx
    host: localhost

redis:
  enabled: true

send_federation: false
federation_sender_instances:
  - federation_sender # the name must match the app_name of your sender

1st) The federation_sender

According to the docs the federation sender

Handles sending federation traffic to other servers. Doesn't handle any REST endpoints itself, but you should set send_federation: False in the shared configuration file to stop the main synapse sending this traffic.

worker_app: synapse.app.federation_sender
worker_name: federation_sender
worker_replication_host: 127.0.0.1
worker_replication_http_port: 9093
worker_log_config: "/srv/matrix/federation_sender_log_config.yaml"

As synapse user save this as federation_sender.yaml in synapse's home directory. We will refer to it in systemd's config file - which by the way looks like this:

[Unit]
Description=Matrix Synapse Federation Sender
After=matrix.service

[Service]
Type=notify
NotifyAccess=main
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/srv/matrix/venv/bin/python -m synapse.app.federation_sender --config-path=/srv/matrix/homeserver.yaml --config-path=/srv/matrix/federation_sender.yaml
ExecStop=/bin/kill $MAINPID
WorkingDirectory=/srv/matrix
SyslogIdentifier=Federation-Sender
User=synapse
Group=synapse
Restart=on-failure

[Install]
WantedBy=multi-user.target

As root user save this in /etc/systemd/system/matrix_federation_sender.service

The last step for this worker is to create a file to configure its logger and refer to it in the worker_log_config setting of the worker configuration setting file. It can look like this:

version: 1

formatters:
  precise:
   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'

handlers:
  file:
    class: logging.handlers.RotatingFileHandler
    formatter: precise
    filename: /srv/matrix/logs/federationsender.log
    maxBytes: 104857600
    backupCount: 10
    level: ERROR
  console:
    class: logging.StreamHandler
    formatter: precise

loggers:
  synapse:
    level: ERROR

  synapse.storage.SQL:
    level: ERROR

root:
  level: ERROR
  handlers: [file]

Now you can enable the service by executing systemctl enable matrix_federation_sender and start it using using systemctl start matrix_federation_sender

Now do this again for one client worker and for one federation worker. It's basically the same for both of them:

  • copy and adjust the synapse worker file
  • copy and adjust the worker log config file
  • copy and adjust the worker systemd service file

2nd) The federation reader

Remember the worker_listeners port number. NginX needs to know where to proxy requests to because the federation reader and client have to handle http REST requests.

worker_app: synapse.app.generic_worker
worker_name: federation_worker # name it as you like

worker_replication_host: 127.0.0.1
worker_replication_http_port: 9093

worker_listeners:
- type: http
  port: 8084 # adjust as needed but remember it!
resources:
  - names:
    - federation

worker_log_config: "/srv/matrix/federation_worker_log_config.yaml"

Save this file as federation_worker.yaml

version: 1

formatters:
  precise:
   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'

handlers:
  file:
    class: logging.handlers.RotatingFileHandler
    formatter: precise
    filename: /srv/matrix/logs/federationreader.log
    maxBytes: 104857600
    backupCount: 10
    level: ERROR
  console:
    class: logging.StreamHandler
    formatter: precise

loggers:
  synapse:
    level: ERROR

  synapse.storage.SQL:
    level: ERROR

root:
  level: ERROR
  handlers: [file]

Save this as federation_worker_log_config.yaml

[Unit]
Description=Matrix Synapse Federation Worker
After=matrix.service

[Service]
Type=notify
NotifyAccess=main
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/srv/matrix/venv/bin/python -m synapse.app.generic_worker --config-path=/srv/matrix/homeserver.yaml --config-path=/srv/matrix/federation_worker.yaml
ExecStop=/bin/kill $MAINPID
WorkingDirectory=/srv/matrix
SyslogIdentifier=Federation-Worker
User=synapse
Group=synapse
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save this as /etc/systemd/system/matrix_federation_worker.service and enable and start the service as shown above.

3rd) The federation client

Again, remember the worker_listeners port number.

worker_app: synapse.app.generic_worker # as above this also is a type of generic_worker
worker_name: worker_client # name it as you like

worker_replication_host: 127.0.0.1
worker_replication_http_port: 9093

worker_listeners:
- type: http
  port: 8083 # adjust as needed but remember it!
resources:
  - names:
    - client

worker_log_config: "/srv/matrix/worker_client_log_config.yaml"

Save this file as client_worker.yaml

version: 1

formatters:
  precise:
    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'

handlers:
  file:
    class: logging.handlers.RotatingFileHandler
    formatter: precise
    filename: /srv/matrix/logs/federationclient.log
    maxBytes: 104857600
    backupCount: 10
    level: ERROR
  console:
    class: logging.StreamHandler
    formatter: precise

loggers:
  synapse:
    level: ERROR

  synapse.storage.SQL:
    level: ERROR

root:
  level: ERROR
  handlers: [file]

Save this as worker_client_log_config.yaml

[Unit]
Description=Matrix Synapse Client Federation
After=matrix.service

[Service]
Type=notify
NotifyAccess=main
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/srv/matrix/venv/bin/python -m synapse.app.generic_worker --config-path=/srv/matrix/homeserver.yaml --config-path=/srv/matrix/client_worker.yaml
ExecStop=/bin/kill $MAINPID
WorkingDirectory=/srv/matrix
SyslogIdentifier=Synapse-Client
User=synapse
Group=synapse
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save this as /etc/systemd/system/matrix_client_worker.service and enable and start the service as above.

Checkpoint #1

You now have a working Synapse configuration with one main process and three additional synapse worker processes

  • one federation sender
  • one federation client and
  • one federation reader (worker)

Now you must tell NginX to proxy Synapses requests to your workers. This is what we do next...

Proxy Synapse requests using NginX

Become root and cd into /etc/nginx. Make a new directory conf and a directory named sites.d - you don't have to but this is what I did... Now cd into conf and create two new files

1: synapse-client-proxypass.conf

proxy_pass http://127.0.0.1:8083;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 600;
client_max_body_size 50M; # max_upload_size defined as in homeserver.yaml

2: synapse-federation-proxypass.conf

proxy_pass http://127.0.0.1:8084;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 600;
client_max_body_size 50M;

Adjust client_max_body_size to your liking. The value should match max_upload_size in homeserver.yaml Now change into the sites.d directory. We create three files there. Of course you have to adjust the server name and the path your Lets Encrypt certificates!

1: the main matrix file

# Federation
server {
    listen 8448 ssl http2 default_server;
    listen [::]:8448 ssl http2 default_server;

    server_name "Your Full Qualified Hostname here";

    ssl_certificate      /full/path/to/your/letsencrypt/fullchain.pem;
    ssl_certificate_key  /full/path/to/your/letsencrypt/privkey.pem;
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
        proxy_pass http://127.0.0.1:8008;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name "Your Full Qualified Hostname here / same as above";

    ssl_certificate      /full/path/to/your/letsencrypt/fullchain.pem;
    ssl_certificate_key  /full/path/to/your/letsencrypt/privkey.pem;
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location /.well-known/matrix/server {
        access_log off;
        add_header Access-Control-Allow-Origin *;
        default_type application/json;
        return 200 '{"m.server": "Your Full Qualified Hostname:443"}';
    }

    location /.well-known/matrix/client {
        access_log off;
        return 200 '{"m.homeserver": {"base_url": "https://Your Full Qualified Hostname"}}';
        default_type application/json;
        add_header Access-Control-Allow-Origin *;
    }

    # Federation stuff
    include /etc/nginx/sites.d/matrix-federation.conf;

    # Client stuff
    include /etc/nginx/sites.d/matrix-client.conf;

    # Fallback. There are still some routes you need to catch such as browsing for rooms
    # which won't work otherwise
    location ~* ^(\/_matrix|\/_synapse\/client) {
        proxy_pass http://127.0.0.1:8008;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $host;
        proxy_read_timeout 600;
        client_max_body_size 50M;
    }
}

2: client worker

# see https://github.com/matrix-org/synapse/blob/develop/docs/workers.md

# Sync requests
location ~* ^/_matrix/client/(v2_alpha|r0)/sync$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|v2_alpha|r0)/events$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0)/initialSync$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

# Client API requests
location ~* ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/devices$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/keys/query$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/versions$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/joined_groups$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/event/ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/joined_rooms$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/search$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

# Registration/login requests

location ~* ^/_matrix/client/(api/v1|r0|unstable)/login$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(r0|unstable)/register$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

# Event sending requests

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/join/ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

location ~* ^/_matrix/client/(api/v1|r0|unstable)/profile/ {
    include /etc/nginx/conf/synapse-client-proxypass.conf;
}

3: federation worker

# see https://github.com/matrix-org/synapse/blob/develop/docs/workers.md
# Federation requests
location ~* ^/_matrix/federation {
   include /etc/nginx/conf/synapse-federation-proxypass.conf;
}

location ~* ^/_matrix/key {
   include /etc/nginx/conf/synapse-federation-proxypass.conf;
}

Remember the federation sender? It does not handle any REST endpoints so we do not need to proxy this.

The last thing we have to do is to activate our matrix.conf file. Open up /etc/nginx/nginx.conf and find the line where your http block closes. Insert include /etc/nginx/sites.d/matrix.conf; right before the closing bracket. The file should basically look like this.

# some
# stuff
http {
  # maybe more sites and other stuff
  include /etc/nginx/sites.d/matrix.conf;
}

Checkpoint #2 reached

Restart nginx. Well done ;)