Install Zitadel with Docker Compose, Nginx and Let’s Encrypt

Follow the Zitadel Docker Compose hardening steps word for word and the API container never starts. The docs have you replace the PostgreSQL admin password, but the connection string in the same .env file still says postgres:postgres, so zitadel-api restarts in a loop with password authentication failed. Fix that and the stack comes up with open self-registration, no account lockout and a Traefik container that holds the Docker socket. None of it is hard to change. You only have to know it is there.

Original content from computingforgeeks.com - post 173067

This guide installs Zitadel with Docker Compose the way you would run it for real users: the official compose file left untouched, Nginx with Let’s Encrypt in place of the bundled Traefik, PostgreSQL behind a role that is not a superuser, the first admin and an automation account created before the first start, forced MFA, lockout, and a device code login to prove an app can sign in. Optional sections add password reset over SMTP, Grafana with role mapping, and a least privilege service account, and the guide ends with a timed restore and a measured upgrade. Everything was run on Ubuntu 26.04 and replayed on Ubuntu 24.04 in October 2026 with Zitadel 4.19.4.

How the pieces fit

Zitadel ships as two containers. zitadel-api serves the API, the OIDC and SAML endpoints and the management console, and speaks HTTP/2 without TLS (h2c) on port 8080. zitadel-login is the newer Next.js login UI on port 3000, and it talks to the API with a token stored in a file. Nginx terminates TLS, sends /ui/v2/login to the login container and everything else to the API. Both containers bind to 127.0.0.1, so Nginx is the only way in. If you are still choosing an identity provider, the Keycloak production setup in Docker and the authentik Docker Compose guide build the same kind of stack on the other two big open source options. The four-way identity provider comparison tested Zitadel against them and Authelia on a single VM.

The PAT files on the left are easy to overlook and the restore drill below shows why they matter. The official pack mounts them into a Docker volume that only the first start ever writes, so they go into a host directory that the backup picks up.

Component	Listens on	Role
Nginx	0.0.0.0:80, 0.0.0.0:443	TLS, routing between the login UI and the API
zitadel-api	127.0.0.1:8080	API, OIDC endpoints, console at /ui/console
zitadel-login	127.0.0.1:3000	Login UI at /ui/v2/login
PostgreSQL	Docker network only	All Zitadel state, as an event store
Grafana (optional example)	127.0.0.1:3001	Example OIDC app with role mapping

The core setup is the same for everyone and needs one server and one DNS name. It ends with a real application login. The optional sections after it are independent, so follow only the ones you need. The last part covers backups, upgrades and the errors we hit.

Section	Needed for	Extra things you need
Core setup	Everyone	zitadel.example.com only
Email and password reset	Self-service password reset, verification mail	An SMTP relay account
Grafana over OIDC	Web apps that log in with OpenID Connect and map roles	A Grafana DNS name if Grafana runs on this host
Service account for scripts	Provisioning users from CI or a script	Nothing extra

Prerequisites and sizing

Login rate decides the CPU. Zitadel hashes passwords with bcrypt at cost 14, and on the test host one password check took about 0.95 seconds of a single Intel i7-6700 core. Fifty people signing in within the same minute keep one core busy for most of that minute, so count cores against peak logins, not total users. Passkey logins skip the hash. Memory is modest: after the full setup the API used 129 MiB, the login UI 127 MiB and PostgreSQL 83 MiB.

The lab VM had 4 vCPU and 8 GB of RAM. That is a comfortable floor for following along, not a production size. The docs ask for at least 2 GB of RAM. For a few thousand users, 2 to 4 cores and 4 GB on one host is a sane start, and the API scales out by adding replicas once you split the init step from the server (the production checklist in the docs covers that).

• A Linux host with Docker Engine and the Compose plugin. Ubuntu 26.04 and 24.04 were both tested. The Docker CE install guide for Ubuntu covers the repository setup.
• A DNS A record for the Zitadel name (zitadel.example.com here) pointing at the host. Zitadel answers only on the name it was set up with, so pick the final name now. Changing it later is a migration, not an edit.
• Inbound TCP 80 and 443 for the HTTP-01 challenge and the site. A host without inbound port 80 can use DNS-01, shown below.
• A TOTP app on your phone (Aegis, 2FAS, Google or Microsoft Authenticator, or your password manager), because the baseline makes MFA mandatory, the admin included.

Set the shell variables

The core setup needs two values: the Zitadel hostname and the email address of the first admin, which is also the login name of that admin and the contact address for Let’s Encrypt. Use a real mailbox, because Let’s Encrypt refuses example.com addresses. Secrets never go into this block. The commands that need a password ask for it at a prompt.

export ZITADEL_DOMAIN="zitadel.example.com"
export ADMIN_EMAIL="[email protected]"
echo "Zitadel on ${ZITADEL_DOMAIN}, first admin ${ADMIN_EMAIL}"

The variables last for the current shell. If you reconnect later, run this block again, plus the ZITADEL_PAT line from the first start section.

Get a Let’s Encrypt certificate

Install Nginx, Certbot with its Nginx plugin, and jq for reading API responses:

sudo apt update
sudo apt install -y nginx certbot python3-certbot-nginx jq

certonly --nginx answers the HTTP-01 challenge through the running Nginx without editing any config, and the deploy hook reloads Nginx after every renewal. The Nginx and Let’s Encrypt guide for Ubuntu 26.04 explains Certbot in more depth:

sudo certbot certonly --nginx -d "${ZITADEL_DOMAIN}" --non-interactive --agree-tos -m "${ADMIN_EMAIL}" \
  --deploy-hook "systemctl reload nginx"

Private network or no port 80: use DNS-01

A host on a private LAN or behind NAT proves ownership with a DNS TXT record instead. Certbot has a plugin per provider (python3-certbot-dns-route53, -digitalocean, -google, -rfc2136 and more). The lab used Cloudflare. The token needs Zone DNS Edit, and the prompt keeps it out of your shell history:

sudo apt install -y python3-certbot-dns-cloudflare
read -rs -p "Cloudflare API token: " CF_TOKEN; echo
echo "dns_cloudflare_api_token = ${CF_TOKEN}" | sudo tee /etc/letsencrypt/cloudflare.ini >/dev/null
sudo chmod 600 /etc/letsencrypt/cloudflare.ini
unset CF_TOKEN

Then request the certificate over DNS:

sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 30 -d "${ZITADEL_DOMAIN}" --non-interactive --agree-tos -m "${ADMIN_EMAIL}" \
  --deploy-hook "systemctl reload nginx"

Either way, Certbot ends with the saved paths:

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/zitadel.example.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/zitadel.example.com/privkey.pem

Download the compose pack and pin the release

The stack lives in /opt/zitadel. Download the official compose file and its example environment file, which becomes your .env. These are the same two files the Zitadel Docker Compose docs start from:

sudo mkdir -p /opt/zitadel && cd /opt/zitadel
sudo curl -fsSLO https://raw.githubusercontent.com/zitadel/zitadel/main/deploy/compose/docker-compose.yml
sudo curl -fsSL -o .env https://raw.githubusercontent.com/zitadel/zitadel/main/deploy/compose/.env.example
grep -E '^(ZITADEL_VERSION|POSTGRES_IMAGE)=' .env

The example file pins its own release, and it lags behind:

ZITADEL_VERSION=v4.16.0
POSTGRES_IMAGE=postgres:17.10-alpine

On the day we tested, that pin was three minor releases behind the latest. Ask GitHub for the current release instead, so a fresh install starts on it:

ZITADEL_VERSION=$(curl -fsSL https://api.github.com/repos/zitadel/zitadel/releases/latest | jq -r .tag_name)
echo "${ZITADEL_VERSION}"

The tag prints with its leading v, which is how the image tags are named:

v4.19.4

Write the secrets and the database connection

The masterkey encrypts secrets at rest and cannot change after the first start, so it is generated once, here. This block also sets the public URL to HTTPS on 443, writes a login cookie secret and two database passwords, and points Zitadel at a database role called zitadel instead of the postgres superuser. Rewriting the connection string in the same command is what the official hardening steps miss:

MASTERKEY=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 32)
PG_ADMIN_PASS=$(openssl rand -hex 24)
PG_ZITADEL_PASS=$(openssl rand -hex 24)
sudo sed -i \
  -e "s|^ZITADEL_DOMAIN=.*|ZITADEL_DOMAIN=${ZITADEL_DOMAIN}|" \
  -e "s|^ZITADEL_EXTERNALPORT=.*|ZITADEL_EXTERNALPORT=443|" \
  -e "s|^ZITADEL_EXTERNALSECURE=.*|ZITADEL_EXTERNALSECURE=true|" \
  -e "s|^ZITADEL_PUBLIC_SCHEME=.*|ZITADEL_PUBLIC_SCHEME=https|" \
  -e "s|^ZITADEL_VERSION=.*|ZITADEL_VERSION=${ZITADEL_VERSION}|" \
  -e "s|^ZITADEL_MASTERKEY=.*|ZITADEL_MASTERKEY=${MASTERKEY}|" \
  -e "s|^LOGIN_SESSION_COOKIE_SECRET=.*|LOGIN_SESSION_COOKIE_SECRET=$(openssl rand -hex 32)|" \
  -e "s|^POSTGRES_ADMIN_PASSWORD=.*|POSTGRES_ADMIN_PASSWORD=${PG_ADMIN_PASS}|" \
  -e "s|^ZITADEL_DATABASE_POSTGRES_DSN=.*|ZITADEL_DATABASE_POSTGRES_DSN=postgresql://zitadel:${PG_ZITADEL_PASS}@postgres:5432/zitadel?sslmode=disable|" \
  .env
echo "POSTGRES_ZITADEL_PASSWORD=${PG_ZITADEL_PASS}" | sudo tee -a .env >/dev/null
sudo chmod 600 .env
unset MASTERKEY PG_ADMIN_PASS PG_ZITADEL_PASS

Check the result. The masterkey must be exactly 32 characters:

sudo grep -E '^(ZITADEL_DOMAIN|ZITADEL_EXTERNAL|ZITADEL_PUBLIC|ZITADEL_VERSION)' .env
sudo awk -F= '/^ZITADEL_MASTERKEY=/{print length($2)}' .env

The four public URL settings and the release match what you set, and the length is 32:

ZITADEL_DOMAIN=zitadel.example.com
ZITADEL_EXTERNALPORT=443
ZITADEL_EXTERNALSECURE=true
ZITADEL_PUBLIC_SCHEME=https
ZITADEL_VERSION=v4.19.4
32

The masterkey and .env are the two things you cannot lose. Without the masterkey, a database backup cannot be decrypted. Copy .env to your password manager or secret store now, before anything depends on it.

Define the first admin and the automation account

Without these settings, the first start creates zitadel-admin with the password Password1!. Set your own admin instead. The password must have at least 8 characters with an uppercase letter, a lowercase letter, a number and a symbol, or the first start fails. Avoid a single quote in it, because .env stores it in single quotes:

read -rs -p "Password for ${ADMIN_EMAIL}: " ZITADEL_ADMIN_PASSWORD; echo
printf '%s\n' "ZITADEL_ORG_NAME=Example" \
  "ZITADEL_ADMIN_EMAIL=${ADMIN_EMAIL}" \
  "ZITADEL_ADMIN_PASSWORD='${ZITADEL_ADMIN_PASSWORD}'" \
  "ADMIN_SA_PAT_EXPIRATION=$(date -u -d '+1 year' +%Y-%m-%dT00:00:00Z)" | sudo tee -a .env >/dev/null
unset ZITADEL_ADMIN_PASSWORD

ZITADEL_ORG_NAME is your first organization, where users and apps live. Use your company name; Example is only a default. The last line sets when the token of the automation account expires, one year from today.

Local changes go into an override file, so the official compose file stays as downloaded and an upgrade never has to merge your edits. Create it:

sudo vim /opt/zitadel/docker-compose.override.yml

The override does four things. It moves Traefik into a profile that never starts, which also removes the only container that mounts the Docker socket. It publishes the API and login UI on localhost only. It passes the admin settings from .env and creates admin-sa, a machine user with the instance owner role whose token is written to secrets/. Last, it replaces the Docker volume for the login token with a host directory, so backups include it:

services:
  proxy:
    profiles: ["traefik"]

  zitadel-api:
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_BASEURI: https://${ZITADEL_DOMAIN}/ui/v2/login/
      ZITADEL_OIDC_DEFAULTLOGINURLV2: https://${ZITADEL_DOMAIN}/ui/v2/login/login?authRequest=
      ZITADEL_OIDC_DEFAULTLOGOUTURLV2: https://${ZITADEL_DOMAIN}/ui/v2/login/logout?post_logout_redirect=
      ZITADEL_SAML_DEFAULTLOGINURLV2: https://${ZITADEL_DOMAIN}/ui/v2/login/login?samlRequest=
      ZITADEL_FIRSTINSTANCE_ORG_NAME: ${ZITADEL_ORG_NAME}
      ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME: ${ZITADEL_ADMIN_EMAIL}
      ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD: ${ZITADEL_ADMIN_PASSWORD}
      ZITADEL_FIRSTINSTANCE_PATPATH: /zitadel/secrets/admin-sa.pat
      ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME: admin-sa
      ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME: Automation admin
      ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE: ${ADMIN_SA_PAT_EXPIRATION}
    volumes:
      - ./bootstrap:/zitadel/bootstrap:rw
      - ./secrets:/zitadel/secrets:rw

  zitadel-login:
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
      - ./bootstrap:/zitadel/bootstrap:ro

admin-sa is your way back in. Every API call in this guide uses its token, and when the human admin loses their phone, this token removes the lost authenticator. It lives in a directory only root can read and only the API container mounts. Create both directories and check what Compose will run:

sudo install -d -m 700 bootstrap secrets
sudo docker compose config --services

Traefik is gone from the list:

postgres
zitadel-api
zitadel-login

Create the database role and start Zitadel

Pull the images first, so the progress shows here and not inside a silent step later:

sudo docker compose pull

Start PostgreSQL alone and create the zitadel role that the connection string expects. The role owns the database and its schema and nothing else:

sudo docker compose up -d --wait postgres
PG_ZITADEL_PASS=$(sudo grep -oP '^POSTGRES_ZITADEL_PASSWORD=\K.*' .env)
echo "CREATE ROLE zitadel LOGIN PASSWORD '${PG_ZITADEL_PASS}'; ALTER DATABASE zitadel OWNER TO zitadel; ALTER SCHEMA public OWNER TO zitadel;" \
  | sudo docker compose exec -T postgres psql -U postgres -d zitadel -v ON_ERROR_STOP=1
unset PG_ZITADEL_PASS

PostgreSQL confirms all three statements:

CREATE ROLE
ALTER DATABASE
ALTER SCHEMA

Now start the rest. The first start creates the schema, the instance, both admins and the login token. It took 27 seconds on the lab VM:

sudo docker compose up -d --wait
sudo docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

All three containers report healthy, and the two Zitadel ports are bound to localhost:

SERVICE         IMAGE                                   STATUS                    PORTS
postgres        postgres:17.10-alpine                   Up 40 seconds (healthy)   5432/tcp
zitadel-api     ghcr.io/zitadel/zitadel:v4.19.4         Up 26 seconds (healthy)   127.0.0.1:8080->8080/tcp
zitadel-login   ghcr.io/zitadel/zitadel-login:v4.19.4   Up 5 seconds (healthy)    127.0.0.1:3000->3000/tcp

The admin password is only read on the first start. Blank it in .env so it does not sit on disk, and lock down the token file:

sudo sed -i "s/^ZITADEL_ADMIN_PASSWORD=.*/ZITADEL_ADMIN_PASSWORD=/" .env
sudo chmod 600 secrets/admin-sa.pat

Load the automation token into your shell and ask the API who it is. Later steps use ZITADEL_PAT, so this is also the line to run again after a reconnect:

export ZITADEL_PAT=$(sudo cat /opt/zitadel/secrets/admin-sa.pat)
curl -s http://127.0.0.1:8080/auth/v1/users/me -H "Host: ${ZITADEL_DOMAIN}" -H "Authorization: Bearer ${ZITADEL_PAT}" | jq '{user: .user.userName, machine: (.user.machine != null)}'

The token belongs to the machine user. The call went straight to the API on localhost with the public name as the Host header, because Nginx is not set up yet:

{
  "user": "admin-sa",
  "machine": true
}

Put Nginx in front

Create the site file:

sudo vim /etc/nginx/sites-available/zitadel.conf

Paste the server blocks. grpc_pass is not only for gRPC clients: Zitadel serves REST, gRPC-Web and the console over the same h2c port, and grpc_pass is how Nginx speaks HTTP/2 to a backend without TLS. The bare domain redirects to the console:

server {
    listen 80;
    listen [::]:80;
    server_name zitadel.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name zitadel.example.com;

    ssl_certificate     /etc/letsencrypt/live/zitadel.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/zitadel.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    add_header Strict-Transport-Security "max-age=31536000" always;

    location = / {
        return 302 /ui/console/;
    }

    location /ui/v2/login {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location / {
        grpc_pass grpc://127.0.0.1:8080;
        grpc_set_header Host $host;
        grpc_set_header X-Forwarded-Proto https;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        grpc_read_timeout 300s;
    }
}

Put your name into it, enable it and remove the default site:

sudo sed -i "s/zitadel.example.com/${ZITADEL_DOMAIN}/g" /etc/nginx/sites-available/zitadel.conf
sudo ln -sf /etc/nginx/sites-available/zitadel.conf /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

On Ubuntu 24.04, nginx -t fails here, because the Nginx in its archive is older than the release that added the http2 on; directive:

2026/10/05 19:16:42 [emerg] 33529#33529: unknown directive "http2" in /etc/nginx/sites-enabled/zitadel.conf:11
nginx: configuration file /etc/nginx/nginx.conf test failed

Move HTTP/2 onto the listen lines there, then test again:

sudo sed -i -e '/^\s*http2 on;/d' -e 's/listen 443 ssl;/listen 443 ssl http2;/' -e 's/listen \[::\]:443 ssl;/listen [::]:443 ssl http2;/' /etc/nginx/sites-available/zitadel.conf
sudo nginx -t && sudo systemctl reload nginx

Check the public side. The health endpoint, the OIDC discovery document and the root redirect all go through Nginx now:

curl -s https://${ZITADEL_DOMAIN}/debug/healthz; echo
curl -s https://${ZITADEL_DOMAIN}/.well-known/openid-configuration | jq '{issuer, authorization_endpoint, device_authorization_endpoint}'
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" https://${ZITADEL_DOMAIN}/

The issuer must be your HTTPS name with no port. If it shows :443 or http://, one of the ZITADEL_EXTERNAL* values is wrong:

ok
{
  "issuer": "https://zitadel.example.com",
  "authorization_endpoint": "https://zitadel.example.com/oauth/v2/authorize",
  "device_authorization_endpoint": "https://zitadel.example.com/oauth/v2/device_authorization"
}
302 https://zitadel.example.com/ui/console/

Open the firewall

Only SSH and Nginx need to be reachable. Allow SSH first, or enabling UFW over an SSH session cuts you off:

sudo ufw allow OpenSSH
sudo ufw allow "Nginx Full"
sudo ufw --force enable
sudo ufw status

The two profiles cover 22, 80 and 443:

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)

Docker publishes container ports through its own iptables rules, which skip UFW. That is why the override binds 8080 and 3000 to 127.0.0.1: a port published on 0.0.0.0 would be reachable from anywhere whatever UFW says.

Sign in for the first time

Open https://zitadel.example.com in a browser. The root redirects to the console, and the console sends you to the login UI:

Note the Register new user link under the field. That is the default: anyone who can reach this page can create an account in your organization. The next section turns it off. Sign in with your admin email and password, and the console opens:

Lock down the defaults

Three defaults are wrong for a server on the internet. We checked each one on the fresh instance before changing it:

Setting	Default	What we set	Why
Self-registration	On	Off	Strangers can create accounts in your organization
Account lockout	Off (no attempt limit)	10 wrong passwords or codes	20 wrong passwords in a row were accepted, and the 21st attempt with the right one still logged in
MFA	Optional	Forced for every user	A leaked password is otherwise a full login
“Forgot password?” link	Shown	Hidden until SMTP exists	Without mail it leads to a page that can never send the code

Forcing MFA changes how the admin signs in. The next time you sign in to the console, Zitadel makes you enroll a TOTP app or a security key before it lets you in. Have the authenticator app ready. If you lose it later, the recovery command is at the end of this section, and it only needs the admin-sa token you already have.

Apply the login policy. The command reads the current instance policy, changes the four fields and writes it back:

curl -s https://${ZITADEL_DOMAIN}/admin/v1/policies/login -H "Authorization: Bearer ${ZITADEL_PAT}" \
  | jq '.policy | del(.details, .isDefault, .secondFactors, .multiFactors, .idps) | .allowRegister=false | .forceMfa=true | .hidePasswordReset=true' \
  | curl -s -X PUT https://${ZITADEL_DOMAIN}/admin/v1/policies/login -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d @- | jq -c .

Then the lockout policy. Note the path: reading it is /admin/v1/policies/lockout, but writing it is /admin/v1/policies/password/lockout, and a PUT to the read path returns Method Not Allowed:

curl -s -X PUT https://${ZITADEL_DOMAIN}/admin/v1/policies/password/lockout -H "Authorization: Bearer ${ZITADEL_PAT}" \
  -H "Content-Type: application/json" -d '{"maxPasswordAttempts":10,"maxOtpAttempts":10}' | jq -c .

Read both back:

curl -s https://${ZITADEL_DOMAIN}/admin/v1/policies/login -H "Authorization: Bearer ${ZITADEL_PAT}" | jq -c '.policy | {allowRegister, forceMfa, hidePasswordReset}'
curl -s https://${ZITADEL_DOMAIN}/admin/v1/policies/lockout -H "Authorization: Bearer ${ZITADEL_PAT}" | jq -c '.policy | {maxPasswordAttempts, maxOtpAttempts}'

allowRegister prints null because the API leaves out fields that are false:

{"allowRegister":null,"forceMfa":true,"hidePasswordReset":true}
{"maxPasswordAttempts":"10","maxOtpAttempts":"10"}

Now restart the login UI. This step is not optional. The login container caches the login policy for 15 minutes and keeps serving the old copy while it refreshes. Before we restarted it, a fresh admin login skipped MFA enrollment entirely and the console then refused the session with mfa required:

cd /opt/zitadel && sudo docker compose restart zitadel-login

Sign out of the console and sign in again. After the password, Zitadel asks for a second factor:

Choose Authenticator App, scan the QR code, and enter the code. Every user gets the same prompt at their next login.

A locked account stays locked until an admin unlocks it, so lockout also lets anyone who knows a login name lock that user out. That is the price of stopping password guessing. In the console, unlock a user from the page of that user. From the shell, set LOCKED_USER to the login name of the locked account:

LOCKED_USER="[email protected]"
USER_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d "{\"queries\":[{\"userNameQuery\":{\"userName\":\"${LOCKED_USER}\"}}]}" | jq -r '.result[0].userId')
curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users/${USER_ID}/unlock -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d '{}' | jq -c .

If the admin loses the authenticator

This is the recovery path. It removes the TOTP factor from the human admin, and the next login asks them to enroll a new one. We tested it on the lab admin:

ADMIN_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d "{\"queries\":[{\"userNameQuery\":{\"userName\":\"${ADMIN_EMAIL}\"}}]}" | jq -r '.result[0].userId')
curl -s -X DELETE https://${ZITADEL_DOMAIN}/v2/users/${ADMIN_ID}/totp -H "Authorization: Bearer ${ZITADEL_PAT}" | jq -c .
curl -s https://${ZITADEL_DOMAIN}/v2/users/${ADMIN_ID}/authentication_methods -H "Authorization: Bearer ${ZITADEL_PAT}" | jq -c .authMethodTypes

Only the password is left:

["AUTHENTICATION_METHOD_TYPE_PASSWORD"]

Renew the admin-sa token before it expires

The admin-sa token was created with a one year lifetime, and every recovery step above depends on it. Put a reminder in your calendar for a month before ADMIN_SA_PAT_EXPIRATION in .env, then mint a fresh token with the current one, swap the file and load it:

SA_ID=$(curl -s https://${ZITADEL_DOMAIN}/auth/v1/users/me -H "Authorization: Bearer ${ZITADEL_PAT}" | jq -r .user.id)
curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/users/${SA_ID}/pats -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d "{\"expirationDate\":\"$(date -u -d '+1 year' +%Y-%m-%dT00:00:00Z)\"}" | jq -r .token | sudo tee /opt/zitadel/secrets/admin-sa.pat.new >/dev/null
sudo test -s /opt/zitadel/secrets/admin-sa.pat.new && sudo mv /opt/zitadel/secrets/admin-sa.pat.new /opt/zitadel/secrets/admin-sa.pat && sudo chmod 600 /opt/zitadel/secrets/admin-sa.pat
export ZITADEL_PAT=$(sudo cat /opt/zitadel/secrets/admin-sa.pat)
sudo sed -i "s|^ADMIN_SA_PAT_EXPIRATION=.*|ADMIN_SA_PAT_EXPIRATION=$(date -u -d '+1 year' +%Y-%m-%dT00:00:00Z)|" /opt/zitadel/.env
curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/users/${SA_ID}/pats/_search -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d '{}' | jq -r '.result[] | .id + " expires " + .expirationDate'

The last call runs with the new token, so a listing proves it works. Both tokens show until the old one expires:

393814567060897795 expires 2027-10-05T00:00:00Z
393817037673070595 expires 2027-10-05T00:00:00Z

Delete the old one in the console (Users, Service Users, admin-sa, Personal Access Tokens) once every script uses the new file, and take a backup so the archive holds the new token too.

Add a user and prove an app can sign in

A login page that works is not the same as an application that gets tokens. This step registers a real OIDC client and signs a user in through it. It uses the OAuth device authorization grant, the flow a CLI tool or a TV app uses, because it needs no second hostname and no web app: you start it with curl on the server and approve it in a browser.

Create a regular user first. The admin account is for administration, so the test runs as alice. Find the ID of your organization, then create her with a password you choose at the prompt:

ORG_ID=$(curl -s https://${ZITADEL_DOMAIN}/auth/v1/users/me -H "Authorization: Bearer ${ZITADEL_PAT}" | jq -r .user.details.resourceOwner)
read -rs -p "Password for alice: " ALICE_PASS; echo
jq -n --arg org "$ORG_ID" --arg pass "$ALICE_PASS" '{organization: {orgId: $org}, username: "[email protected]", profile: {givenName: "Alice", familyName: "Ops"}, email: {email: "[email protected]", isVerified: true}, password: {password: $pass, changeRequired: false}}' \
  | curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users/human -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d @- | jq -c .
unset ALICE_PASS

The response carries her user ID:

{"userId":"393810228304412675","details":{"sequence":"2","changeDate":"2026-10-05T18:33:08.705174Z","resourceOwner":"393809981629071363"}}

isVerified: true marks the email as checked, because no mail server exists yet to send a verification link. Set changeRequired to true when you hand out temporary passwords.

Applications live in projects. Create a project and a native app that may use the device code grant. A native app has no client secret, which is correct for software running on the machine of a user:

PROJECT_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/projects -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"name":"Internal tools"}' | jq -r .id)
CLIENT_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/projects/${PROJECT_ID}/apps/oidc -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"name":"cli-device","responseTypes":["OIDC_RESPONSE_TYPE_CODE"],"grantTypes":["OIDC_GRANT_TYPE_DEVICE_CODE","OIDC_GRANT_TYPE_REFRESH_TOKEN"],"appType":"OIDC_APP_TYPE_NATIVE","authMethodType":"OIDC_AUTH_METHOD_TYPE_NONE","accessTokenType":"OIDC_TOKEN_TYPE_JWT"}' | jq -r .clientId)
echo "client_id ${CLIENT_ID}"

Start the device flow the way a CLI would. The response holds a short user code and a secret device code:

curl -s -X POST https://${ZITADEL_DOMAIN}/oauth/v2/device_authorization -d client_id=${CLIENT_ID} \
  -d scope="openid profile email offline_access" | tee /tmp/device.json | jq '{user_code, verification_uri, expires_in}'

You have five minutes from here:

{
  "user_code": "NSMW-NNXK",
  "verification_uri": "https://zitadel.example.com/device",
  "expires_in": 300
}

Do not open that verification_uri. Zitadel still points it at the old login UI, which works but looks and behaves differently from everything else your users see. The new login UI has its own device page. Print its address with the code filled in, and open it in a private browser window:

echo "https://${ZITADEL_DOMAIN}/ui/v2/login/device?user_code=$(jq -r .user_code /tmp/device.json)"

Click Continue on the code page. The consent page comes next, before any sign in:

The two buttons show raw translation keys on this release. The blue device.device.request.submit button is Allow. Click it, sign in as alice, and because MFA is now forced, enroll her authenticator. The page ends with “You are signed in” and tells you to return to the device. Back in the shell, collect the tokens and ask Zitadel who they belong to:

curl -s -X POST https://${ZITADEL_DOMAIN}/oauth/v2/token -d grant_type=urn:ietf:params:oauth:grant-type:device_code \
  -d client_id=${CLIENT_ID} -d device_code=$(jq -r .device_code /tmp/device.json) > /tmp/tokens.json
jq -c '{token_type, expires_in, refresh: (.refresh_token != null), id_token: (.id_token != null)}' /tmp/tokens.json
curl -s https://${ZITADEL_DOMAIN}/oidc/v1/userinfo -H "Authorization: Bearer $(jq -r .access_token /tmp/tokens.json)" | jq .
rm -f /tmp/device.json /tmp/tokens.json

An access token valid for 12 hours, a refresh token for offline_access, an ID token, and the profile of alice:

{"token_type":"Bearer","expires_in":43199,"refresh":true,"id_token":true}
{
  "sub": "393810228304412675",
  "name": "Alice Ops",
  "given_name": "Alice",
  "family_name": "Ops",
  "locale": null,
  "updated_at": 1791225294,
  "preferred_username": "[email protected]",
  "email": "[email protected]",
  "email_verified": true
}

If you run the token request before approving, it answers authorization_pending, which is the normal answer while the user has not finished. That is the core setup done: Zitadel runs behind Nginx with a real certificate, the defaults are locked down, MFA is enforced, and an OIDC client has signed in a user. The next three sections are optional and independent of each other.

Optional: email and password reset

Without a mail relay, Zitadel works fine. Users sign in with a password plus MFA, and an admin sets a new password for them in the console (Users, then the user, then the password option) when someone forgets theirs. Self-service password reset and email verification need SMTP. Skip this section until you have a relay.

You need the relay hostname with its port, a username and password, and a sender address the relay accepts. Your mail provider, Amazon SES, Mailgun or a company relay all work. Set the three values that are not secret:

export SMTP_HOST="smtp.example.com:587"
export SMTP_USER="[email protected]"
export SMTP_FROM="[email protected]"

Add the relay as an email provider and activate it. The password goes from a prompt straight into the request:

read -rs -p "SMTP password: " SMTP_PASS; echo
SMTP_ID=$(jq -n --arg host "$SMTP_HOST" --arg user "$SMTP_USER" --arg from "$SMTP_FROM" --arg pass "$SMTP_PASS" \
  '{senderAddress: $from, senderName: "Example SSO", tls: true, host: $host, user: $user, plain: {password: $pass}}' \
  | curl -s -X POST https://${ZITADEL_DOMAIN}/admin/v1/email/smtp -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d @- | jq -r .id)
unset SMTP_PASS
curl -s -X POST https://${ZITADEL_DOMAIN}/admin/v1/email/${SMTP_ID}/_activate -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d '{}' | jq -c .
curl -s -X POST https://${ZITADEL_DOMAIN}/admin/v1/email/_search -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d '{}' \
  | jq -c '.result[] | {state, host: .smtp.host, tls: .smtp.tls, from: .smtp.senderAddress}'

The provider shows as active:

{"state":"EMAIL_PROVIDER_ACTIVE","host":"smtp.example.com:587","tls":true,"from":"[email protected]"}

tls: true works with a STARTTLS relay on 587. Zitadel tries implicit TLS first, logs could not connect using normal tls. trying starttls instead... as a warning, and then upgrades the connection. The console has a test button for this and the API has a test call, but on this release the API answered method TestEmailProviderSMTPById not implemented. A real password reset mail is the better test anyway. Turn the reset link back on, restart the login UI so it picks up the policy, and send alice a reset mail:

curl -s https://${ZITADEL_DOMAIN}/admin/v1/policies/login -H "Authorization: Bearer ${ZITADEL_PAT}" \
  | jq '.policy | del(.details, .isDefault, .secondFactors, .multiFactors, .idps) | .hidePasswordReset=false' \
  | curl -s -X PUT https://${ZITADEL_DOMAIN}/admin/v1/policies/login -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d @- | jq -c .
cd /opt/zitadel && sudo docker compose restart zitadel-login
USER_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"queries":[{"userNameQuery":{"userName":"[email protected]"}}]}' | jq -r '.result[0].userId')
curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users/${USER_ID}/password_reset -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d '{"sendLink":{}}' | jq -c .

The mail arrived in the mailbox of alice with the subject “Reset password” from [email protected]. Its link opens /ui/v2/login/password/set with the code filled in, and after setting a new password the login UI still asked for her TOTP code, so a reset does not skip MFA.

Optional: Grafana over OIDC with role mapping

This section connects Grafana and maps Zitadel roles to Grafana roles: admins get Admin, everyone else with access gets Viewer, and users without a Grafana role cannot sign in at all. It runs Grafana on the same host. If your Grafana runs elsewhere, skip the container and Nginx steps and use the environment settings in your existing setup.

Grafana gets its own name, DNS record and certificate. Create the A record first, then request the certificate with the same method as the core, using this name:

export GRAFANA_DOMAIN="grafana.example.com"

Then request the certificate over HTTP-01:

sudo certbot certonly --nginx -d "${GRAFANA_DOMAIN}" --non-interactive --agree-tos -m "${ADMIN_EMAIL}" --deploy-hook "systemctl reload nginx"

If you used DNS-01 for the core certificate, request this one over DNS too:

sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 30 -d "${GRAFANA_DOMAIN}" --non-interactive --agree-tos -m "${ADMIN_EMAIL}" \
  --deploy-hook "systemctl reload nginx"

In Zitadel, roles belong to a project. Create a Grafana project that asserts roles in tokens and refuses users without a role, then add the two roles:

GRAFANA_PROJECT=$(curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/projects -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"name":"Grafana","projectRoleAssertion":true,"projectRoleCheck":true}' | jq -r .id)
curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/projects/${GRAFANA_PROJECT}/roles/_bulk -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"roles":[{"key":"grafana-admin","displayName":"Grafana Admin"},{"key":"grafana-viewer","displayName":"Grafana Viewer"}]}' | jq -c .

Register Grafana as a web app with a client secret. idTokenRoleAssertion puts the roles into the ID token, and Grafana needs that. Without it the roles only appear in the userinfo response, Grafana evaluates the ID token first, the role expression falls through to Viewer there, and every admin lands in Grafana as a Viewer. That is exactly what happened in the lab until we set it:

jq -n --arg d "$GRAFANA_DOMAIN" '{name: "grafana", redirectUris: ["https://\($d)/login/generic_oauth"], postLogoutRedirectUris: ["https://\($d)/login"], responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE"], appType: "OIDC_APP_TYPE_WEB", authMethodType: "OIDC_AUTH_METHOD_TYPE_BASIC", idTokenRoleAssertion: true}' \
  | curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/projects/${GRAFANA_PROJECT}/apps/oidc -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d @- > /tmp/grafana-app.json
jq -r '"client_id " + .clientId' /tmp/grafana-app.json

Give the admin the grafana-admin role and alice grafana-viewer. A role assignment is called a user grant in the API:

for pair in "${ADMIN_EMAIL}:grafana-admin" "[email protected]:grafana-viewer"; do
  U=$(curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
    -d "{\"queries\":[{\"userNameQuery\":{\"userName\":\"${pair%%:*}\"}}]}" | jq -r '.result[0].userId')
  curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/users/${U}/grants -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
    -d "{\"projectId\":\"${GRAFANA_PROJECT}\",\"roleKeys\":[\"${pair##*:}\"]}" | jq -c .
done

Grafana runs from its own directory. Port 3000 is already taken by zitadel-login, so Grafana publishes on 3001. Write its client credentials into a private .env and remove the temporary file:

sudo mkdir -p /opt/grafana && cd /opt/grafana
printf '%s\n' "GF_AUTH_GENERIC_OAUTH_CLIENT_ID=$(jq -r .clientId /tmp/grafana-app.json)" \
  "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=$(jq -r .clientSecret /tmp/grafana-app.json)" | sudo tee .env >/dev/null
sudo chmod 600 .env && rm -f /tmp/grafana-app.json
sudo vim compose.yaml

The endpoints are the fixed Zitadel paths. The role expression reads the project roles claim and falls back to an empty object when a token has no roles:

services:
  grafana:
    image: grafana/grafana:13.2.3 # https://github.com/grafana/grafana/releases
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3000"
    env_file: .env
    environment:
      GF_SERVER_ROOT_URL: https://grafana.example.com/
      GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
      GF_AUTH_GENERIC_OAUTH_NAME: Zitadel
      GF_AUTH_GENERIC_OAUTH_SCOPES: openid profile email
      GF_AUTH_GENERIC_OAUTH_AUTH_URL: https://zitadel.example.com/oauth/v2/authorize
      GF_AUTH_GENERIC_OAUTH_TOKEN_URL: https://zitadel.example.com/oauth/v2/token
      GF_AUTH_GENERIC_OAUTH_API_URL: https://zitadel.example.com/oidc/v1/userinfo
      GF_AUTH_GENERIC_OAUTH_USE_PKCE: "true"
      GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH: preferred_username
      GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: contains(keys("urn:zitadel:iam:org:project:roles" || `{}`), 'grafana-admin') && 'Admin' || 'Viewer'
      GF_AUTH_SIGNOUT_REDIRECT_URL: https://zitadel.example.com/oidc/v1/end_session
      GF_AUTH_OAUTH_AUTO_LOGIN: "true"
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  grafana-data:

Fill in both names and start it:

sudo sed -i -e "s/grafana.example.com/${GRAFANA_DOMAIN}/g" -e "s/zitadel.example.com/${ZITADEL_DOMAIN}/g" compose.yaml
sudo docker compose up -d --wait

The Grafana Nginx site is a plain proxy with WebSocket headers for Grafana Live:

sudo vim /etc/nginx/sites-available/grafana.conf

Paste the server blocks:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name grafana.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name grafana.example.com;

    ssl_certificate     /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_pass http://127.0.0.1:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

The map block at the top defines $connection_upgrade. If another site on this host already defines it, nginx -t complains about a duplicate variable, so delete the four map lines here in that case. On Ubuntu 24.04, move HTTP/2 onto the listen lines first, as in the Zitadel site:

sudo sed -i -e '/^\s*http2 on;/d' -e 's/listen 443 ssl;/listen 443 ssl http2;/' -e 's/listen \[::\]:443 ssl;/listen [::]:443 ssl http2;/' /etc/nginx/sites-available/grafana.conf

Then fill in the name and enable it on either release:

sudo sed -i "s/grafana.example.com/${GRAFANA_DOMAIN}/g" /etc/nginx/sites-available/grafana.conf
sudo ln -sf /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Open https://grafana.example.com. Grafana sends you straight to the Zitadel login, and after password and TOTP you land on the Grafana home page. We checked the role Grafana assigned through its API for each test account:

Account	Zitadel role	Grafana role
[email protected]	grafana-admin	Admin
[email protected]	grafana-viewer	Viewer
[email protected] before the grant	none	Refused at login

The refused login shows only “Unknown error occurred” in the login UI. The real reason is in the API log: Errors.User.GrantRequired, the project role check doing its job. The English message for it is missing on this release, so the UI falls back to the generic text.

Optional: a service account for scripts

admin-sa can do anything, which is right for a break-glass account and wrong for a provisioning script or a CI job. Give those their own machine user with only the role they need. This one may manage users in your organization and nothing else, and it signs in with the client credentials grant instead of a long-lived token file:

BOT_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/users/machine -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"userName":"provisioner","name":"User provisioning bot","accessTokenType":"ACCESS_TOKEN_TYPE_JWT"}' | jq -r .userId)
curl -s -X PUT https://${ZITADEL_DOMAIN}/management/v1/users/${BOT_ID}/secret -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" -d '{}' > /tmp/bot.json
curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/orgs/me/members -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d "{\"userId\":\"${BOT_ID}\",\"roles\":[\"ORG_USER_MANAGER\"]}" | jq -c .
jq . /tmp/bot.json

The last command prints the client ID (provisioner) and the secret. Zitadel shows the secret only this once, so store both in your CI secret store now. A script then gets a token with the client credentials grant. Test it from the same shell, which reads the secret from the file, then delete the file:

BOT_SECRET=$(jq -r .clientSecret /tmp/bot.json)
BOT_TOKEN=$(curl -s -X POST https://${ZITADEL_DOMAIN}/oauth/v2/token -u "provisioner:${BOT_SECRET}" \
  -d grant_type=client_credentials -d scope="openid urn:zitadel:iam:org:project:id:zitadel:aud" | jq -r .access_token)
curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${BOT_TOKEN}" -H "Content-Type: application/json" \
  -d '{"queries":[{"typeQuery":{"type":"TYPE_HUMAN"}}]}' | jq -r '.result[].preferredLoginName'
curl -s -X PUT https://${ZITADEL_DOMAIN}/admin/v1/policies/password/lockout -H "Authorization: Bearer ${BOT_TOKEN}" \
  -H "Content-Type: application/json" -d '{"maxPasswordAttempts":0}' | jq -c .
rm -f /tmp/bot.json; unset BOT_SECRET BOT_TOKEN

The scope with zitadel:aud is what lets the token call the Zitadel API at all. The token lists users, which its role allows, and the attempt to switch off the lockout policy is refused:

[email protected]
[email protected]
{"code":7,"message":"No matching permissions found (AUTH-5mWD2)","details":[{"@type":"type.googleapis.com/zitadel.v1.ErrorDetail","id":"AUTH-5mWD2","message":"No matching permissions found"}]}

Back up and restore

A Zitadel backup is three things: a PostgreSQL dump, .env with the masterkey, and the two token directories. The masterkey is the one people forget. Every client secret, SMTP password and signing key in the database is encrypted with it, so a dump without it restores into an instance that cannot use its own secrets.

Create the backup script:

sudo vim /usr/local/bin/zitadel-backup.sh

The dump runs as the zitadel role inside the PostgreSQL container, so no extra credentials are needed. Backups older than 14 days are deleted:

#!/usr/bin/env bash
set -euo pipefail

STACK=/opt/zitadel
DEST=/var/backups/zitadel
KEEP_DAYS=14
STAMP=$(date +%Y%m%d-%H%M%S)

install -d -m 700 "$DEST"
cd "$STACK"

docker compose exec -T postgres pg_dump -U zitadel -d zitadel -Fc > "$DEST/zitadel-db-$STAMP.dump.tmp"
mv "$DEST/zitadel-db-$STAMP.dump.tmp" "$DEST/zitadel-db-$STAMP.dump"
tar -czf "$DEST/zitadel-files-$STAMP.tar.gz" .env docker-compose.yml docker-compose.override.yml bootstrap secrets
chmod 600 "$DEST"/zitadel-*-"$STAMP".*

find "$DEST" -name 'zitadel-*' -mtime +"$KEEP_DAYS" -delete
echo "backup ok: $(du -ch "$DEST"/zitadel-*-"$STAMP".* | tail -1 | cut -f1) in $DEST"

A systemd timer runs it every night. Create the service unit:

sudo vim /etc/systemd/system/zitadel-backup.service

Add the unit:

[Unit]
Description=Zitadel database and file backup
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/zitadel-backup.sh

Then the timer:

sudo vim /etc/systemd/system/zitadel-backup.timer

It fires at 02:30 with up to 15 minutes of random delay, and Persistent=true catches up on a run the host missed while it was off:

[Unit]
Description=Nightly Zitadel backup

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

Make the script executable, enable the timer and take the first backup now:

sudo chmod 750 /usr/local/bin/zitadel-backup.sh
sudo systemctl daemon-reload
sudo systemctl enable --now zitadel-backup.timer
sudo systemctl start zitadel-backup.service
sudo ls -l /var/backups/zitadel

A fresh install with a few users makes a small backup:

-rw------- 1 root root 394040 Oct  5 19:00 zitadel-db-20261005-190024.dump
-rw------- 1 root root   4691 Oct  5 19:00 zitadel-files-20261005-190024.tar.gz

Copy these files off the host. A backup on the same disk as the database does not survive the disk.

Restore drill

We restored from these files after deleting every Docker volume. That covers a broken database or a botched upgrade. If the whole host is gone, first install Docker, Nginx and the certificate as in the core setup, create /opt/zitadel, copy the backup files into /var/backups/zitadel, then run the first two lines of the block below, extract the file archive with sudo tar -xzf "$FILES" -C /opt/zitadel and start the database with sudo docker compose up -d --wait postgres, so the readability check has a PostgreSQL to talk to. The order of the steps below matters, and one mistake in it is silent. If the database restore fails and you start the full stack anyway, Zitadel finds an empty database, creates a brand new instance with a new admin and a new admin-sa token, and overwrites your token file. Nothing looks broken until you notice your users are gone. So the API starts only after the restore is checked.

Pick the newest backup and prove the dump is readable while the old database still exists. pg_restore -l only lists the archive and changes nothing:

cd /opt/zitadel
FILES=$(sudo find /var/backups/zitadel -name 'zitadel-files-*.tar.gz' | sort | tail -1)
DUMP=$(sudo find /var/backups/zitadel -name 'zitadel-db-*.dump' | sort | tail -1)
echo "restoring ${FILES} and ${DUMP}"
sudo test -s "$FILES" && sudo cat "$DUMP" | sudo docker compose exec -T postgres pg_restore -l >/dev/null && echo "backup readable"

Continue only when it prints backup readable. The next block deletes the live database. Then restore the files, start PostgreSQL alone and recreate the role:

sudo docker compose down -v
sudo tar -xzf "$FILES" -C /opt/zitadel
sudo docker compose up -d --wait postgres
PG_ZITADEL_PASS=$(sudo grep -oP '^POSTGRES_ZITADEL_PASSWORD=\K.*' .env)
echo "CREATE ROLE zitadel LOGIN PASSWORD '${PG_ZITADEL_PASS}'; ALTER DATABASE zitadel OWNER TO zitadel; ALTER SCHEMA public OWNER TO zitadel;" \
  | sudo docker compose exec -T postgres psql -U postgres -d zitadel -v ON_ERROR_STOP=1
unset PG_ZITADEL_PASS

find runs under sudo because the backup directory is readable only by root. A plain shell glob there expands as your own user, matches nothing, and leaves DUMP empty, which is how our first drill restored nothing. Load the dump and count the events before starting anything else:

sudo cat "$DUMP" | sudo docker compose exec -T postgres pg_restore -U zitadel -d zitadel --no-owner --exit-on-error && echo "pg_restore ok"
sudo docker compose exec -T postgres psql -U zitadel -d zitadel -tAc "select count(*) from eventstore.events2"

A count in the hundreds or more means the data is there:

pg_restore ok
1309

Only now start the stack and check that the users came back:

sudo docker compose up -d --wait
curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"queries":[{"typeQuery":{"type":"TYPE_HUMAN"}}]}' | jq -r '.result[] | .preferredLoginName + " " + .state'

Both accounts are back and active:

[email protected] USER_STATE_ACTIVE
[email protected] USER_STATE_ACTIVE

The admin signed in with the TOTP enrolled before the backup, and Grafana still logged in as Admin, which proves the encrypted client secret survived with the masterkey.

Upgrade Zitadel

Upgrades are a version bump in .env. Each start of zitadel-api runs the database migrations for the new release before it serves traffic, and both images move together because the login image uses the same tag. Take a backup first, then pull and restart:

cd /opt/zitadel
sudo /usr/local/bin/zitadel-backup.sh
ZITADEL_VERSION=$(curl -fsSL https://api.github.com/repos/zitadel/zitadel/releases/latest | jq -r .tag_name)
sudo sed -i "s|^ZITADEL_VERSION=.*|ZITADEL_VERSION=${ZITADEL_VERSION}|" .env
sudo docker compose pull
sudo docker compose up -d --wait
sudo docker compose ps --format "{{.Service}} {{.Image}} {{.Status}}"

Both Zitadel images now run the new tag:

postgres postgres:17.10-alpine Up About a minute (healthy)
zitadel-api ghcr.io/zitadel/zitadel:v4.19.4 Up 11 seconds (healthy)
zitadel-login ghcr.io/zitadel/zitadel-login:v4.19.4 Up 5 seconds (healthy)

We started the replay VM one minor release behind on purpose and upgraded it with exactly these commands. A probe calling /debug/ready through Nginx every half second could not reach the API for about 3 seconds while the container was replaced, and the new release ran its migrations inside that window. Users, TOTP enrollments and the Grafana login all worked afterwards. A release with heavier migrations takes longer, so upgrade in a quiet hour.

Read the release notes between your version and the new one before you pull. The Zitadel releases page lists breaking changes per release, and the compose file on the main branch changes too. Compare it with your copy once in a while with curl -fsSL https://raw.githubusercontent.com/zitadel/zitadel/main/deploy/compose/docker-compose.yml | diff docker-compose.yml -, because a new environment variable there does nothing until your copy has it.

Errors we hit, and the fix for each

“password authentication failed for user “postgres” (SQLSTATE 28P01)”

This is the first one most people see. Following the official hardening steps, the API crash-looped with:

Error: initialize database failed: failed to connect to `user=postgres database=zitadel`: 172.18.0.2:5432 (postgres): failed SASL auth: FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01)

The steps change POSTGRES_ADMIN_PASSWORD, which PostgreSQL uses, but Zitadel connects with ZITADEL_DATABASE_POSTGRES_DSN, which still holds the old password. The POSTGRES_ZITADEL_PASSWORD the same steps add is not read by the compose file at all. Put the password into the DSN, as the secrets section above does.

“Instance not found. Make sure you got the domain right.”

Zitadel picks the instance from the host name of each request. Any other name, including the server IP, gets this answer:

unable to set instance using origin &{10.0.1.50  https} (ExternalDomain is zitadel.example.com): ID=QUERY-1kIjX Message=Instance not found. Make sure you got the domain right. Check out https://zitadel.com/docs/apis/introduction#domains

Use the configured name. If it happens on the configured name, Nginx is not passing the Host header, or ZITADEL_DOMAIN, ZITADEL_EXTERNALPORT and ZITADEL_EXTERNALSECURE do not match what the browser uses.

“[permission_denied] mfa required (AUTHZ-Kl3p0)”

The console showed this, followed by “Your authorization token has expired” with a Login button that went in a circle. It happened right after forcing MFA for an admin who had no second factor. The login UI was still serving its cached policy without forced MFA, issued a session without one, and the console rejected it. Restart zitadel-login after every login policy change.

“Unknown error occurred” after entering the TOTP code

The login UI shows this for an error it has no English text for. Look in the API log, sudo docker compose logs zitadel-api | grep -i warn. In our case it was Errors.User.GrantRequired: the user had no role in a project with the role check on.

zitadel-login stays unhealthy: “Awaiting file and reading token”

After the first restore drill, the login container never became healthy and its log stopped at:

ZITADEL_SERVICE_USER_TOKEN_FILE=/zitadel/bootstrap/login-client.pat is set. Awaiting file and reading token.

Only the first start writes that token, and the official pack keeps it in a Docker volume that docker compose down -v deletes and no file backup sees. The override in this guide keeps it in /opt/zitadel/bootstrap, which the backup includes. If you lost it anyway, mint a new one for the login-client machine user with the admin-sa token and restart:

LC_ID=$(curl -s -X POST https://${ZITADEL_DOMAIN}/v2/users -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d '{"queries":[{"userNameQuery":{"userName":"login-client"}}]}' | jq -r '.result[0].userId')
curl -s -X POST https://${ZITADEL_DOMAIN}/management/v1/users/${LC_ID}/pats -H "Authorization: Bearer ${ZITADEL_PAT}" -H "Content-Type: application/json" \
  -d "{\"expirationDate\":\"$(date -u -d '+1 year' +%Y-%m-%dT00:00:00Z)\"}" | jq -r .token | sudo tee /opt/zitadel/bootstrap/login-client.pat >/dev/null
cd /opt/zitadel && sudo docker compose up -d --wait

The API keeps working while the login UI is down, so the admin-sa token still gets through even though nobody can sign in to the console. That is one more reason to keep that token in your backup and your password manager.