Install an edge node

This guide takes one box from a stock nginx (or Angie) package to a managed edge node serving your zones with issued certificates and per-request decisions. Read the concepts page first if the words zone, decision service or fast/slow split are new.

You need: a brain (the kapkan daemon) reachable from the box over HTTPS, an agent token on it, a Linux box with nginx ≥ 1.22 or Angie installed, TCP 80 and 443 open to the world — and UDP 443 too, on the box, the security group and the upstream ACL, if any zone will speak HTTP/3 — and DNS for the zones pointing at the box (or at the anycast/VIP the boxes share).

1. On the brain: zones and nodes

Write the zones file — it is tenant data, kept separate from kapkan.yaml — and point the brain at it:

# /etc/kapkan/zones.yaml
zones:
  - name: shop.example.com
    origins: ["10.0.0.10:8080"]
    policy: { mode: decide, failure_mode: open, rate: { rps: 50, concurrency: 20 } }
  - name: static.example.com
    origins: ["10.0.0.20:80"]
    policy: { mode: none }             # proxy only, no decisions

# /etc/kapkan/config.yaml (the brain)
edge:
  zones_file: /etc/kapkan/zones.yaml   # absolute; the whole reload fails on a broken file
  nodes:
    - name: edge-1                     # must equal the node's controller.name
    - name: edge-2
  stale_after_seconds: 15              # a node unheard-from this long counts as lost
api:
  tokens:
    - { name: edge-1, token_env: KAPKAN_EDGE_1_TOKEN, role: agent, node: edge-1 }
    - { name: edge-2, token_env: KAPKAN_EDGE_2_TOKEN, role: agent, node: edge-2 }

Scope zones to nodes when the fleet has PoPs: give each node the hostgroups whose zones it serves (edge.nodes[].hostgroups: [edge-us], with the literal global for the fleet-wide zones) and each zone its hostgroup; a node without a scope serves the global zones only, a zone without a hostgroup is global. The rule, the global literal and the runbook are in Placing zones on nodes; the duplicate-certificate ceiling is counted per placement, not per fleet (see Limits). Scoping any node requires every agent token to be bound, as above.

kapkan -check-config /etc/kapkan/config.yaml validates both files (it follows edge.zones_file), and kapkan -dump-zones-schema prints the zones file's JSON schema for your editor. Give each node its own agent token, bound to it with node: a bound token is refused as any other node before the brain does anything with the name, so a token stolen from one box can no longer speak for any other node — its presence, reports and ACME slots stay its own — and only that one token needs rotating. A zones file that fails to read or validate on a reload keeps the previous zones live — the brain never serves a half-broken document. Every key is in the zones reference.

iThe agent token is a certificate-issuing credential — bind it

An edge node authenticates with an agent token (see authentication). Besides polling the document, it uses that token to coordinate certificate issuance through the brain — so an unbound token can act as any node, and rotating it on a compromise means rotating the whole fleet's. Bind each token to its node (api.tokens[].node); until every agent token is bound, -check-config, the daemon's log and the inventory's unbound_agent_tokens say so. What binding buys is impersonation and the rotation radius; what limits the set of zones a stolen bound token can issue for is the node's placement scope: acting as its own node it can take the issuance slot and publish an ACME key authorization only for the zones that scope covers (every zone on a fleet without scopes) — treat a node compromise as a certificate exposure for its node's zones. Migrating a fleet from one shared token: add a bound token per node with its own secret (a value shared with the unbound entry is ambiguous and refused with 401 for every node presenting it), switch each node's environment file and watch the inventory's last_token flip, then remove the shared entry last — an unbound token keeps working meanwhile. Keep the token in the node's environment file, never in edge.yaml.

2. On the box: nginx or Angie

Kapkan owns one include directory and drives nginx -t and reloads; it does not ship or supervise nginx. Make your main configuration include the live generation once, inside http {}:

http {
    # ... your defaults ...
    include /var/lib/kapkan-edge/conf/live/*.conf;
}

Remove (or plan to keep, see below) any default_server on :80/:443 — Kapkan renders its own catch-all that refuses unknown hosts. Debian's default site declares one; either disable the site or set omit_catch_all: true in edge.yaml. Then confirm the include compiles before anything is rendered (a wildcard include of a directory that does not exist yet is fine for nginx):

nginx -t

live itself is a symlink the node creates and retargets on every generation — do not create it by hand: a directory in its place stops every install (apply: rename … live in the journal) until it is removed.

Find the worker's group — nginx for nginx.org packages and the Docker image, www-data for Debian's own package, angie for Angie. The decision and log sockets are created 0660 for that group.

For HTTP/3, the build must carry the module and the address must accept UDP:

nginx -V 2>&1 | grep -o -- --with-http_v3_module     # Angie: angie -V

Debian 12's nginx 1.22 has none (zones asking for h3 are served over TCP there and the node says so); Debian 13's nginx 1.26.3, nginx.org's packages from 1.25 and Angie do. Open UDP 443 wherever TCP 443 is open, give UDP flows a conntrack timeout on any stateful firewall in the path, and keep the path MTU at 1280 bytes or more — QUIC never fragments, so a GRE/IPIP diversion with a smaller MTU breaks h3 while TCP survives through MSS clamping. If your nginx.conf sets ssl_protocols at the http level, it must include TLSv1.3 — QUIC speaks nothing else (Kapkan's own QUIC listeners say so themselves). kapkan edge -check prints the node's HTTP/3 readiness once edge.yaml is in place.

3. The node: `edge.yaml`, the token, the unit

Install the same kapkan binary as on the brain, then the node's own configuration:

# /etc/kapkan/edge.yaml
dry_run: true                          # watch-only until the counts look right
controller:
  url: https://kapkan.example.net:8443 # the brain's API base, https outside a lab
  token_env: KAPKAN_EDGE_TOKEN         # the agent token's environment variable
  name: edge-1                         # must equal an edge.nodes[] entry on the brain
  report_interval_seconds: 10
state_dir: /var/lib/kapkan-edge        # cached document, generations, keys, certificates
sockets_dir: /run/kapkan-edge          # the four unix sockets nginx talks to
socket_group: nginx                    # the terminator's worker group
terminator:
  binary: nginx                        # or angie
  main_conf: /etc/nginx/nginx.conf     # passed as -c to `nginx -t` and `-s reload`
  reload: exec                         # exec | signal | command
  pid_file: /run/nginx.pid             # liveness check; required for reload: signal
acme:
  directory: ""                        # Let's Encrypt production when empty
  fallback: ""                         # optional second CA after repeated failures
  contact: ["mailto:ops@example.net"]
status_listen: 127.0.0.1:9102          # /healthz + /metrics; keep it private

install -m 600 /dev/null /etc/kapkan/edge.env
echo 'KAPKAN_EDGE_TOKEN=...the agent token...' > /etc/kapkan/edge.env
kapkan edge -config /etc/kapkan/edge.yaml -check

-check validates the file and what it names on this box: the socket group must exist and any External Account Binding key must be well-formed (problems), while an absent token, binary or main_conf is a warning — the check may run before the environment file or the terminator is in place. When the binary is on PATH it also prints what nginx -V says — kind, version, nginx core, TLS library, whether the build has the HTTP/3 module — and the node's HTTP/3 readiness; a build whose nginx core falls in a published QUIC advisory's range gets a warning naming the CVE (check your distribution's changelog: backports do not move the version). Set quic.h3: off in edge.yaml to keep HTTP/3 off on that node; -check then states the readiness as node_off and stops repeating the advisory, while the node's report keeps carrying it as a fact about the build.

Install the shipped unit, deploy/kapkan-edge.service, and start it:

install -m 644 deploy/kapkan-edge.service /etc/systemd/system/
systemctl daemon-reload && systemctl enable --now kapkan-edge
journalctl -u kapkan-edge -f

The unit runs as root, like certbot and every configuration manager that drives nginx: nginx -t opens nginx's own log and pid paths and nginx -s reload signals a root master. It is hardened with ProtectSystem=strict and explicit write paths (/var/lib/kapkan-edge, /run, and the nginx or Angie log directory — paths prefixed - are skipped when absent, so one unit serves either terminator), and its state directory is 0711 so the worker can traverse to the empty try_files root while the ACME keys stay in 0700 directories of their own. The role's directories are its own — never the brain's /var/lib/kapkan and /run/kapkan — so a brain and an edge may share a host.

!Startup order does not matter, but the sockets do

The node brings its four unix sockets up first, so nginx may already be running and probing them; then it probes the terminator, recovers a generation a crashed predecessor left untested, renders the last document it had on disk, and only then polls the brain. A component that cannot start — a socket another process already serves, a group that does not exist — ends the process with that error and systemd restarts it.

4. Check

journalctl -u kapkan-edge shows kapkan edge running, then configuration installed with the generation number and the zone count.
curl -s 127.0.0.1:9102/healthz answers 200 with "healthy":true, "converged":true and the rendered ETag; ls /var/lib/kapkan-edge/conf/live/ lists kapkan_00_common.conf and one kapkan_zone_<name>.conf per zone.
The brain's inventory, GET /api/v1/edge/nodes, lists the node alive with its report, and the console's Edge view appears in the navigation once a node is configured.
Certificates: the first order needs port 80 reachable from the CA for every zone name. Watch certificate issued in the journal, then ls /var/lib/kapkan-edge/certs/<zone>/current/. Until a zone has a certificate its :80 listener answers 503 for everything but ACME challenges — Kapkan never proxies a zone over cleartext.
Requests: with dry_run: true a client over the zone's rate still reaches the origin, with X-Kapkan-Mark: would-deny:rate. Watch kapkan_edge_decisions_total{result="would_deny"} on /metrics and your origin's logs.
The rung: give a zone policy.challenge: auto (its challenge_options.dry_run is true by default, so nobody is challenged yet) and read who would be challenged — GET /api/v1/edge/zones/status on the brain, or the console's Edge view — while a would-be challenge reaches the origin marked X-Kapkan-Mark: would-challenge:table:flood. Put your health checks, API clients and webhooks under challenge_options.exempt_paths before the rung goes live.
HTTP/3: the inventory's terminator.h3.state is ready on the nodes you expect; with a zone on tls.h3: true, terminator.h3.serving names it and listening is true, and curl --http3-only https://<zone>/ from outside answers 200 — then kapkan_edge_requests_total{protocol="h3"} starts climbing once advertise is on. A node in no_module, node_off or unknown lists the zone under unsupported and serves it over TCP.

5. Go live

Set dry_run: false in edge.yaml and restart the unit. From then on a client over its ceiling is answered 429 with Retry-After: 1, and a source the rollups promoted to a deny gets 403. Tighten a zone's policy.rate on the brain when under attack — that change reaches the decision service on the next poll and never reloads nginx.

The rung goes live per zone, in the zones file: challenge_options.dry_run: false once the would-be set names the right clients. From then on a flooder in an auto zone is served the clearance page before it is blocked, and a zone under attack can be put under challenge at once — policy.challenge: manual in the file, or, for a bounded time and without an edit, the lever:

curl -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  https://kapkan.example.net:8443/api/v1/edge/zones/shop.example.com/challenge \
  -d '{"mode":"manual","ttl_seconds":1800,"reason":"login flood"}'

To stage zones one at a time, policy.dry_run: true keeps a zone watch-only on every node while its siblings enforce.

Several nodes and a shared address

Point every node's controller.name at its own edge.nodes[] entry and put the same zones in front of all of them — every node behind one shared address must serve the zone, that is, list its hostgroup in edge.nodes[].hostgroups (or leave the zone global on unscoped nodes). Under anycast or a shared VIP the CA's HTTP-01 validation may land on any node behind that address: the ordering node publishes its challenge answer to the brain, which fans it out in the document of every node that serves the zone, and each of them answers it; a node the zone is not placed on refuses the Host and cannot answer. Issuance is serialised per zone through the brain's slot so a fleet does not burn a CA's duplicate-certificate ceiling in one afternoon, and renewal is jittered per node. All of it is advisory: with the brain gone each node still renews on its own.

Troubleshooting

Symptom	Cause and fix
`nginx -t` fails with a duplicate default server	Your `nginx.conf` already declares a `default_server` on `:80` or `:443`. Set `omit_catch_all: true` or remove yours.
`[crit] stat() ... failed` in nginx's error log on every request	The worker cannot traverse to `state_dir/empty`. The unit's `StateDirectoryMode=0711` does that; a hand-made `state_dir` needs `chmod 711`.
`connect() to unix:/run/kapkan-edge/edge-decide.sock failed (13: Permission denied)`	`socket_group` is not the worker's group. Fix it and restart the unit.
`healthy: false`, `generation: 0`	No tested generation of the node's own is live: the very first candidate failed `nginx -t` (read `test_error`), the renderer refused the document (read `last_error`), a recovered generation failed — or the node has never received a document at all (`brain_seen` absent; read the journal's poll errors: `controller.url`, the token, `controller.name`).
`apply: rename … live: file exists` (or `is a directory`) on every apply	A directory sits where the node's `live` symlink goes — created by hand, or by an old install step. Move its contents away and `rmdir` it; the next attempt installs.
`converged: false` with `test_error`	The latest document was refused; the previous one serves. Fix the cause (an origin that does not resolve, a directive in `extra_directives_file`) and the node retries on its own.
Zone stays on `:80` with `503`	No certificate yet. Check port 80 reachability from the internet for that name, `kapkan_edge_acme_attempts_total`, and the journal's `certificate order failed` line.
A zone you moved between hostgroups is unreachable, or never gets a certificate	Its address still points at a node the new placement does not cover. That node no longer serves the name, so it closes the connection instead of answering — including on the challenge path, which is why the order fails with a connection error rather than a `404`. Move the DNS record (or the anycast prefix) to the nodes that now list the zone's hostgroup; `GET /api/v1/edge/zones/status` names them under `placement`.
Node shown lost on the brain while running	The poll is the liveness signal and a poll parked in the brain's hold counts as alive, so a node shown lost has really stopped reaching the brain. Check `controller.url`, the token (`401`/`403` in the journal), that `controller.name` is declared under `edge.nodes[]` (`404`), and that the node is not sharing a token with more than four others (`429`).
`would-challenge` marks, but nobody is ever challenged	One of the three watch-only layers is on: the node's `dry_run` (`edge.yaml`), the zone's `policy.dry_run`, or the rung's own `challenge_options.dry_run` — `true` by default. `GET /api/v1/edge/zones/status` names the nodes a zone is watch-only on and whether a zone-wide challenge is previewing (`dry_run: true`).
Every request to a zone gets the clearance page	The zone is under challenge: `policy.challenge: manual` in the file, a lever (`override` in `GET /api/v1/edge/zones/status`), or the zone-wide trigger (`challenge_active` with `reason: zone-rps`). A client that clears it is not asked again for `cookie_ttl_seconds`; clients that cannot keep cookies see it every time.
Clients see `503` (or pass unchallenged) on a zone whose rung is on	nginx could not reach the clearance page and followed `failure_mode`. Look for `connect() to unix:/run/kapkan-edge/edge-clearance.sock failed` in nginx's error log: the socket is `0660` for `socket_group`, like the decision socket, and the node logs `clearance page listening` when it is up.
`nginx -t` fails with duplicate listen options for 0.0.0.0:443	Your `nginx.conf` (with `omit_catch_all: true`) already has a `listen 443 quic` with socket options (`reuseport`, `rcvbuf`, …) and Kapkan's QUIC anchor carries its own `reuseport`. Set `quic.omit_anchor: true` and give your server `reuseport` and `ssl_protocols TLSv1.3` yourself.
A zone with `tls.h3: true` is served over TCP only on one node	That node's `terminator.h3.state` is not `ready`: `no_module` (the build lacks `--with-http_v3_module`), `node_off` (`quic.h3: off`) or `unknown` (the probe failed — read the journal). The zone is listed under `terminator.h3.unsupported`; the other nodes serve it over h3.
Browsers are slow to load a zone after `h3: true`	UDP 443 is blocked upstream while `Alt-Svc` announces h3: every browser races the QUIC attempt and times out before falling back. Fix the firewall, or set `tls.h3_options.advertise: false` until you have; `terminator.h3.listening` only tells you the box's half.
`curl --http3-only` fails while TCP works, with a tunnel or an overlay in the path	Path MTU under 1280 bytes: QUIC never fragments. Raise the MTU or keep the zone on TCP (MSS clamping saves TCP, not QUIC).
Every QUIC handshake fails while `nginx -t` passes and `Alt-Svc` is sent	An http-level `ssl_protocols` in your `nginx.conf` without `TLSv1.3`, under `omit_catch_all` with `quic.omit_anchor: true` (Kapkan's own catch-all and anchor pin `TLSv1.3`; your default server must too). Look for `SSL_do_handshake() failed … unsupported protocol` in nginx's error log.

Edge nodes — how it works and what fails how
Zones reference
kapkan edge — flags and every edge.yaml key
The edge channel

Install an edge node#

1. On the brain: zones and nodes#

2. On the box: nginx or Angie#

3. The node: edge.yaml, the token, the unit#

4. Check#

5. Go live#

Several nodes and a shared address#

Troubleshooting#

Related#