GitHub

Troubleshooting

Find your symptom below. Each entry says how to confirm the cause and what to change. If you are mid-attack, start with the Under attack now runbook instead.

I don't see any attacks

GET /api/v1/attacks is empty even though traffic is clearly flooding in. Work down this list:

  • Are flows arriving at all? Check GET /api/v1/traffic and the kapkan_ingest_flows_total metric (curl -s localhost:8080/metrics | grep kapkan_ingest_flows). If it is not climbing, no telemetry is reaching Kapkan.
  • Are your exporters pointed at the right ports? sFlow → UDP :6343, NetFlow/IPFIX → UDP :2055. Confirm the router is exporting to Kapkan's address and that no firewall drops the UDP.
  • Is the sampling rate right? If the exporter does not report its rate, Kapkan uses sampling.default_rate. A wrong rate makes real traffic look tiny. See Configuration and How it works.
  • Is the target inside your networks? Kapkan only raises attacks for destinations inside your configured prefixes — traffic to anything else is counted but never alerted. See the Safety model.
  • Are your thresholds too high? Compare the rates in GET /api/v1/traffic against your configured thresholds; lower them or use baselines if real attacks sit below the limit.

Detection fires but nothing is announced

You see attacks in the API and logs, but no routes reach your routers. This is almost always dry-run — Kapkan's default:

curl -s localhost:8080/api/v1/status | jq '.dry_run'

If this is true, Kapkan is working exactly as designed: it detects and records but announces nothing. To start announcing, follow Going live.

I can't find the dry-run line in the logs

Dry-run mitigations are logged at warn level with this exact message:

DRY-RUN: would announce mitigation (not sent)

Grep for that string (journalctl -u kapkan | grep "would announce mitigation"). The matching withdrawal is DRY-RUN: would withdraw mitigation (not sent). If you see neither, no mitigation decision has been made yet — check that detection is firing (above).

BGP won't reach ESTABLISHED

Peering happens even in dry-run, so you can fix this before going live. Watch the state:

journalctl -u kapkan -f | grep "bgp peer state"

If a neighbor never reaches ESTABLISHED, the cause is on one of two sides:

  • Kapkan's config — bgp.local_asn, the neighbor's remote_asn, and its address must match what the router expects. A mismatched ASN is the most common cause.
  • The router / host side — the router must be configured to accept the session, the source address must be reachable, and no ACL or firewall may block TCP 179.

GET /api/v1/status shows your BGP configuration and neighbor addresses, but not live session state — peer state lives only in the logs.

Routes are announced but attack traffic still arrives

Kapkan announces the route (you see it in GET /api/v1/bans with state: "active"), but the flood does not stop:

  • Does your upstream honour the blackhole? A /32 or /128 blackhole only works if the router or upstream provider that carries your traffic accepts the route and its blackhole community. Confirm the community in bgp.community matches what your provider expects, and that they accept host-length prefixes from you.
  • Source-based attack? RTBH drops traffic to the victim. To also drop traffic from a blackholed source you need uRPF on your edge, or use FlowSpec to match the attack precisely.
  • FlowSpec not taking effect? The peer must have negotiated the FlowSpec capability; a peer that does not support it simply ignores the rules. See FlowSpec.

The XDP data plane isn't dropping anything

You enabled dataplane and traffic that a static rule should drop is still arriving. Ask the box directly — this command is read-only, needs no running daemon, and cannot disturb what it reports:

sudo kapkan dataplane status

The first two lines answer the question; the rest is detail. The states it can report, and what each one means:

  • ENFORCING with DRY-RUN on the first line — the datapath is rewriting every drop verdict into a pass. This is the most common answer, and it reflects what the kernel is doing, which can differ from what the config file now says. Set dry_run: false and reload.
  • ENFORCING with GENERIC on the first line — the program is on the generic (skb) path rather than the driver path, costing roughly an order of magnitude of capacity. Either the NIC driver has no native XDP support or dataplane.xdp_mode: generic is set.
  • no_pin_path / no_program — nothing is attached and nothing ever was here. Check that dataplane.enabled is true, and that the pin path in the report is the one in your config (the report always says where it got the path from).
  • detached — the program and maps are pinned and intact but no attachment is bound to a live netdevice. Confirm the configured interfaces exist (ip link), then restart Kapkan.
  • not_bpffs — the pin path is not on a bpffs, so nothing can ever be pinned there. Mount it (mount -t bpf bpffs /sys/fs/bpf), and if your unit sets ProtectKernelTunables=yes, remove it — that mounts /sys/fs/bpf read-only.
  • torn / schema_skew — the pin set is incomplete, or the pinned maps are a different map schema version than the binary (an upgrade that has not restarted yet). Restart Kapkan: it refuses to adopt either and rebuilds. Dynamic mitigation rules from the old process are lost in the rebuild; static policy is not, it comes from the config file.
  • attach_unknown — you are not running as the user the daemon runs as. Reading a bpf_link pin needs write permission on it, so an unprivileged reader sees the maps and no attachments. Re-run with sudo.

If it says ENFORCING, not dry-run, on the right interface, then the program is running and the rules are the next thing to look at: the VERDICTS block shows which branch actually decided the traffic's fate. Remember that the OBSERVATIONS block is bumped alongside those verdicts, not instead of them, so it must not be added into the total. Full flag and exit-code reference: CLI.

A static rule's counter never moves

The program is ENFORCING on the right interface and not in dry-run, but one static rule's counter sits at zero while the traffic it names is plainly arriving. A zero counter is ambiguous on its own — it looks exactly like a correct rule on a quiet day — so first ask whether the rule can fire at all:

kapkan -check-config /etc/kapkan/config.yaml

A rule that can never fire is printed as a WARNING after the OK line, naming the rule and what takes its packets. The exit code stays 0: this is legal config that the daemon will run, not a validation failure. A running daemon reports the same finding as a WARN on every policy apply, as the policy_shadowed condition on /healthz and /api/v1/status, and as kapkan_dataplane_shadowed_static_rules > 0.

Two things can take a rule's packets before it is reached:

  • the allowlist, which is evaluated first and stops evaluation on a hit — a prefix in dataplane.allowlist disables every static drop aimed at a source inside it;
  • an earlier static rule, because static rules are first match wins — a broad rule written above a narrower one makes the narrower one dead policy.

Move the specific rule above the general one, or narrow the general one or the allowlist entry. See Rules that can never fire.

If the check is silent, the rule is reachable and the question is whether the traffic matches it: match fields are ANDed, and a src_port or dst_port on the wrong side of the flow is the usual reason a rule that looks right never matches anything.

A ban was rejected (HTTP 409)

A manual POST /api/v1/ban returned 409, or you see rejections on the kapkan_mitigate_bans_rejected_total metric. This is a safety guard doing its job. The reason tells you which:

  • the target is in protected_whitelist;
  • the target is outside your configured networks;
  • max_active_bans is already reached;
  • an optional blast-radius guard (max_banned_fraction / max_bans_per_window) tripped.

Raise the relevant limit only if you understand why it was set.

A config reload failed

A POST /api/v1/config/reload (or SIGHUP) had no effect, or the daemon logged a validation error. Validate the file before reloading:

kapkan -check-config /etc/kapkan/config.yaml   # exit 0 = valid, 1 = invalid

The error message names the offending key. A reload that fails validation is rejected and the previous good config keeps running — Kapkan does not load a broken config.

The update check exits 1

kapkan -check-update returns exit 1 with a "no such file" error. The check loads your config (to read update_check), and -config defaults to the development path configs/dev.yaml. On a real host, point it at your file:

kapkan -check-update -config /etc/kapkan/config.yaml

Exit codes: 0 = up to date, 10 = a newer release exists, 1 = error. See Upgrading.

ClickHouse history is empty

You enabled storage but GET /api/v1/attacks history or your ClickHouse tables stay empty:

  • Confirm the ClickHouse server is running and reachable from the Kapkan host.
  • Watch kapkan_storage_rows_total by result: rows under written should climb; rows under dropped or error mean Kapkan cannot write (check the address, credentials and the logs).