Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Troubleshooting

Start with the process logs, database health, and federation status:

curl http://127.0.0.1:8786/health
birdd --config birdd.toml federation-status --json

On NixOS:

systemctl status birdd
journalctl -u birdd
birdd-admin federation-status --json

The service does not start

Check that the configuration, database directory, and private key are readable and writable by the service account. Configuration rejects malformed TOML and unknown top-level fields. Empty credential files are also rejected.

The NixOS service has a condition on services.birdd.flockKeyFile. If the unit is inactive instead of failed, inspect its conditions:

systemctl show birdd -p Conditions

Credential paths in the NixOS module must be absolute, outside /nix/store, and configured in required pairs. Use birdd-admin instead of running ordinary CLI commands as root against the service database.

A port is already in use

The default loopback ports are:

  • 8786 - node admin API
  • 8787 - Grist webhook and media adapter
  • 8788 - operator portal
  • 8789 - Signal health and metrics

The admin, Signal metrics, and operator portal listeners reject non-loopback addresses. Put an authenticated reverse proxy in front of services that need a public route.

Federation is not moving

Run one explicit sync and inspect its output:

birdd --config birdd.toml sync

With no configured relays, sync is a successful no-op. With relays configured, at least one must be reachable. birdd run remains healthy and retries after a relay error, so /health alone does not detect federation failure.

Use federation-status --json to check missing sequences, conflicts, manifest completeness, and recipient delivery states. A restricted delivery stuck in wrapped often means at least one configured relay has not accepted it.

An observation cannot be corrected

Corrections and deletions fail when the observation is remote-owned, already deleted, stale, or missing a representation for a currently selected audience. Refresh stale data before retrying.

If the error says an observation predates a selected audience and explicit backfill is required, there is currently no supported backfill command. Do not modify signed history by hand.

Signal is disconnected

Check both services and the Signal-specific health endpoint:

systemctl status signal-cli birdd-signal
curl http://127.0.0.1:8789/health

The Signal base URL must be credential-free loopback HTTP. Attachment failures can also come from unsupported formats, configured size limits, missing vipsthumbnail or ffmpeg, or unavailable object storage.

Grist edits fail

Reconciliation requires a base URL, document ID, and API key. Webhook requests need the configured bearer secret. A stale Revision, remote observation, or invalid editable value is rejected by birdd. Changes to managed fields are not applied to observations.

Check the reconciliation timer and webhook service separately:

systemctl status birdd-grist-reconcile.timer birdd-grist-webhook
journalctl -u birdd-grist-reconcile -u birdd-grist-webhook

Rebuild carefully

birdd rebuild regenerates derived observations and metadata from retained events. It is not a SQLite repair command and is not a substitute for restoring a known-good backup.