birdd

birdd (bir(distributed)d) is for distributed communitities of observers. It provides resilient logging of observations that are federated across sub-communities.

flowchart TD
    obsP[Observations] --> pallet[Pallet Town]
    obsC[Observations] --> cerulean[Cerulean City]
    obsS[Observations] --> sunyshore[Sunyshore City]

    pallet <-.-> cerulean
    cerulean <-.-> sunyshore
    sunyshore <-.-> pallet

    classDef palletTown fill:#4caf50,stroke:#1b5e20,color:#fff
    classDef ceruleanCity fill:#2196f3,stroke:#0d47a1,color:#fff
    classDef sunyshoreCity fill:#fdd835,stroke:#f57f17,color:#222
    class pallet palletTown
    class cerulean ceruleanCity
    class sunyshore sunyshoreCity

It was built for local bird watching communities to share sightings, but it’s useful for all kind of communities-of-observation.

Examples:

• Birding
• Where’s George currency circulation tracking
• Seasonal blooms
• Weather
• Subaru Bajas
• Many more!

A birdd node, and its members is called a flock. Your flock can install extensions to add extra trackable information to your observations for whatever thing you enjoy observing.

Hopefully you find it useful and fun! birdd is open source! Please fork. Patches are welcome. birdd is a labor of love. If you enjoy birdd and want to support my work, please consider making a donation. All funds go directly to the cost of hosting. Any extra funds will be donated to The Sameer Project.

Using birdd

Your community’s birdd node records observations as signed events and can share them with other community nodes. Start with Trust, federation, and audiences to understand where observations can travel and what the system’s trust and privacy guarantees mean.

birdd currently has three supported interfaces:

1. Signal - record and query observations through a bot in a Signal group
2. Grist - browse observations and make bulk additions or corrections in a spreadsheet-like editor
3. Command-line interface - create and query observations directly on a birdd node

Trust, federation, and audiences

birdd is distributed: your community’s node can exchange observations with other communities instead of relying on one central database. Three related ideas govern that exchange:

• Identity and trust answer which flock produced an event and which other flocks your node recognizes.
• Federation copies signed events between flock nodes through relays.
• Audiences determine which representation of a local event is sent and who can receive it.

Flock identity

Each flock has a cryptographic key pair. Its public key is its stable identity, and its familiar name is a mutable label. A flock can change their display name without changing their stable identity, defined by the public key.

The private key signs observations, corrections, deletions, and metadata. A valid signature establishes that an event was signed by the holder of that flock’s key and has not been modified since.

The node operator is responsible for protecting the flock’s private key. If it is lost, there is currently no identity recovery or key rotation mechanism.

Trust is local

Your node operator decides which public keys to trust and assigns local names to them. Another flock cannot make your node trust it by publishing its name as someone you already trust.

Signal queries include local observations and observations signed by directly trusted flocks. They exclude observations from other flocks, as well as observations dated in the future. Trusting a flock identifies an accepted source.

This filter is specific to Signal, not a universal access boundary. The CLI and Grist’s Following_Remote table can include valid remote observations retained by the node regardless of trust status. Access to media has additional authorization rules described below.

Trust and permission to use a restricted audience are separate decisions. A node can receive and retain a valid event without making that event visible in every user interface.

How federation works

An observation is recorded as a signed event in its origin flock’s log. Corrections and deletions append new signed events rather than silently changing the original history. User interfaces normally show the latest valid state.

Nodes publish and retrieve events through Nostr-compatible relays:

flowchart LR
    origin[Origin flock] -->|signed event| relay[Relay]
    relay -->|sync| neighbor[Receiving flock]

Relays help disconnected flocks catch up and allow more than one relay to carry the same deterministic event.

Federation is not necessarily immediate. A node or relay may be offline, and a restricted audience may deliberately delay publication. Search results can therefore differ between flocks or catch up later.

Audiences

An audience is an operator-defined sharing policy. It answers:

1. Which fields and timestamp precision leave the node?
2. Is that representation public or encrypted for named recipients?

An observation can be represented in more than one audience. For example, a flock could publish a minimal public representation and send a more detailed version to a restricted group of neighboring flocks.

The current public projection includes the subject, exact observation time, notes, tags, media references, and extensions. It does not include location. A media reference does not make the media bytes public: fetching them requires a signed request from a flock directly trusted by the media’s origin.

A restricted projection can coarsen the observation time to an hour or UTC day, delay publication, omit notes, tags, location, or media, and allow only selected extensions. Extension selection includes each chosen envelope in full, not only the fields shown in Signal or Grist. The subject and observation identity are always included.

Timestamp coarsening applies to when the observation reportedly occurred. It does not hide the exact time the event was created, which remains visible to an authorized recipient. Delayed publication likewise does not conceal that original creation time after delivery.

Important

A restricted audience limits delivery; it cannot control what an authorized recipient does after decryption. Recipients can retain or copy data. Do not record sensitive information solely because the audience is restricted.

Audience policy is selected by the node operator for local events. Signal, Grist, and CLI users do not currently choose an audience for each observation. Ask your operator which audiences are active before recording sensitive notes, tags, media, or extension data.

Receiving and trusting are different

Restricted federation has separate inbound and outbound lists:

• Recipients are flocks that receive this node’s restricted events.
• Allowed senders are flocks whose restricted events this node accepts.
• Directly trusted flocks are sources included by Signal queries and are eligible for media access.

These lists do not imply one another. Being trusted does not automatically make a flock a restricted recipient, and receiving an encrypted event does not automatically make its signer trusted. Media access also requires a signed request and a live observation that disclosed the media reference. A restricted recipient must have received a published projection that included it.

Corrections, deletion, and privacy

Corrections and deletions are signed history. A deletion tells conforming nodes to remove an observation from their current view, but it cannot guarantee that relay operators or prior recipients erase copies they already possess.

Changing audience policy affects future representations and can permanently suppress restricted deliveries that were queued but not yet published. It does not recall published events or automatically share older history with a newly added audience.

An older observation cannot be corrected or deleted when it lacks a representation for every audience that is currently selected. The operator must backfill the missing representation first, but this version of birdd does not yet provide a backfill command. Treat publication, especially public publication, as irreversible.

Questions for your operator

Before contributing observations, it is reasonable to ask:

• Does this flock publish to a public audience?
• Which fields are included in each restricted audience?
• Which flocks receive restricted observations?
• Is publication delayed or are timestamps made less precise?
• Which remote flocks are directly trusted in search results?
• Who can use Grist or other administrative interfaces to make corrections?

Node operators can find implementation and configuration details in Identity and trust and Federation and audiences.

Signal

birdd can join a Signal group and create observations from chat messages.

Commands

!saw

Records an observation. !saw takes a required subject and supports trailing tags and a note. Structured extension fields can be provided after ::. Any media attached to the message is stored with the observation when your flock has media enabled.

!saw <subject> [#tag...] [-- note] [:: field=value...]

Examples:

!saw Red-tailed Hawk
!saw creek flooding #water #trail -- near the footbridge

The fields accepted after :: depend on the extension profiles installed by your node operator. For example, the optional bird count profile accepts:

!saw Common Loon :: count=3

!today

Shows up to ten observations from today (by calendar day as configured on your flock’s node). Results include the local flock and directly trusted remote flocks. Add here to include only observations by the local flock. See Trust, federation, and audiences for how remote visibility works.

!today
!today here

!recent

Show the newest observations. The default limit is five.

!recent
!recent 69

Note

birdd has an arbitrary 4096-byte message limit to avoid signal errors/abuse filters. birdd truncates results at the message limit. Interested in contributing to birdd? I’d love a patch to paginate long result sets!

!search

Find up to ten observations using a case-insensitive subject search.

!search loon
!search red-tailed hawk

!obs

List observations whose subject exactly matches after normalizing case and whitespace. If no exact subject exists, Birdd suggests grouped partial matches.

!obs Common Loon
!observations common loon

!show

Show one observation and any media available on this node. Use a short reference returned by another command or the full observation UUID.

!show 0198abcd
!show 0198abcd-1234-7000-8000-123456789abc

!profiles

List the structured metadata profiles installed on this node, or show the fields and example for one profile. !profile is an alias. The profile names below require the optional bird count manifest.

!profiles
!profiles wildlife

Profile fields follow :: in a !saw command. The profile name can be explicit, or Birdd can infer it from its fields.

!saw Common Loon :: wildlife count=3
!saw Common Loon :: count=3

!help

List all commands or show focused help for a command or profile.

!help
!help saw
!help wildlife

For profile inference, validation, receipts, and extension output behavior, see Signal integration.

Grist

Grist is a third-party open-source software we use for admin-editing birdd observations. Only observations made directly on a birdd community node can be edited with Grist. Observations from neighbors are not editable.

Note

Grist is a node-operator only capability. Your node-admin may give you credentials to use Grist and edit observations. That’s a conversation between you and your node-admin!

You might find the Grist documentation useful for general grist information.

Tables

Our_Observations

Shows observations made by your flock. Add one row or paste many rows to create observations in bulk. Edit a supported cell just like any other spreadsheet; birdd will record the change as a correction to the original observation.

You can edit:

• Subject - the name of the thing observed
• Observed At - a Unix timestamp; blank rows default to the current time
• Text - notes about the observation
• Tags - a comma-separated list of tags
• Editable extension fields exposed as columns by birdd
• Delete - check this box to delete the observation

For example, a node with the optional bird manifests installed may expose:

Subject: Common Loon
Count: 3
Tags: lake, migration
Text: Seen near the north shore

Columns such as Observation ID, Revision, Media, Origin, and Sync Status are managed by birdd; changes to them are not applied to the observation.

Extension columns depend on the manifests installed by the node operator. The read-only Extensions JSON column preserves the complete envelope map even when no dedicated column exists. See Grist integration for projection, conversion, and unknown-version behavior.

Note

Use the Delete checkbox rather than deleting the Grist row directly. Grist does not emit webhooks for row removal, so removing a spreadsheet row cannot delete its signed observation. The checkbox is an irreversible action, not an editable deleted state: once birdd receives it, a signed deletion is recorded and reconciliation removes the row from Grist. Unchecking the box cannot restore the observation. To recover its content, create a new row, which produces a new observation with a new ID.

Following_Remote

Shows remote observations retained by your node, including observations that Signal hides because their flock is not directly trusted. These observations are read-only because corrections must be signed by the flock that originally made them. See Trust, federation, and audiences for the difference between receiving and trusting an event.

Use this table to browse, sort, and filter observations from your neighbors.

Corrections

When you edit a supported field in Our_Observations, Grist sends the changed row back to your flock’s node.

If someone else corrected the same observation first, your edit may be rejected as stale. Refresh the table to get the newest revision, then make the correction again.

Syncing

Your node periodically reconciles Grist with its observation log. New local and remote observations appear automatically, and corrected values are updated in the spreadsheet. Sync Status shows the federation state reported by your node.

Command-line reference

birdd includes commands for operating a node, creating and querying observations, managing trust, syncing federation data, and running adapters.

Use --help to see the commands and options supported by your installed version:

birdd --help

Pass --help to a command for its arguments and usage examples:

birdd create --help
birdd run --help
birdd grist-provision --help

Use --config before the command to select a configuration file. It defaults to birdd.toml and can also be set with BIRDD_CONFIG.

birdd --config /etc/birdd/birdd.toml list

Running a node

A birdd node is home base for a flock. It stores the flock’s observations, signs new events with the flock identity, and exchanges permitted observations with other nodes through relays.

At minimum, running a node means taking care of three pieces of persistent state:

• The flock’s private key
• The SQLite database
• The node configuration

Important

Keep the private key private and back it up. The key is your flock’s identity; losing it means losing the ability to sign as that flock.

What Runs on a Node

birdd is the core of a node, but a typical end-to-end deployment includes the services that let flock members submit observations and operators administer the system:

• birdd stores observations, signs events, and synchronizes with federation relays.
• signal-cli connects the flock’s Signal account to the Signal adapter.
• Grist provides the operator-facing spreadsheet used to review and edit local observations.
• Garage provides private S3-compatible storage for sanitized image and audio attachments.
• birdd-admin-web provides the node administration portal.
• Authelia authenticates access to Grist, the administration portal, and the metrics interface.
• Caddy terminates HTTPS and routes requests to those local services.
• Prometheus and Grafana collect and display node metrics. Alloy, Loki, and Tempo support metric ingestion, logs, and traces used by the operator portal.

This is the recommended reference deployment, not a required architecture. The core birdd service can run independently, and operators may enable only the adapters they need or use their own object storage, ingress, authentication, administration, and observability stack. The NixOS full-node role is an opinionated serving suggestion that provides a complete system with known integration points.

These services normally communicate over loopback and are not each exposed to the internet. The NixOS full-node role configures their service accounts, startup ordering, local endpoints, HTTPS ingress, and restart behavior. It also means a full node has persistent state beyond the three core files, including Signal account state, Grist data, Garage objects and metadata, Authelia data, and local observability data. See Backups and recovery for what must be preserved.

Get Running

Note

This is not necessary information if using the reference stack.

Install birdd, choose a persistent location for its files, and initialize the node:

birdd --config /var/lib/birdd/birdd.toml init \
  --node-name pallet \
  --flock-name "Pallet Town Birders" \
  --database /var/lib/birdd/birdd.sqlite \
  --key-file /var/lib/birdd/flock.key \
  --relay wss://relay.example.com

Start the node after reviewing the generated configuration:

birdd --config /var/lib/birdd/birdd.toml run

run keeps federation syncing and serves a read-only admin API on loopback. Signal, Grist, and other adapters run alongside the node as separate processes. For a long-running deployment, use a service manager so each process starts on boot and restarts after a failure.

Operator Guide

• Installation covers installing and updating the node.
• Configuration documents every birdd.toml setting.
• Relays explains how nodes exchange events.
• Identity and trust covers flock keys and choosing which neighbors to trust.
• Federation and audiences covers public and restricted sharing policies.
• Media storage covers observation attachments.
• Backups and recovery covers the persistent files you need to protect.
• Monitoring covers health and metrics.
• Troubleshooting collects common operator problems.
• The Birdd reference stack covers the recommended NixOS stack and third-party service setup.

Installation

birdd can run as a standalone binary or as a NixOS service. The project flake supports Linux and macOS on x86-64 and ARM64.

Note

The quickest way to get started with birdd is to use the reference stack.

NixOS

The flake exports nixosModules.birdd. A minimal service configuration looks like this:

{inputs, pkgs, ...}: {
  imports = [inputs.birdd.nixosModules.birdd];

  services.birdd = {
    enable = true;
    package = inputs.birdd.packages.${pkgs.system}.birdd;
    nodeName = "pallet";
    flockDisplayName = "Pallet Town Birders";
    flockKeyFile = "/var/lib/private/birdd/flock.key";
    relays = ["wss://relay.example.com"];
  };
}

The key file must already exist at an absolute path outside the Nix store. The service will not start when it is missing. Node state defaults to /var/lib/birdd, and the module installs a birdd-admin wrapper for running CLI commands with the service identity and permissions.

Build with Nix

Build the node from the repository:

nix build .#birdd
./result/bin/birdd --help

Install it into your user profile:

nix profile install .#birdd

Use birdd-with-docs instead when the admin portal should serve an embedded copy of this book:

nix build .#birdd-with-docs

Build from source

Source builds require a Rust toolchain supporting edition 2024, OpenSSL, and pkg-config. SQLite is bundled by the Rust dependency.

cargo build --release --locked
./target/release/birdd --help

The build produces birdd, the main node and CLI; and birdd-admin-web, the local operator portal.

Continue with Node configuration after installation.

Node configuration

birdd reads TOML configuration from birdd.toml by default. Select another file with --config or BIRDD_CONFIG:

birdd --config /etc/birdd/birdd.toml run

Unknown fields are rejected in the root, federation, audience, projection, and extensions tables. Signal, Grist, and media tables currently ignore unknown fields, so check those names carefully.

NixOS mappings

The NixOS module generates birdd.toml. Common node settings have dedicated options; other non-secret values use services.birdd.settings with the same shape as the TOML.

For example, these are equivalent:

[federation.audiences.neighbors]
transport = "restricted"
recipients = ["<public-key>"]

services.birdd.settings.federation.audiences.neighbors = {
  transport = "restricted";
  recipients = ["<public-key>"];
};

Secret values should use the dedicated NixOS *File options. Those files are passed to the required process as systemd credentials and override values from TOML.

Node identity

node_name

Stable operator-facing name used in logs and telemetry. This is not the flock’s cryptographic identity.

• Type: string
• Required
• NixOS: services.birdd.nodeName

flock_display_name

Human-readable flock name published in signed metadata.

• Type: string
• Required
• NixOS: services.birdd.flockDisplayName

time_zone

IANA time zone used to interpret local dates and display observation times.

• Type: string, such as UTC or America/New_York
• Default: UTC
• NixOS: time.timeZone

The time zone is resolved when Signal formats a command response. An invalid name fails at that point rather than during configuration loading.

database

Path to the node’s SQLite database. Relative paths are resolved from the process working directory. Parent directories are created when needed.

• Type: path string
• Required
• NixOS: generated as ${services.birdd.dataDir}/birdd.sqlite

key_file

Path to the flock’s private Nostr key. An existing key is reused. A standalone node creates a new mode-0600 key file when the path does not exist.

• Type: path string
• Required
• Environment override: BIRDD_FLOCK_KEY_FILE
• NixOS source: services.birdd.flockKeyFile

The NixOS module makes the source key available as a per-service systemd credential, so the effective runtime path is under /run/credentials.

Important

Keep this file secret and backed up. A new key creates a new flock identity.

relays

Nostr-compatible WebSocket relays used to publish and receive federated events. URLs are accepted by the Nostr client when synchronization begins.

• Type: array of URL strings
• Default: []
• NixOS: services.birdd.relays

An empty list keeps the node local-only. See Relays for synchronization behavior.

Federation

federation.default_audiences

Audience policies applied to new local mutations. Every entry must name a policy under federation.audiences, and duplicates are rejected.

• Type: non-empty array of audience names
• Default: ["public"]
• NixOS: services.birdd.settings.federation.default_audiences

federation.audiences

Map of available audience policies. Defining this map replaces the default map; add the public policy explicitly if you still want it.

• Type: map containing 1-32 policies
• Default: one audience named public
• NixOS: services.birdd.settings.federation.audiences

Audience names contain 1-32 lowercase letters, digits, or hyphens. Replace <name> below with an audience name.

federation.audiences.<name>.transport

Determines whether events are published openly or encrypted for named recipients.

• Type: enum
• Values: public, restricted
• Required
• NixOS: services.birdd.settings.federation.audiences.<name>.transport

federation.audiences.<name>.allowed_senders

Nostr public keys allowed to send events into a restricted audience. Public audiences must leave this empty. This is independent of the local trust store.

• Type: array of unique Nostr public keys
• Default: []
• NixOS: services.birdd.settings.federation.audiences.<name>.allowed_senders

federation.audiences.<name>.recipients

Nostr public keys that receive encrypted events for a restricted audience. Restricted audiences require at least one recipient; public audiences must leave this empty. A recipient can belong to only one configured audience.

• Type: array of unique Nostr public keys
• Default: []
• NixOS: services.birdd.settings.federation.audiences.<name>.recipients

Audience projection

Projection controls change the representation shared with an audience. Public audiences must use every default below; custom projections are restricted-only.

federation.audiences.<name>.projection.observed_at_precision

Reduces timestamp precision before publication. hour and day use UTC boundaries.

• Type: enum
• Values: exact, hour, day
• Default: exact
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.observed_at_precision

federation.audiences.<name>.projection.publication_delay_seconds

Minimum delay before a restricted recipient’s projected event becomes eligible for publication.

• Type: unsigned integer from 0 through 31536000
• Default: 0
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.publication_delay_seconds

federation.audiences.<name>.projection.notes

Controls whether observation notes are included.

• Type: boolean
• Default: true
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.notes

federation.audiences.<name>.projection.tags

Controls whether observation tags are included.

• Type: boolean
• Default: true
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.tags

federation.audiences.<name>.projection.location

Controls whether location data is included.

• Type: boolean
• Default: false
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.location

federation.audiences.<name>.projection.media

Controls whether signed media references are included.

• Type: boolean
• Default: true
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.media

federation.audiences.<name>.projection.extensions

Selects which namespaced extension envelopes are included.

• Type: non-empty array of strings
• Values: ["*"] or an allowlist of namespaced extension IDs
• Default: ["*"]
• NixOS: services.birdd.settings.federation.audiences.<name>.projection.extensions

Projection selects complete envelopes, not only fields declared by a local manifest. Review the extension’s actual data before adding its ID to an audience. See Federation and audiences for the disclosure model.

Extensions

Extensions are disabled by default. Birdd loads an ordered list of declarative TOML manifests at startup:

[extensions]
manifests = ["extensions/birds-count.toml"]

extensions.manifests

• Type: array of filesystem paths
• Default: []
• NixOS: services.birdd.settings.extensions.manifests

Relative paths are resolved from the directory containing the selected birdd.toml, not from the process working directory. Use absolute paths when a manifest is installed separately from the configuration.

All paths are loaded as one registry. A missing or invalid file, invalid Signal template, duplicate extension ID or profile name, generated Grist column collision, or invalid measure reference prevents startup. Restart the node, Signal adapter, Grist webhook, and other long-running consumers after changing the list.

The Nix package includes optional examples under share/birdd/extensions. A NixOS configuration can reference one directly from the selected package:

services.birdd.settings.extensions.manifests = [
  "${inputs.birdd.packages.${pkgs.system}.birdd}/share/birdd/extensions/birds-count.toml"
];

For a source checkout, the equivalent file is extensions/birds-count.toml. No example manifest is loaded automatically. See Extensions for the data model and Authoring extensions for the manifest schema.

Signal

The Signal adapter connects to the loopback HTTP API provided by signal-cli. All three connection values are required by birdd signal-run.

signal.base_url

Base URL for the Signal event stream and JSON-RPC API. It must be credential-free loopback HTTP with no path, query, or fragment.

• Type: URL string
• Default: unset
• NixOS: services.birdd.settings.signal.base_url

signal.account

Signal account whose inbound messages are accepted and whose identity sends bot responses.

• Type: secret string
• Default: unset
• Environment file: BIRDD_SIGNAL_ACCOUNT_FILE
• NixOS: services.birdd.signalAccountFile

signal.group_id

The only Signal group authorized to send commands to this flock.

• Type: secret string
• Default: unset
• Environment file: BIRDD_SIGNAL_GROUP_ID_FILE
• NixOS: services.birdd.signalGroupIdFile

On NixOS, signalAccountFile and signalGroupIdFile must be configured together.

Signal search view

signal.views.search optionally customizes !search output. A list view needs only a layout and header:

[extensions]
manifests = ["/etc/birdd/extensions/birds-count.toml"]

[signal.views.search]
layout = "list"
header = "Found {observation_count} {observation_count:plural(observation)}"

signal.views.search.layout

• Values: list, grouped
• Required when signal.views.search is present
• NixOS: services.birdd.settings.signal.views.search.layout

signal.views.search.header

Header template, up to 512 printable bytes. Supported placeholders are {observation_count} and {observation_count:plural(observation)}.

• Type: string
• Required when signal.views.search is present
• NixOS: services.birdd.settings.signal.views.search.header

Grouped search

A grouped view combines case-insensitive subject matches and can show the last date, latest note, and loaded extension measures:

[signal.views.search]
layout = "grouped"
header = "Found {observation_count} {observation_count:plural(observation)}"

[signal.views.search.group]
by = "subject"
normalize = "casefold"
label = "latest"
heading = "{group_label:highlight}"
sort_by = "core.last_seen"
sort_order = "descending"

[[signal.views.search.group.fields]]
label = "Last seen"
measure = "core.last_seen"
format = "date"

[[signal.views.search.group.fields]]
label = "Latest note"
measure = "core.latest_note"
format = "text"

[[signal.views.search.group.fields]]
label = "Seen"
measure = "fyi.birdd.birds/count.seen"
format = "number"
singular = "time"
plural = "times"

signal.views.search.group

Required for layout = "grouped" and forbidden for layout = "list".

• NixOS: services.birdd.settings.signal.views.search.group

The current grouped implementation requires these exact declarations:

signal.views.search.group.heading accepts {group_label} or one {group_label:highlight} placeholder and is limited to 512 printable bytes. Its NixOS path is services.birdd.settings.signal.views.search.group.heading.

signal.views.search.group.fields

Non-empty ordered list of values shown under each group. Measures cannot be repeated.

• NixOS: services.birdd.settings.signal.views.search.group.fields

Each field supports:

Supported core measure/format pairs are:

• core.last_seen with date
• core.latest_note with text

Loaded extension manifests may add numeric measures. The example above requires the configured birds-count.toml manifest, which declares fyi.birdd.birds/count.seen. An unknown measure prevents startup.

In Nix, fields is a list of attribute sets; the * above represents one list element rather than a literal attribute name.

Grist

Grist configuration is optional for the core node. Reconciliation requires base_url, document_id, and api_key. Handling existing-row webhook edits requires webhook_secret; handling new rows also requires the reconciliation settings so Birdd can write generated fields back to Grist.

grist.base_url

Base URL of the Grist API.

• Type: URL string
• Default: unset
• NixOS: services.birdd.settings.grist.base_url

grist.document_id

ID of the provisioned Grist document.

• Type: secret string
• Default: unset
• Environment file: BIRDD_GRIST_DOCUMENT_ID_FILE
• NixOS: services.birdd.gristDocumentIdFile

grist.api_key

Document-scoped API key used by reconciliation and new-row webhook writeback.

• Type: secret string
• Default: unset
• Environment file: BIRDD_GRIST_API_KEY_FILE
• NixOS: services.birdd.gristApiKeyFile

grist.webhook_secret

Bearer secret required to authenticate the Grist edit webhook.

• Type: secret string
• Default: unset
• Environment file: BIRDD_GRIST_WEBHOOK_SECRET_FILE
• NixOS: services.birdd.gristWebhookSecretFile

The NixOS reconciliation timer is created when both gristDocumentIdFile and gristApiKeyFile are configured. The webhook service is created independently when gristWebhookSecretFile is configured. That independent service can process corrections and deletions, but new-row creation cannot complete its Grist writeback unless the document ID and API key credentials are also available.

Media

Media storage is disabled by default. When enabled, birdd uses static credentials with an S3 or S3-compatible object store.

media.enabled

• Type: boolean
• Default: false
• NixOS: services.birdd.settings.media.enabled

When disabled, other media settings are not validated.

media.endpoint

Private S3 API endpoint. Leave unset for the normal regional AWS endpoint. Plain HTTP is allowed only for loopback hosts.

• Type: optional HTTP or HTTPS URL
• Default: unset
• NixOS: services.birdd.settings.media.endpoint

media.public_endpoint

Public birdd media route used by other flocks, not the S3 endpoint. It must be HTTPS, end in /_birdd/media, and contain no credentials, query, or fragment.

• Type: optional HTTPS URL
• Default: unset
• NixOS: services.birdd.settings.media.public_endpoint

media.region

• Type: non-empty string
• Default: us-east-1
• NixOS: services.birdd.settings.media.region

media.bucket

• Type: string
• Required when media is enabled
• NixOS: services.birdd.settings.media.bucket

media.access_key

• Type: secret string
• Required when media is enabled
• Environment file: BIRDD_MEDIA_ACCESS_KEY_FILE
• NixOS: services.birdd.mediaAccessKeyFile

media.secret_key

• Type: secret string
• Required when media is enabled
• Environment file: BIRDD_MEDIA_SECRET_KEY_FILE
• NixOS: services.birdd.mediaSecretKeyFile

The NixOS media credential options must be configured together.

media.max_attachments

• Type: unsigned integer from 1 through 16
• Default: 4
• NixOS: services.birdd.settings.media.max_attachments

media.max_attachment_bytes

Maximum input, sanitized output, stored object, and remote download size for one attachment.

• Type: positive unsigned integer
• Default: 15728640 (15 MiB)
• Must not exceed media.max_total_bytes
• NixOS: services.birdd.settings.media.max_attachment_bytes

media.max_total_bytes

Maximum combined attachment size for one Signal message.

• Type: unsigned integer
• Default: 31457280 (30 MiB)
• Range: media.max_attachment_bytes through 268435456 (256 MiB)
• NixOS: services.birdd.settings.media.max_total_bytes

media.max_image_dimension

Maximum width and height used when sanitizing images.

• Type: unsigned integer from 256 through 8192
• Default: 4096
• NixOS: services.birdd.settings.media.max_image_dimension

media.processing_timeout_seconds

Timeout applied separately to each image or audio processor invocation.

• Type: unsigned integer from 1 through 300
• Default: 30
• NixOS: services.birdd.settings.media.processing_timeout_seconds

See Media storage for supported formats, delivery authorization, and operational behavior.

Credential files

Each environment variable below contains a path to a credential file, not the credential value itself. File contents are trimmed and empty files are rejected.

Secret values are protected in memory with the secrecy crate and omitted when Rust serializes the effective configuration.

NixOS-only options

These service options affect deployment but are not written as Rust TOML settings:

• services.birdd.enable: enables the node service
• services.birdd.package: package containing the birdd binary
• services.birdd.dataDir: state directory; default /var/lib/birdd
• services.birdd.syncIntervalSeconds: sync interval; default 5
• services.birddGarage.enable: enables the reference Garage object store and configures Birdd media storage
• services.birddGarage.capacity: capacity assigned to the local Garage node; default 10G

The generated base values for node_name, flock_display_name, time_zone, database, key_file, and relays take precedence over values placed in services.birdd.settings.

Relays

birdd uses Nostr-compatible WebSocket relays to exchange signed events. A relay transports events.

Configure one or more relay URLs at the top level of birdd.toml:

relays = [
  "wss://relay-one.example",
  "wss://relay-two.example",
]

An empty list is valid and keeps the node local-only. Use multiple independent relays when federation availability matters.

Synchronizing

Run one synchronization pass with:

birdd --config birdd.toml sync

sync publishes retained local events, fetches remote history, applies valid new events, and prints the number of newly received transport events. Duplicate publication and reception are safe because Nostr event IDs are deterministic.

The long-running node repeats this process every five seconds by default:

birdd --config birdd.toml run --interval-seconds 5

If a sync fails, run logs the error and retries. Local commands and SQLite writes remain available while relays are down.

Failure behavior

At least one configured relay must be reachable for a sync to succeed. Fetch failures from one relay do not prevent other connected relays from being used. Outgoing events are attempted against every connected relay.

Restricted deliveries are considered published only after every configured relay has accepted the recipient’s encrypted event. A relay that stays offline can therefore leave restricted deliveries in the wrapped state even when another relay has accepted them.

Inspect relay and stream state with:

birdd --config birdd.toml federation-status
birdd --config birdd.toml federation-status --json

Note

Relays can replay, censor, duplicate, or lose events. Signatures prevent a relay from forging your flock, but relay history is not a backup.

Identity and trust

Every flock has a secp256k1 key pair. The private key signs observations, corrections, deletions, and metadata. The public key is the stable identity of the flock; display names are just labels.

Print your public key with:

birdd --config birdd.toml identity

When a standalone node opens without an existing key file, it creates one and uses mode 0600 on Unix. NixOS deployments require the configured key to exist before the service starts.

Important

Keep the private key secret and backed up. birdd does not currently provide key rotation or identity recovery. Starting with a new key creates a different flock identity.

Metadata

Flock metadata associates the public key with the configured display name and extension capabilities. init creates the first metadata event. Publish a new metadata event after changing those values:

birdd --config birdd.toml metadata
birdd --config birdd.toml sync

Direct trust

Trust is local policy stored in your node’s database. Verify a public key with the other operator before trusting it:

birdd --config birdd.toml trust \
  <public-key> "Cerulean City Birders"

The supplied name is your local label for that key. It does not need to match the flock’s self-published metadata.

Check a relationship with:

birdd --config birdd.toml trust-status <public-key>

Only trusted grants direct trust behavior. There is currently no supported CLI workflow to revoke trust.

Trust and restricted-audience authorization are separate. Direct trust controls which remote observations appear in user-facing Signal queries and which flocks may fetch media. allowed_senders controls who may send into a restricted audience, while recipients controls who receives it.

Federation and audiences

An audience answers two questions: what version of an event may leave this node, and who may receive it? Every new local mutation is projected into each audience named by federation.default_audiences.

Important

Selected audiences are parallel outputs, not layers of access control. If public and neighbors are both selected, the public representation remains public. A restricted representation does not make fields in the public representation private. Evaluate the most permissive selected audience when explaining the policy to flock members.

Public federation

The default configuration selects one public audience:

[federation]
default_audiences = ["public"]

[federation.audiences.public]
transport = "public"

Public events are signed by the flock and readable by anyone with relay access. The public projection uses exact timestamps, includes notes, tags, media, and all extensions, and excludes location.

Restricted federation

Restricted audiences sign a projected inner event and encrypt a separate NIP-59 gift wrap for each recipient.

[federation]
default_audiences = ["public", "neighbors"]

[federation.audiences.public]
transport = "public"

[federation.audiences.neighbors]
transport = "restricted"
recipients = [
  "<cerulean-public-key>",
  "<sunyshore-public-key>",
]
allowed_senders = [
  "<cerulean-public-key>",
  "<sunyshore-public-key>",
]

[federation.audiences.neighbors.projection]
observed_at_precision = "day"
publication_delay_seconds = 3600
notes = false
tags = true
location = false
media = false
extensions = ["fyi.birdd.birds/count"]

recipients and allowed_senders are intentionally independent. Recipients receive your events. Allowed senders may send events into that audience. A restricted audience can be outbound-only by leaving allowed_senders empty. These lists do not inherit from the trust database.

What leaves the node

Use this matrix when describing the active policy to operators and flock members. It covers observation content, not all federation protocol metadata:

Timestamp precision changes observed_at, not the exact event creation time. Publication delay does not hide that creation time after delivery.

Restricted encryption hides projected content from relays, but not all traffic metadata. A relay can see the recipient key on each gift wrap and when encrypted messages are published. After decryption, a recipient learns the sender key, audience name, sequence, policy epoch, and revision identity in addition to the projected observation.

A media reference identifies an object but does not make the media bytes public. Fetching media requires a signed request from a flock directly trusted by the media’s origin. Restricted media access additionally requires a published representation that disclosed the reference to that recipient.

Extension allowlists include the complete envelope for each selected extension ID. Review the extension’s schema and actual values; an innocent-looking column name does not guarantee that the envelope contains no other sensitive keys. A local extension manifest is not a projection filter; see Extensions.

Corrections and deletions follow the selected audiences too. A deletion updates the current state on conforming nodes but cannot force relays or prior recipients to erase copies. Recipients can also retain or redistribute decrypted restricted data.

Projection controls

A restricted projection can:

• Reduce observation timestamps to the start of an hour or UTC day
• Delay publication for up to one year
• Remove notes, tags, location, or media
• Include all extensions or an explicit allowlist of extension IDs

The subject and observation identity are always included. Public audiences use the default projection and cannot customize these controls.

An enabled field is preserved only when present on an observation. In particular, current local create and correction commands do not accept location, even though restricted projection policy has a location control.

Preview the current projection of a locally owned observation without publishing it:

birdd --config birdd.toml federation-preview <observation-id>

Preview representative observations containing notes, tags, location, media, and every installed extension. Do not preview only an empty test observation.

Choose a privacy posture

Open public sharing

Use the default public audience only when members understand that observation subjects, exact times, notes, tags, media references, and every extension are publicly readable:

[federation]
default_audiences = ["public"]

[federation.audiences.public]
transport = "public"

Public projections cannot be reduced. If any of those fields should be limited to named flocks, remove public from default_audiences and use a restricted audience instead.

Named recipients with full detail

Select only a restricted audience to preserve the normal observation detail while limiting relay-readable content and delivery to named recipient keys:

[federation]
default_audiences = ["neighbors"]

[federation.audiences.neighbors]
transport = "restricted"
recipients = [
  "<cerulean-public-key>",
  "<sunyshore-public-key>",
]
allowed_senders = []

The default restricted projection still sends exact observed times, notes, tags, media references, and every extension. It omits location. Encryption limits delivery but does not minimize what recipients learn.

Leaving allowed_senders empty makes this audience outbound-only. Add keys only for flocks whose restricted events this node should accept; this does not happen automatically when a key is a recipient or is directly trusted.

Minimized and delayed sharing

For sensitive observations, reduce the data before encryption and delay its release:

[federation]
default_audiences = ["regional-summary"]

[federation.audiences.regional-summary]
transport = "restricted"
recipients = ["<regional-coordinator-public-key>"]
allowed_senders = []

[federation.audiences.regional-summary.projection]
observed_at_precision = "day"
publication_delay_seconds = 86400
notes = false
tags = false
location = false
media = false
extensions = ["fyi.birdd.birds/count"]

This sends the subject, observation ID, exact event creation time, day-level observed time, and the count envelope when present. It does not provide full anonymity: subjects and creation times can still be identifying.

The extension projection must be ["*"] or a non-empty allowlist. There is no explicit deny-all setting. An allowlist containing a reserved namespaced ID that the node never writes, such as invalid.example/not-installed, effectively omits current extensions, but would disclose that ID if it were ever added to an observation. Treat this as a workaround and audit it when installing extensions.

Different policies for different recipients

Define multiple restricted audiences when recipient groups should receive different projections, then select all of them in default_audiences. A recipient key can belong to only one restricted audience, so groups cannot overlap.

Every local observation and later correction or deletion goes to every selected audience. Users cannot select an audience per observation. There is no local-only audience transport. Leaving the relay list empty only postpones delivery: selected representations remain stored and can publish if relays are configured later. A durable “never federate” policy therefore requires a separate operationally isolated deployment and is not enforced by the current audience model.

Changing policy

Projection, transport, and recipient changes create a new stream policy epoch. Pending deliveries made obsolete by the new policy are marked suppressed; already published events cannot be recalled.

Warning

Adding an audience does not backfill existing history. An older observation without a representation for every currently selected audience cannot be corrected or deleted. birdd reports that explicit backfill is required, but this MVP does not yet include a backfill command.

Inspect stream positions, missing sequences, conflicts, manifests, and restricted delivery queues with:

birdd --config birdd.toml federation-status --json

Restricted delivery states are delayed, due, wrapped, published, and suppressed. See Node configuration for every audience option.

Media storage

Media is optional. When enabled, Signal !saw commands can attach images and audio to observations. Events contain signed content references; the media bytes remain in S3-compatible object storage and never pass through relays.

Configuration

birdd currently supports S3 and S3-compatible storage:

[media]
enabled = true
endpoint = "http://127.0.0.1:3900"
public_endpoint = "https://birds.example.org/_birdd/media"
region = "garage"
bucket = "birdd-media"
access_key = "..."
secret_key = "..."

endpoint is the private S3 API used by birdd. Plain HTTP is accepted only for a loopback endpoint. Leave it unset to use the normal regional AWS endpoint.

public_endpoint is the HTTPS birdd media route other flocks use; it is not the S3 endpoint. It must end in /_birdd/media. Run the adapter server behind that route:

birdd --config birdd.toml serve --listen 127.0.0.1:8787

For production, keep credentials out of generated configuration. The NixOS module accepts mediaAccessKeyFile and mediaSecretKeyFile; standalone processes can use BIRDD_MEDIA_ACCESS_KEY_FILE and BIRDD_MEDIA_SECRET_KEY_FILE.

The NixOS full-node role enables the reference Garage object store, generates its application credentials, and configures the endpoint, region, bucket, and public media URL automatically. Set services.birddGarage.capacity to size its single-node storage allocation. Use a standalone birdd-garage module when adding the same managed storage to a custom role, or configure the media fields and credentials directly to use another S3-compatible service.

Limits

The defaults allow:

• max_attachments = 4 attachments per observation
• max_attachment_bytes = 15728640 bytes per attachment
• max_total_bytes = 31457280 bytes for the complete attachment set
• max_image_dimension = 4096 pixels per dimension after processing
• processing_timeout_seconds = 30 seconds for each media processor

Sanitization

Accepted images are JPEG, PNG, WebP, HEIC, and HEIF. They are decoded, auto-oriented, resized when needed, stripped of metadata, and stored as JPEG.

Accepted audio includes AAC, MP4 audio, MP3, Ogg, Opus, WAV, and M4A. It is decoded, stripped of metadata and video, and stored as Opus in an Ogg container.

Non-NixOS installations must provide vipsthumbnail and ffmpeg on the path of the Signal adapter. Unsupported formats, view-once messages, expiring messages, and attachment sets over the configured limits are rejected.

Federation

Media references use a SHA-256 digest. Remote requests are authorized with a short-lived NIP-98 event signed by the requesting flock. The origin serves only media referenced by a live local observation and disclosed to that requester. The receiver verifies the MIME type, byte length, and digest before caching it.

Only directly trusted flocks may fetch media. Restricted-audience recipients must also have a published event that disclosed that media reference. Set projection.media = false to remove media references from an audience.

Without public_endpoint, local media still works, but remote flocks cannot fetch it.

Note

The object store has no built-in garbage collection. Correcting or deleting an observation does not delete its old objects. Include object storage in your backup and retention plan.

Backups and recovery

A relay is not a backup. The node’s authoritative state includes its private key and SQLite database, plus any state owned by configured adapters.

What to back up

Always protect:

• The flock private key
• birdd.toml, or the NixOS configuration used to generate it
• The SQLite database
• Runtime credential files

When enabled, separately back up Signal account state, the Grist database, and media object storage. None of those are contained in birdd.sqlite.

Important

Test that a restored key produces the public key you expect. Restoring the database with a different key creates an identity mismatch that federation cannot repair.

Consistent SQLite backups

SQLite runs in WAL mode. A live node may have -wal and -shm files beside the main database, so copying only birdd.sqlite while a writer is active can lose committed data.

The simplest safe filesystem backup is to stop every process that can write the database, copy the complete state directory, and then restart them. This can include the main node, Signal adapter, Grist webhook, Grist reconciliation job, and operator commands.

For an online database backup, use SQLite’s backup command instead of copying the live database file:

sqlite3 /var/lib/birdd/birdd.sqlite \
  ".backup '/var/backups/birdd/birdd.sqlite'"

Copy the private key and configuration through a secret-aware backup mechanism, preserving restrictive ownership and permissions.

Restore

1. Stop all birdd processes that use the database.
2. Restore the database, private key, configuration, credentials, and enabled integration stores.
3. Restore ownership and restrictive permissions for the service account.
4. Start the node and verify its identity and health.
5. Run a synchronization pass and inspect federation status.

birdd --config birdd.toml identity
birdd --config birdd.toml list --json
birdd --config birdd.toml federation-status --json
birdd --config birdd.toml sync

On NixOS, use birdd-admin for these checks.

Export and rebuild

birdd export writes retained signed events as newline-delimited JSON. It does not include the private key, trust decisions, restricted delivery state, credentials, or adapter data, so it is not a complete backup format.

birdd rebuild regenerates materialized observations and flock metadata from retained events. It does not repair a damaged SQLite file or recreate all operational tables. Use it when retained events are intact but a derived view needs to be regenerated.

Monitoring

birdd run serves a read-only operational API on 127.0.0.1:8786 by default. It has no HTTP authentication and must remain on loopback or behind a trusted local operator portal.

Health

curl http://127.0.0.1:8786/health

A healthy response is HTTP 200 and confirms that the process can query SQLite. Database failure returns HTTP 503. Health does not prove that relays are reachable or federation is current.

The adapter processes have separate health endpoints:

• birdd serve: http://127.0.0.1:8787/health
• birdd-signal: http://127.0.0.1:8789/health

The Signal endpoint returns HTTP 503 while its signal-cli event stream is disconnected. The generic adapter endpoint only confirms that its HTTP server is running.

Metrics

Prometheus metrics are available from:

curl http://127.0.0.1:8786/metrics
curl http://127.0.0.1:8789/metrics

Useful signals include:

• Process and database availability
• Local and remote observation totals
• Synchronization successes, failures, and last-success time
• Per-relay connect, fetch, and publish state
• Federation gaps and conflicts
• Delayed, due, wrapped, published, and suppressed restricted deliveries
• Signal stream, command, response, and attachment failures

Runtime counters reset when a process restarts. Database-backed observation and federation gauges persist.

Federation status

Use the CLI when you need the complete stream and recipient view:

birdd --config birdd.toml federation-status
birdd --config birdd.toml federation-status --json

Pay attention to increasing gaps or conflicts, stale sync-success timestamps, all relays becoming unreachable, and growing due or wrapped delivery queues.

Admin API

The loopback API also exposes:

/admin/v1/overview
/admin/v1/federation
/admin/v1/trust
/admin/v1/extensions
/admin/v1/config

The configuration response is redacted, but the API should still be treated as operator-only. birdd-admin-web provides a UI for these endpoints when deployed behind an authenticated reverse proxy.

Logs and telemetry

Logs are structured JSON. Set RUST_LOG to adjust filtering:

RUST_LOG=birdd=debug birdd --config birdd.toml run

OpenTelemetry export is enabled when an OTEL_EXPORTER_OTLP_*_ENDPOINT environment variable is present. NixOS operators can inspect services with systemctl status and journalctl.

Troubleshooting

Start with the process logs, database health, and federation status:

curl http://127.0.0.1:8786/health
birdd --config birdd.toml federation-status --json

On NixOS:

systemctl status birdd
journalctl -u birdd
birdd-admin federation-status --json

The service does not start

Check that the configuration, database directory, and private key are readable and writable by the service account. Configuration rejects malformed TOML and unknown top-level fields. Empty credential files are also rejected.

The NixOS service has a condition on services.birdd.flockKeyFile. If the unit is inactive instead of failed, inspect its conditions:

systemctl show birdd -p Conditions

Credential paths in the NixOS module must be absolute, outside /nix/store, and configured in required pairs. Use birdd-admin instead of running ordinary CLI commands as root against the service database.

A port is already in use

The default loopback ports are:

• 8786 - node admin API
• 8787 - Grist webhook and media adapter
• 8788 - operator portal
• 8789 - Signal health and metrics

The admin, Signal metrics, and operator portal listeners reject non-loopback addresses. Put an authenticated reverse proxy in front of services that need a public route.

Federation is not moving

Run one explicit sync and inspect its output:

birdd --config birdd.toml sync

With no configured relays, sync is a successful no-op. With relays configured, at least one must be reachable. birdd run remains healthy and retries after a relay error, so /health alone does not detect federation failure.

Use federation-status --json to check missing sequences, conflicts, manifest completeness, and recipient delivery states. A restricted delivery stuck in wrapped often means at least one configured relay has not accepted it.

An observation cannot be corrected

Corrections and deletions fail when the observation is remote-owned, already deleted, stale, or missing a representation for a currently selected audience. Refresh stale data before retrying.

If the error says an observation predates a selected audience and explicit backfill is required, there is currently no supported backfill command. Do not modify signed history by hand.

Signal is disconnected

Check both services and the Signal-specific health endpoint:

systemctl status signal-cli birdd-signal
curl http://127.0.0.1:8789/health

The Signal base URL must be credential-free loopback HTTP. Attachment failures can also come from unsupported formats, configured size limits, missing vipsthumbnail or ffmpeg, or unavailable object storage.

Grist edits fail

Reconciliation requires a base URL, document ID, and API key. Webhook requests need the configured bearer secret. A stale Revision, remote observation, or invalid editable value is rejected by birdd. Changes to managed fields are not applied to observations.

Check the reconciliation timer and webhook service separately:

systemctl status birdd-grist-reconcile.timer birdd-grist-webhook
journalctl -u birdd-grist-reconcile -u birdd-grist-webhook

Rebuild carefully

birdd rebuild regenerates derived observations and metadata from retained events. It is not a SQLite repair command and is not a substitute for restoring a known-good backup.

The Birdd reference stack

The Birdd reference stack is the deployment used by the project nodes and the recommended starting point for a new operator. It is an opinionated serving suggestion, not a required architecture: the core birdd service does not depend on Grist, Signal, Garage, Authelia, Caddy, or the included observability tools.

The NixOS full-node role enables the complete stack, but it deliberately does not create third-party accounts, DNS records, or most secrets. It does initialize the local Garage service and its Birdd application credentials. This page describes the remaining operator setup. Third-party account procedures can change independently of Birdd; follow the linked upstream documentation when it differs from this overview.

NixOS configuration

A full node needs four public hostnames pointing to the VPS. The examples on this page use:

All four names must fit within the configured cookie domain. Allow inbound TCP ports 80 and 443 so Caddy can obtain certificates and serve HTTPS.

The following is a complete configuration skeleton. Replace every example domain, identity, path, and relay with deployment-specific values:

{inputs, pkgs, ...}: {
  imports = [inputs.birdd.nixosModules.full-node];

  time.timeZone = "America/Toronto";
  networking.firewall.allowedTCPPorts = [80 443];

  services.birdd = {
    package = inputs.birdd.packages.${pkgs.system}.birdd-with-docs;
    nodeName = "pallet";
    flockDisplayName = "Pallet Town Birders";
    flockKeyFile = "/var/lib/private/birdd/flock.key";
    relays = ["wss://relay.example.com"];

    signalAccountFile = "/var/lib/private/signal-cli/account";
    signalGroupIdFile = "/var/lib/private/birdd/signal-group-id";
    gristApiKeyFile = "/var/lib/private/birdd/grist-api-key";
    gristDocumentIdFile = "/var/lib/private/birdd/grist-document-id";
    gristWebhookSecretFile = "/var/lib/private/birdd/grist-webhook-secret";

    settings = {
      signal.base_url = "http://127.0.0.1:7583";
      grist.base_url = "http://127.0.0.1:8484";
    };
  };

  services.birddSignalCli.accountFile =
    "/var/lib/private/signal-cli/account";

  # Garage and Birdd media are enabled by full-node. Size this for the VPS.
  services.birddGarage.capacity = "50G";

  services.gristContainer = {
    domain = "office.example.org";
    authDomain = "auth.example.org";
    environmentFile = "/var/lib/private/grist/environment";
  };

  services.birddOperatorPortal = {
    siteName = "Pallet Town Birders";
    officeDomain = "office.example.org";
    authDomain = "auth.example.org";
    adminDomain = "admin.example.org";
    metricsDomain = "metrics.example.org";
    cookieDomain = "example.org";
    grafanaSecretFile = "/var/lib/private/grafana/secret-key";
  };
}

Keep credential files out of the Nix store. Garage creates its RPC and Birdd application credentials under /var/lib/private on first start. The other services expect root-managed files there and do not start when required files are absent. It is safe to apply the NixOS configuration before provisioning every integration; the affected units remain inactive until their files exist.

signal-cli

signal-cli owns the Signal account and protocol state. The birdd-signal adapter consumes its local event stream, accepts commands from one configured group, and sends responses through the same account. The NixOS module keeps the unauthenticated signal-cli HTTP API on loopback and installs signal-cli-admin for account administration under the correct service user.

The paved-road setup uses a dedicated phone number registered directly with signal-cli:

1. Apply the NixOS configuration while the account and group ID files are still absent. The Signal units will remain inactive.

2. Create services.birddSignalCli.accountFile containing the account number in E.164 format, such as +15551234567, with root ownership and mode 0600.

3. Register the number and complete its SMS or voice verification:

   signal-cli-admin register
   signal-cli-admin verify 123-456
   

4. Add that account to the flock’s Signal group. Receive once to update its local group state, then list the known groups as JSON:

   signal-cli-admin receive --timeout 10
   signal-cli-admin --output json listGroups
   

5. Find the intended group by name and copy its exact id value into services.birdd.signalGroupIdFile. The opaque base64 group ID, not the group name, is Birdd’s authorization boundary. Protect the file with root ownership and mode 0600.

6. Start the daemon and adapter, then check the adapter health endpoint:

   systemctl start signal-cli birdd-signal
   curl http://127.0.0.1:8789/health
   

Registration commonly requires a CAPTCHA token, and linking signal-cli as a secondary device uses a different procedure. See the upstream signal-cli quick start and device-linking guide. Do not run a separate root-owned signal-cli instance against the service data directory.

Once the health check succeeds, send !help in the authorized group. See Using Birdd with Signal for the member commands and Configuration for adapter settings.

Garage

Garage stores sanitized image and audio attachments behind an S3-compatible API. It is not exposed publicly: Birdd reads and writes objects through 127.0.0.1:3900, while authorized remote flocks fetch media through Birdd’s HTTPS /_birdd/media route.

The full-node role enables a single-node Garage deployment, generates its RPC secret and Birdd access key, assigns the configured storage capacity, creates the birdd-media bucket, and grants the application key read and write access. The generated credentials are connected to the Signal and Grist adapters through systemd credentials. Check provisioning with:

systemctl status garage birdd-garage-provision
birdd-garage-admin status
birdd-garage-admin bucket info birdd-media

Set services.birddGarage.capacity to the disk space Garage may use. The defaults are otherwise aligned with the Birdd media configuration: region garage, bucket birdd-media, and loopback endpoint http://127.0.0.1:3900. See Media storage for attachment limits and federation behavior.

The reference setup has replication factor one because it runs on one VPS. Garage is object storage, not a backup: losing that VPS loses the only copy of its media unless /var/lib/garage is backed up elsewhere. Consult Garage’s operations documentation before changing topology, replication, or storage layout.

Authelia

Authelia is the authentication boundary for Grist, birdd-admin-web, and Grafana. The reference configuration requires two-factor authentication and uses a local YAML user database.

Create these files under services.birddOfficeAuth.secretsDir, which defaults to /var/lib/private/authelia-birdd:

Generate password hashes with authelia crypto hash generate argon2 and follow Authelia’s user database format. The email address assigned to an operator must match the identity used when provisioning Grist. Keep all three files root-owned and private.

After deployment, sign in at the authentication hostname and request WebAuthn enrollment. The reference stack uses Authelia’s filesystem notifier, so retrieve the registration link from /var/lib/authelia-birdd/notifications.txt on the node instead of waiting for email. Authelia maintains its own persistent database, so include it in the node backup plan.

Grist

Grist provides the spreadsheet interface for reviewing and editing local observations. It runs from a digest-pinned upstream image, stores its data under /var/lib/grist, and trusts only identities supplied by the Authelia and Caddy boundary.

Create services.gristContainer.environmentFile with a random session secret and the email address of the initial operator:

GRIST_SESSION_SECRET=replace-with-a-long-random-secret
GRIST_DEFAULT_EMAIL=operator@example.org

The module supplies the remaining reference-stack environment, including the public URL, forced login, forwarded authentication, service accounts, formula sandboxing, and webhook restrictions. See Grist’s self-hosting documentation before changing those defaults.

Create a separate random bearer secret at services.birdd.gristWebhookSecretFile. Once Grist and Authelia are reachable, run birdd grist-provision from the same reviewed Birdd package revision as the deployed node:

birdd grist-provision \
  --base-url http://127.0.0.1:8484 \
  --admin-email operator@example.org \
  --public-webhook-url https://office.example.org/_birdd/grist-webhook \
  --webhook-secret-file /var/lib/private/birdd/grist-webhook-secret \
  --api-key-output-file /var/lib/private/birdd/grist-api-key \
  --document-id-output-file /var/lib/private/birdd/grist-document-id

Run this command as root because it writes root-managed credential files. It creates or updates the Birdd workspace, document schema, service account, and webhook. Start reconciliation after provisioning and verify both integration units:

systemctl start birdd-grist-reconcile.service birdd-grist-webhook
systemctl status birdd-grist-reconcile.timer birdd-grist-webhook

Re-run provisioning after an upgrade that changes the generated Grist schema or after changing loaded extension projections. See Using Grist for editor behavior.

birdd-admin-web

birdd-admin-web is the operator portal. It reads Birdd’s loopback-only admin API, presents node and federation status, and embeds the Grist and Grafana interfaces. It receives neither the flock key nor adapter credentials.

The services.birddOperatorPortal configuration starts it automatically. There is no separate account setup: Caddy passes the authenticated Authelia identity to the portal. Open the configured administration hostname after Authelia is working.

Caddy

Caddy is the only public HTTP entry point in the reference stack. It obtains TLS certificates and routes the office, authentication, administration, and metrics hostnames to loopback services. The Grist webhook and media paths are routed separately from browser authentication and enforce their own request authorization.

Create DNS records for all four hostnames before starting Caddy, permit inbound ports 80 and 443, and make sure no other service occupies those ports. Do not expose the loopback application ports directly.

Prometheus and Grafana

Prometheus collects Birdd, adapter, host, and service metrics. Grafana serves a pre-provisioned node dashboard through the authenticated metrics hostname. Alloy sends metrics to Prometheus, traces to Tempo, and systemd journal logs to Loki; the operator portal uses the resulting data without a separate setup wizard.

Create services.birddOperatorPortal.grafanaSecretFile containing a strong random value with root ownership and mode 0600. Grafana’s own login form is disabled because the reference stack uses the Authelia identity supplied by Caddy.

Use the checks in Monitoring to verify each local target. The Garage, Prometheus, Grafana, Loki, Tempo, and Authelia data directories are persistent operational state; decide how much history to preserve when following Backups and recovery.

Extensions

Extensions attach domain-specific data to generic observations without changing the event protocol or requiring every node to understand that data.

Envelopes

An observation stores a map from namespaced extension IDs to versioned JSON envelopes:

{
  "org.example.observations/rainfall": {
    "version": 1,
    "data": {
      "millimeters": 12,
      "condition": "steady"
    }
  }
}

Every envelope has a stable extension ID, a positive version, and an object-shaped data value. An ID must contain /, have non-empty text on both sides, use a namespace containing . or :, and contain no whitespace. Reverse-domain namespaces are conventional but not centrally registered:

org.example.observations/rainfall
com.example.community/trail-condition

An extension publisher must not change the meaning of an existing ID and version. Publish an incompatible interpretation under a new version.

Opaque compatibility

Every node validates the envelope structure. A node does not need a manifest to store, export, rebuild, or forward a valid envelope. Unknown IDs, unsupported positive versions, and unknown keys remain opaque data.

A local manifest adds typed adapter behavior:

• Signal input profiles
• Signal detail and creation-receipt templates
• Dedicated Grist columns and edit conversion
• Numeric measures for grouped Signal search
• Extension support in flock metadata

Manifests are local configuration, not federated schemas. Flock metadata advertises only extension IDs and versions. Each receiving node independently chooses which manifests to trust and install. Advertised support never gates replication.

Removing a manifest does not remove extension data. Its envelopes become opaque again and continue to be stored and forwarded.

Install manifests

Birdd loads no extensions implicitly. Configure an ordered list of strict TOML manifest files and restart the process:

[extensions]
manifests = ["extensions/rainfall.toml"]

Relative paths are resolved from the directory containing birdd.toml. Birdd validates the complete registry before opening the node. Invalid manifests, duplicate extension IDs or profile names, generated Grist column collisions, and invalid measure references prevent startup.

The source tree includes optional bird examples:

• extensions/birds-taxon.toml projects common_name as a Species column.
• extensions/birds-count.toml provides a positive count input profile, Count column, and seen measure.

Nix packages install these examples under share/birdd/extensions. See Node configuration for standalone and NixOS paths.

Create opaque data

The CLI can create arbitrary envelopes whether or not their manifests are loaded:

birdd --config birdd.toml create "Creek flooding" \
  --extension '{"org.example.observations/water":{"version":1,"data":{"depth_cm":12}}}'

The argument must be a JSON object mapping extension IDs to complete envelopes. A manifest is needed only for typed adapter behavior.

Disclosure

Audience projection copies complete extension envelopes. Field declarations in a local manifest are not a privacy filter. Restricted audiences can include all extensions or exact IDs, while public audiences always include every envelope. Review the actual envelope data before publishing it.

The authoritative projection behavior is in Federation and audiences, and accepted configuration values are in Node configuration.

Continue with Authoring extensions, Signal integration, or Grist integration.

Authoring extensions

An extension manifest declares local typed behavior using data, not executable code. Adding an extension does not require rebuilding birdd.

Read Extensions first for envelope identity, compatibility, installation, and federation semantics.

Complete example

This manifest declares a rainfall envelope, Signal presentation, two Grist columns, a numeric search measure, and a Signal input profile:

manifest_version = 1
extension_id = "org.example.observations/rainfall"
supported_versions = [1]

[signal]
detail_template = """
Rainfall: {{ data.millimeters }} mm
{% if data.condition is defined %}Condition: {{ data.condition }}{% endif %}
"""
receipt_template = "{{ data.millimeters }} mm"

[[projected_fields]]
json_key = "millimeters"
grist_column_label = "Rainfall mm"
grist_column_type = "Int"
editable = "editable"

[[projected_fields]]
json_key = "condition"
grist_column_label = "Condition"
grist_column_type = "Text"
editable = "editable"

[[measures]]
name = "total"
json_key = "millimeters"
aggregate = "sum"
missing = 0

[[profiles]]
name = "rainfall"
summary = "Rainfall observation with a measured depth"
example = "!saw Rain gauge :: rainfall millimeters=12"

[[profiles.fields]]
name = "millimeters"
extension_id = "org.example.observations/rainfall"
extension_version = 1
json_key = "millimeters"
value_type = "positive_u32"
required = true

Save it as extensions/rainfall.toml, add it to the node configuration, and restart birdd:

[extensions]
manifests = ["extensions/rainfall.toml"]

The profile can then create the extension:

!saw Rain gauge :: rainfall millimeters=12

It produces:

{
  "org.example.observations/rainfall": {
    "version": 1,
    "data": {
      "millimeters": 12
    }
  }
}

The optional condition key can be added through its editable Grist column or through a complete CLI envelope.

Manifest fields

Manifests are strict TOML. Unknown fields and unsupported values prevent node startup.

The manifest ID is both the wire identity and the registry collision boundary. Only one loaded manifest may declare an extension ID.

Signal templates

The optional [signal] table accepts one or both MiniJinja templates:

Templates receive only this context:

{
  "extension": {
    "id": "org.example.observations/rainfall",
    "version": 1
  },
  "data": {
    "millimeters": 12,
    "condition": "steady"
  }
}

Access envelope fields through data and metadata through extension:

Rainfall: {{ data.millimeters }} mm
Version: {{ extension.version }}

Undefined values are strict. Guard optional keys explicitly:

{% if data.condition is defined %}Condition: {{ data.condition }}{% endif %}

Birdd compiles templates once when loading the registry. A syntax error, blank template, template longer than 4096 bytes, or empty [signal] table prevents startup. Rendering uses a 10,000-instruction fuel limit and recursion limit of 32. Detail output is limited to 1024 bytes per extension and receipt output to 256 bytes.

Signal is plain text, so auto-escaping is disabled. Before insertion, Birdd replaces control characters, collapses whitespace within each non-empty output line, and removes blank lines. A receipt is collapsed to one line. The final message remains subject to Signal’s overall message limit.

Formatting is applied only to versions listed in supported_versions. Missing templates and unsupported versions fall back to the extension ID in detail output. A runtime template error, including a missing unguarded key or exhausted resource limit, is logged and uses the same fallback instead of failing the Signal command. Receipt errors omit only the receipt text.

Templates are local presentation policy. They are not included in observation events or flock metadata, and a receiving node uses its own installed manifest. See the MiniJinja template syntax for expressions, conditions, loops, and filters.

Projected fields

Each [[projected_fields]] entry maps one top-level extension data key to one Grist column:

Text accepts JSON strings. Int accepts non-negative integers that fit in a u32. An editable field can be created or corrected from Grist; a read-only field is projection-only.

Birdd generates a column ID by replacing every non-ASCII-alphanumeric character in the extension ID and JSON key with _, then joining them with __:

org.example.observations/rainfall + millimeters
-> org_example_observations_rainfall__millimeters

Generated IDs must be unique across the complete loaded registry.

Measures

Each [[measures]] entry exposes a numeric value to grouped Signal search:

The configuration reference is the extension ID, a ., and the measure name:

org.example.observations/rainfall.total

Use a simple dot-free measure name because references split at the final .. Malformed fields and unsupported envelope versions contribute nothing; a missing envelope contributes the configured missing value.

Profiles

Each [[profiles]] entry declares a Signal input profile:

Each [[profiles.fields]] entry accepts:

A profile may populate multiple loaded extensions. Birdd groups fields by extension ID and version when constructing envelopes. Declare at least one field; an empty profile has no useful Signal input and can make inference ambiguous.

Use lowercase ASCII token names for profiles and fields. Avoid whitespace and = because Signal input is whitespace-delimited name=value text. Use names that distinguish profiles by their required fields when relying on inference.

Version limitations

One manifest can advertise several versions, but projections and measures are shared across all of them. List multiple versions only when their projected keys, types, and measure meanings remain compatible.

Grist creates new envelopes using the first supported version. Since versions must be ascending, this is the lowest listed version. There is no migration or per-version projection mechanism.

Unknown positive versions remain valid opaque envelopes. Typed Signal and Grist behavior ignores versions that are not listed.

Validation boundary

Manifests describe the typed boundaries Birdd currently implements:

• Signal profile input validates positive_u32 values.
• Signal templates render locally interpreted envelope data.
• Grist projection and edits validate Text and unsigned Int values.
• Measures extract unsigned integers from declared Int fields.

Manifests are not JSON Schema. They do not validate arbitrary envelopes created through the CLI or received through federation, and they cannot express ranges, cross-field rules, free-form parsers, or executable transformations. Such envelopes remain subject only to core ID, version, and object-shape validation.

Test and deploy

Before enabling a manifest, check:

• Every supported version and malformed typed value
• Generated Grist column IDs and collisions
• Signal profile parsing, inference, and generated envelopes
• Signal detail and receipt rendering, including missing and malformed values
• Measure behavior for values, missing envelopes, malformed fields, and unsupported versions
• Grist projection, creation, editing, and unknown-key preservation
• Restricted audience allowlists and the complete envelope contents

Restart every process that consumes the node configuration. Then inspect and publish the resulting capabilities:

birdd --config birdd.toml grist-schema
birdd --config birdd.toml metadata
birdd --config birdd.toml sync

If projected Grist columns changed, run grist-provision before editors use them. See Grist integration.

Signal integration

Signal uses locally loaded profiles for typed extension input, MiniJinja templates for presentation, and measures for grouped search. It never executes or renders definitions received through federation.

Input profiles

Members can inspect the profiles installed on their node:

!profiles
!profiles rainfall

A profile turns whitespace-delimited name=value fields into extension envelopes:

!saw Rain gauge :: rainfall millimeters=12

When the supplied field names identify exactly one profile, its name can be omitted:

!saw Rain gauge :: millimeters=12

Unknown fields, missing required fields, invalid values, duplicate fields, and ambiguous inference fail without creating an observation. A plain !saw command creates no extension envelopes.

After profile-based creation, each populated extension may append its optional receipt_template output. For example, the bird count manifest declares x{{ data.count }}, producing:

Recorded: Common Loon x3

The complete command syntax belongs in Using Signal. The manifest declarations belong in Authoring extensions.

Observation presentation

A manifest’s detail_template renders in observation cards returned by !today, !recent, and !obs, and in the detailed !show response. The bird count manifest renders:

Count: 3

Compact list search remains compact, while grouped search uses configured measures rather than detail templates.

Signal falls back to an Extensions: ... line for unknown IDs, unsupported versions, missing detail templates, empty output, and rendering errors. A bad extension value therefore cannot fail an observation query or hide the fact that an envelope exists. Use event export or Grist’s read-only Extensions JSON column to inspect the complete envelopes.

Templates are compiled and bounded when the local registry loads. The context, syntax, limits, and failure behavior are defined in Authoring extensions.

Grouped-search measures

A loaded manifest can expose a numeric measure. For the rainfall example, the qualified reference is:

org.example.observations/rainfall.total

The node configuration may include it as a numeric grouped-search field:

[extensions]
manifests = ["extensions/rainfall.toml"]

[signal.views.search]
layout = "grouped"
header = "Found {observation_count} {observation_count:plural(observation)}"

[signal.views.search.group]
by = "subject"
normalize = "casefold"
label = "latest"
heading = "{group_label:highlight}"
sort_by = "core.last_seen"
sort_order = "descending"

[[signal.views.search.group.fields]]
label = "Rainfall"
measure = "org.example.observations/rainfall.total"
format = "number"
singular = "millimeter"
plural = "millimeters"

Birdd filters observations for local visibility and direct trust before grouping. It then sums supported numeric values in each normalized subject group. A missing envelope contributes the manifest’s missing value. An unsupported version or malformed field contributes nothing. Arithmetic overflow fails the view rather than wrapping.

Core date and text measures can be included alongside extension measures. The authoritative grouped-view configuration is in Node configuration.

Grist integration

Grist combines typed columns generated from locally loaded manifests with a read-only JSON fallback containing every extension envelope.

Inspect the schema for the configured node with:

birdd --config birdd.toml grist-schema

Without configured manifests, the schema contains no extension-specific columns.

Projected columns

Each [[projected_fields]] declaration maps one top-level extension data key to a Grist column. For example, the optional bird manifests generate:

These are examples, not built-in columns. Column IDs are derived deterministically from the extension ID and JSON key as described in Authoring extensions.

Projection fails closed. A projected cell is null when:

• The envelope is absent
• Its version is not supported by the local manifest
• Its value does not match the declared Grist type

Text requires a JSON string. Int requires a non-negative integer that fits in a u32.

Extensions JSON

Extensions JSON contains the canonical JSON representation of the complete extension map, including supported envelopes, unsupported versions, unknown extension IDs, and keys without projected columns.

The column is read-only. Editing it does not update an observation. Add an editable projected field when a key needs a typed Grist edit path.

Creation and editing semantics

Editable projected cells in a new Our_Observations row create extension envelopes. Birdd uses the first version in the manifest’s ascending supported_versions list. Blank projected cells do not create keys, and an extension with no populated keys is omitted.

For an existing local observation, changing an editable projected cell creates a signed correction. Setting a cell to null removes only that JSON key. Birdd clones the complete extension map before editing, preserving unknown keys and unrelated envelopes.

If an existing envelope uses an unsupported version, Birdd ignores projected edits to it rather than rewriting data it cannot interpret. Remote observations in Following_Remote remain read-only.

These rules describe extension conversion. General row creation, deletion, stale revisions, and table usage are documented in Using Grist.

Webhook requirements

The managed webhook watches core editable columns and extension columns marked editable. A no-op edit creates no event, and stale revisions are rejected.

Handling corrections requires the webhook secret. Supporting new-row creation also requires grist.base_url, grist.document_id, and grist.api_key because Birdd writes the generated observation ID, revision, canonical extension JSON, and sync status back to the row after recording the observation.

Schema changes

Changing the loaded manifest set changes the deterministic Grist schema. After deploying manifests and restarting the relevant process, run birdd grist-provision to apply additive changes:

• Missing tables and columns are created.
• Changed column labels are updated.
• Existing columns with incompatible types stop provisioning.
• Extra tables and columns are retained.
• The managed webhook’s watched columns are updated.

Removing a manifest does not delete old Grist columns or stored envelope data. The columns are no longer managed by the generated schema.

Test generated IDs and forward/reverse conversion before editors use a changed field. Run reconciliation after successful provisioning.