Storage backends

TopGun keeps your data in an embedded store inside the server: there is no database to set up, run or connect to. Data survives restarts on the path you give it. The backend is selected at startup via STORAGE_BACKEND; two values are accepted: redb (the default) and null (ephemeral, tests only).

Choosing a backend

redb (embedded default)

Your data stays on a path you choose and survives restarts, with no separate database, no Docker and no connection pool. The server keeps the store in a data directory (./topgun-data by default), served in-process by redb, and keeps acknowledged writes that are not yet in the store in a write-ahead log directory (./topgun-wal by default).

Starting with redb

# Default — no configuration needed
pnpm start:server

# Or with explicit paths
TOPGUN_DATA_DIR=/var/lib/topgun/data \
TOPGUN_WAL_DIR=/var/lib/topgun/wal \
pnpm start:server

Configuration

The container image sets TOPGUN_DATA_DIR=/data/topgun-data and TOPGUN_REDB_PATH=/data/topgun.redb, and its write-ahead log lives in /data/topgun-wal, so a volume mounted at /data holds everything. Every start logs store location: data directory … with the directory it uses and where that came from (data_dir_source).

Upgrading from a store file

Earlier versions kept the store as one file at TOPGUN_REDB_PATH. The first start of this version moves it into the data directory by itself: nothing to run by hand, and the old file is kept.

• Before upgrading, copy the TOPGUN_REDB_PATH file to another disk. The migration keeps it, but only a copy elsewhere survives the loss of this disk.

• The migration runs at start-up, before the server listens. It needs 1.5 × the size of the old file plus 64 MiB free on the data directory’s file system, and otherwise refuses to start without changing anything. It logs store format: migrating … into … from the table-per-map format, a store migration progress line at least every 10 s, and store format: migration complete.

• It is resumable: if the process is stopped or killed during the migration, the next start continues from where it got to. A health probe that restarts the server converges as long as each run lives a few seconds; for a large store, give the start-up probe enough time.

• The copy is verified before anything is renamed. Then the old file is kept as {TOPGUN_REDB_PATH}.pre-migration — every start logs store format: the pre-migration store file is kept at …; delete it once this store is confirmed — and a small marker file is written at TOPGUN_REDB_PATH.

• Do not delete the marker file while an older server binary may be started with that path. An older binary refuses to start on it (it exits with an error before it listens, and changes no file); without it, an older binary would create an empty store and serve empty maps.

• Rolling back is only a return to the kept file, and it loses on the server every write the new version acknowledged:

  1. Stop the new server. It must exit 0 and log write-behind shutdown: rollback hints written with drained=true; that stop writes, into the WAL directory, the hints the old server reads to know how far the log was stored. After a crash, a non-zero exit or drained=false (the stop could not store every buffered write in time), start the new server again and stop it until the line reads drained=true.
  2. Move the data directory away. Leave the WAL directory in place.
  3. Delete the marker file at TOPGUN_REDB_PATH.
  4. Rename {TOPGUN_REDB_PATH}.pre-migration back to TOPGUN_REDB_PATH.
  5. Start the old server.

  There is no tool that exports the new format back. Never point an older binary at the data directory’s engine.redb itself: it would open it as a store with no maps and serve empty maps.

• Not a rollback target: the first version with a data directory (built from 6a030bb6) refuses a data directory this version has opened — it exits with store format: refusing to open: unknown metadata … before it listens and changes no file of the WAL directory. There is no tool to go back to it.

• If the server refuses to start with a store location: or store migration: line, see Troubleshooting.

Notes

• Data persists across server restarts as long as the data directory and the WAL directory exist.
• Single-process: not shared between multiple server instances.
• For Docker deployments, mount the path as a volume so data survives container recreation.

Backup and restore

• Back up: stop the server (SIGTERM). The stop must exit 0 and log write-behind shutdown: rollback hints written with drained=true; after drained=false, start the server again and stop it until the line reads drained=true. Then copy both directories together: the whole data directory (TOPGUN_DATA_DIR) recursively and the WAL directory (TOPGUN_WAL_DIR). In the container image these are /data/topgun-data and /data/topgun-wal.
• Restore: with the server stopped, put both back from the same backup, then start the server. A data directory older than the WAL directory beside it is refused at start with nothing written (WAL directory check: refusing to start: the data directory is older than the WAL directory …, see Troubleshooting).
• Not supported: a copy of a running server’s directory. The data directory alone is not the server’s state while it runs, because acknowledged writes that are not yet in it live in the WAL directory, and a file copied while the server writes to it can mix pages from different moments.

Postgres (removed)

The Postgres backend has been removed. There is no migration tool: data kept by the Postgres backend remains available with the server built from the tag postgres-backend-last. See the changelog.

null (ephemeral, test only)

# In-memory ephemeral backend — data lost on restart
# Use in unit tests / CI only
STORAGE_BACKEND=null pnpm start:server

The null backend is used in the TopGun integration test suite for deterministic, stateless test runs. "null" is one of the two accepted values.

Tombstone reclamation and legacy stores

When an OR-Map entry is removed, the server keeps a small tombstone so the deletion converges across every replica. Tombstones are reclaimed automatically once every tracked client has synced past them — no operator action is required on a store created by a current server.

Stores first written by a pre-epoch server may still carry a legacy tombstone corpus in an older on-disk form. That corpus is excluded from garbage collection by design: it is never eligible for automatic reclamation, so it can never resurrect a deleted entry. It is inert and safe to leave in place. On startup the server logs a one-line WARN reporting how many legacy tombstone rows it found.

The supported way to reclaim a legacy corpus is to recreate the datastore: stop the server, start it against a fresh store (a new, empty TOPGUN_DATA_DIR with a TOPGUN_REDB_PATH that names no file, and a new TOPGUN_WAL_DIR), and let clients reconnect. Because every client holds its own authoritative local state, CRDT delta-sync restores the live data on reconnect while the stale legacy tombstones are simply not recreated. There is intentionally no in-place “drop legacy tombstones” switch — dropping a tombstone while a client still holds the matching add is exactly the resurrection hazard the tombstone exists to prevent.