Troubleshooting

This guide covers common issues you may encounter when using TopGun, organized by category.

CRDT Merge Edge Cases

Data reverted after reconnect

Symptom: After going offline and reconnecting, some of your local changes appear to revert to an older state.

Cause: TopGun uses Hybrid Logical Clocks (HLC) to order writes. Each write carries a timestamp of {millis, counter, nodeId}. When two clients write to the same field, the write with the higher HLC timestamp wins (Last-Write-Wins semantics). If your local clock is behind the server or another client, your writes may have lower timestamps.

Solutions:

Ensure system clocks are reasonably synchronized (NTP). TopGun tolerates moderate clock skew, but large drift (minutes+) can cause unexpected ordering
Check that nodeId is unique per client instance — duplicate node IDs cause HLC collisions. A custom nodeId must also not contain :, | or # — the client throws at construction if it does
Remember that LWW is per-field, not per-record. Only the conflicting fields revert; other fields retain their values

Concurrent edits lost

Symptom: Two users edit different items in a collection simultaneously, but one user’s additions disappear after sync.

Cause: If you are using LWWMap for a collection where items are added and removed concurrently, LWW semantics may cause one write to overwrite another when they modify the same key.

Solutions:

Use ORMap (Observed-Remove Map) for collections where concurrent additions and removals are expected. ORMap uses unique tags to track each add operation, allowing concurrent adds to coexist
Reserve LWWMap for single-document fields where last-write-wins is the desired behavior
See the Data Structures guide for details on when to use each CRDT type

Stale data after merge

Symptom: After a sync cycle completes, some records still show old values even though newer writes exist on the server.

Cause: MerkleTree-based delta sync compares hash digests to determine which records need syncing. If the sync was interrupted or only partially completed, some records may not have been transferred yet.

Solutions:

Wait for the sync cycle to fully complete. Check the SyncEngine connection state — it should transition to connected and then synced
For large datasets, sync happens in batches. Initial sync after a long offline period may take several seconds
If the issue persists, the client’s local MerkleTree may be corrupted. Clearing IndexedDB storage and re-syncing will rebuild it from the server. Note: call LWWMap.prune() to remove tombstones if the map has accumulated many deleted entries before clearing

WebSocket Connection Issues

Connection drops repeatedly

Symptom: The client connects briefly then disconnects in a loop, or never establishes a stable connection.

Solutions:

Verify the server address format: use ws:// for local development and wss:// for production (TLS)
Check that the server is actually running and listening on the expected port
If behind a reverse proxy (nginx, Cloudflare), ensure WebSocket upgrade headers are forwarded. Common nginx configuration:
```
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```
Increase proxy timeout settings. Many proxies close idle WebSocket connections after 60 seconds. TopGun sends heartbeats, but aggressive proxy timeouts can still cause drops

Sync not working behind corporate proxy

Symptom: The application works on home WiFi but fails to sync in a corporate environment.

Cause: Many corporate proxies block WebSocket upgrade requests (HTTP 101 Switching Protocols) or filter non-HTTP traffic.

Solutions:

Use wss:// (WebSocket over TLS) which is harder for proxies to inspect and block
If the proxy terminates TLS, you may need to configure the proxy to allow WebSocket upgrades
Check with your network administrator whether WebSocket traffic is explicitly blocked
TopGun clients automatically queue writes when offline and sync when connectivity is restored, so the application remains usable even if sync is intermittently blocked

Authentication failed

Symptom: Connection is rejected with an authentication error, or the server returns a 401/403 status.

Cause: TopGun uses JWT tokens for authentication. The token must contain a sub (subject) claim per RFC 7519. The server validates the sub claim is present and non-empty; no custom userId field is required or supported.

Solutions:

Verify your JWT token has not expired. Check the exp claim
Ensure the token contains a sub claim with the user ID. TopGun uses only the standard sub claim — there is no custom userId field
Verify the token is signed with the same secret or key pair configured on the server (JWT_SECRET env var)

Check that the token is supplied via setAuthToken or setAuthTokenProvider after construction:

import { IDBAdapter } from '@topgunbuild/adapters';

const client = new TopGunClient({
  serverUrl: 'wss://your-server.com',
  storage: new IDBAdapter(),
});
client.setAuthToken('your-jwt-token');
// Or for refreshing tokens:
// client.setAuthTokenProvider(async () => fetchFreshToken());

IndexedDB Persistence

Data gone after browser update

Symptom: After a browser update or clearing browser data, locally stored records are gone.

Cause: IndexedDB storage can be evicted by the browser under storage pressure, especially if the site is not installed as a PWA or if the user has not interacted with it recently.

Solutions:

Request persistent storage to reduce eviction risk:

if (navigator.storage && navigator.storage.persist) {
  const granted = await navigator.storage.persist();
  console.log('Persistent storage granted:', granted);
}

Inform users that critical data is synced to the server. Local storage is a cache for offline access, not the sole source of truth
Consider using a Service Worker to register the app as a PWA, which gives it higher storage priority

Slow startup with large dataset

Symptom: The application takes several seconds to become interactive when there is a lot of data in IndexedDB.

Cause: On startup, the IDBAdapter loads stored records into memory to hydrate the local CRDT state.

Solutions:

Use selective sync to limit which data is loaded at startup. Only subscribe to the maps and queries your initial view needs
Paginate large collections using query limits rather than loading everything at once
Consider using fields projection in queries to load only the fields you need for list views, then fetch full records on demand

Server Startup

Port already in use

Symptom: The server fails to start with “Address already in use” error.

Solutions:

Check if another TopGun instance or other service is using the port:
```
lsof -i :8080
```

Kill the existing process or choose a different port:

PORT=8081 cargo run --bin topgun-server --release

If running in development, ensure previous server processes are terminated before restarting

Server refuses to start: removed storage setting

Symptom: The server exits at startup with an error whose message begins with one of these sentences:

STORAGE_BACKEND=postgres is no longer supported: the Postgres backend has been removed.
DATABASE_URL is set, but the Postgres backend has been removed and this server does not read it.

Cause: The server keeps its data in an embedded store, because the Postgres backend has been removed. It refuses STORAGE_BACKEND=postgres, and it refuses to start while DATABASE_URL is set (an empty value counts as set), so a leftover variable cannot silently send your data to a different store than you expect.

Solutions:

Unset DATABASE_URL (an empty value counts as set), and remove STORAGE_BACKEND=postgres if you set it. The server then starts on the embedded store (STORAGE_BACKEND=redb, the default). Check your shell, .env files, compose files and container environment.
Data kept by the old Postgres backend is not read by this server. There is no migration tool: it remains available with the server built from the tag postgres-backend-last.
See Storage Backends for the embedded store, its paths and backup
If the server starts on the default embedded store and you see write failures, check that the data directory is writable (defaults to ./topgun-data; override with TOPGUN_DATA_DIR)

Server refuses to start with a store location or store migration line

Symptom: The server exits with code 1 at startup, before it listens, with an error that begins with store location: or store migration: (a refusal is also logged as one ERROR line).

Cause: The server keeps its store in a data directory (TOPGUN_DATA_DIR) and, once, migrates the store file of earlier versions (TOPGUN_REDB_PATH) into it. When the two paths do not say unambiguously which store to serve, it refuses rather than guess. A refusal creates, moves and deletes nothing: the old store file, the kept .pre-migration file and the marker file stay exactly as they were, and the data directory’s records are unchanged. Fix the cause and start again. Below, {D} is the data directory, {L} the path TOPGUN_REDB_PATH names (default ./topgun.redb), and {K} the kept file {L}.pre-migration. The store location: data directory {D} line logged just before the refusal shows the paths the server used (data_dir, data_dir_source, legacy_path).

Solutions, by the line the server logged:

store location: {VAR} is set but empty — TOPGUN_DATA_DIR or TOPGUN_REDB_PATH is set to an empty value. Unset it to use the default, or set a path.
store location: TOPGUN_DATA_DIR ({D}) and TOPGUN_REDB_PATH ({L}) name the same place or one lies inside the other — give the data directory and the old store path different places, neither inside the other, for example /data/topgun-data and /data/topgun.redb.
store location: {D} is not a store directory, an empty directory or absent — the data directory holds files but no store (engine.redb), or is a file. Point TOPGUN_DATA_DIR at an empty or absent directory, or at the directory that holds your store.
store location: two stores found: {D} holds a store in the data-directory format and {L} is a table-per-map store file; the server serves only one. Move one of them aside: rename {L} to keep the data-directory store, or move {D} away to migrate {L} again — an old store file was put back beside an existing data directory (for example restored from a backup, or after a rollback that left the data directory in place). Decide which one holds the data you want. To keep the data directory, rename the file at {L}; the server then writes its marker file there. To serve the old file, move {D} away; the next start migrates {L} into a new data directory.
store location: {L} is an empty file; an older binary would open it as an empty store. Delete it, and this server writes the marker file there — delete the empty file; the next start writes the marker file in its place.
store location: {L} says the store was moved to a data directory, but {D} holds no store — {L} holds the marker file, but TOPGUN_DATA_DIR points at a directory without a store. Point TOPGUN_DATA_DIR at the directory the marker file names (its second line says which), or restore that directory from your backup. The marker file is never a store, so the server does not migrate it.
store location: a pre-migration store file is kept at {K}, but {D} holds no store; restore the data directory, or rename {K} to {L} to migrate it again — the data directory is missing after a migration. Restore it from your backup, or rename {K} to {L} so the next start migrates that file again (writes acknowledged after the first migration are not in it).
store migration: a pre-migration file already exists at {K}; move it away before migrating {L} — a previous migration’s kept file is in the way. Move {K} somewhere else (keep it until the new store is confirmed), then start again.
store migration: {L} is not a table-per-map store file — the file at {L} is neither a store of an earlier version nor the marker file (an empty file counts). Point TOPGUN_REDB_PATH at your old store file, or move the file away to start with an empty store.
store migration: refusing to migrate {L}: … table(s) are not table-per-map store tables: … — the old file holds tables that no TopGun version created (at most 20 are named). The file is left as it was. Check that TOPGUN_REDB_PATH names the TopGun store file and not another redb database.
store migration: refusing to migrate {L}: its path is not UTF-8 — move the file to a path whose name is valid UTF-8 and point TOPGUN_REDB_PATH there.
store migration: refusing to start: {available} bytes are free on the file system of {D}, {required} are needed (1.5 × the size of {L}, plus 64 MiB); nothing was changed — free disk space on the data directory’s file system (or point TOPGUN_DATA_DIR at a larger one) and start again. While the kept file exists the node needs about twice the disk of the store.
store migration: refusing to resume the migration from {source}: {reason} — a migration was interrupted, and what is at the paths now no longer matches it. By reason:
- the store is being migrated from that file, not from {L} — TOPGUN_REDB_PATH changed during the migration. Set it back to {source}; the next start resumes.
- {L} is not the table-per-map store file the copy reads — the old file was moved or replaced during the migration. Put it back at {L}; the next start resumes.
- a pre-migration file already exists at {K} — a kept file appeared during the migration. Move {K} away; the next start resumes.
- both {L} and {K} hold something; the copy is verified and only one of them may — move aside whichever of the two is not the file the migration started from; the next start finishes the migration.
- neither {L} nor {K} holds the legacy file the verified copy was taken from — put the old store file back at {L} or {K}, or move {D} away and start from your backup.
store migration: refusing to resume the migration from an unknown source: the mark cannot be read (…) — the data directory holds an unfinished migration this server cannot read. Move {D} away and start again: the old file at {L} is migrated anew.
store migration: verification failed for {table}: {difference}; {D} is left unfinished and is refused until it is removed — the copy in the data directory does not match the old file. The old file is untouched. Move {D} away and start again to migrate anew; if the failure repeats, keep the logs and the old file and report it.
store location: cannot write the stub at {L} — the server could not write its marker file at {L}: check that the directory of {L} is writable by the server.
store open: {file} does not open with this version: redb panicked while opening it (…), as it does on a file left by a crash while the file grew; do not delete the file, its records are probably intact; to bring the node back now, restore the data directory and the WAL directory from the same backup and keep this file — the embedded store failed inside its own open on this file, and it fails the same way at every start. The known cause is a crash (power loss, kill -9) while an earlier version of the server was growing the store file: the store’s header reached the disk and the file’s growth did not. The current version makes the growth durable before the header can name it, so it no longer leaves such a file, but it still refuses one left by an earlier version. The records in the file are probably intact, but this version of the store cannot open it. Do not delete or edit the file. To bring the node back now, restore the data directory and the WAL directory from the same backup, taken with the server stopped, and keep the refused file aside. The store may have rewritten the file’s header before it failed, so this refusal is the one that can change {file}.

Not a refusal: store location: TOPGUN_REDB_PATH names {L}, which is not read: TOPGUN_DATA_DIR ({D}) already holds a store is a warning; the server starts on the data directory and leaves the file at {L} alone. Remove it, or point TOPGUN_REDB_PATH elsewhere, to silence it. store format: the pre-migration store file is kept at {K}; delete it once this store is confirmed is logged at every start while the kept file exists; delete {K} once you no longer need a way back (see Storage backends).

Server refuses to start with a WAL directory check line

Symptom: The server exits with code 1 at startup, before it listens, with FATAL: Cannot initialise WAL: followed by a line that begins with WAL directory check: refusing to start:, or with FATAL: WAL recovery failed: followed by a line that begins with WAL recovery: refusing to start:.

Cause: The data directory (TOPGUN_DATA_DIR) records, for each partition of the write-ahead log, how far the log has been stored (its applied watermark); the WAL directory (TOPGUN_WAL_DIR) records how far it has collected the log (wal-collected). The two directories are one unit. When they do not belong together, or a file in the WAL directory cannot be read, the server refuses rather than serve a store that may be missing acknowledged writes. A WAL directory check: refusal writes nothing to either directory. Below, {wal_dir} is the WAL directory.

Solutions, by the line the server logged:

WAL directory check: refusing to start: the data directory is older than the WAL directory {wal_dir}: partition {p} has applied watermark {V} there, but the WAL was collected up to {G}; frames {V+1}..={G} are no longer in the WAL and may be missing from the data directory. Restore the data directory and the WAL directory from the same backup, taken with the server stopped (when more partitions are behind, (and {n} more partitions) follows {G}) — the data directory was restored from an older backup than the WAL directory, or the WAL directory belongs to another node. Restore both directories from the same backup, taken after a stop that logged write-behind shutdown: rollback hints written with drained=true.
WAL directory check: refusing to start: the data directory is older than the WAL directory {wal_dir}: the data directory holds no applied watermarks, but the WAL directory has served a store that keeps them. Restore the data directory and the WAL directory from the same backup, taken with the server stopped — the data directory was written by a version that kept the applied watermark in the WAL directory (or is a fresh, empty one), but the WAL directory has already been used by this version. Restore both directories from the same backup. To start a node from scratch, give it an empty data directory and an empty WAL directory.
WAL directory check: refusing to start: {wal_dir}/wal-collected is malformed: {reason} — the record of how far the log was collected is damaged (wrong size, wrong header or checksum). Restore the WAL directory together with the data directory from the same backup.
WAL directory check: refusing to start: {file} holds {n} bytes, an applied watermark is 8; it cannot be imported — at its first start over a WAL directory written by an earlier version, the server imports each partition-{p:03}.applied file once; this one is damaged. Restore the WAL directory and the data directory from the same backup taken under the earlier version, and start again.
WAL directory check: refusing to start: {file} names no WAL partition 0..271; its applied watermark cannot be imported — a file named like partition-*.applied that is not one of the 271 partitions lies in the WAL directory. Move it out of the WAL directory and start again.
WAL recovery: refusing to start: partition {p} holds frame {sequence} at or below its applied watermark {W}; a frame below the watermark is never replayed — the log hands recovery a write the data directory already counts as stored. This is not a state the server produces; keep the logs and both directories, and report it. Restoring both directories from the same backup lets the node start.

Server refuses to start with an encryption line

Symptom: The server exits with code 1 at startup, before it accepts a connection, with an error that names one of the encryption variables, begins with encryption: refusing to start: (also logged as one ERROR line), or begins with store format: refusing to open: and names the keyring or the seal pass.

Cause: With encryption at rest, the data keys that seal your records are kept in the data directory only wrapped by your master key. A store that has ever had encryption on needs that master key at every start, also after encryption was turned off again. Without it the server refuses before it opens the write-ahead log, and the refusal writes nothing: no record, no metadata and no WAL file. The server never prints or logs the key. Below, {fp} is the fingerprint of a master key, 16 hexadecimal characters that identify it without revealing it.

Solutions, by the line the server logged:

{VAR} is set but empty — one of TOPGUN_ENCRYPTION_AT_REST, TOPGUN_MASTER_KEY_PROVIDER, TOPGUN_MASTER_KEY_FILE or TOPGUN_MASTER_KEY_PREVIOUS_FILE is set to an empty value. Unset it, or give it a value.
TOPGUN_ENCRYPTION_AT_REST={value} is not `on` or `off` — use exactly on or off, in lower case.
TOPGUN_ENCRYPTION_AT_REST=on with STORAGE_BACKEND=null: the null backend stores nothing to seal — the null backend keeps nothing on disk. Remove STORAGE_BACKEND=null to use the embedded store, or turn encryption off.
TOPGUN_ENCRYPTION_AT_REST=on needs TOPGUN_MASTER_KEY_PROVIDER=file — set TOPGUN_MASTER_KEY_PROVIDER=file and TOPGUN_MASTER_KEY_FILE.
TOPGUN_MASTER_KEY_PROVIDER=file needs TOPGUN_MASTER_KEY_FILE — set TOPGUN_MASTER_KEY_FILE to the path of the key file.
{VAR} is set, but TOPGUN_MASTER_KEY_PROVIDER is not — a key file is named but no provider. Set TOPGUN_MASTER_KEY_PROVIDER=file, or unset the key file variable.
TOPGUN_MASTER_KEY_PROVIDER={value} is not a provider (expected: file) — file is the only provider. AWS KMS and Vault Transit are not available yet.
encryption: refusing to start: provider file: the master key file {path} cannot be read: {error} — the file is missing or the server’s user cannot read it. Mount or restore the key file at that path. The line names the file, not a key id: no key is known until the file is read.
encryption: refusing to start: provider file: the master key file {path} is accessible by group or others (mode {mode}); allow its owner only (chmod 600) — run chmod 600 (or 400) on the file, owned by the user the server runs as. The mode is checked on the target of a symlink, so in Kubernetes mount the Secret with defaultMode: 0400.
encryption: refusing to start: provider file: the master key file {path} does not hold exactly 64 hexadecimal characters — the file must hold the 64 hexadecimal characters of a 256-bit key and at most one trailing newline. A file saved with Windows line endings (\r\n) is refused: convert it to a plain \n ending, or remove the ending.
encryption: refusing to start: the store is sealed; set TOPGUN_MASTER_KEY_PROVIDER to the key that sealed it (fingerprint {fp}) — the store has had encryption on and no provider is configured. This also happens after encryption was turned off: the store keeps its keys. Set TOPGUN_MASTER_KEY_PROVIDER=file and TOPGUN_MASTER_KEY_FILE to the key with that fingerprint.
encryption: refusing to start: provider {name}: key {id}: the keyring is wrapped by the master key with fingerprint {expected}, but the configured key has fingerprint {configured}; … — the configured key is not the one the store expects, for example after a backup taken before a key rotation was restored. Point TOPGUN_MASTER_KEY_FILE at the key with fingerprint {expected}, or, to move the store to the configured key, set TOPGUN_MASTER_KEY_PREVIOUS_FILE to the key with fingerprint {expected}: the start then re-wraps the keys under the configured one.
encryption: refusing to start: provider {name}: key {id}: the wrap does not authenticate under the configured master key (fingerprint {fp}) (or the previous master key) — the fingerprint matches, but the wrapped data key in the data directory was changed or damaged. Restore the data directory and the WAL directory from the same backup.
store format: refusing to open: … followed by one of the keyring row cannot be decoded, the keyring is version …, the keyring holds key id 0, which is never issued, the keyring holds key id … twice, the keyring holds … active keys, the keyring's next key id … is not above its largest id …, a seal pass without a keyring, the seal pass direction … is unknown, a seal pass with no active key, an unseal pass with an active key or a seal pass cursor outside the record keyspaces — the store holds a keyring or seal-pass row that this server does not write. The data directory was damaged or changed by something else. Restore the data directory and the WAL directory from the same backup; keep the refused directory and report it.
value authentication failed for map {map_id} partition {p} — a sealed value does not open: it was changed, or copied to another record, outside the server. A read of it fails instead of returning it; during the start (the seal or unseal pass) the start fails. Restore the data directory and the WAL directory from the same backup. The record key is not logged.

Server refuses to start with a WAL segment line

Symptom: The server exits with code 1 at startup, before it accepts a connection, with a FATAL: line about the WAL followed by a line that begins with WAL segment or WAL file.

Cause: Each write-ahead-log segment starts with a header that names its partition, its first sequence number, its format and, when encryption at rest is on, the data key that seals its frames. A segment whose header, name or frames do not agree is refused rather than replayed. Below, {file} is the segment.

Solutions, by the line the server logged:

WAL segment {file} has a header for partition {p} from sequence {s}, which its name does not carry; refusing to start — the file was renamed or copied between partitions. Restore the WAL directory together with the data directory from the same backup.
WAL segment {file} has a header that is refused: {reason}; refusing to start — with unknown header version, unknown partition layout, unknown payload schema or unknown cipher, the segment was written by a newer version of the server: run that version. With header CRC … does not match its content or key id … does not fit cipher …, the header is damaged: restore the WAL directory together with the data directory from the same backup.
WAL segment {file} has a truncated tail but is not the active segment; a torn non-last segment is corruption — refusing to start — a segment that is not the last one in its partition is cut short. Restore both directories from the same backup.
WAL segment {file} is sealed with data key {id}, which is not in the keyring; refusing to start — the WAL directory does not belong to this data directory: no key the data directory holds opens it. Restore the data directory and the WAL directory from the same backup; they are one unit.
WAL segment {file} has a frame at offset {offset} that fails authentication: its bytes were changed; refusing to start — a sealed frame was changed on disk although its checksum is intact. It is not a torn write from a crash, which the server tolerates at the end of the last segment. Restore both directories from the same backup, and keep the refused files.
WAL file {file} has wrong magic at offset 0; refusing to start to avoid replaying corrupt data from a server older than WAL format v2 — the WAL directory was written by a newer version, which no older version reads. Start the newer version again and stop it until its stop logs drained=true: that stop removes every segment the store holds. An older version that refuses with store format: refusing to open: unknown metadata "keyring" cannot open a store that has had encryption on; see rolling back.

A store file that the embedded store cannot open at all is a different refusal, store open: … redb panicked while opening it; see the store location section above.

Cluster nodes not discovering each other

Cluster mode is on the roadmap

Multi-node cluster discovery is on the roadmap. Single-node deployment is recommended pre-1.0. See /docs/roadmap for the cluster-safety status.

Symptom: Multiple server nodes are running but not forming a cluster. Each node operates independently.

The following steps are forward-compatible troubleshooting guidance for when full cluster discovery ships:

Verify that all nodes are configured with the same cluster name/identifier
Ensure the cluster port (separate from the client-facing WebSocket port) is accessible between nodes. Check firewall rules
Verify that nodes can resolve each other’s hostnames or IP addresses
Check that the seed node addresses are correctly configured in each node’s cluster configuration
Review server logs with RUST_LOG=topgun_server=debug to see cluster protocol messages and identify where discovery fails