Synchronization Protocol
Understanding how TopGun moves data between clients and servers efficiently using Merkle Trees and WebSockets.
Sync vs. Request
Instead of writing thousands of API endpoints (GET /users, POST /todos), TopGun synchronizes state.
Traditional REST/GraphQL
- • Fetch data explicitly
- • Send requests to mutate
- • Handle loading/error states
- • Manual retry logic
TopGun Sync
- • Subscribe to datasets
- • Mutate local objects directly
- • Automatic retries & conflict resolution
- • Real-time updates pushed to you
Merkle Tree Synchronization
To keep bandwidth usage extremely low, TopGun doesn’t re-download the whole dataset every time you reconnect. It uses Merkle Trees (hash trees) to efficiently detect differences.
The Exchange Process
- 1
Handshake: Client and Server exchange the Root Hash of their respective Merkle Trees.
- 2
Comparison: If hashes match, data is identical. Sync complete (0 bytes transferred).
- 3
Drill Down: If hashes differ, they request hashes of child nodes (buckets) to pinpoint the exact difference.
- 4
Patch: Only the modified records (leaves) are transmitted over the wire.
HTTP Sync
For serverless environments where persistent WebSocket connections are unavailable, TopGun provides a stateless HTTP sync transport via POST /sync.
Unlike WebSocket sync, which maintains a persistent connection for real-time push updates and Merkle tree delta exchange, HTTP sync uses a polling-based request-response model. The client accumulates operations locally and periodically sends them to the server along with sync timestamps. The server processes the operations, computes deltas from its in-memory LWWMap for records newer than the client’s lastSyncTimestamp (using HLC comparison), and returns acknowledgments and delta records in a single response.
Each request carries full client context — clientId, clientHlc, and syncMaps with per-map timestamps — so any server node behind a load balancer can handle any request without session affinity. Authentication is performed via an Authorization: Bearer <token> header on every request.
The HTTP Sync Exchange
- 1
Accumulate: Client writes operations to local state (LWWMap + OpLog) and queues them for sync.
- 2
Poll: At the configured poll interval (default 5 seconds), client sends
POST /syncwith queued operations andsyncMaptimestamps indicating the last known server state per map. - 3
Process: Server applies client operations using CRDT merge semantics, then computes deltas by iterating in-memory maps and filtering records newer than the client’s
lastSyncTimestamp. - 4
Respond: Server returns acknowledgments for processed operations, delta records for requested maps, and any one-shot query results — all in a single response.
- 5
Apply: Client applies delta records to local state and updates
lastSyncTimestampper map for the next poll cycle.
When to Use Which Transport
| Criterion | WebSocket | HTTP Sync | AutoConnectionProvider |
|---|---|---|---|
| Real-time updates | Pushed instantly | Polled (configurable interval) | WS when available, HTTP fallback |
| Serverless compatible | No (needs persistent connection) | Yes (stateless requests) | Yes (auto-detects) |
| Live query subscriptions | Yes | No (one-shot queries only) | Depends on active transport |
| Bandwidth efficiency | Merkle tree delta sync | Timestamp-based delta sync | Best available |
| Connection cost | Per-connection billing | Per-request billing | Adapts to environment |
| Recommended for | Real-time apps, VPS/container deployments | Serverless functions, edge functions | Unknown deployment target |
Server Architecture
While clients are “Local-First”, TopGun is backed by a robust server cluster for persistence and scalability.
1. Gateway Node
Handles WebSocket connections and routes traffic.
2. Partition Engine
In-memory sharding logic that distributes data across the cluster.
3. Persistence Layer
Async write-behind to PostgreSQL (or in-memory for development).
4. Pub/Sub Bus
Broadcasts updates to other connected clients in real-time.