Clustering

Run multiple Meridian nodes for horizontal scaling and high availability.

Meridian supports multinode deployments with zero distributed consensus overhead — because CRDTs converge by definition, nodes don’t need Raft or Paxos. Each node independently applies operations and convergence is guaranteed by the CRDT lattice properties.

Two transport modes are available depending on your infrastructure:

Mode	Feature flag	Requires	Best for
Redis Pub/Sub	cluster	Redis	Low latency, recommended
HTTP push	cluster-http	Nothing extra	PostgreSQL-only deployments

---

How it works

Real-time fan-out

When a node receives a write (HTTP or WebSocket), it:

1. Applies the op locally
2. Persists to WAL
3. Broadcasts the delta to all peer nodes via the transport layer

Peers receive the delta and forward it to their local WebSocket subscribers — clients see the update in real time regardless of which node they’re connected to.

Anti-entropy gossip

Every 30 seconds (configurable), each node replays its recent WAL entries and re-broadcasts any deltas that peers may have missed. This guarantees convergence after:

• Redis disconnections
• Network partitions
• Node restarts

Anti-entropy is the safety net — real-time fan-out is the fast path.

---

Redis Pub/Sub transport (recommended)

Build with:

cargo build --release --features cluster

Set environment variables:

REDIS_URL=redis://your-redis:6379
MERIDIAN_NODE_ID=1          # optional — auto-derived from hostname+port if unset
MERIDIAN_ANTI_ENTROPY_SECS=30  # optional — default: 30

Each node connects to the same Redis instance. Deltas are published to meridian:delta:{namespace} channels and consumed by all other nodes via PSUBSCRIBE meridian:delta:*.

Example: 3-node Docker Compose

A ready-to-use compose file is included in the repository:

MERIDIAN_SIGNING_KEY=your-secret docker compose -f docker-compose.cluster.yml up

This starts 3 Meridian nodes (ports 3000–3002) and a Redis instance. All nodes share the same signing key and connect to the same Redis for fan-out. Point your load balancer at ports 3000–3002 — any node can serve any client.

---

HTTP push transport (PostgreSQL-only)

Use this when you have PostgreSQL but no Redis. Each node directly POSTs deltas to its peers over HTTP.

Build with:

cargo build --release --features cluster-http

Set environment variables:

# Comma-separated URLs of all OTHER nodes in the cluster
MERIDIAN_PEERS=http://node-b:3000,http://node-c:3000

# Port for the internal cluster API (receives incoming deltas from peers)
# Default: 0.0.0.0:3001 — keep this port internal, not exposed to clients
MERIDIAN_INTERNAL_BIND=0.0.0.0:3001

MERIDIAN_NODE_ID=1                 # optional
MERIDIAN_ANTI_ENTROPY_SECS=30      # optional

Each node exposes POST /internal/cluster/delta on MERIDIAN_INTERNAL_BIND. This port should not be exposed to the public internet — use firewall rules or a private network.

Example: 3-node setup

Node A (node-a:3000 public, node-a:3001 internal):

MERIDIAN_PEERS=http://node-b:3001,http://node-c:3001
MERIDIAN_INTERNAL_BIND=0.0.0.0:3001

Node B (node-b:3000 public, node-b:3001 internal):

MERIDIAN_PEERS=http://node-a:3001,http://node-c:3001
MERIDIAN_INTERNAL_BIND=0.0.0.0:3001

Node C similarly.

MERIDIAN_PEERS must point to each peer’s internal port (MERIDIAN_INTERNAL_BIND), not the public client-facing port.

---

Environment variable reference

Variable	Default	Description
REDIS_URL	—	Redis connection URL (cluster feature)
MERIDIAN_PEERS	—	Comma-separated peer URLs (cluster-http feature)
MERIDIAN_NODE_ID	auto	Unique node ID (u64). Auto-derived from hostname:port hash if unset
MERIDIAN_INTERNAL_BIND	0.0.0.0:3001	Internal cluster API bind address (cluster-http only)
MERIDIAN_ANTI_ENTROPY_SECS	30	Anti-entropy gossip interval in seconds

---

Load balancing

Clients can connect to any node — all nodes share the same state via the transport layer. Use any L4/L7 load balancer (nginx, Caddy, AWS ALB, etc.) in round-robin or least-connections mode.

WebSocket connections are sticky per-session by nature (the connection persists), but there is no requirement for session affinity — a client reconnecting to a different node will receive the current CRDT state immediately.

---

Known limitations

Node restart catch-up (Redis mode)

In Redis Pub/Sub mode, each node has its own WAL and anti-entropy pushes its WAL entries to peers — but there is no pull mechanism. A node that was down for an extended period will not automatically pull the ops it missed from peers. It will catch up as new writes arrive and as its own WAL is replayed to peers on restart. For most deployments (short outages, active traffic) this is acceptable.

In HTTP push mode (cluster-http), pull anti-entropy is fully implemented: on startup and periodically, each node pulls WAL entries it missed from each peer via GET /internal/cluster/wal. A restarted node catches up completely within one anti-entropy interval (MERIDIAN_ANTI_ENTROPY_SECS).

Shared PostgreSQL and concurrent writes

Concurrent writes to the same CRDT from different nodes on a shared PostgreSQL database are safe. The PgStore implementation uses SELECT FOR UPDATE inside a transaction for all write operations (merge_put_with), which prevents lost updates and ensures linearizable per-key updates.

If you use separate PostgreSQL databases per node (or sled), the cluster transport layer handles convergence via delta fan-out — nodes converge within one gossip interval.

Redis as a single point of failure

If Redis goes down, real-time delta propagation stops. Nodes continue to serve local clients normally, but peers stop receiving updates. When Redis recovers, the next anti-entropy tick (within MERIDIAN_ANTI_ENTROPY_SECS, default 30s) replays all missed WAL entries to peers. No data is lost as long as the WAL is intact.

Clock skew and Presence TTL

Presence TTL expiry is evaluated using each node’s local clock. With significant clock skew between nodes, a presence entry may expire on one node before another. This is acceptable for typical NTP-synchronized infrastructure (< 10ms skew). Ensure NTP is configured on all cluster nodes.

---

Single-node mode

If neither REDIS_URL nor MERIDIAN_PEERS is set at startup, Meridian runs in single-node mode. No cluster code is active, no background tasks are spawned. The cluster/cluster-http feature flags only add overhead when clustering is actually configured.