> ## Documentation Index
> Fetch the complete documentation index at: https://chahine.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Clustering

> Run multiple Meridian nodes for horizontal scaling and high availability.

Meridian supports multinode deployments with **zero distributed consensus overhead** — because CRDTs converge by definition, nodes don't need Raft or Paxos. Each node independently applies operations and convergence is guaranteed by the CRDT lattice properties.

Two transport modes are available depending on your infrastructure:

| Mode | Feature flag | Requires | Best for |
| - | - | - | - |
| Redis Pub/Sub | `cluster` | Redis | Low latency, recommended |
| HTTP push | `cluster-http` | Nothing extra | PostgreSQL-only deployments |

***

## How it works

### Real-time fan-out

When a node receives a write (HTTP or WebSocket), it:

1. Applies the op locally
2. Persists to WAL
3. Broadcasts the delta to all peer nodes via the transport layer

Peers receive the delta and forward it to their local WebSocket subscribers — clients see the update in real time regardless of which node they're connected to.

### Anti-entropy gossip

Every 30 seconds (configurable), each node replays its recent WAL entries and re-broadcasts any deltas that peers may have missed. This guarantees convergence after:

* Redis disconnections
* Network partitions
* Node restarts

Anti-entropy is the safety net — real-time fan-out is the fast path.

***

## Redis Pub/Sub transport (recommended)

Build with:

```bash theme={null}
cargo build --release --features cluster
```

Set environment variables:

```bash theme={null}
REDIS_URL=redis://your-redis:6379
MERIDIAN_NODE_ID=1          # optional — auto-derived from hostname+port if unset
MERIDIAN_ANTI_ENTROPY_SECS=30  # optional — default: 30
```

Each node connects to the same Redis instance. Deltas are published to `meridian:delta:{namespace}` channels and consumed by all other nodes via `PSUBSCRIBE meridian:delta:*`.

### Example: 3-node Docker Compose

A ready-to-use compose file is included in the repository:

```bash theme={null}
MERIDIAN_SIGNING_KEY=your-secret docker compose -f docker-compose.cluster.yml up
```

This starts 3 Meridian nodes (ports 3000–3002) and a Redis instance. All nodes share the same signing key and connect to the same Redis for fan-out. Point your load balancer at ports 3000–3002 — any node can serve any client.

***

## HTTP push transport (PostgreSQL-only)

Use this when you have PostgreSQL but no Redis. Each node directly POSTs deltas to its peers over HTTP.

Build with:

```bash theme={null}
cargo build --release --features cluster-http
```

Set environment variables:

```bash theme={null}
# Comma-separated URLs of all OTHER nodes in the cluster
MERIDIAN_PEERS=http://node-b:3000,http://node-c:3000

# Port for the internal cluster API (receives incoming deltas from peers)
# Default: 0.0.0.0:3001 — keep this port internal, not exposed to clients
MERIDIAN_INTERNAL_BIND=0.0.0.0:3001

MERIDIAN_NODE_ID=1                 # optional
MERIDIAN_ANTI_ENTROPY_SECS=30      # optional
```

Each node exposes `POST /internal/cluster/delta` on `MERIDIAN_INTERNAL_BIND`. This port should **not** be exposed to the public internet — use firewall rules or a private network.

### Example: 3-node setup

Node A (`node-a:3000` public, `node-a:3001` internal):

```bash theme={null}
MERIDIAN_PEERS=http://node-b:3001,http://node-c:3001
MERIDIAN_INTERNAL_BIND=0.0.0.0:3001
```

Node B (`node-b:3000` public, `node-b:3001` internal):

```bash theme={null}
MERIDIAN_PEERS=http://node-a:3001,http://node-c:3001
MERIDIAN_INTERNAL_BIND=0.0.0.0:3001
```

Node C similarly.

`MERIDIAN_PEERS` must point to each peer's **internal** port (`MERIDIAN_INTERNAL_BIND`), not the public client-facing port.

***

## Environment variable reference

| Variable | Default | Description |
| - | - | - |
| `REDIS_URL` | — | Redis connection URL (`cluster` feature) |
| `MERIDIAN_PEERS` | — | Comma-separated peer URLs (`cluster-http` feature) |
| `MERIDIAN_NODE_ID` | auto | Unique node ID (u64). Auto-derived from `hostname:port` hash if unset |
| `MERIDIAN_INTERNAL_BIND` | `0.0.0.0:3001` | Internal cluster API bind address (`cluster-http` only) |
| `MERIDIAN_ANTI_ENTROPY_SECS` | `30` | Anti-entropy gossip interval in seconds |

***

## Load balancing

Clients can connect to any node — all nodes share the same state via the transport layer. Use any L4/L7 load balancer (nginx, Caddy, AWS ALB, etc.) in round-robin or least-connections mode.

WebSocket connections are sticky per-session by nature (the connection persists), but there is **no requirement** for session affinity — a client reconnecting to a different node will receive the current CRDT state immediately.

***

## Known limitations

### Node restart catch-up (Redis mode)

In Redis Pub/Sub mode, each node has its own WAL and anti-entropy pushes its WAL entries to peers — but there is no pull mechanism. A node that was down for an extended period will not automatically pull the ops it missed from peers. It will catch up as new writes arrive and as its own WAL is replayed to peers on restart. For most deployments (short outages, active traffic) this is acceptable.

In **HTTP push mode** (`cluster-http`), pull anti-entropy is fully implemented: on startup and periodically, each node pulls WAL entries it missed from each peer via `GET /internal/cluster/wal`. A restarted node catches up completely within one anti-entropy interval (`MERIDIAN_ANTI_ENTROPY_SECS`).

### Shared PostgreSQL and concurrent writes

Concurrent writes to the same CRDT from different nodes on a **shared** PostgreSQL database are safe. The `PgStore` implementation uses `SELECT FOR UPDATE` inside a transaction for all write operations (`merge_put_with`), which prevents lost updates and ensures linearizable per-key updates.

If you use separate PostgreSQL databases per node (or sled), the cluster transport layer handles convergence via delta fan-out — nodes converge within one gossip interval.

### Redis as a single point of failure

If Redis goes down, real-time delta propagation stops. Nodes continue to serve local clients normally, but peers stop receiving updates. When Redis recovers, the next anti-entropy tick (within `MERIDIAN_ANTI_ENTROPY_SECS`, default 30s) replays all missed WAL entries to peers. No data is lost as long as the WAL is intact.

### Clock skew and Presence TTL

Presence TTL expiry is evaluated using each node's local clock. With significant clock skew between nodes, a presence entry may expire on one node before another. This is acceptable for typical NTP-synchronized infrastructure (\< 10ms skew). Ensure NTP is configured on all cluster nodes.

***

## Single-node mode

If neither `REDIS_URL` nor `MERIDIAN_PEERS` is set at startup, Meridian runs in single-node mode. No cluster code is active, no background tasks are spawned. The `cluster`/`cluster-http` feature flags only add overhead when clustering is actually configured.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.