catamaran

A fault-tolerant distributed key-value store built on the Raft consensus algorithm.

Architecture

                +----------+
    HTTP/REST   |  Client  |
  +------------>+----------+
  |                 |
  v                 v
+----------+    +----------+    +----------+
|  Node 0  |<-->|  Node 1  |<-->|  Node 2  |
| (Leader) |    |(Follower)|    |(Follower)|
+----------+    +----------+    +----------+
  |   ^             |   ^            |   ^
  v   |             v   |            v   |
 AppendEntries  AppendEntries   AppendEntries
  |   |             |   |            |   |
  +---+             +---+            +---+
  Log               Log              Log
  |                 |                |
  v                 v                v
 State Machine   State Machine    State Machine
  |                 |                |
  v                 v                v
  KV Store         KV Store         KV Store

Features

• Leader election with randomized timeouts (150-300ms)
• Log replication with fast conflict resolution (skip-back by term)
• HTTP REST API with automatic leader redirect (307)
• Exactly-once client semantics via session IDs and sequence numbers
• ReadIndex protocol for linearizable reads without log writes
• Log compaction with InstallSnapshot RPC
• Prometheus metrics and pre-configured Grafana dashboard
• Crash-safe persistence via atomic file writes
• Comprehensive chaos test suite

Quick Start

# Start a 3-node cluster
docker compose -f deploy/docker-compose.yml up

# Register a session (exactly-once semantics)
CLIENT=$(curl -s -X POST http://localhost:8080/session | jq -r .client_id)

# Write with exactly-once guarantee
curl -X PUT http://localhost:8080/kv/hello \
  -H "X-Client-ID: $CLIENT" -H "X-Seq-Num: 1" \
  -d '{"value":"world"}'

# Linearizable read (ReadIndex protocol)
curl http://localhost:8080/kv/hello

# Fast stale read (no quorum confirmation)
curl "http://localhost:8080/kv/hello?stale=true"

# Check cluster status
curl http://localhost:8080/status

# Delete
curl -X DELETE http://localhost:8080/kv/hello \
  -H "X-Client-ID: $CLIENT" -H "X-Seq-Num: 2"

# Monitor
curl http://localhost:9090/metrics

Observability

Grafana dashboard at http://localhost:3000 (anonymous access enabled).

Dashboard panels:

• Term over time (spikes indicate elections)
• Current leader (which node holds leadership)
• Commit latency (p50/p95/p99)
• Replication lag per peer
• Election rate (cluster stability)
• Log length per node
• RPC success rate

Running Tests

go test -race -timeout 120s ./...

Chaos Test Suite

Test	Setup	Action	Assert
Leader Election	3 nodes	Wait 2s	Exactly 1 leader
Leader Crash	3 nodes	Kill leader	New leader within 1s
Stale Leader	3 nodes	Isolate leader, reconnect	Leader steps down
Minority Partition	5 nodes	Isolate 2 nodes	Majority commits succeed
Partition Heal	5 nodes	Isolate then heal	All nodes converge
Concurrent Writes	3 nodes	10 goroutines x 50 writes	No data loss, all identical
Persistence	3 nodes	Kill all, restart	All keys preserved
Log Replication	3 nodes	50 writes	All nodes identical

API Reference

Endpoint	Method	Description
/status	GET	Node state, term, leader
/session	POST	Register client session
/kv/{key}	GET	Linearizable read
/kv/{key}?stale=true	GET	Fast local read
/kv/{key}	PUT	Write with exactly-once headers
/kv/{key}	DELETE	Delete with exactly-once headers

Write Headers:

• X-Client-ID: <uuid> - Client session ID
• X-Seq-Num: <uint64> - Monotonic sequence number

Known Limitations

• No leader leases or clock-based read optimization
• No joint-consensus membership changes (nodes cannot be added/removed from a live cluster)
• Snapshot serialized with encoding/gob; production systems would use Protobuf
• No AppendEntries pipelining (leader waits for reply before next batch to same peer)
• HTTP-based RPC transport; production would use gRPC for lower overhead