⚠️ Multi-region content superseded by ADR-0050 / [`multi-region-ha.md`](https://github.com/Stellar-Index/StellarIndex/blob/main/docs/architecture/multi-region-ha.md) (2026-08-21). This plan's multi-region framing (and its "active/active out of scope for v1" stance) is overturned. The single-region HA design below (HAProxy / Patroni / Redis-Sentinel) remains current and is Phase 1 of the multi-region plan — read it for that, not for the multi-region shape.

DEPLOYMENT STATE (audit 2026-07-16): the HAProxy/Patroni/Redis-Sentinel HA below is a ratified DESIGN, not deployed. The roles exist under configs/ansible/roles/{haproxy,patroni,redis-sentinel}/ but NO playbook invokes them — R1 runs a single unclustered Postgres and a single internal-bind no-AUTH Redis today. Any operator/runbook that assumes Patroni or multi-node Redis on R1 is describing the target, not reality.

⚠️ ARCHITECTURE-STALENESS AMENDMENT (2026-07-18). This plan predates the ADR-0034 ClickHouse tier-1 lake — now the largest store (8.6 TiB) and the primary serving path. §4.3 (storage/capacity) and §8 (backup) have been rewritten to current reality. Still stale / TODO: §3 (component-by-component HA) has no ClickHouse tier — deploying the HA design as-written would leave ClickHouse a SPOF. The HA/DR re-evaluation (bootstrap-from-snapshot model, R2/R3 sequencing) is in docs/operations/production-readiness-master-plan-2026-07-18.md §6b (the campaign source of truth) + docs/operations/off-site-backup-plan.md.

High-Availability Infrastructure Plan

Owner: @ash (arch) + @alex (ops). Ratification: binding decisions accepted as ADR-0008 on 2026-04-27. Per-component implementation lives in the relevant ansible roles (configs/ansible/roles/{patroni,redis-sentinel,haproxy,prometheus,loki}/) and the per-region storage strategy is captured in ADR-0016. This plan is the umbrella that binds them.

The HA story is constrained by three non-negotiable service targets:

• p95 ≤ 200 ms, p99 ≤ 500 ms (Performance SLA).
• ≥ 99.9 % responsiveness — we commit to 99.99 %.
• ≤ 30 s data freshness.

Every topology decision below is traced back to which of these numbers it protects.

1. Design principles

1. Single-region HA first; multi-region DR second. The initial

build window forces us to ship a strong single-region deployment at launch, with cold DR in cloud. Multi-region active/ active is explicitly out of scope for v1.

1. Ingest must never block serving. If the ingestion plane slows,

the serving plane serves stale-marked responses — it does not error.

1. Decouple hot from cold. Anything in the 30-second serving

hot path lives in Redis; everything older reads from TimescaleDB; archive + replay source is MinIO. Three tiers, three failure domains.

1. No single machine's failure takes us below SLA. Redundancy is

N+1 at minimum for every stateful component; the Tier-1 three-validator aspiration (ADR-0004) provides N+2 for the validator / history-archive layer. > Update (2026-04-23): stellar-rpc was removed from > production ingest and is now diagnostics-only (rpc-probe + > fixture capture). The N+2 redundancy goal applies to the > validators per ADR-0004; it does not describe stellar-rpc, > which is no longer on the ingest path. The §3.2 sizing below > predates this removal — see docs/operations/r1-deployment-state.md.

1. Every component has a "degraded mode" defined up front. If

Aquarius ingestion dies, what does /v1/price?asset=… return? The answer is in §9, not invented during an incident.

1. We own the hardware we need to own. Captive-core + Galexie on

colocated R640s (per ADR). Everything stateless lives in cloud. Cloud is DR target; colo is primary.

2. Physical topology

Cross-region context: this section shows the per-region layout — one full stack, as deployed in each of three regions. The 3-region architecture (primary / sync-replica / async-replica, graceful degradation across regions) lives in infrastructure/multi-region-topology.md. The phased 1 → 3 validator rollout lives in infrastructure/validator-rollout.md. Per-node hardware spec is in infrastructure/archival-node-spec.md. At launch we run exactly one region (R1, Hetzner FSN1 in Falkenstein, DE) with this topology; R2 (AWS us-east-1) and R3 (Vultr Singapore) join post-launch with the same per-region shape, modified per ADR-0016 for each provider's storage economics.

2.1 Primary (colo)

                          ┌──────────────────────────────┐
                          │ Internet (Anycast + CDN/WAF) │
                          └──────────────┬───────────────┘
                                         │
                ┌────────────────────────┴─────────────────────────┐
                │                                                  │
         ┌──────┴──────┐                                    ┌──────┴──────┐
         │  HAProxy-A  │                                    │  HAProxy-B  │
         │  (keepalived VIP)                                │             │
         └──────┬──────┘                                    └──────┬──────┘
                │                                                  │
                └─────────────────────┬────────────────────────────┘
                                      │
            ┌─────────────────────────┴─────────────────────────┐
            │           stellarindex-api pool (N=3)              │   stateless
            └─────────────────────────┬─────────────────────────┘
                                      │
       ┌──────────────────────────────┼────────────────────────────┐
       │                              │                            │
┌──────┴───────┐          ┌───────────┴──────────┐       ┌─────────┴────────┐
│ Redis cluster│          │ TimescaleDB          │       │ MinIO (erasure)  │
│ (3 masters,  │          │ Patroni-managed HA:  │       │ EC(6+3) on 9     │
│  3 replicas, │          │ 1 primary +          │       │ hosts; bucket    │
│  Sentinel)   │          │ 2 sync replicas      │       │ versioning on.   │
└──────┬───────┘          └───────────┬──────────┘       └─────────┬────────┘
       │                              │                            │
       └──────────────────────────────┼────────────────────────────┘
                                      │
                      ┌───────────────┴──────────────┐
                      │   stellarindex-aggregator     │   one active, one
                      │   (leader-elected via Redis) │   standby
                      └───────────────┬──────────────┘
                                      │
                      ┌───────────────┴──────────────┐
                      │   stellarindex-indexer pool   │   one shard per source
                      │   (per-source orchestration) │
                      └───┬──────────┬──────────┬────┘
                          │          │          │
                    ┌─────┴───┐ ┌────┴────┐ ┌───┴─────┐
                    │ core A  │ │ core B  │ │ rpc A/B │
                    │+galexie │ │+galexie │ │(captive)│
                    │  (R640) │ │ (R640)  │ │ (R640)  │
                    └─────────┘ └─────────┘ └─────────┘

Component counts in §3. Each box is ≥ N+1; the stellar-core nodes are N+2 because the Tier-1 aspiration (ADR-0004) requires three independent archives post-launch.

2.2 DR (cloud — AWS primary)

• Stateless services (stellarindex-api, stellarindex-aggregator)

warm-standby in AWS. Scale-to-zero when not failing over; scale-out on DNS flip.

• TimescaleDB async logical replica (not streaming — crosses a WAN,

pg_logical is resilient) at AWS RDS with 5-minute RPO budget.

• Redis not replicated cross-region: warm-standby is cold.

Acceptable because Redis is cache; re-hydrates from Timescale after failover within minutes.

• MinIO Veeam-style replicated to S3 via mc mirror --overwrite;

RPO 1h for the archive bucket. galexie-live/ replicated at 5 min.

• stellar-core and stellar-rpc not replicated to cloud — they are

rebuilt from our own MinIO archive on DR activation (~4 hours to CATCHUP_RECENT). This is intentional: running captive-cores in AWS violates our cost envelope.

2.3 Why colo primary, cloud DR

Already ratified in ADR-0002 alternatives: captive-cores at 8 vCPU / 32 GB / large NVMe scale are ~3× cheaper on dedicated hardware than cloud IOPS-matched instances. The colocated R640 fleet is already provisioned; cloud is pay-as-you-use for DR.

3. Component-by-component HA

3.1 stellar-core + galexie fleet

• Instances: 3 × R640 (core-01, core-02, core-03). Each

runs captive-core in CATCHUP_RECENT mode + galexie in live-export mode writing to the shared MinIO galexie-live/.

• Quorum set: our 3 nodes vote with the public SDF + 3 Tier-1

organisations (SDF, LOBSTR, Satoshipay) per the Tier-1 aspiration (ADR-0004).

• Failure mode: 1 captive-core down → aggregator continues

reading from the surviving 2 nodes' Galexie output (dedup by (ledger, hash) on ingest). 2 captive-cores down → SEV-1; the third + SDF public quorum keeps us writing ledgers but freshness degrades toward 30 s ceiling.

• RPO: 0 for live ledger (captive-core replays on restart from

local state); 5 min for galexie export (configurable).

• RTO: 2 min failover (the ingester stops reading node X and

starts reading node Y). Captive-core restart: 10–20 min from cold to catchup-recent if local state is intact; 4 h if it has to rebuild from our MinIO archive.

3.2 stellar-rpc

• Instances: 2 × stellar-rpc on core-01 and core-02 (one

captive-core each).

• Why 2 not 3: stellar-rpc's SQLite is not a cluster; each

instance is independent. Two is enough for serving live-event subscriptions.

• Failure mode: 1 rpc down → stellarindex-indexer switches its

getEvents stream to the survivor via configured [stellar_rpc].endpoints array.

• Historic event retention: we do not rely on stellar-rpc

for historic event queries. Historic goes through Galexie → our own events hypertable. This sidesteps the SQLite ceiling flagged in the adversarial audit §6c.

3.3 TimescaleDB cluster

• Topology: Patroni-managed primary + 2 synchronous replicas on

3 separate R640s (db-01, db-02, db-03). synchronous_commit=remote_apply, synchronous_standby_names='ANY 1 (db-02, db-03)'.

• Etcd quorum: 3 nodes for Patroni leader election.
• PgBouncer: a pair with keepalived VIP in front of the

Patroni cluster. Transaction-mode pooling. Pool size sized against PostgreSQL max_connections and the api-pod count.

• Hypertables:

- trades — partitioned by ts daily; raw rows kept forever (migration 0031 removed the old 90-day retention — invariant 8: storage is not a constraint, and Postgres is the served tier, not the full archive). Compression still applies for space. - oracle_updates — same shape as trades, smaller volume. - prices_1m, prices_15m, prices_1h, prices_4h, prices_1d, prices_1w, prices_1mo — continuous aggregates (CAGGs) with add_continuous_aggregate_policy. - soroban_events — the ADR-0029 Soroban-event landing zone the projector tails (compressed after a window). - asset_supply_history — supply-history hypertable; retention indefinite. - asset_metadata — ordinary table (small).

• Retention policy (invariant 8 — ADR-0034):

- trades raw: indefinite (no drop_after; migration 0031 removed the rogue 90-day policy — if you see one on trades, it's drift, remove it). - prices_1m, prices_15m: retention also removed — indefinite. - prices_1h, prices_4h, prices_1d, prices_1w, prices_1mo: indefinite (daily OHLC spans back to 2015).

• Backup (target design; see §8's ⛔ block for what is actually

provisioned): - pgBackRest with WAL-stream, --type=full weekly, --type=diff daily, --type=incr hourly. - RPO 5 min (WAL archiving lag SLA). - Restore test: automated on a timer, evidence appended to docs/operations/drills/ per ADR-0043 §3 (that ADR says *monthly*; the shipped restore-drill.timer template is *weekly* — unreconciled drift, do not treat either as a live guarantee). There is no ops dashboard for drill results, here or anywhere in the repo. > Reality check (2026-07-25, audit-2026-07-23 DOC-06). On R1 > today: the repo is repo1 at /var/lib/pgbackrest — a local ZFS > dataset on the same data pool as the DB, not MinIO and not > off-site (§8). pgbackrest-backup.timer is enabled and runs > full Sunday / diff Mon–Sat at 02:00 UTC — there is no hourly > incr. WAL archiving is continuous (archive-async=y), so the > 5-min RPO holds *for repo1 only*. The restore drill is not > automated: restore-drill.timer (Sat 04:00 UTC) is installed > but deliberately left disabled while the pool is tight > (tasks/18-pgbackrest-backup.yml), and exactly one drill has > ever run — 2026-07-03, repo1, manual > (docs/operations/drills/restore-drills.md).

• Failover: Patroni leader election. Target RTO 60 s.
• Connection secret: read via secret manager at startup, never

on disk.

• Cross-region consistency: API endpoints reading from the CAGG

tables serve only closed buckets per ADR-0015; the in-progress window is never exposed. This makes "all 3 regions return the same rate" a property of the design rather than a hopeful side-effect of replication latency. See the ADR for trade-offs and the ≤30 s freshness contract it implies.

3.4 Redis Sentinel cluster

Amended 2026-05-01 to remove the Cluster-vs-Sentinel contradiction the original draft had. Ratified by ADR-0024. The redis-sentinel ansible role (role docs) deploys this exact topology.

• Topology: 1 primary + 2 replicas, Redis Sentinel mode (no

sharding). 3 Sentinels co-located on the same 3 cache hosts; Sentinel quorum = 2.

• Why Sentinel, not Cluster: our hot-set is small

(~few GB across all categories below); sharding adds operational tax without solving capacity. Sentinel is simpler at SEV-1 time and the migration to Cluster, if we ever need it, is a one-time cost rather than an ongoing tax. Full reasoning in ADR-0024.

• Client connection model: clients use go-redis/v9's

NewFailoverClient and ask any Sentinel for the current primary. There is no VIP or HAProxy in front of Redis — the client SDK does the discovery itself. (This is why the Redis sub-role of Task #72 ships standalone: no companion HAProxy role is required for cache, only for Postgres.)

• Data categories:

- Hot prices — key price:<asset> → latest aggregated price JSON, TTL 60 s (refreshed by aggregator). - VWAP precompute — key vwap:<pair>:<tf> → value+ computed-at, TTL matches the window. - Rate-limit buckets — key rl:<api_key>:<min>, TTL 120 s. - SEP-1 / home-domain cache — key toml:<domain>, TTL 15 min. - Asset-metadata cache — key meta:<asset>, TTL 5 min. - SSE subscriber registry — key sub:<channel>, no TTL (heartbeat).

• Failure mode: master down → Sentinel failover, 15–30 s

window. During the window stellarindex-api returns stale_flag=true on affected keys (pulls from Timescale as fallback).

• Persistence: AOF every-second. RDB nightly. We do not

rely on Redis persistence for correctness — a wiped Redis re-hydrates from Timescale within 2 min (the stellarindex-aggregator re-warms).

• Caveat: token-bucket rate-limit resets on a wipe (users get

a grace minute). Acceptable.

3.5 MinIO

• Topology: 9 nodes with EC(6+3) erasure coding. Tolerates 3

node failures before losing availability, 6 node failures before losing data.

• Buckets:

- galexie-live/ — current captive-core exports, versioning on, object-lock mode COMPLIANCE disabled (we may re-export). - galexie-archive/ — immutable past exports, object-lock COMPLIANCE on, 1-year retention. - backups/ — Timescale pgBackRest, object-lock off. - docs/ — docs site build artefacts, public-read.

• Replication: bucket-level replication to AWS S3 via mc mirror

every 5 min for live, 1 h for archive. Runs from ops-01 with circuit-breaker (if replication lag > 30 min, page).

• Upgrade strategy: rolling, one node at a time; Galexie

retries transient writes with exponential backoff.

3.6 stellarindex-api pool

• Instances: 3 pods on 3 hosts, stateless, behind HAProxy.
• Health checks: HTTP /healthz (shallow: process up) and

/readyz (deep: every registered ReadyChecker polled in parallel; wave-110 split into critical (Postgres → 503) vs non-critical (Redis → 200 with status="degraded"). HAProxy routes only to readyz=200, so a Redis-only outage no longer drains the pool — cache misses fall through to Timescale per ADR-0007.

• Autoscaling: static 3 at launch; target 50% CPU. Scale-up

requires an operational decision; we do not let the autoscaler paper over a bug.

• Rolling deploy: 1-at-a-time, 60 s drain, 30 s settle.
• Graceful shutdown: 30 s for in-flight requests + SSE

connections (SSE peers re-connect to the new pod automatically).

3.7 stellarindex-aggregator

• Instances: 2, leader-elected via Redis SET key NX EX 30

with periodic renewal.

• Role: reads trades hypertable, computes running VWAP/TWAP,

writes to Redis hot-key + Timescale precompute tables.

• Why leader-elected instead of sharded: our aggregation compute

load is small (< 1 core per second on current market volume); a single active instance is simpler and preserves strict ordering.

• Failure mode: leader dies → standby acquires lock within 30 s;

Redis hot keys stale-flag for ≤ 30 s until the new leader writes fresh values.

3.8 stellarindex-indexer fleet

• Topology: one stellarindex-indexer process per source (SDEX,

Soroswap, Aquarius, Phoenix, Comet, Blend, Reflector, Redstone, Band, CEX, FX — roughly 11 processes).

• Cursors: persisted in Timescale per-source

(cursor(<source_id>)). On restart the indexer resumes from the saved cursor.

• Backfill: triggered via stellarindex-ops backfill subcommand;

writes into the same hypertable with idempotent upserts keyed on (source, ledger, tx_hash, op_index, ts).

• Failure mode: one source dies → others continue. The dead

source's freshness timer in Prometheus breaches the 60 s alarm; /v1/price for pairs that rely on that source sets reduced_redundancy=true in the envelope.

3.9 stellarindex-migrate

Not a long-running process. Runs before each deploy in a pre-start job. Uses PostgreSQL advisory lock pg_try_advisory_lock(...) to prevent two migrators from racing.

3.10 stellarindex-ops

Admin CLI. Runs from an operator's SSH session on ops-01. Top-level subcommands cover backfill, gap-detection, archive-completeness verify/check/fix, source decoder verification, RPC probe, archive hash-walking, and supply-snapshot generation. The authoritative list is the binary's own help output (stellarindex-ops --help) and the source at `cmd/stellarindex-ops/main.go`; operator runbooks under `docs/operations/runbooks/` cite the specific subcommand each playbook needs (e.g. runbooks/all-ingestion-down.md references stellarindex-ops backfill).

4. Capacity planning — napkin math

These are lower-bound estimates. Week 9 load-test supersedes them.

4.1 Traffic envelope

Assume 50 wallets × 200 active users each = 10 000 daily actives. A typical wallet asset-detail page makes ~5 API calls per render. Assume 10 renders per user per active day.

• Baseline: 10 000 × 10 × 5 = 500 000 requests/day = ~6 rps.
• Peak (everyone checks during a market move): ~60 rps.
• Service requirement: 1 000 req/min per client = ~17 rps per client.

Capacity target: 500 rps sustained, 2 000 rps burst. That is ~30× baseline; headroom protects us through a year of growth.

4.2 Per-component headroom

Single-pod Go net/http routinely serves 10 000 rps on a modern host for lightweight handlers. Our handlers are mostly "Redis GET → JSON encode → return." Hitting 2 000 rps per pod with 3 pods is comfortable.

4.3 Storage growth — REWRITTEN 2026-07-18, corrected 2026-07-24 (the original was pre-ClickHouse and wrong; the 2026-07-18 rewrite then re-asserted a hardware fix that's since been explicitly ruled out — audit-2026-07-23 DOC-05)

⚠️ The original estimate here ("~500 GB/year, a single TB NVMe lasts 2 years, storage is not a constraint") sized storage off the Timescale trades table and predates the ADR-0034 ClickHouse tier-1 lake. It is the likely root cause of R1 reaching 94% unplanned — the capacity model was never redone after the architecture changed. Real, live-verified numbers:

Actual footprint (R1, 2026-07-18): ZFS pool data = 27.7 TB raw (4× 7.68 TB NVMe, raidz1), 94% full. Datasets: ClickHouse 8.6 TiB (the tier-1 lake — the dominant store), MinIO galexie-archive 5.56 TiB, Postgres 676 GiB, pgBackRest 2.6 TiB, rest small. Storage is the binding production constraint, not an afterthought. Update: Phase A of the capacity-relief campaign (ZSTD-recompressing the four largest ClickHouse tables) has since completed, reclaiming ~3.8 TiB and taking the pool from 94% to ~75% before Phase D's comprehensive backfill began drawing that headroom back down — see docs/operations/production-readiness-master-plan-2026-07-18.md §0 for the current live number; this architecture doc intentionally doesn't chase the day-to-day figure.

Growth: live ingest adds ~10–20 GiB/day (full-fidelity ledger_entry_changes dominate) + a one-time multi-TiB draw from the Phase-D comprehensive backfill.

Headroom levers — software-only. Hardware expansion is ruled out, not a lever: docs/operations/production-readiness-remaining.md §4 states it plainly — *"⛔ R1 is NOT hardware-upgradeable. Fixed 4× 7.68 TB NVMe, no 5th drive, no raidz expansion. Never propose a drive upgrade."* — a standing operator constraint also recorded verbatim in ADR-0027: *"I cannot expand the capacity of this server."* Correction: this section previously listed "(3) A 5th NVMe … the durable fix" as a live option. That was wrong and is removed. The real levers, in priority order:

1. `galexie-archive` → cold S3, then trim local — the biggest single lever (~5.5 TiB). Already config-supported (s3_cold_bucket_archive; reads fall through transparently, cold default = the free AWS public dataset) via ADR-0027's dual-source hot/cold tiering — implemented and in-tree, gated behind an operator flag not yet enabled in production. Scheduled for after the Phase-D backfill completes.
2. `tx_hash` → `FixedString(32)` (~0.6–1 TiB) — stored as 64-char hex today; a Stellar tx hash is 32 raw bytes, so this is a real schema + binary migration, not yet started.
3. ZSTD recompress of the CH XDR columns — measured 1.75× on entry_xdr; the four biggest tables are done (~3.8 TiB reclaimed, see the Update above); tx_hash/tx_hash_index and other still-LZ4 columns remain (~0.3–0.5 TiB more).
4. TimescaleDB compression policies (~0.2–0.4 TiB) — 19 hypertables are compression-eligible but have no policy attached yet.
5. pgBackRest diff-retention prune (~1 TiB, deferred until off-site backup exists — see §8).
6. Horizontal growth — a second server, not a bigger R1. R1 is one region of the multi-region design in §2; the durable answer to R1 filling up is R2 coming online (per ADR-0016; production-readiness-remaining.md §5c: "R2 provisioning … unblocks HA/DR *and* capacity") and eventual R1 retirement — not more drives in this chassis.

Detail + live capacity table: docs/operations/runbooks/phase-a-capacity-relief-2026-07-18.md (its own "5th NVMe" line carried the same stale claim and has since been corrected the same way — the runbook is safe to follow) + docs/operations/production-readiness-remaining.md §4 (the current, hardware-ruled-out version). Campaign source of truth: docs/operations/production-readiness-master-plan-2026-07-18.md.

5. Failure matrix

No single-component failure breaches 99.9% monthly (≤ 43 min/month). Two-component failures can breach; catalogued above with response times.

6. Security posture

Not the full threat model (that lives in docs/operations/threat-model.md, Week 9), but the HA-relevant items:

• Secrets: Vault (colo) + AWS Secrets Manager (cloud), cross-

replicated via periodic sync. Application reads at startup via a sidecar; no secret ever on a disk outside Vault.

• TLS everywhere internal and external. Internal: mTLS between

api↔pgbouncer↔timescale and api↔redis. External: Let's Encrypt + HSTS.

• Network segmentation: Management VLAN, data VLAN, DMZ for

HAProxy. api pool has no egress except to Timescale, Redis, and logging. Indexers have egress only to pinned CEX/FX IP ranges + stellar-rpc.publicnode.com fallback.

• HSM for validator keys (ADR-0004) — YubiHSM-2 on two physical

hosts.

• Audit log: every stellarindex-ops command recorded to an

append-only bucket. Admin surface requires 2FA via the jump host.

7. Observability

• Metrics: Prometheus pair (primary + replica); federated from

cloud Prometheus for DR. Retention: 30 d local, 1 y downsampled to MinIO via Thanos.

• Dashboards: Grafana — one dashboard per component + one

"Golden Signals" board (latency p50/p95/p99, error rate, saturation, traffic).

• Alerts: AlertManager → PagerDuty. Tiers:

- P1: 99.9 % SLA-breaking; pages immediately. - P2: degraded; pages during business hours + daily summary. - P3: informational; ticketed.

• Tracing: OpenTelemetry → Tempo. Sampling 100 % at development,

10 % at production, 100 % on errors.

• Logs: structured JSON via zerolog; shipped to Loki with

14-day retention + 1 y cold.

Alerts already sketched in docs/operations/alerts-catalog.md (Week 9).

8. Backup & restore — TARGET DESIGN (rewritten 2026-07-18; deployment status verified 2026-07-25)

⛔ NOT DEPLOYED — none of the off-site streams in the table below exist yet (verified against config AND against the live host, 2026-07-25).

>

Live evidence from R1, not inference from the repo: ``` $ grep repo /etc/pgbackrest/pgbackrest.conf repo1-path=/var/lib/pgbackrest # no repo2, no S3, no azure/gcs

>

$ df -h /var/lib/pgbackrest /var/lib/postgresql data/pgbackrest 2.0T 1019G 948G 52% /var/lib/pgbackrest data/postgres 1.7T 695G 948G 43% /var/lib/postgresql `` Identical Avail on both rows is the point: **the only backup shares one ZFS pool with the database it protects.** systemctl list-timers shows pgbackrest-backup.timer and nothing else — no clickhouse-backup, no restore-drill. During an incident, plan recovery from what is actually on the box, not from this table. **What exists today on R1:** pgBackRest **repo1 only**, at /var/lib/pgbackrest — a ZFS dataset on the *same* data pool as the database it protects (templates/pgbackrest.conf.j2`), full Sunday + differential Mon–Sat at 02:00 UTC with continuous WAL archiving. **A pool/box loss loses the data *and* every backup of it.** Specifically not provisioned:

>

- ClickHouse — no backup of any kind. clickhouse-backup appears nowhere in the repo outside design docs. The 8.6 TiB lake's only recovery path today is re-derivation from the galexie archive (~1–2 weeks). - Postgres off-site (`repo2`) — not configured. configs/ansible/inventory/r1.yml sets pgbackrest_offsite_ack: true, the explicit acknowledgement of this gap that tasks/18-pgbackrest-backup.yml demands in lieu of an S3 repo. - Galexie archive off-site — not configured. Galexie exports to the box's own MinIO (galexie_s3_endpoint: http://127.0.0.1:9000) and the only mc mirror job, galexie-archive-fill.sh, runs *inbound* (AWS public blockchain dataset → local/galexie-archive). Nothing copies the archive out, and ADR-0027's cold-S3 tier is not enabled on r1 (s3_cold_bucket_archive unset). - Config / vault / secrets tarball — no such job exists.

>

Design, provider choice, cost and sequencing: `docs/operations/off-site-backup-plan.md` (status: proposed — execute after Phase A/D). Anything below is the target that plan builds toward.

⚠️ The original table (in git history) had two gaps that make it unsafe as-is: (1) no ClickHouse — the largest store and primary serving path, omitted because this predates ADR-0034; **(2) backups landed on the *same box's* MinIO**, so a single ZFS-pool/box loss takes the data *and* its backups. The design below corrects both; note that gap (2) is still the LIVE situation until the streams above are provisioned.

Design principle — back up the source of truth + the RTO tiers. The archive is irreplaceable; PG + CH are the served/RTO tiers. The CH lake is *also* re-derivable from the archive, but that path is weeks, so it's backed up in full for a ~half-day RTO. The off-site CH snapshot doubles as the region-bootstrap + cross-region consistency baseline — R2/R3 restore from it and keep up live, rather than each independently re-deriving (see the ADR-0008 amendment / §2). RTOs: Postgres PITR ~1–3 h; full CH restore ~½ day; total-loss (both copies gone) → re-derive from a Stellar history archive, days.

Better than backups for downtime: the HA warm standby (§2, §3.3). An S3 restore is still hours dark; a warm standby (PG Patroni + a CH replica that bootstraps from the snapshot) fails over in minutes and removes the single-box SPOF. Full S3 backup = the durable floor; the standby = the real production target.

9. Degradation modes (what we promise under failure)

We document "what happens when prices become unavailable, sources start to differ, etc." The API envelope (to be specified in api-design.md §Error envelope) carries four boolean flags:

No flag is a response-level error; they're advisory. Clients decide whether to accept. The price value is always best-available; stale_flag=true means "here's the last known good, fix your decision-making accordingly."

Specific "everything is on fire" scenarios:

10. Launch checklist (HA subset)

• [ ] All 3 stellar-core + galexie instances running stably for 7 days with no crashes.
• [ ] Patroni failover drilled end-to-end in staging (simulate primary OOM).
• [ ] Redis Sentinel failover drilled (kill master during load).
• [ ] Load test hits 2 000 rps with p95 ≤ 200 ms on cached endpoints.
• [ ] Restore drill: point-in-time recovery to 24 h ago, < 2 h wall-clock.
• [ ] DR drill: DNS-flip to cloud, serve for 1 h, flip back.
• [ ] Alerts catalogue reviewed — every alert has a runbook link.
• [ ] SEV-1 + SEV-2 playbooks rehearsed with a tabletop exercise.

None of these are green today (Week 1). Every line becomes a PR checklist at its owning week.

11. Open questions — closed

The Week-1 plan called for these to land as ADRs or design docs by end of Week 2. They have:

1. Colo provider + physical locations — Hetzner FSN1 (Falkenstein, DE)

for R1; AWS for R2; Vultr for R3. See r1-deployment-state.md + ADR-0016.

1. Patroni vs Stolon vs native TimescaleDB HA — Patroni; landed

as configs/ansible/roles/patroni/.

1. MinIO EC(6+3) vs EC(4+2) — EC(6+3); fixed in ADR-0008 §2.
2. Cloud DR region — AWS eu-west-1 (matching the colo latency

profile for European users); ADR-0008 §5.

1. Secret-manager choice — Ansible Vault for inventory secrets;

configs/ansible/inventory/r1.secrets.yml is the source of truth, per the playbook README.

1. Observability stack — self-hosted Prometheus + Grafana +

Loki; ansible roles configs/ansible/roles/{prometheus,loki}/ deploy them.

Anything new that surfaces post-ratification gets a fresh ADR rather than an entry here.

12. Cost envelope

Order-of-magnitude; concrete per-line numbers live in the operator's own cost spreadsheet (not checked into the repo). Below is the shape used to size hardware in ADR-0008.

Revenue model is out of scope (free public API; SDF grant funds). Cost envelope checked against the infrastructure budget.

13. Appendix — tooling

• HAProxy — 2.9 LTS.
• keepalived — for VRRP VIPs.
• Patroni — 3.x with etcd3 DCS.
• PgBouncer — transaction mode.
• Redis — 7.x with Sentinel.
• MinIO — current RELEASE.* on the docker-compose profile;

baremetal RPMs in production.

• pgBackRest — with MinIO as the repo backend.
• Prometheus + AlertManager + Grafana + Loki + Tempo — "grafana

stack." Possibly replaced with Grafana Cloud depending on cost model.

All tools are Apache-2.0 / MIT / PostgreSQL / BSD-compatible. No copyleft dependencies in the serving path.