first commit

docs/api-stack/api/api-2pc/src/recovery/recovery.md

@@ -0,0 +1,133 @@
+# `recovery/`
+
+<!-- begin:auto-docs -->
+_(no submodules)_
+<!-- end:auto-docs -->
+
+## Purpose
+
+Boot-time orphan resolution for the 2PC coordinator. When a host
+crashes between phase 1 and phase 2, prepared transactions remain
+held in `pg_prepared_xacts` on every writer participant. The recovery
+runner sweeps those rows on the next boot, consults the host-owned
+ledger, and either commits or rolls back each one — guaranteeing the
+"once prepared, eventually resolved" half of the 2PC durability
+property.
+
+The runner is the only path that should ever resolve an in-doubt
+transaction. The coordinator's hot path NEVER auto-recovers; it
+escalates to `TWOPC_IN_DOUBT` and lets the next-boot recovery sweep
+handle it. That separation keeps the hot path's failure handling
+small and the recovery path testable in isolation.
+
+## Features
+
+- `resolveRecovery({ hostInstanceId, ledger, probes, onResolved })` —
+  the entry point. Returns a `RecoveryReport` enumerating committed,
+  rolled-back, skipped, and errored gids.
+- Idempotent — re-running the function on a host with no outstanding
+  prepared transactions is a no-op (the inner `queryOrphans` returns
+  an empty list).
+- Treats Postgres `42704` ("does not exist") as success during
+  rollback — covers the race where another process already resolved
+  the orphan.
+- Marks rows as `in_doubt` on unresolvable errors so the next sweep
+  can retry; errored gids are surfaced on the report rather than
+  thrown.
+- `buildProbeFromRegistry(registry)` — convenience adapter that
+  spins up one probe per writer participant.
+
+## Usage: Basic
+
+```ts
+import {
+    buildProbeFromRegistry,
+    resolveRecovery,
+} from "@suaveplan/api-2pc/recovery";
+
+const report = await resolveRecovery({
+    hostInstanceId: "host-01",
+    ledger,
+    probes: buildProbeFromRegistry(registry),
+});
+
+console.log(
+    `committed=${report.committed.length} ` +
+        `rolledBack=${report.rolledBack.length} ` +
+        `errors=${report.errors.length}`
+);
+```
+
+## Usage: Advanced
+
+Wire a custom probe to feed an alternate identifier convention or a
+replicated-read connection — useful when the recovery sweep should
+hit a read replica to avoid contending with the live writer:
+
+```ts
+import { gidPrefixFor } from "@suaveplan/api-2pc/phases";
+import { resolveRecovery, type RecoveryProbe } from "@suaveplan/api-2pc/recovery";
+
+const probes: RecoveryProbe[] = [
+    {
+        participantId: "billing",
+        async queryOrphans(prefix) {
+            const rows = await readReplica.unsafe<{ gid: string }[]>(
+                `SELECT gid FROM pg_prepared_xacts WHERE gid LIKE $1`,
+                [`${prefix}%`]
+            );
+            return rows.map((r) => r.gid);
+        },
+        async commit(gid) {
+            await primary.unsafe(`COMMIT PREPARED $1`, [gid]);
+        },
+        async rollback(gid) {
+            await primary.unsafe(`ROLLBACK PREPARED $1`, [gid]);
+        },
+    },
+];
+
+const report = await resolveRecovery({
+    hostInstanceId: "host-01",
+    ledger,
+    probes,
+    onResolved: (e) => log.info(e),
+});
+```
+
+## API Reference
+
+- `interface RecoveryProbe`
+  - `participantId: string`
+  - `queryOrphans(prefix: string): Promise<readonly string[]>`
+  - `commit(gid: string): Promise<void>`
+  - `rollback(gid: string): Promise<void>`
+- `interface RecoveryReport`
+  - `committed: readonly string[]`
+  - `rolledBack: readonly string[]`
+  - `skipped: readonly string[]`
+  - `errors: readonly { gid; error: TwoPcError }[]`
+- `interface RecoveryInput`
+  - `hostInstanceId, ledger, probes, onResolved?`
+- `function resolveRecovery(input): Promise<RecoveryReport>`
+- `function buildProbeFromRegistry(registry): readonly RecoveryProbe[]`
+
+## Implementation Notes
+
+The ledger lookup uses the gid's parsed `txId` as the primary key —
+the gid format from `phases/gid.ts` is what makes that mapping
+unambiguous. Rows whose ledger state is `committing` OR `committed`
+trigger `COMMIT PREPARED`; everything else (including missing rows)
+triggers `ROLLBACK PREPARED`. That asymmetric default ("rollback by
+default") is the safe choice — committing a row whose ledger does
+not explicitly say "all voted commit" would violate the design
+guarantee.
+
+The `42704` swallow during rollback is what makes the runner safe to
+re-execute: if the previous sweep crashed AFTER rolling back the
+prepared transaction but BEFORE updating the ledger, the second
+sweep will see the orphan listed in `pg_prepared_xacts` (or not, if
+truly gone) and will not throw on the "does not exist" follow-up.
+Errors that cannot be classified as idempotency are captured per-gid
+and surfaced on the report — operators page-on `errors.length > 0`,
+not on a generic throw.