Benchmarks

Copy-on-write fork cost (v0.2.x)

Measured numbers for the claim the copy-on-write fork actually makes: forking is storage-O(1) — a shared fork writes two tiny objects (the child lineage's base.json + the branch ref), regardless of database size — and a child pays only for what it changes. Everything in this section was re-measured for v0.2.12 on the machine below; every section below the --- rule predates copy-on-write (v0.1.x measurements) and describes the materialize path — see the version note below it.

Machine: darwin/arm64, Apple M5 (10 cores), macOS 27.0, local APFS disk, Go 1.27.1. Local directory store backend (the default make bench target's backend). No other load. Measured 2026-09-30 at 751ecb7 (v0.2.12) — the "after" — and, for the "What changed in v0.2.12" comparisons, at 97320cc (the commit before this work: v0.2.11 plus the design spec) on the same machine the same day — the "before".

Reproduce: make bench-cow (benchmarks live in internal/ops/cow_bench_test.go; the target's invocations are exactly the ones below). SQLite 3.53.4, as bundled by github.com/mattn/go-sqlite3 v1.14.52 (the version in go.mod), with the default 4096-byte page size. Latency numbers are the median of 5 samples (-count=5 -benchtime=3x, each sample the mean of 3 iterations); byte numbers are exact object-store accounting (sum of stored object sizes, diffed before/after — what an S3 backend would bill), identical across repeat runs.

Shared fork latency vs database size

The share path itself (at=checkpoint) is near-constant, ~9–10 ms from 12 MB to 1 GB — it writes two small objects and never touches the database's bytes. Fork at head (at="") now is too, once the checkout has been quiet for a second: its uncheckpointed-changes check proves the checkout clean from the .sum sidecar's size/mtime/change-counter fingerprint instead of SHA-256-hashing it, and 1 GB forks at head in 9.5 ms. The 12 MB row and the first three 100 MB samples are the exception, and they are the honest caveat: the fingerprint is trusted only when the file's mtime is more than 1 s older than the sidecar stamp (git's racily-clean rule), and those samples ran inside that second after the seed's checkpoint, so they hashed. Raw samples:

BenchmarkCoWSharedFork/size=12MB/at=head-10          3  13671333 ns/op ... (5 samples: 13.7/15.3/14.9/15.6/15.0 ms)
BenchmarkCoWSharedFork/size=12MB/at=checkpoint-10    3   8664708 ns/op ... (8.7/9.6/9.6/7.4/8.2 ms)
BenchmarkCoWSharedFork/size=100MB/at=head-10         3  43763459 ns/op ... (43.8/43.7/43.6/20.4/9.1 ms)
BenchmarkCoWSharedFork/size=100MB/at=checkpoint-10   3   9182083 ns/op ... (9.2/8.8/10.6/8.3/8.7 ms)
BenchmarkCoWSharedFork/size=1GB/at=head-10           3   8684486 ns/op ... (8.7/8.9/9.5/10.1/10.5 ms)
BenchmarkCoWSharedFork/size=1GB/at=checkpoint-10     3   9441958 ns/op ... (9.4/10.1/11.4/10.4/11.2 ms)

Added object-store bytes per fork (100 MB database)

A shared fork of a 100 MB database adds 377 bytes — one base.json (the durable base pointer) plus one branch ref — about 280,000× less than a materialized fork or a cp, and flat from N=1 to N=100. For latency context, plain cp of the same file took a median 18.5 ms (5.7 GB/s) vs the share path's ~9 ms.

BenchmarkCoWSharedForkAddedBytes-10         100  16794541 ns/op  376.9 addedBytes/fork  2.000 addedObjects/fork
BenchmarkCoWMaterializedForkAddedBytes-10    10  44540458 ns/op  105883916 addedBytes/fork  2.000 addedObjects/fork
BenchmarkCoWCpBaseline-10                     3  18654527 ns/op  5627.83 MB/s   (5 samples: 18.7/18.5/18.3/18.7/18.4 ms)

Divergence cost: what a shared child pays as it writes

Fork a 100 MB database (share), pre-materialize the child's checkout (so the settling-flush suppression arms and flushes are segments — the steady-state, see cow_divergence_test.go), then write k single-row transactions with one Flush each at the default snapshot cadence (SnapshotEvery=16). Identical across 3 repeat runs:

Under the cadence, divergence is O(changed pages): ~761 B per single-row transaction against a 100 MB database — segments, the ref update, nothing size-proportional. (The 6,089 B numerator also includes the fork's own one-time 377 B of base.json + ref; the pure per-segment cost is ~714 B/txn — 761 is the conservative all-in figure.) The honest second row: every SnapshotEvery-th flush (16th, by default) writes a full self-snapshot (the divergence floor that keeps read chains bounded), which is O(DB) — ~105.9 MB here. Steady-state cost is therefore ~714 B/txn plus one full snapshot per 16 flushes.

These two rows are from a re-run, not from make bench-cow's own paste. In the make bench-cow run for this section the same benchmark reported 104,990,993 B at k=8 and 210,883,507 B at k=20 — each exactly one ~105 MB file more than the rows above. That file was not a store object: the benchmark's byte accounting (cowStoreUsage) walked the whole store directory except checkouts/, and since v0.2.12 the child's checkout builds a by-chain cache entry under checkouts-ro/ (the local clone source, never uploaded). cowStoreUsage now skips checkouts-ro/ too, and the added-bytes steps of make bench-cow were re-run with that one change; the fork rows above came out identical to the first run, and k=8/k=20 came out as shown, matching the before run at 97320cc (6,088 B and 105,898,603 B).

BenchmarkCoWDivergenceAddedBytes/k=8-10    1   669633334 ns/op       6089 addedBytes/child     761.1 addedBytes/txn
BenchmarkCoWDivergenceAddedBytes/k=20-10   1  1303934417 ns/op  105898603 addedBytes/child   5294930 addedBytes/txn

Read-path sanity: checking out a shared fork

Checkout from scratch (checkout file + sidecar deleted each iteration; 100 MB database; median of 5):

Since v0.2.12 neither is a materialization any more: deleting the checkout leaves the by-chain entry for its chain in checkouts-ro/, so each iteration is a clone of that entry plus the sidecar and shadow stamps. The shared child's chain is the parent's snapshot, so both clone the same entry; the ~1 ms difference is the child's extra base.json read and chain resolution. The before run, where every iteration decoded the snapshot, is in "What changed in v0.2.12" below.

BenchmarkCoWCheckoutAfterFork/branch=parent-10         3   8756458 ns/op ... (8.8/8.8/8.8/9.1/9.1 ms)
BenchmarkCoWCheckoutAfterFork/branch=shared-child-10   3   9687597 ns/op ... (9.7/10.0/11.2/9.9/9.4 ms)

make bench: fork at head, clean checkout, session open

make bench (-count=3 -short, internal/ops/fork_bench_test.go) runs the three benchmarks the pre-copy-on-write sections below were built on. Median of 3, same machine, before and after:

Before, all three were dominated by one SHA-256 pass over the checkout (~3 GB/s here). After, a clean checkout is proven clean from its fingerprint, so CheckoutCleanSkip and SessionOpen are flat in size. ForkAtHead at 64MB is the racily-clean caveat again: its seed finishes inside the 1 s window, so its early iterations hash. SessionOpen's settleSnapshotBytes (67,759,458 and 542,254,275) is unchanged: that metric forces one settling flush per size by design.

# after
BenchmarkForkAtHead/size=64MB-10              51    19754832 ns/op    3401.23 MB/s
BenchmarkForkAtHead/size=512MB-10            100    11698501 ns/op   45943.40 MB/s
BenchmarkCheckoutCleanSkip/size=64MB-10     3921      312501 ns/op  215009.76 MB/s
BenchmarkCheckoutCleanSkip/size=512MB-10    3236      323870 ns/op 1659519.69 MB/s
BenchmarkSessionOpen/size=64MB-10            180     6239043 ns/op   10769.41 MB/s   67759458 settleSnapshotBytes
BenchmarkSessionOpen/size=512MB-10           216     5688384 ns/op   94485.35 MB/s  542254275 settleSnapshotBytes
# before (97320cc)
BenchmarkForkAtHead/size=64MB-10              39    31093848 ns/op    2160.90 MB/s
BenchmarkForkAtHead/size=512MB-10              6   183046076 ns/op    2936.25 MB/s
BenchmarkCheckoutCleanSkip/size=64MB-10       54    21897294 ns/op    3068.45 MB/s
BenchmarkCheckoutCleanSkip/size=512MB-10       6   176981750 ns/op    3036.86 MB/s
BenchmarkSessionOpen/size=64MB-10             42    27756484 ns/op    2420.72 MB/s   67759458 settleSnapshotBytes
BenchmarkSessionOpen/size=512MB-10             6   180073896 ns/op    2984.71 MB/s  542254275 settleSnapshotBytes

(Each raw line is the median sample of its three. make bench also runs the CoW suite at its cheap sizes with Go's default benchtime; those lines are in the full output and not repeated here — the make bench-cow rows above are the ones this section quotes.)

A re-run of make bench after the local-store List fix (see "BranchBench topologies" below) moved only ForkAtHead, and by about a tenth: 64MB 19.8 → 17.7 ms, 512MB 11.7 → 10.4 ms median; CheckoutCleanSkip (0.31 / 0.32 ms) and SessionOpen (6.4 / 5.8 ms) were unchanged. The paste above is the run the table was built from.

What changed in v0.2.12

Before (97320cc) and after (751ecb7), same machine, same day, both from make bench-cow / make bench:

• Fork at head, 1 GB: 376.7 ms → 9.5 ms. The share never cost anything; the uncheckpointed-changes hash did, and a quiet checkout no longer needs one. Inside the 1 s racily-clean window it still hashes (100 MB at head: 45.4 ms → 43.6 ms median, because three of the five samples ran in that window).
• Fork at a named checkpoint: unchanged (8.7 ms → 10.4 ms at 1 GB, 9.2 ms → 8.7 ms at 12 MB — inside each run's sample spread).
• Checkout of a state some branch already materialized: 262.8 ms → 8.8 ms (parent) and 262.1 ms → 9.9 ms (shared child), 100 MB — a clone of the by-chain entry instead of a decode of the snapshot.
• Clean checkout and session open: O(size) → O(1) — 177.0 ms → 0.32 ms and 180.1 ms → 5.7 ms at 512MB.
• Object-store bytes: unchanged. 377 B per shared fork, 105,883,916 B per materialized fork, ~761 B per diverging transaction, in both runs — the v0.2.12 changes to the store side (segment checkpoints, shared rollback/promote) are not on these benchmarks' paths.

Caveats (read before quoting these numbers)

• Local backend, APFS, one machine. S3 adds per-request network latency to every number here, but the object math is identical by construction: a shared fork issues the same two small writes (base.json + ref) against S3 — no snapshot copy, no data-plane traffic — so the "377 B / 2 objects per fork" accounting carries over even though the milliseconds do not.
• Added bytes are logical stored-object bytes (what S3 would bill). On APFS the local backend's materialize path clones (clonefile), so the materialized fork's physical local disk usage is far below its 105.9 MB logical footprint — the logical number is the one that transfers to a real object store.
• The clone and fingerprint wins need a filesystem that can clone (APFS here). On ext4 or tmpfs there are no by-chain entries, so a checkout decodes the snapshot as in the before column; the fingerprint check does not need clones and applies everywhere.
• The 1 s racily-clean window: a fork at head or a checkout within a second of the checkout's last write hashes it, as before v0.2.12.
• A shared child's write path snapshots in full every SnapshotEvery (default 16) flushes — divergence is O(changed pages) between those floors, not unconditionally.
• Seeded databases are freshly checkpointed single-snapshot chains — the share path's common case. Forking a branch whose resolved chain has reached the fork-time floor (ForkShareMaxDepth, or the configured session cadence) materializes instead, at the "Materialized" row's cost.

Older measurements. Everything below this line was measured before copy-on-write existed, on a different machine (Apple M4, Go 1.26; the sections above used an Apple M5, Go 1.27), and is preserved as published. The MinIO numbers below are no longer reproducible: MinIO withdrew its community images, and make bench-s3 now targets RustFS.

The next section's version note scopes what still applies.

Measured baselines for ops.Workspace.Fork, Checkout's clean-skip fast path (Task 1 of Milestone 2), and session.Open — before and after Task 6a's fast-path fork. The point of this document is to make the before/after comparison honest: these are real numbers from real runs, not estimates. The "before" column is exactly what was measured and published prior to Task 6a; it is kept, not replaced, so the comparison is checkable.

Version note — every number below was measured on a pre-copy-on-write build (the v0.1.x line, 2026-08-06), and is kept as measured rather than re-invented. Since v0.2.0 the default fork shares (a base pointer, no object copy at all), so the ForkAtHead numbers below no longer describe the common-case fork's storage work — they describe the materialize path: a fork that trips the fork-time snapshot floor, and every promote/rollback/compact, all of which still use exactly the snapshot-copy machinery measured here. The O(size) uncheckpointed-changes check these numbers are dominated by (see "Isolating the fast path itself") still runs on a shared fork at head too. The CheckoutCleanSkip and SessionOpen numbers are unaffected by the copy-on-write change. Also superseded since measurement: S3 CopyObject's 5 GiB ceiling is now a strategy boundary, not a limit — v0.2.4 added multipart UploadPartCopy for sources up to S3's 5 TiB per-object ceiling, so the ">5GB falls back to materialize" behavior described below no longer applies (only >5 TiB falls back). And since v0.2.12, promote and rollback share by default too (only --materialize, the depth floor and compact still take the copy machinery), and the O(size) SHA-256 check that dominates ForkAtHead, CheckoutCleanSkip and SessionOpen below is skipped for a quiet, clean checkout — see "make bench: fork at head, clean checkout, session open" above for those three re-measured.

Benchmarks live in internal/ops/fork_bench_test.go. Run them with make bench (local store) or make bench-s3 (real MinIO in Docker). The benchmark code itself is unchanged by Task 6a — same subtest names, same seeding, same at="" (fork at branch head, which runs Fork's normal uncheckpointed-changes check; see "What's still O(size) after the fast-path fork" below) — so before/after numbers are a like-for-like comparison of the same call, not two different things being measured.

Per-test isolation primitives (v0.2.12)

An eval harness needs a fresh, isolated database per test (see docs/eval-harness.md). This section puts offshoot's own fork side by side with the plain-file and Postgres alternatives a harness author might reach for instead, measured on this machine with scripts/bench-isolation.py (make bench-isolation; stdlib + sdk/python on sys.path, docker CLI only for the Postgres rows — skipped cleanly, with a note, when Docker or a local postgres:16 image isn't available; --dry-run validates tooling and prints the plan without doing any timed work).

What each row is:

• offshoot fork + open + close (SDK): the full per-test cost via the Python SDK, against an eval-bench database seeded once via Client.create(db, from_path=...) (Task 1's SQLite import) — fork a fresh branch, open a live session on it, close it. This is what a test pays if it opens a session at all (e.g. via offshoot_fork() / the pytest fixture), even if it never writes.
• offshoot fork alone: the same fork, no session opened — isolates the pure branch-creation cost from what open adds on top.
• sqlite3.Connection.backup(): Python's built-in online-backup API, copying the seed file into a fresh file.
• shutil.copyfile: a plain byte-for-byte copy of the seed file.
• Postgres CREATE DATABASE trial TEMPLATE seed: clones a database from a template inside a local docker run -d postgres:16 container (DROP DATABASE between iterations); the seed is built with generate_series to approximately the stated size, verified via pg_total_relation_size.
• Postgres cold container start: docker run -d postgres:16 until pg_isready, then docker rm -f — the cost of not keeping a container warm between tests. Measured separately, only 3 iterations (it's slow, and independent of seed size).

Machine: darwin/arm64, Apple M5, macOS 27.0, Docker 29.8.0 (Postgres runs inside Docker Desktop's Linux VM here, not natively), Go 1.27.1, local-directory store backend, no other load, no network. Measured 2026-09-30 at 751ecb7 (v0.2.12). Raw output of make bench-isolation (--sizes 10,100 --iters 20), pasted verbatim:

Postgres cold container start (docker run -d postgres:16 until pg_isready; N=3, size-independent): 347.26 / 427.50 ms (median / p95)

docker exec no-op (SELECT 1) overhead (same docker exec ... psql round trip the template-clone row above pays; N=20, size-independent): 34.70 / 38.10 ms (median / p90)

pg_isready -> first successful SELECT 1 gap (measured inside the cold-container start above; N=3, size-independent): 77.16 / 166.92 ms (median / p90)

Values are median / p95 milliseconds over 20 iterations (3 for the cold-container row), except the docker-exec-no-op and pg_isready-gap lines above, which report median / p90. Postgres seed sizes (pg_total_relation_size): 10 MB target -> 10.6 MiB actual, 100 MB target -> 105.2 MiB actual. SQLite seed sizes (actual file size): 10 MB target -> 10.0 MiB actual, 100 MB target -> 100.1 MiB actual.

Caveats:

• Single host, single run, not a fleet average: all rows were measured back-to-back on the same machine (see the machine line above this table in the docs) with no other load; run-to-run variance on a laptop is real.
• Apple Silicon macOS: the offshoot daemon and SQLite rows run natively, but Postgres only runs in Docker Desktop's Linux VM here — its numbers include a virtualization/VM-boundary tax a native Linux host would not pay.
• offshoot uses its local-directory store backend (not S3) for this run.
• No network access is used or required by this script; the postgres:16 image must already be present locally or the Postgres rows are skipped.

Why the SQLite rows are near-constant in size and the Postgres row is not: offshoot fork alone stays flat (~10 ms either way) because a shared CoW fork writes two small objects — a base pointer and a branch ref — regardless of database size (see "Copy-on-write fork cost" above); nothing about the fork touches the database's bytes. sqlite3.Connection. backup() and shutil.copyfile scale close to linearly with size, as expected, because both physically copy every byte (backup(): 10.39 ms -> 98.66 ms, roughly the 10x size ratio; copyfile is under a millisecond at 10 MB, so page-cache noise dominates its p95 there). The two rows are not an apples-to-apples I/O comparison, though: backup() opens its destination connection with SQLite's default synchronous=FULL, so every .backup() call fsyncs the destination as part of committing — it asks the OS for durability. shutil.copyfile never fsyncs, so it asks for none. That asymmetry accounts for part of the gap between the two rows, on top of the difference in what each one physically does. Postgres's CREATE DATABASE ... TEMPLATE also physically copies the template's files — PostgreSQL 16's default STRATEGY = WAL_LOG copies the template block by block through WAL regardless of filesystem, rather than taking a filesystem-level reflink/CoW shortcut — but the row's absolute numbers at these sizes are dominated by the fixed cost of the docker exec ... psql round trip itself: a no-op SELECT 1 through the same path, measured directly above, costs 34.70 ms median (p90 38.10 ms) alone on this host. Only the roughly 68 ms difference between the two sizes' medians (50.61 -> 118.80 ms) is the actual template-clone work; that delta does scale with size, it's just swamped by per-command overhead at 10 MB.

The honest fork-versus-open statement: offshoot fork by itself is the flat, near-constant row above (~10 ms). The fork + open + close row is roughly 4.6x slower at 10 MB and roughly 20.7x slower at 100 MB — not because forking got more expensive, but because of what open (and close) does that fork doesn't: a real daemon round trip, and — since a freshly forked branch has no checkout locally yet (the fork itself only wrote a base pointer + ref) — Open's first CheckoutProven call has to put the branch's database bytes on local disk before handing back a live session, on top of the settling-flush checksum machinery described in "Settling-flush cost" below. Since v0.2.12 that step is a clone of the by-chain cache entry the first fork's open built, not a decode of the snapshot, which is most of the drop in "What changed" below. The row is still NOT size-independent (55 ms at 10 MB vs. 200 ms at 100 MB); this benchmark does not break down which part of open/close the remaining size-dependent time is (the settling flush and Session.Close's drain and sidecar re-stamp are the candidates — see docs/status.md's sidecar-refresh row). A test that only forks and never opens a session pays the flat ~10 ms row instead of this one.

A note on the cold-container number: the Postgres image's entrypoint briefly runs a temporary, setup-only server on the same socket before the real server starts, and pg_isready (used here, per this row's own definition) can report ready during that window. A follow-up check this script uses elsewhere, before it runs any real SQL against a container it just started (an actual SELECT 1, retried until it succeeds), measured directly above as its own line, took another 77.16 ms median (p90 166.92 ms) past pg_isready's signal on this host — note that this figure is measured via _wait_pg_queryable, which re-checks pg_isready (already satisfied at that point) before it starts polling SELECT 1, so the measured gap includes one redundant pg_isready docker-exec round trip and slightly overstates the pure readiness-to-queryable gap. Both numbers are still comfortably sub-second here — this container was already locally cached (no image pull) and mounts no volume; a colder path (a network pull, a mounted volume, a busier host) would cost meaningfully more.

What changed in v0.2.12. From the v0.2.11 paste this section carried before (same machine, measured 2026-09-26) to the run above: fork + open + close at 100 MB went from 420.87 to 199.78 ms median (p95 435.87 -> 230.44), and from 68.34 to 54.89 ms at 10 MB; fork alone stayed flat (9.41 -> 9.63 ms at 100 MB). The difference is the open's checkout: a clone of the by-chain entry rather than a snapshot decode (see "Read-path sanity" above). The copy baselines moved by run-to-run noise only (backup() 93.06 -> 98.66 ms, Postgres template clone 119.91 -> 118.80 ms at 100 MB). One caveat on provenance: this run is the second make bench-isolation of the day — the first ran with Docker Desktop stopped, skipped the Postgres rows, and measured the offshoot rows at 55.63 / 201.43 ms (fork + open + close) and 9.70 / 8.80 ms (fork alone), within noise of the run above.

Re-run after the local-store List fix (d8f50dd, see "BranchBench topologies" below), same machine, 2026-09-30, Docker not reachable so the Postgres rows were skipped and the paste above is kept rather than replaced: fork alone 8.79 / 8.62 ms median at 10 / 100 MB (p95 10.24 / 9.71), against 11.96 / 9.63 ms above; fork + open + close 52.64 / 198.33 ms, within noise of the 54.89 / 199.78 ms above.

BranchBench topologies (v0.2.12)

BranchBench ("Aligning Database Branching with Agentic Demands" — Elaine Ang, In Keun Kim, Sam Weldon, Kevin Durand, Kostis Kaffes and Eugene Wu, of Columbia University's DAPLab, arXiv:2604.17180) is a benchmark for database branching under agentic workloads: five macrobenchmark workflows, each a parameter tuple over one tree-building loop — fork a branch, mutate it, evaluate it, sometimes prune it — run against hosted branchable databases. Its harness is at github.com/ElaineAng/db-fork.

What we ran. cmd/branchbench (make bench-branchbench) re-runs those five parameter tuples against a local offshoot store. The harness repository carries no license file, so none of it is vendored here: we took the numbers below — BranchBench's own notation: T workers, S steps per worker, F_r root fanout, F_i inner fanout, D max depth, C cross-branch queries for the run, γ prune probability, M_s schema changes per step, M_d data mutations per step, Q_v eval queries per step — and wrote our own generator, schema and SQL around them.

That is 2,310 forks in all (one per worker-step). The whole table takes about 2 minutes on the machine below (under 3 before v0.2.12 — see "What changed" below) and needs about 30 GB of free space in $TMPDIR — see the per-workflow stores below. go run ./cmd/branchbench -quick is a seconds-scale smoke run of the same five topologies (it is what go test ./cmd/branchbench runs).

• Every workflow gets its own store. Each one runs against a fresh store directory with its own ops.Init and its own seed, and that directory is removed before the next workflow starts (-keep keeps them; an interrupted run removes them on the way out). The rows are therefore independent: no workflow's latencies or bytes are measured against the branches an earlier one left lying around. It also bounds the disk needed to the largest single workflow — mcts peaked at 26.5 GiB before v0.2.12, hence the ~30 GB above, against well over 45 GB if the five shared one store. (Its Store peak now reads 43.0 GiB, but that is a logical sum over clones — see Store peak below. On a filesystem that cannot clone there are no shadows or by-chain entries; that case was not re-measured, so ~30 GB stays the figure to plan for.)
• The seed is our own CH-benCHmark-shaped generator (cmd/branchbench/seed.go), not BranchBench's SQL dump: TPC-C's transactional tables (warehouse, district, customer, item, stock, orders, order_line, new_order, history) plus TPC-H's region/nation/supplier dimension tables, at the scale BranchBench's configs describe (10 warehouses, 1K items, 100 customers per district — 10,000 customers, 10,000 orders, ~100,000 order lines). It is deterministic (math/rand seeded 1), which is what makes the per-workflow stores comparable — every workflow starts from a byte-identical 17 MiB database — and the program fails loudly if two seeds ever come out different sizes. Within a workflow, the root of the tree is chbench@main, whose head never moves after the seed checkpoint.
• No daemon, no network. Each step is Fork → Checkout → plain database/sql work on the checkout file → Checkpoint, straight against internal/ops (the in-process API the CLI uses). The checkpoint is what makes the loop a tree: a child forks from its parent's published state, so the parent's writes have to be snapshotted into the store before any child can fork from them. A node becomes eligible as a parent only once its own step has finished. Workers are goroutines through a semaphore (-concurrency, default 8); one mutex guards the tree.
• Parent selection, and what happens when a tuple outgrows its tree. BranchBench's rule is: the worker's current node if it still has depth and fanout left, else a uniformly random node that does, else the root. Two of these tuples ask for more branches than their own fanouts and depth allow — data_cleaning runs T×S = 200 steps against a tree that F_r=10, F_i=3, D=3 caps at 130 nodes — so the rule's last clause fires by construction and the surplus forks off the root. That is the specified behaviour, not a fallback of ours, but it does mean such a workflow ends with more children at depth 1 than F_r and more live branches than its fanouts alone would allow; the per-workflow lines print how many forks took that path.
• Cross-branch queries are aggregated in the driver — for each live branch, check it out, sum ol_amount read-only, total it up in Go — which is what BranchBench's harness does too. DoltHub's critique of the benchmark makes the point that this "live[s] as global knowledge in the driver rather than leveraging Dolt's native SQL-layer cross-branch support"; offshoot has no cross-branch SQL to undersell, so the driver-side shape is a fair match here — on a system that has one, it would not be. This pass is timed separately (the per-workflow lines) and counts in wall time, but not as branch management.
• Branch management means fork + checkout + checkpoint + destroy. The "Branch overhead" column is that time summed across workers over the run's worker-time budget: wall × effective concurrency, where effective means min(-concurrency, T) — a workflow with fewer workers than -concurrency can never use the whole semaphore, and dividing by the flag would understate its overhead badly (failure_repro has one worker). At concurrency 1 this is exactly BranchBench's "fraction of wall-clock time spent on branch management". synchronous=OFF is set on the step connections, which makes the productive SQL cheaper and so the overhead ratio larger: the conservative direction for us.
• Latencies are p50/p99 per operation at depth 1 and at the deepest depth the workflow reached, so a depth penalty would be visible. With these tuples most (operation, depth) cells hold well under 100 samples, and there a p99 is simply the slowest sample — the counts are on the per-workflow lines, and the note under the table repeats the caveat.
• Store peak is the largest the workflow's store directory got, sampled while it ran and once at the end, as the sum of the files' logical sizes. For a local store the checkouts live inside that directory, so it counts store objects and, per live branch, the checkout file — and since v0.2.12 also its .shadow and the checkouts-ro/~by-chain/ entries, each at full logical size although they are clones sharing blocks. Peak live counts live forked branches, excluding the root; for the γ=1.0 workflows, which prune every branch immediately after evaluating it, that number measures in-flight concurrency rather than accumulation — worth keeping in mind next to Neon's 20-live-branch ceiling quoted below.

A single pasted run can't separate a release-to-release shift from run-to-run noise; make bench-ab can, for a large enough shift: it reports a shift of more than 15% on fork, checkout or checkpoint p50 when none of the before and after runs overlap. make bench-ab [BEFORE=<ref>] (cmd/bench-ab) builds cmd/branchbench twice — once from git archive <ref> (make supplies the latest release tag when BEFORE is unset; the tool itself has no default), once from the working tree — and alternates the two binaries for 4 rounds per workflow on one machine, with one discarded warm-up run of the after build first, so drift during the series (thermal state, background load, page cache) moves both sides together instead of separating one make bench-branchbench run from the last one pasted below. It flags a regression only on a gating metric's depth-1 p50 (fork, checkout, checkpoint — wall and eval are informational and never affect the exit code) when the after median exceeds the before median by more than 15% and every after sample exceeds every before sample; the mirrored rule reports an improvement. The built binary exits 0, 1 (a gating regression) or 2 (a tool error); go run and make fold those codes into their own, so under make bench-ab a regression shows as make's exit 2 and the report tells the two apart. The nightly perf-ab job (.github/workflows/nightly.yml) builds the binary and runs the same comparison against git describe --tags --abbrev=0 --match 'v[0-9]*' on days main moves (and on its Sunday leg and manual dispatch), with -workflows failure_repro:1,simulation:4. The rule was tuned on the maintainer's machine, where the v0.2.15 → v0.2.16 per-checkpoint regression below measured +23% with non-overlapping before/after samples, while noise within a series stayed under 5%. The first run on the shared runner (a manual dispatch on 2026-10-09, run 37931344620, ubuntu-latest, v0.2.19 against main at 58ece9b, then still with simulation:8) took 29 min 21 s for the job and exited 0: on failure_repro the checkpoint p50 read 204.3 / 203.7 / 174.5 / 204.6 ms before and 173.9 / 159.9 / 159.7 / 170.6 ms after (medians 204.0 → 165.2 ms, −19.0%, "improved"); simulation's fork p50 read 17.1 → 14.4 ms (−15.7%, "improved"); every other gating row read "ok". The only runtime change between v0.2.19 and 58ece9b is the dependency bump of pierrec/lz4/v4 4.1.31 → 4.1.33 and aws-sdk-go-v2/service/s3 (the rest is the A/B tool itself and docs), so the two improvements are either the LZ4 bump, which the checkpoint's segment encoding exercises, or drift on the shared runner across the series; this run does not separate the two. The runner's checkpoint p50 is about four times the maintainer machine's for the same workflow. failure_repro's fork p50 before samples were 1.5 / 14.7 / 14.1 / 15.6 ms against 1.8 / 1.5 / 1.6 / 1.6 ms after, which the rule reports "ok" because one before run sits below the after runs, the conservatism it is meant to have. The eight simulation runs took 172 to 194 s each and drove the one-minute load average from 0.96 to 9.05 on the four-vCPU runner, so the job now runs simulation at four workers; its timing at four has not been measured yet.

Machine: as the header line below reports — darwin/arm64, Apple M5, macOS 27.0.1, local APFS disk, local-directory store backend, no network, measured 2026-10-08. The machine was not idle: uptime read load averages 3.39 3.79 3.48 just before the run and 4.03 3.94 3.58 just after it. Raw stdout of make bench-branchbench with its defaults (all five workflows, concurrency 8, 10 warehouses, 2h per-workflow cap), pasted verbatim, minus the two lines make prints first (the target's notice and its echoed go run line) and the per-workflow progress lines on stderr:

darwin/arm64, Apple M5, 10 cores, Go 1.27.1, offshoot v0.2.18-5-ge503ddf, measured 2026-10-08, seed 17 MiB, concurrency 8

Latencies are milliseconds. p99 is the maximum sample wherever a cell has fewer than 100 samples at that depth, which is most of them — the per-workflow lines below give the counts. Branch overhead is fork+checkout+checkpoint+destroy time summed over workers, over wall x effective concurrency (min(-concurrency, T workers)). Peak live counts live forked branches, excluding the root.

• simulation (flat star; T=1000, S=1, F_r=1000, F_i=0, D=1, C=1, γ=1.0, M_s=0, M_d=50, Q_v=1): max depth reached 1; 1 cross-branch query over 1 live branch (the root included) in 3 ms; all 1000 steps landed at d=1; store ended at 429 MiB of which 17 MiB is the seed; branch-management time 220.9 s summed over workers (4.7x wall at effective concurrency 8 of 8 requested); 0 CAS retries; store sizes approximate: 6 entries vanished or were unreadable during the size walks
• data_cleaning (wide shallow; T=10, S=20, F_r=10, F_i=3, D=3, C=2, γ=0.0, M_s=1, M_d=1, Q_v=1): max depth reached 3; 2 cross-branch queries over 201 live branches (the root included) in 2.7 s; 17 steps landed at d=1 and 132 at d=3 (the sample counts behind those two p50/p99 pairs); store ended at 8.3 GiB of which 17 MiB is the seed; 7 forks fell back to the root after the tree filled; branch-management time 36.0 s summed over workers (3.4x wall at effective concurrency 8 of 8 requested); 0 CAS retries; store sizes approximate: 1 entries vanished or were unreadable during the size walks
• software_dev (bushy; T=5, S=20, F_r=5, F_i=3, D=4, C=1, γ=0.1, M_s=1, M_d=1, Q_v=2): max depth reached 4; 1 cross-branch query over 84 live branches (the root included) in 886 ms; 5 steps landed at d=1 and 51 at d=4 (the sample counts behind those two p50/p99 pairs); store ended at 3.7 GiB of which 17 MiB is the seed; branch-management time 14.1 s summed over workers (2.4x wall at effective concurrency 5 of 8 requested); 0 CAS retries
• mcts (deep narrow; T=10, S=100, F_r=10, F_i=10, D=25, C=0, γ=0.1, M_s=0, M_d=1, Q_v=1): max depth reached 25; no cross-branch queries (C=0); 10 steps landed at d=1 and 73 at d=25 (the sample counts behind those two p50/p99 pairs); store ended at 30.8 GiB of which 17 MiB is the seed; branch-management time 271.9 s summed over workers (5.5x wall at effective concurrency 8 of 8 requested); 0 CAS retries; store sizes approximate: 11 entries vanished or were unreadable during the size walks
• failure_repro (flat, 1 worker; T=1, S=10, F_r=10, F_i=0, D=1, C=0, γ=1.0, M_s=5, M_d=45, Q_v=1): max depth reached 1; no cross-branch queries (C=0); all 10 steps landed at d=1; store ended at 82 MiB of which 17 MiB is the seed; branch-management time 829 ms summed over workers (0.5x wall at effective concurrency 1 of 8 requested); 0 CAS retries

(v0.2.18-5-ge503ddf is git describe for the branch that makes the checkpoint's stamp hash once, five commits past the v0.2.18 tag, with a clean tree; "The stamp hashes once again", below, is that change.)

Every workflow completed every step: 2,310/2,310 worker-steps, 114.5 s of workflow wall time in total (the five Wall cells summed: 47.0 + 10.7 + 5.9

• 49.2 + 1.7), about 1 min 58 s for the whole run including go run's build and a seed build per workflow (from the output file's creation and last-write times, 04:15:25 to 04:17:23), no CAS retries, nothing aborted or timed out.

Previous run (v0.2.13-11-gf182bb4, 2026-10-02)

Machine: as the header line below reports — darwin/arm64, Apple M5, macOS 27.0, local APFS disk, local-directory store backend, no other load, no network, measured 2026-10-02. Raw stdout of make bench-branchbench with its defaults (all five workflows, concurrency 8, 10 warehouses, 2h per-workflow cap), pasted verbatim, minus make's own echoed go run line and the per-workflow progress lines on stderr:

darwin/arm64, Apple M5, 10 cores, Go 1.27.1, offshoot v0.2.13-11-gf182bb4, measured 2026-10-02, seed 17 MiB, concurrency 8

Latencies are milliseconds. p99 is the maximum sample wherever a cell has fewer than 100 samples at that depth, which is most of them — the per-workflow lines below give the counts. Branch overhead is fork+checkout+checkpoint+destroy time summed over workers, over wall x effective concurrency (min(-concurrency, T workers)). Peak live counts live forked branches, excluding the root.

• simulation (flat star; T=1000, S=1, F_r=1000, F_i=0, D=1, C=1, γ=1.0, M_s=0, M_d=50, Q_v=1): max depth reached 1; 1 cross-branch query over 1 live branch (the root included) in 34 ms; all 1000 steps landed at d=1; store ended at 429 MiB of which 17 MiB is the seed; branch-management time 229.7 s summed over workers (4.9x wall at effective concurrency 8 of 8 requested); 0 CAS retries; store sizes approximate: 8 entries vanished or were unreadable during the size walks
• data_cleaning (wide shallow; T=10, S=20, F_r=10, F_i=3, D=3, C=2, γ=0.0, M_s=1, M_d=1, Q_v=1): max depth reached 3; 2 cross-branch queries over 201 live branches (the root included) in 2.5 s; 18 steps landed at d=1 and 130 at d=3 (the sample counts behind those two p50/p99 pairs); store ended at 8.3 GiB of which 17 MiB is the seed; 8 forks fell back to the root after the tree filled; branch-management time 29.9 s summed over workers (3.1x wall at effective concurrency 8 of 8 requested); 0 CAS retries
• software_dev (bushy; T=5, S=20, F_r=5, F_i=3, D=4, C=1, γ=0.1, M_s=1, M_d=1, Q_v=2): max depth reached 4; 1 cross-branch query over 84 live branches (the root included) in 710 ms; 5 steps landed at d=1 and 53 at d=4 (the sample counts behind those two p50/p99 pairs); store ended at 3.8 GiB of which 17 MiB is the seed; branch-management time 12.8 s summed over workers (2.4x wall at effective concurrency 5 of 8 requested); 0 CAS retries
• mcts (deep narrow; T=10, S=100, F_r=10, F_i=10, D=25, C=0, γ=0.1, M_s=0, M_d=1, Q_v=1): max depth reached 25; no cross-branch queries (C=0); 10 steps landed at d=1 and 69 at d=25 (the sample counts behind those two p50/p99 pairs); store ended at 30.7 GiB of which 17 MiB is the seed; branch-management time 239.9 s summed over workers (5.4x wall at effective concurrency 8 of 8 requested); 0 CAS retries; store sizes approximate: 4 entries vanished or were unreadable during the size walks
• failure_repro (flat, 1 worker; T=1, S=10, F_r=10, F_i=0, D=1, C=0, γ=1.0, M_s=5, M_d=45, Q_v=1): max depth reached 1; no cross-branch queries (C=0); all 10 steps landed at d=1; store ended at 82 MiB of which 17 MiB is the seed; branch-management time 780 ms summed over workers (0.5x wall at effective concurrency 1 of 8 requested); 0 CAS retries

(v0.2.13-11-gf182bb4 is git describe for the launch-hardening branch, eleven commits past the v0.2.13 tag, with a clean tree. An earlier version of this table came from a pre-release build with uncommitted changes; this one replaces it.)

Every workflow completed every step: 2,310/2,310 worker-steps, 107.9 s of workflow wall time in total (1 min 52 s for the whole run, including a seed build per workflow), no CAS retries, nothing aborted or timed out.

Against that run, the checkpoint column is the one that moved in the same direction everywhere: every p50 is higher, at depth 1 (simulation 109.2 → 121.8 ms, data_cleaning 73.4 → 90.4, software_dev 50.8 → 81.0, mcts 82.9 → 129.3, failure_repro 48.3 → 51.4) and at the deepest depth (data_cleaning 63.1 → 82.7, software_dev 53.4 → 61.5, mcts 64.6 → 84.4), and every p99 is higher except data_cleaning's at depth 1 (117.4 → 112.2). Wall time rose in four workflows (simulation 46.7 → 47.0 s, data_cleaning 9.7 → 10.7, software_dev 5.2 → 5.9, mcts 44.6 → 49.2) and held at 1.7 s in failure_repro. The fork, checkout and eval columns moved both ways (simulation's fork p50 41.1 → 31.0 ms, mcts's depth-1 fork p50 53.9 → 62.8 ms), and the stamp change below does not touch those operations. Between the two builds lie v0.2.14 through v0.2.18, including the checkpoint lease below (one more durable ref write per checkpoint), as well as the stamp change. The two runs were also on different days under different load, the earlier recorded as having no other load and this one at one-minute load averages of 3.39 before it and 4.03 after it, so no single change accounts for the difference between the two tables: only the alternating pairs below are a controlled comparison.

Measured changes and context

Since the 2026-10-02 table: at-rest checkpoints take the branch lease (v0.2.17). An at-rest checkpoint now writes the ref twice instead of once: it acquires the branch lease before it uploads, and releases it in the write that advances the head (reference). On a local store every ref write is fsynced (F_FULLFSYNC on macOS), so that is one more flush per checkpoint, and that table's checkpoint column reads low for code with the lease. Measured 2026-10-03 on the machine above, alternating a build of c9ca45d (before) with one of the change (after) run by run, at load average 2 to 6 (other processes were running):

Every after run's checkpoint p50 sits above every before run's: by the medians, +11.3 ms (+10%) eight at a time and +3.3 ms (+7%) one at a time. The full runs' wall time does not move outside run-to-run noise; the sequential quick runs take about 0.1 s more of 2.6 s, roughly twenty checkpoints' worth (the 5.0 s before run stalled once, which is also its 129.4 ms p99). On a heavily loaded machine (load average 10 and above) the eight-way p50 gap measured 20 to 40 ms, as concurrent fsyncs queue behind one another. On a local store that is the cost of the design: the acquire is what makes the object key private to the call, and it must be durable to fence a concurrent writer, while a ref read costs 0.06 to 0.16 ms there (timed per request in an eight-way burst). The build measured above also read the ref once more, before the acquire; the checkpoint now runs its refusals on the acquire's own read, which saves nothing measurable locally. The 2026-10-08 table at the head of this section was measured on a build with the lease, and with the one-pass stamp below, so it carries both.

On S3 every request is a round trip, so there the request count is the cost. Before the lease, an at-rest checkpoint read and wrote the ref once each, and a segment checkpoint also made two LISTs probing for another checkpoint's snapshot. Now it reads and writes the ref twice each (the acquire, then the head write), in sequence, and makes no probe: a snapshot checkpoint adds one GET and one conditional PUT, and a segment checkpoint makes the same number of requests as before, with a GET and a durable PUT in place of the two LISTs. A renewal adds a GET and a PUT per 10 s of upload. This was not measured against a real S3 endpoint.

The stamp hashes once again. After its head write, an at-rest checkpoint stamps the checkout's .sum sidecar, and the stamp is trusted only when the checkout holds what the store's head holds. Before v0.2.16 it took that from a shortcut: the encode's checksum, whenever the checkout's size/mtime fingerprint still matched the one taken right after quiesce. A same-size write in the same mtime tick could keep that match with the content changed, so v0.2.16 (6714394) applied checkout's 1 s racily-clean margin to the shortcut: from v0.2.16 through v0.2.18 the stamp trusted a matching fingerprint only once it was settled, and otherwise checksummed the live database a second time (changelog). A checkout written within the second before its checkpoint — every BranchBench checkpoint, which follows its step's writes — is not settled, so those stamps read the whole checkout twice: once for the sidecar's SHA-256 and once more for ltxio.ChecksumDatabase.

Earlier the same day, on the machine described under the 2026-10-08 table, each tag (and 6714394 and its parent 160cc8c) was built from a git archive of it into a clean directory and run with alternating builds within each series; the figures below are the checkpoint column of the harness's own rows. This alternating-build comparison is now the nightly perf-ab job's to run, through cmd/bench-ab instead of by hand; its first run is pending (see the BranchBench method notes above). The raw rows and their uptime readings are archived outside the repository, with the other raw benchmark outputs, and are not part of what this document publishes. Single-worker failure_repro checkpoint p50 was 45.5 / 45.5 / 45.7 ms on v0.2.15 and 56.2 / 55.5 / 56.7 ms on v0.2.16, alternating with v0.2.14 in the same series, at load averages 2.04 before and 2.50 after it; and 45.9 / 46.4 / 45.1 ms on 160cc8c against 57.4 / 55.3 / 56.7 ms on 6714394, in a series run directly after it for which no uptime was recorded. By the medians, v0.2.16 added 10.7 ms (56.2 − 45.5) and 6714394 alone added 10.8 ms (56.7 − 45.9). Eight-way simulation checkpoint p50 was 110.9 and 98.4 ms on v0.2.13 against 126.2 and 126.2 ms on v0.2.18 (wall 47.1 / 43.6 s against 47.0 / 47.6 s), at load averages that read 2.97 before the first run, 3.30, 5.00 and 5.35 between runs, and 4.57 after the last; that span also includes the lease above.

The stamp now reads the checkout once: ltxio.StreamChecksum folds the LTX page checksum over the same read that feeds the sidecar's SHA-256, the two digests computed concurrently, and the stamp is trusted only when the file's fingerprint did not change across that read, that checksum equals the head's, and no WAL frames sit beside the file. There is no shortcut on the stamp any more, so no margin either (checkout's fingerprint fast path keeps its margin), and the content check runs on every stamp. Measured 2026-10-08 on the machine described under the 2026-10-08 table, alternating a build of d8c81f8 (v0.2.18, before) with one of e503ddf (after) run by run. uptime read load averages 4.30 3.89 3.39 before and 3.94 3.83 3.38 after the failure_repro series, and 3.79 3.79 3.37 before and 3.82 3.88 3.51 after the simulation series:

One at a time, every after run's checkpoint p50 sits below every before run's: by the medians, 51.8 ms against 60.5 ms, 8.7 ms lower (60.5 − 51.8; about 14%: 8.7 / 60.5), and the p99 is lower in every round. The 10.7 and 10.8 ms above come from different series than these pairs, so the deltas are not directly comparable, and neither is the after build's 51.8 ms with v0.2.15's 45.5 ms, with the lease between them besides. Eight at a time, these two rounds do not resolve the change: each after run's p50 is below its before run's (by 6.1 ms and 0.4 ms), but the two before runs differ from each other by 17.8 ms, far more than either pair, and round 2's wall time went the wrong way (46.5 → 47.8 s). What remains is one pass: the stamp still reads the whole checkout for the sidecar's hash, now carrying both digests concurrently. In memory, that pass is slower than the SHA-256-only pass the stamp made when it took the shortcut (before v0.2.16 on any matching fingerprint, from v0.2.16 through v0.2.18 on a settled one): a 64 MiB micro-benchmark run the same day (random bytes, 4 KiB pages, archived in the same file) printed 3484.77 MB/s for SHA-256 alone, 2389.76 MB/s for the LTX page checksum alone, 1413.46 MB/s for the two one after the other and 2357.63 MB/s for the two concurrently. The page checksum is the slower digest, so StreamChecksum now folds each block's pages in two halves on two hashers at once (the rolling checksum is an XOR fold, so the halves recombine in any order). Measured afterwards on a 64 MiB file read from the page cache rather than in memory (BenchmarkStreamChecksum, go test ./internal/ltxio -run '^$' -bench StreamChecksum -benchtime=20x -count=3, load average 2.9 to 3.4): the one pass with both digests went from 32955077 / 32452812 / 32356846 ns/op (2036.37 / 2067.89 / 2074.02 MB/s) to 24638827 / 24537867 / 24467535 ns/op (2723.70 / 2734.91 / 2742.77 MB/s), against a SHA-256-only read of the same file at 22491600 / 22384619 / 22415335 ns/op before and 22064252 / 22031402 / 22104710 ns/op after (BenchmarkStreamChecksumSHAOnly). The pass now sits about 2.5 ms per 64 MiB above the SHA-256-only floor (24.54 ms against 22.06 ms by the medians) instead of about 10 ms.

On a local store, chain resolution slows as at-rest checkpoints accumulate on one lineage. Each checkpoint writes under the epoch its lease acquire minted, so it leaves one more data/<lineage>/<epoch>/ directory, and a local listing of the lineage, which every chain resolution not served by the recorded-chain cache below makes (checkout, a fork below head, materialize, and every descendant resolving through the lineage), reads every one. A segment checkpoint's own resolve and a fork at head skip this listing when the cache applies ("What changed in this work", below). Before the lease, at-rest checkpoints all wrote under the branch's one epoch. BenchmarkChainAfterCheckpoints takes n sequential at-rest checkpoints on one branch and then times Store.Chain at its head (2026-10-03, the machine above, load average 2 to 3, go test ./internal/ops -run '^$' -bench ChainAfterCheckpoints -benchtime=50x, before = c9ca45d):

Seeding the 1,000 took about 25 s before and about 45 s after, the difference being the growing resolve inside each segment checkpoint plus the extra fsync. compact makes its result self-contained, so it truly starts fresh. rollback and promote, which share via a base pointer by default since v0.2.12, start a new lineage ID that Store.Chain still resolves through the old lineage's directories (the base pointer it follows) until the new lineage writes its own snapshot or --materialize copies it forward, so for checkout, a fork below head and materialize the count moves, it does not reset, in that default case. Resolving a shared lineage's own head no longer pays that listing every time: the first checkpoint after a checkout lists once and records the chain across both lineages in the sidecar, and a later segment checkpoint, a fork at its head, a promote of it and a rollback to its head reuse that record, checked against the ref's base pointer with no store read ("A shared child's checkpoints", below). A daemon session, which writes all of its flushes under one epoch, adds one directory per session, as before. An S3 listing of a lineage is flat, so the count does not slow it. Removing an epoch directory once GC or a failed checkpoint empties it would bound the count for objects that are gone, but not for checkpoints the branch keeps.

What changed in this work (the two candidates named above). The .sum sidecar now records the resolved chain's member keys at the head identity it stamps; a segment checkpoint and a fork at head reuse that recorded chain instead of calling Store.Chain when the sidecar matches the ref's head, is snapshot-anchored, contiguous and ends at the head (any mismatch resolves as before). Separately, the local backend now removes an epoch directory, then its lineage directory, once a delete empties it, so a directory GC or a failed checkpoint attempt leaves behind no longer survives to be listed next time. Neither change touches what Store.Chain itself walks: checkout, a fork below head, materialize, and every descendant resolving through the lineage as a base (a descendant's own checkpoints only until its recorded chain serves them, from its second checkpoint after a checkout to its own snapshot, "A shared child's checkpoints" below) still list every epoch directory a kept checkpoint left, so the table above is unchanged by this work, confirmed by re-running the same benchmark on this branch:

Machine/date: darwin/arm64, Apple M5, macOS 27.0, Go 1.27.1, local APFS, load average 2 to 3, 2026-10-06. Before = main 9aca5e3, same machine, same day, in a throwaway worktree.

go test ./internal/ops -run '^$' -bench ChainAfterCheckpoints -benchtime=50x:

Single run, load average 2 to 3. The n=1 and n=100 points move more than expected on a single sample (2.0x and 1.19x respectively) — both are sub-millisecond, where scheduling noise dominates, and this was not investigated further. n=1,000, the only point large enough to read past that noise, is within 1.4% (14.2 → 14.4 ms): caching the chain a checkpoint or a fork at head needs does not change what Store.Chain itself walks for everyone else, and removing an emptied directory does not touch a directory a kept checkpoint still holds.

What does change is the segment checkpoint's own resolve, since it is now one of the callers that skips Store.Chain when the recorded chain applies:

go test ./internal/ops -run '^$' -bench SegmentCheckpointAfterCheckpoints -benchtime=20x:

Before, the segment checkpoint's own cost grew 26.3 → 36.4 ms as n went 1 → 1,000 (+10.1 ms), partly because it listed the lineage to resolve its own chain. After the recorded-chain cache removed that listing, the same growth became 25.5 → 30.3 ms (+4.8 ms) — a 6.1 ms saving at n=1,000 (36.4 → 30.3 ms). That 6.1 ms saving is smaller than the 14.2 ms a standalone Store.Chain walk costs at n=1,000 in the Chain table above; the two numbers come from separate runs over different operations and are not reconciled against each other. A smaller climb (+4.8 ms, against +10.1 ms before), not a flat one, is what the listing removal bought here.

A shared child's checkpoints. Until this change the recorded chain served only a lineage whose every key was its own, so a shared child (a fresh fork, or a lineage a shared rollback or promote started) listed its base lineage's epoch directories and its own on every segment checkpoint until it wrote its own snapshot. The recorded chain now serves it too: the keys on other lineages must start with a snapshot and end exactly at ref.Base.TXID, and the child's own keys must be a contiguous run of segments starting at ref.Base.TXID + 1, which the ref alone can confirm. BenchmarkSharedChildCheckpointAfterParentCheckpoints seeds n at-rest checkpoints on app@main and then, per iteration and untimed, forks a fresh child at main's head, checks it out and takes the child's first checkpoint; it times the child's second checkpoint, a one-row segment. It uses a fresh child per iteration because one child checkpointing over and over writes its own snapshot within the snapshot cadence (after main's 1,000 checkpoints, at its 8th checkpoint, since main's head chain is 9 deep), after which both builds resolve only the child's lineage. A first version of the benchmark that timed 50 checkpoints on one child measured mostly that path and was discarded.

Machine/date: darwin/arm64, Apple M5, macOS 27.0.1, Go 1.27.1, local APFS, 2026-10-08. Before = 8da5121 (the branch's merge-base with main), extracted with git archive into a scratch directory with the benchmark file copied in. The machine was not idle. Four series ran one after another, not alternating, in this order, with one-minute load averages read just before and just after each: after 1 (3.01, 4.19), before 1 (4.19, 2.96), before 2 (2.62, 2.72), after 2 (2.70, 4.54).

go test ./internal/ops -run '^$' -bench SharedChildCheckpointAfterParentCheckpoints -benchtime=50x (ns/op as printed, and divided by 1,000,000 for ms):

Every timed checkpoint in every series wrote a segment (segments 1.000). From n=1 to n=1,000, using the ms values in the table, each series changes by: before 1, 34.23 − 68.68 = −34.45 ms; before 2, 34.87 − 23.87 = +11.00 ms (+4.24 ms from 1 to 100, 28.11 − 23.87, then +6.76 ms from 100 to 1,000, 34.87 − 28.11); after 1, 24.84 − 22.36 = +2.48 ms (22.36 → 24.24 → 24.84 ms, rising at each step); after 2, 19.99 − 24.70 = −4.71 ms (24.70 → 25.40 → 19.99 ms). At n=1,000 the two before series read 34.23 and 34.87 ms and the two after series 24.84 and 19.99 ms. Before 1's n=1 and n=100 points (68.68 and 77.07 ms) are 2.88 and 2.74 times before 2's (68.68 / 23.87 and 77.07 / 28.11) and did not recur in before 2; they were not investigated, and before 1's n=1,000 point agrees with before 2's. The saving is bounded. A shared child's first checkpoint after a checkout still lists, since its sidecar holds no recorded chain yet. The checkpoints that take the recorded chain instead run from the second up to and including the one that writes the child's own snapshot, because a checkpoint resolves its head chain before it decides between a segment and a snapshot. That snapshot comes when the chain reaches the bound, which is SnapshotEvery when a daemon sets it and ForkShareMaxDepth (16) otherwise, so the deeper the parent's head chain, the fewer such checkpoints there are: after main's 1,000 checkpoints, the child's 2nd through 8th. Once that snapshot exists a child's checkpoints took the recorded chain before this change too.

Diagnostic: isolating the ref's own cost. BenchmarkSegmentCheckpointRefGrowth checks whether the ref's own Checkpoints map — which grows by one entry per checkpoint named uniquely, and is read, json-decoded, mutated and written back twice per at-rest checkpoint (the lease acquire, then the head write) — explains any of a segment checkpoint's growth with n. It seeds n checkpoints exactly as the benchmark above does (so epoch directories and checkout size match at each n), then times one more checkpoint twice: once letting the ref keep growing ("grown", as every other benchmark here does), and once forcing the ref's map back to empty immediately before each timed checkpoint via store.Store.PutRef directly, outside the timed region ("trimmed").

Machine/date: darwin/arm64, Apple M5, macOS 27.0, Go 1.27.1, local APFS, load average 2 to 3, 2026-10-06.

go test ./internal/ops -run '^$' -bench SegmentCheckpointRefGrowth -benchtime=20x:

Attributed within this diagnostic run. One more checkpoint after 1,000 prior checkpoints costs about 7.3 ms more than after 1 (grown: 23.85 → 31.17 ms). Keeping the ref's map at one entry throughout (trimmed: 22.08 → 25.88 ms) removes about 3.5 ms of that — the gap between grown and trimmed widens from 1.77 ms at n=1 to 5.29 ms at n=1,000, with epoch-directory count and checkout size held equal between the two variants at each n. The growing ref is the largest measured contributor to this diagnostic's own climb — a ref that grows by one entry per named checkpoint is a real product cost, read and written on every at-rest checkpoint afterward, not a benchmark artifact — and roughly 3.8 ms of the diagnostic's own +7.3 ms remains unexplained; candidates: the checkout and its reflinked .shadow, which the segment diff reads in full and which grows by one 200-byte row per seeded checkpoint here (documented as O(size) local I/O); or the .sum sidecar's own per-checkpoint write, including the chain field Task 1 added (though its length should be bounded by the snapshot cadence, not by n, making this the weaker candidate). Neither was isolated further.

This diagnostic's own climb (+7.3 ms, grown, 1 → 1,000) and the main benchmark's residual above (+4.8 ms, 25.5 → 30.3 ms) come from separate runs — their absolute levels were not reconciled against each other; only each benchmark's own n=1 → n=1,000 delta is used in this section.

On S3, TestAtRestCheckpointStoreRequests (internal/ops/rpc_count_test.go) counts backend calls against a local store directly (the fastest way to pin the exact count; it was not run against a real S3 endpoint): a segment checkpoint's lineage prefix is listed 0 times, "want 0 (the recorded chain)" — one fewer LIST per at-rest segment checkpoint than before this work, which listed it once. A local List call of a lineage prefix is exactly what the S3 backend turns into one LIST request, so the count carries over unchanged.

What changed in v0.2.12. The before is the same target run at 97320cc (the commit before this work) on the same machine the same day; its wall times reproduce the v0.2.11 table this section used to carry to within a few percent:

• All five topologies got faster end to end: simulation 72.2 → 45.7 s, data_cleaning 17.8 → 9.8 s, software_dev 7.6 → 5.4 s, mcts 63.3 → 41.9 s, failure_repro 1.9 → 1.6 s, and branch management's share of the time dropped by 26 to 32 points in each. The changes they exercise are clone checkouts, segment checkpoints and a fix to how a local store lists a lineage's objects (next bullet).
• mcts was 3x slower (197.2 s) partway through this work, and the cause was a store listing, not cloning. A local store's List walked the whole store directory — which also holds every checkout, its .shadow and the by-chain entries — and filtered by prefix. Resolving a chain Lists one lineage per base hop (up to 16 since forks share), and each fork, checkout and checkpoint resolves about one chain. Instrumented (table 1 under "Diagnostic instrumentation" below), one List cost 23.8 ms early in an mcts run and 132.0 ms late (with ~890 live checkouts on disk), one chain resolution 1,099.5 ms, and that was the ~0.7 s p50 every operation had gained at depth 25; the clone calls themselves stayed at 0.22-0.47 ms plus a 3.90-4.65 ms fsync. List now reads only the directory its prefix names: 0.06 ms per call late in an instrumented run with the fix. At depth 25, fork / checkout / checkpoint p50 went 703.8 / 734.7 / 730.6 ms (the regressed run) → 32.6 / 78.5 / 67.3 ms, against 84.1 / 290.1 / 103.1 ms before v0.2.12. The whole-directory walk predates v0.2.12; it only became expensive once chains recursed across base pointers and the directory filled with shadows and by-chain entries.
• Eval (read) latency rose where checkouts are fresh clones, and that is the page cache. simulation 11.5 → 56.4 ms, data_cleaning 7.0 → 50.3 ms and mcts 8.2 → 68.4 ms at depth 1; software_dev rose far less (12.9 → 15.6 ms) and failure_repro is unchanged (5.0 → 5.1 ms). A clone does not share its source's cached pages. Measured on its own (table 2 under "Diagnostic instrumentation" below): the eval query on a fresh clone of a 17 MiB mcts state took 50.6-54.9 ms, the same query on a freshly written copy 6.3-6.4 ms, and a second query on the clone 6.2-6.3 ms. So a checkout that got hundreds of ms cheaper hands its first query a cold cache; the step as a whole is still faster (the wall times above). mcts at depth 25 reads 103.0 ms; that is not the file's depth — fresh clones of kept mcts checkouts queried one at a time took 37.1 ms p50 at depth 1 and 39.7 ms at depth 25 (8 each, table 2) — and is attributed, by elimination rather than measurement, to 8 workers' cold reads and branch I/O sharing one disk.
• Checkout p50 at depth 1 rose for two topologies — software_dev 158.0 → 225.3 ms and mcts 348.6 → 405.1 ms — while it fell 7x to 8x for the other three. No diagnostic run isolated a depth-1 checkout, so this is not attributed here. What the runs above do show is where such a checkout can spend time that a pre-v0.2.12 one did not: a first miss on a chain builds the by-chain entry before cloning the checkout from it (see operations), and whatever the step reads from a fresh clone reads it with a cold page cache (table 2). Table 1's clone and fsync spans are single-digit milliseconds, so the clone call itself does not account for a 57-67 ms rise.
• Store objects shrank; the logical Store peak did not. simulation fell from 12.0 GiB to 650 MiB because its checkpoints are now segments of the pages its 50 mutations touched, not 17 MiB snapshots, and failure_repro fell from 167 to 82 MiB (no diagnostic run broke that store down, so this page does not attribute it). data_cleaning, software_dev and mcts grew because Store peak now also counts each live branch's .shadow and by-chain entries at full size. A diagnostic mcts run (before the List fix) kept its store to look (table 3 below): of 43.2 GiB logical, 0.64 GiB was store objects under data/; the rest was 14.55 GiB of checkouts, 14.55 GiB of shadows and 13.44 GiB of by-chain entries — clones of one another. The disk's used space grew by about 1.8 GiB over that run (df, so approximate — other processes wrote too).

Diagnostic instrumentation (2026-09-30, not a make target). The figures the three bullets above quote that are not in the pasted table. Machine: darwin/arm64, Apple M5, 10 cores, macOS 27.0, local APFS disk, Go 1.27.1, no other load, measured 2026-09-30.

Table 1 — wall-clock profile of mcts. Temporary instrumentation (a wall-clock span at the top of ~40 functions in internal/store, internal/ops, internal/reflink and internal/ltxio, bucketed by 30 s of run time; removed before committing) during go run ./cmd/branchbench -workflows mcts. Run 1 is the code before the List fix (216.1 s); its depth-25 samples fall in the late bucket. Run 2 has the fix (42.8 s). Means per call; the raw dumps were not kept, and these are the figures transcribed from them the same day.

Table 2 — cold page cache on a clone. The eval query (SELECT ol_w_id, SUM(ol_amount) FROM order_line GROUP BY ol_w_id, Python's sqlite3) against files from a kept mcts store (after the fix), one query at a time. Rows A: a by-chain entry cloned with cp -c, or written out byte-for-byte, then queried; three trials. Rows B: a fresh clone of each of 8 kept checkouts per depth, p50.

Table 3 — what the logical Store peak holds. A kept mcts store (before the fix, 214.8 s run), logical sizes by area (os.walk, st_size):

df used space grew 1,897,452 KiB (1.81 GiB) over that run.

What BranchBench found on hosted systems. For context — these are the authors' numbers on hosted Postgres-family systems, not ours:

• From the abstract: "systems optimized for fast branching suffer up to 5-4000x slower reads as branches deepen, while systems optimized for fast data operations incur 25-1500x higher branch creation and switching latency." The paper states those ranges across systems and topologies; it does not attribute either end of either range to a particular topology (the two figures below do name theirs).
• Dolt, on the deep-narrow MCTS topology (10 workers × 100 steps), completed 170/1000 worker-steps before the harness's 2-hour cap — its published run_stats_final records timed_out: true at elapsed_sec: 7581.85.
• Neon's cloud service "does not support more than 20 concurrent 'live' branches", which is what ends its MCTS run rather than the clock: DAPLab's write-up reports 33/1000 worker-steps. (The harness's raw neon_full MCTS stats file sums its per-worker completed_steps to 23, not 33. We cite the published figure and flag the discrepancy rather than reconcile it.)
• And the headline: "no system was able to fully complete the five agentic applications within the 2 hours."

Not apples to apples. Those are the authors' runs on hosted, network- attached, multi-tenant Postgres-family services. Branch "creation" there includes control-plane work and compute provisioning, and every measurement carries WAN round trips, TLS, connection pooling and provider-side rate limiting — BranchBench's own "mini" configs exist to stay inside Neon's 20-branch and API-rate ceilings. Ours is a local file store: no network, no compute to provision, no other tenants, a different seed, a different concurrency model (goroutines over one process's checkout files, not clients over a server), and a storage engine whose byte accounting is not on the same curve as either reference system's. Our numbers sit next to theirs because the shape of the workload is the same, not because the systems are comparable. Finishing all five in minutes is not evidence that offshoot "beats" Dolt or Neon at anything they were measured on; it is what these five topologies cost on a local copy-on-write SQLite store.

What the numbers do and do not show. These bullets were written against the 2026-09-30 run of this table (v0.2.11-15-g67cc6b1-dirty), which neither table above replaces cell for cell: the cells they quote, and "the table" and "this run" in them, are that run's.

• Reads do not get slower with depth. That is the axis BranchBench's "5-4000x slower reads as branches deepen" finding lives on, and it is why the table reports every latency at depth 1 and at the deepest depth reached. A branch's checkout is a plain SQLite file, so once it is on disk the store is not in the query path at all and depth cannot enter the read cost: software_dev reads 15.6 → 13.3 ms. The table's data_cleaning (50.3 → 66.9 ms) and mcts (68.4 → 103.0 ms) cells do rise with depth; queried one at a time, fresh clones of mcts checkouts read the same at depth 1 and depth 25 (37.1 vs 39.7 ms p50, table 2 under "Diagnostic instrumentation"), so the rise in the table is the cold-cache first read under 8-way load described under "What changed", not depth in the file.
• Depth does not cost branch operations. At depth 25, mcts fork, checkout and checkpoint are 32.6 / 78.5 / 67.3 ms p50, against 61.0 / 405.1 / 79.2 ms at depth 1 (the first steps, whose checkouts build the by-chain entries from the store) and 84.1 / 290.1 / 103.1 ms at depth 25 before v0.2.12. By design, offshoot forks by sharing the parent's chain until the fully-resolved chain reaches 16 members, at which point the next fork materializes a fresh floor snapshot instead (ops.ForkShareMaxDepth; offshoot compact does the same on demand), and an at-rest checkpoint writes a snapshot rather than a segment at the same bound — so a deep spine crosses that floor repeatedly and pays an O(size) copy each time; that is in these p50s and p99s.
• These are latencies under 8-way concurrency, not per-op costs in isolation. failure_repro (T=1, no other worker sharing this run's disk and page cache) gives a control point from inside this same table: its uncontended checkout is 8.1 ms p50, against 40.1-405.1 ms at depth 1 in the 8-way topologies on the same 17 MiB seed. The difference is queueing: eight workers deep on one laptop's disk and page cache. Fork at head also quiesces and checks the parent's checkout; instrumented in an mcts run that cost 4.6-10.8 ms mean per fork (table 1 under "Diagnostic instrumentation"), so it is not what separates these cells from failure_repro — 8-way I/O contention and the floor materialization above are what is left, and this run does not split them. Read these columns as what a branch-heavy agentic workload costs end to end, not as primitive latencies; the two sections above are where the primitives are measured.
• Branch management dominates most of these topologies. With the denominator built from the concurrency each workflow can actually use, the spread is 40% (data_cleaning) to 65% (mcts) — failure_repro 47%, software_dev 48%, simulation 60% — down from 71-96% before v0.2.12. failure_repro, whose step does 5 schema changes and 45 mutations, now spends under half its single worker's time forking, checking out, checkpointing and destroying. data_cleaning counts its two cross-branch passes (2.4 s of its 9.8 s wall) in wall time but not as branch management. The honest reading is still that BranchBench's tuples are deliberately branch-heavy — the ratio is a statement about the workload's mix at least as much as about offshoot, and the way to move it is to do more work per branch.
• Store peak is "what the run held", not a steady state, and not disk used. It sums logical file sizes: store objects, and per live branch the checkout, its .shadow and the by-chain entries — the last three clones of one another on APFS (the mcts breakdown under "What changed" is the worked example). destroy is a metadata operation — it tombstones, it does not reclaim, because reclaiming is offshoot gc's job and nothing runs gc during the benchmark; simulation still holds 429 MiB after pruning all 1,000 of its branches (γ=1.0). BranchBench measures reclaim as its own metric; this table does not measure it at all.
• Rows are independent, and that mattered. Each workflow gets its own fresh store (see "What we ran"). An earlier version of this benchmark reused one store across all five: the same topologies reported substantially higher fork latencies there, growing with how much the earlier workflows had already written. Anyone porting this methodology should isolate the stores rather than assume the store's size is irrelevant.
• One host, one run. These are quantiles within a single run, not a median-of-N across runs, on a laptop. Repeat runs of the whole table have reproduced the completion counts exactly and the wall times within a few percent (mcts after the List fix: 41.9 s here, 42.4 s and 42.8 s in two mcts-only runs the same day), but run-to-run variance on a laptop is real and no fleet average is implied.

Method

• Each subtest seeds a fresh SQLite database of the target size (bulk INSERT of blob rows into one table, single transaction, PRAGMA synchronous=OFF) and checkpoints it once — outside the timed loop, once per size, per the brief's "build once per size, fork/checkout/open b.N times" shape. Content is reused pseudo-random bytes (not unique per row): nothing in the write path compresses page content, so uniqueness has no bearing on the measurement and generating fresh randomness at the 4GB size would dominate seed time for no benefit.
• BenchmarkForkAtHead forks to a fresh, uniquely-named branch on every iteration and destroys it (b.StopTimer'd) before the next, so peak extra storage during a run stays close to one seeded database's worth rather than growing with b.N. Every seeded database checkpoints cleanly, so its chain is always exactly one snapshot — the fast path's precondition (see "The reflink / clonefile fork fast path: what changed" below) holds for every iteration in this suite.
• BenchmarkCheckoutCleanSkip calls Checkout repeatedly against a checkout that never dirties between calls (the clean fast path performs no write), so every iteration measures the same "already correct, don't rebuild" path Task 1 added. Unaffected by Task 6a; re-measured anyway since make bench runs the whole suite together.
• BenchmarkSessionOpen measures session.Open's latency (AcquireLease + Checkout + starting the capture engine and waiting for its resume-or-rebase verdict to settle — not the full async startup rebase, which does not gate Open's return). It also forces one settling flush per size (not per iteration) and reports the stored snapshot's byte size as a custom settleSnapshotBytes metric — see "Settling-flush cost" below. Also unaffected by Task 6a (session.Open never calls Fork).
• Every subtest calls b.SetBytes(dbSize) with the actual on-disk checkout size (not the nominal seed target — SQLite page/header overhead makes them close but not identical), so both ns/op and the reported MB/s are meaningful.
• make bench runs -count=3 -short. -short excludes the size=4GB Fork case from the routine sweep; see "The 4GB case" below for why and how it was measured instead.

Machine

• Host: darwin/arm64, Apple M4, 10 cores, 16 GiB RAM, macOS 26.0/27.0 (build 26A5388g), local disk (APFS, ~48 GiB free at measurement time), Go 1.26.5. Same physical machine for both the "before" and "after" numbers below.
• Linux container: golang:1.24-bookworm under Docker Desktop on the same host. Docker Desktop's Linux VM on Apple Silicon runs linux/arm64 containers, not linux/amd64 — these numbers show the container/cgroup overhead and a different libc/allocator, not a different CPU architecture from the host. sqlite3 3.40.1 + gcc 12.2.0 installed in-container via apt-get before running go test. The container's writable layer (where t.TempDir()/b.TempDir() land, since no GOTMPDIR is set) is backed by Docker Desktop's Linux VM disk — confirmed via go test ./internal/reflink -run TestCopyFileClonedFlagOnClonableFS -v inside the container, which skips loudly ("temp filesystem does not support reflink/clonefile"). This is the "ext4-without-reflink" case referenced throughout: Task 6a's fast path still fires (one object copy instead of decode+re-encode), it just can't clone, so it falls back to reflink.CopyFile's plain-byte-copy path — see "The reflink / clonefile fork fast path: what changed" below for what that's still worth.
• S3 path: minio/minio:latest in Docker on the same host (make bench-s3 now runs RustFS instead, since MinIO's images were withdrawn; the numbers below were measured against MinIO and not re-measured), reached over loopback (127.0.0.1, host-mapped port). This is a real S3-API round trip (HTTP, request signing, PutIf-style conditional writes) but not a real network path — no WAN latency, no TLS. Treat it as "S3 API overhead measured locally," not "S3 performance from a real client location." store.S3.CopyObject returned the ErrCopyUnsupported sentinel unconditionally in Task 6a; Task 6b (this update) replaces that with a real server-side CopyObject call (gated to objects at or under S3's 5GB single-request CopyObject limit — see "The S3 server-side copy fork: what changed" below). All MinIO-local numbers throughout this section are exactly that: MinIO-local. They say nothing about a real AWS S3 endpoint's network latency, throughput, or server-side copy performance at scale — do not read them as an AWS claim.
• "Before" (Task 6a's numbers) measured 2026-08-06 (pre-Task-6a, main); "after Task 6a" measured 2026-08-06 (post-Task-6a, this branch), same day, same machine. "After Task 6b" (S3 server-side copy) measured 2026-08-06, same machine, separate section below.

The reflink / clonefile fork fast path: what changed

ops.Workspace.copySnapshotToNewLineage (the primitive behind Fork, Rollback, and Promote) now checks, before doing anything else, whether the source checkpoint's chain (store.Chain) resolves to exactly one member, and that member is a snapshot. When it does — the common case: a freshly checkpointed branch, or any at-rest fork of a branch that has never been flushed through a live session's segment cadence — the child's seed is a direct backend-level object copy (store.Backend.CopyObject) of the source snapshot object, verified by resolving the child's own chain afterward, rather than a materialize-to-temp-file-then-ltxio.EncodeSnapshot round trip. That post-copy verification (tryFastForkCopy's second call to store.Chain, which is itself one Backend.List call) is a real extra round trip the fast path pays that the slow path doesn't need — it's what's included in every ForkAtHead number in this document (the numbers were never measured with it stripped out), and on the S3 backend it's a real extra HTTP request per fork alongside the HEAD/CopyObject/PutRef sequence described in "Results: S3 path" below. internal/reflink backs the local implementation: a filesystem clone (clonefile(2) on darwin, the FICLONE ioctl on Linux) when the filesystem supports it, silently falling back to a plain byte copy otherwise. S3 returns store.ErrCopyUnsupported unconditionally in this slice (Task 6b adds S3 server-side CopyObject, gated to objects ≤5GB — since v0.2.4 larger objects copy server-side too, via multipart UploadPartCopy), so Fork against S3 is unaffected — see "Results: S3 path" below.

The precondition does NOT hold once a daemon session has flushed even one segment past the branch's last snapshot (SnapshotEvery cadence): a multi-member chain has no single source object to copy, so Fork falls through to exactly the pre-6a materialize-and-re-encode path, unchanged (internal/ops/gc_chain_test.go's TestForkFastPathSkipsMultiMemberChains covers this).

The S3 server-side copy fork: what changed

store.S3.CopyObject (internal/store/s3.go) now issues a real server-side copy — S3's CopyObject API (a PUT carrying an X-Amz-Copy-Source header, no request body, no download to this process and no re-upload) — instead of returning ErrCopyUnsupported unconditionally. ops.Fork's fast path (Task 6a, unchanged by this slice) now actually fires against an S3 backend for any single-snapshot-chain checkpoint, the same way it already did against the local backend.

The 5GB guard (as measured then; superseded in v0.2.4 — see the version note at the top: CopyObject now multipart-copies sources up to S3's 5 TiB per-object ceiling, and only >5 TiB falls back): S3's CopyObject supports source objects up to 5 GiB in a single request; anything larger needed the multipart UploadPartCopy API, which this backend did not implement at measurement time (out of scope for that slice — see copyObjectMaxBytes in s3.go). CopyObject HEADs the source first — one request, not a download — reads ContentLength, and returns the ErrCopyUnsupported sentinel for anything over 5 GiB before ever attempting the copy. That sentinel is the exact signal ops.Fork's fast path (Task 6a) was already wired to treat as "fall back to the slow, materialize-and-re-encode path," so an oversized checkpoint still forks correctly, just without the server-side-copy win — pinned directly at the store.S3.CopyObject boundary by TestS3CopyObjectOverSizeLimitFallsBack (internal/store/s3_test.go), using a fake-S3 hook (FakeS3.SetSizeOverride) to make a small stored object report a >5GB size on HEAD without actually allocating or uploading one.

The shared storetest.RunConformance CopyObject subtest (added in Task 6a, at the one place backend conformance tests live) now runs for real against S3 — both the in-process fake (TestS3Conformance/CopyObject) and the MinIO-gated real-provider suite (TestS3RealProvider/Conformance/ CopyObject, make test-s3) — instead of skipping on the sentinel, exactly as the brief's placement of that subtest was meant to let 6b inherit for free. The in-process fake (internal/store/storetest/fakes3.go) gained CopyObject-request handling (an X-Amz-Copy-Source-carrying PUT) to make that possible — without it, the fake would have silently overwritten the destination with an empty body instead of a copy, since a CopyObject request has no body of its own.

Results: local store (host, make bench, -count=3)

ForkAtHead is ~11.7x faster at 64MB and ~14.5x faster at 512MB. The other two benchmarks are within run-to-run noise of their pre-6a numbers, as expected — Task 6a touches only Fork's (and Rollback's/Promote's) object-copy path.

Raw "after" output:

BenchmarkForkAtHead/size=64MB-10             40    30251214 ns/op   2221.09 MB/s
BenchmarkForkAtHead/size=64MB-10             40    30296356 ns/op   2217.78 MB/s
BenchmarkForkAtHead/size=64MB-10             39    30357906 ns/op   2213.29 MB/s
BenchmarkForkAtHead/size=512MB-10             6   198433674 ns/op   2708.56 MB/s
BenchmarkForkAtHead/size=512MB-10             6   197286291 ns/op   2724.31 MB/s
BenchmarkForkAtHead/size=512MB-10             6   198909368 ns/op   2702.08 MB/s
BenchmarkCheckoutCleanSkip/size=64MB-10      48    24529970 ns/op   2739.13 MB/s
BenchmarkCheckoutCleanSkip/size=512MB-10      6   191314104 ns/op   2809.35 MB/s
BenchmarkSessionOpen/size=64MB-10            39    29178191 ns/op   2302.77 MB/s   settleSnapshotBytes=67710606
BenchmarkSessionOpen/size=512MB-10            6   197517292 ns/op   2721.12 MB/s   settleSnapshotBytes=541863202

(Original "before" raw output is preserved in this file's git history — commit that added "Task 6a" to the log, or git show 2ba8fdb:docs/benchmarks.md.)

Isolating the fast path itself

ForkAtHead's at="" call still pays for Fork's pre-existing uncheckpointed-changes check (warnIfUncheckpointed → checkoutState → a full SHA-256 hash of the checkout file, Task 1 machinery, unrelated to Task 6a) on every call. That check is itself O(size) — at ~2.7 GB/s on this machine, it accounts for essentially all of the 198.2ms measured at 512MB above (compare CheckoutCleanSkip's ~192ms, which is dominated by the same hash). To see the fast path's own cost in isolation, forking at a named checkpoint (at="seed") skips that check entirely:

go test ./internal/ops -bench 'ForkAtHead' -benchmem -run '^$' -short
# (fork_bench_test.go temporarily edited: w.Fork(db, "main", branch, "seed", 0))
BenchmarkForkAtHead/size=64MB-10            146     9290401 ns/op    7232.28 MB/s
BenchmarkForkAtHead/size=512MB-10           148     9254606 ns/op   58075.83 MB/s

Each line above is a single run (go test's default -count=1), not this document's usual -count=3-and-average the tables further up use — labeled that way here so it isn't mistaken for the same kind of measurement. It is a spot-check, taken once per size with the temporary edit noted above, not a repeated-and-averaged number.

~9.3ms regardless of size — the 64MB and 512MB single-run figures land within a few percent of each other. This is the number that matters for the design spec's "~40ms" figure: it's in the same regime, on this machine, in this fast path's applicability window. It is not the number the committed benchmark suite reports (that suite deliberately measures Fork's actual default behavior, at="", which still includes Task 1's separate O(size) check), and it required a one-line, temporary, uncommitted edit to produce — it's reported here for honesty about what's actually constant-time and what isn't, not as a replacement for the table above.

Results: Linux container (golang:1.24-bookworm, linux/arm64, -count=3)

This container's writable layer does not support FICLONE (verified — see "Machine" above), so every one of these ForkAtHead numbers is the plain-copy fallback, not a real clone. It is still 7.7x faster at 64MB and 5.9x faster at 512MB than before Task 6a: skipping the decode-into-SQLite-pages-then-re-encode-into-LTX round trip and doing one io.Copy instead is a real win on its own, exactly as expected for "ext4-without-reflink" — a plain copy is still O(size), just a much smaller constant than decode+re-encode, and the fallback triggers silently with no special-casing needed anywhere in ops.

Raw "after" output:

BenchmarkForkAtHead/size=64MB-10             21    50917202 ns/op   1319.61 MB/s
BenchmarkForkAtHead/size=64MB-10             25    45323748 ns/op   1482.46 MB/s
BenchmarkForkAtHead/size=64MB-10             26    46263989 ns/op   1452.33 MB/s
BenchmarkForkAtHead/size=512MB-10             3   495316153 ns/op   1085.10 MB/s
BenchmarkForkAtHead/size=512MB-10             2   598705230 ns/op    897.72 MB/s
BenchmarkForkAtHead/size=512MB-10             3   443778612 ns/op   1211.12 MB/s
BenchmarkCheckoutCleanSkip/size=64MB-10      45    23315294 ns/op   2881.83 MB/s
BenchmarkCheckoutCleanSkip/size=512MB-10      6   177494146 ns/op   3028.09 MB/s
BenchmarkSessionOpen/size=64MB-10            44    24421828 ns/op   2751.26 MB/s   settleSnapshotBytes=67710606
BenchmarkSessionOpen/size=512MB-10            6   187563653 ns/op   2865.53 MB/s   settleSnapshotBytes=541863202

Within noise of the host numbers for the two unaffected benchmarks — unsurprising, since both run on the same physical machine (the container is not a different CPU or a network hop). The value of this table remains process isolation and confirming the code builds and runs unmodified under a plain golang:*-bookworm image, plus — new for Task 6a — direct evidence that the fast path's fallback behaves correctly and still helps on a filesystem with no clone support.

Results: S3 path (make bench-s3, MinIO in Docker, -count=1)

This table is Task 6a's measurement, kept as published: at that point store.S3.CopyObject still returned ErrCopyUnsupported immediately, with no HTTP call, before copySnapshotToNewLineage fell through to the unchanged materialize-and-re-encode path. The 64MB numbers landed within 1% of the pre-6a run; 512MB was ~12% slower, which read as single-sample Docker/MinIO variance (-count=1, no repeats to average, a different container/network warm-up state than the original run) rather than a regression — there was no code-path difference on this backend for Task 6a to have introduced. CheckoutCleanSkip and SessionOpen are, as before, close to the local numbers because their dominant cost never touches the backend at all — Task 6b doesn't change that either.

S3 server-side CopyObject (make bench-s3, MinIO in Docker, -count=1)

ForkAtHead is ~3.5x faster at 64MB and ~4.4x faster at 512MB: the server-side copy means this process never downloads the source snapshot object, never re-encodes it, and never re-uploads it — MinIO copies the object on its own side of the wire, so the round trips this process pays for are the HEAD (size gate), the CopyObject request itself, one List call to verify the child's chain resolves after the copy (see "The reflink / clonefile fork fast path: what changed" above — this is extra work the fast path pays that the slow path doesn't, not free), and the same PutRef/GetRef calls every fork makes regardless of path. It is NOT as fast as the local numbers (1.03s vs. ~198ms at 512MB): Fork's uncheckpointed-changes SHA-256 check (see "Isolating the fast path itself" above) still runs against the LOCAL checkout file either way (OFFSHOOT_CHECKOUTS is local disk regardless of backend), so that ~192ms cost is present in both the local and S3 numbers; the remaining gap is real HTTP round trips (HEAD, CopyObject, PutRef) plus MinIO's own server-side copy time, which is real disk I/O on MinIO's side even though it never crosses the network back to this process. CheckoutCleanSkip and SessionOpen are unaffected, as expected (neither calls CopyObject).

Raw "after Task 6b" output:

BenchmarkForkAtHead/size=64MB-10              7   152972149 ns/op    439.24 MB/s
BenchmarkForkAtHead/size=512MB-10             1  1030185084 ns/op    521.72 MB/s
BenchmarkCheckoutCleanSkip/size=64MB-10      39    27249625 ns/op   2465.75 MB/s
BenchmarkCheckoutCleanSkip/size=512MB-10      6   194418798 ns/op   2764.49 MB/s
BenchmarkSessionOpen/size=64MB-10            34    30661629 ns/op   2191.36 MB/s   settleSnapshotBytes=67710606
BenchmarkSessionOpen/size=512MB-10            5   202717683 ns/op   2651.32 MB/s   settleSnapshotBytes=541863202

Same caveat as every other S3 number in this document: MinIO-local, -count=1, no WAN latency, no TLS — this is "S3 API overhead measured locally," not an AWS performance claim, and a real network hop to a real AWS region would add real latency Task 6b's design does nothing to hide (one HEAD + one CopyObject round trip per fork, same as any other S3 API call this codebase makes).

The 4GB case

BenchmarkForkAtHead/size=4GB exists in the source but is skipped under -short (make bench's default) — the PM's amendment for this task timeboxed a 4GB run to a single attempt, not a routine -count=3 sweep costing several GB of RAM and disk on every make bench invocation. Before and after Task 6a, run once, directly, host-only:

go test ./internal/ops -bench 'ForkAtHead/size=4GB' -benchmem -run '^$' -benchtime=1x -timeout 20m

# before Task 6a
BenchmarkForkAtHead/size=4GB-10   1   83385791875 ns/op   51.56 MB/s   18172542944 B/op   15762922 allocs/op

# after Task 6a
BenchmarkForkAtHead/size=4GB-10   1    3409868667 ns/op 1260.96 MB/s      89224 B/op        489 allocs/op

83.4s → 3.41s, a ~24.5x speedup, and the allocation profile tells the real story: before Task 6a, Fork's slow path buffered the entire re-encoded snapshot in memory (bytes.Buffer in ltxio.EncodeSnapshot) before writing it out, so a 4GB database drove ~18GB of allocation over the call (~4.5x the database size) — GC pressure at that scale, not disk or network, was the likely reason it didn't scale linearly from the smaller sizes. After Task 6a, that same fork allocates 489 times and 89KB total: the fast path never materializes the content in Go memory at all, it's a backend-level object copy end to end. The 4GB fork is still slower than the 512MB fork (3.41s vs. 198ms is more than the 8x size ratio would predict) — consistent with the "Isolating the fast path itself" finding above: the remaining time is dominated by Fork's separate O(size) uncheckpointed-changes SHA-256 check, which does scale with size, not by the object copy itself.

What's still O(size) after the fast-path fork

• Fork's uncheckpointed-changes check (warnIfUncheckpointed → checkoutState, Task 1 machinery, not touched by Task 6a): every default Fork(..., at="", ...) call quiesces and SHA-256-hashes the whole checkout file to decide whether to print a "forking last committed state" warning. This is now the dominant cost in every ForkAtHead number above — see "Isolating the fast path itself".
• Checkout's clean-skip path (Task 1): avoids the rebuild, but checkoutState still quiesces the WAL and computes a fresh SHA-256 over the whole checkout file on every call to confirm it's actually still clean. Cheaper than a rebuild (no temp file, no rename, no LTX encode) — the ~2.7-3.0 GB/s throughput above is close to this machine's raw SHA-256 rate — but not free, and still O(size). Unchanged by Task 6a.
• session.Open: dominated by the same Checkout clean-skip cost, since Open calls Checkout internally. Unchanged by Task 6a.
• Fork itself, when the fast-path precondition doesn't hold: a multi-member chain (post-segment-flush branch) still takes the full materialize-and-re-encode path, unchanged from before Task 6a. At measurement time a single-snapshot checkpoint whose object was over S3's 5GB single-request CopyObject limit also fell back this way; since v0.2.4 that case is instead a multipart server-side copy (see the version note at the top). Before Task 6b, EVERY S3 fork took the slow path regardless of chain shape or size (store.S3.CopyObject returned ErrCopyUnsupported unconditionally) — see TestForkFastPathSkipsMultiMemberChains.

Task 6a's fast path applies only when the checkpoint being forked resolves to a chain of exactly one member, and that member is itself a snapshot (store.Chain returns a single snapshot entry, no segments layered after it). That's the common case for the benchmarks above — a freshly seeded-and-checkpointed database — and for any at-rest fork of a branch that has never been flushed through a live session with segment cadence (SnapshotEvery > 1). It stops applying the moment a daemon session has flushed even one segment past the branch's last snapshot: forking from a multi-member chain still requires replaying segments to materialize before anything can be re-encoded, so the pre-6a numbers in this document remain the correct comparison baseline for that case. The design spec's "forking a 10GB database took ~40ms" figure describes the reflink/clonefile mechanism this task implements, in exactly this single-snapshot-chain window, on a clone-capable local filesystem — see "Isolating the fast path itself" for the closest this document gets to checking that figure directly (~9.3ms, in the same regime); it is cited here only as the target this benchmark suite exists to check against, never as a claim about ForkAtHead's own reported numbers, which also include Task 1's separate O(size) check.

Settling-flush cost

Update (Milestone 2 follow-up, shipped): the measurements below still describe the upload's size accurately for the case where it happens, but it is no longer unconditional. rebaseline now skips this flush entirely when BOTH the checkout Open received was already proven byte-identical to the branch's head at open time AND the checksum recorded in that checkout's own .sum sidecar exactly matches what the checkout actually contains once the session's real startup rebase finishes — see docs/status.md's "Settling-flush checksum-compare suppression" row and internal/session/session.go's rebaseline doc comment for exactly why both conditions matter. A read-only agent reopening an unmodified checkout uploads nothing at all — AND, just as important, the checksum comparison itself costs no store read: it's read straight out of the local sidecar Checkout/Checkpoint/Rollback/ Promote already stamp, never fetched fresh from the store. An earlier, since-reverted version of this fix DID fetch it fresh on every Open, which meant downloading the entire head object whenever it happened to be a full snapshot — the exact case a permanently-idle read-only session always hits, since it never advances past its first (snapshot) head. internal/session/flush_test.go's TestReadOnlySessionWithCleanCheckoutMakesNoStoreWrites asserts this at the backend-call level (exactly 2 Gets during Open — both tiny ref reads, zero for the head object — and zero of anything at all across the idle settle window). BenchmarkSessionOpen, re-run against the final code:

size	ns/op	throughput	B/op	allocs/op
64MB	29,996,972	2239.92 MB/s	62,634	394
512MB	199,496,139	2694.13 MB/s	62,906	393

(-benchtime=3x, local filesystem backend — the default this benchmark falls back to without OFFSHOOT_S3_TEST_BUCKET set; B/op/allocs/op stay flat across a 8x size increase, consistent with no per-byte store traffic on this path — the unit test above is the byte-level proof, this is its latency consequence.) A session whose checkout had to be (re)materialized first (first-ever open, a dirty/stale checkout, or one whose sidecar predates checksum-recording — see status.md's row for the exact fail-toward-settling case) still pays the settling-flush cost measured below, once.

Every daemon session's first auto-flush tick after Open uploads a full snapshot — the forceSnapshot path — even for a session that never writes anything (a read-only agent that only queries). This was a ledgered tradeoff from Task 2: closing a startup-rebase race required forcing that first flush to be a full re-baseline rather than a segment.

BenchmarkSessionOpen measures this directly rather than only describing it: settleSnapshotBytes above is the actual stored object size of that forced snapshot, read back from the store after triggering it. It tracks the database size almost exactly — 67,710,606 bytes for the 64MB seed (a ~67MB source database) and 541,863,202 bytes for the 512MB seed — i.e. the settling flush is O(size), not a fixed background cost. A daemon serving many idle read-only sessions against large databases pays one full-snapshot upload per session, once, ~30s after each Open (the daemon's default -flush-every), not per query and not repeatedly. This call never goes through Fork's fast path (it's a session flush, not a fork) and is unaffected by Task 6a.