Contributing to offshoot

offshoot is at 0.x, single-maintainer, and moving fast. That means: ask before investing a lot of time in something structural (see "what needs an issue first" below), but small, well-scoped PRs are welcome and get reviewed quickly.

Dev setup

You need:

• Go 1.26+ with cgo enabled, which means a C compiler on PATH (offshoot uses github.com/mattn/go-sqlite3, which is cgo-backed — there's no pure-Go build). macOS: xcode-select --install; Debian/Ubuntu: apt-get install build-essential.
• The sqlite3 and sqldiff CLIs on PATH. The test suite shells out to sqlite3 as a stock foreign writer and for .dump equivalence checks, and offshoot diff's default mode (and its tests) call sqldiff.
  • macOS: brew install sqlite sqldiff, then export PATH="$(brew --prefix sqlite)/bin:$PATH" — the sqlite formula is keg-only, so brew does not link its sqlite3 into PATH for you (and the stock macOS image may not ship one at all).
  • Debian/Ubuntu: apt-get install sqlite3 sqlite3-tools (sqldiff is in sqlite3-tools).
• Linux or macOS — there are no Windows code paths, and none are planned for v1

Tests gated on sqlite3/sqldiff skip when the binary is missing locally, but fail when the CI environment variable is set (GitHub Actions sets it automatically; see internal/testutil/require.go). So a green go test ./... on a machine without sqldiff proves less than it looks like; run CI=1 go test ./... to turn those skips into failures and confirm your setup is complete.

Optional, only needed for the SDK test tier:

• Python 3.10+ (Python SDK tests and tooling; macOS: brew install python@3.12, Debian/Ubuntu: apt-get install python3). The Makefile runs python3 by default and refuses an older one with a message; when the platform's python3 is older (macOS ships 3.9), pass the interpreter explicitly: make PYTHON=python3.12 test-sdks.
• Node 20+ (TypeScript SDK tests; macOS: brew install node, elsewhere via nodejs.org or your package manager)

Optional, only needed for the pytest fixture plugin's OWN test tier (sdk/python/offshoot/pytest_plugin.py — the offshoot-db[pytest] extra; the base SDK's plain-unittest suites above need none of this):

pip install -e "sdk/python[pytest]" pytest-xdist

Optional, only needed for the LangGraph companion's end-to-end test tier (sdk/python-langgraph; kept separate so the base SDK test still proves a stdlib-only install):

python3 -m venv .venv-langgraph
.venv-langgraph/bin/pip install -e sdk/python -e "sdk/python-langgraph[test]"

go build -o offshoot ./cmd/offshoot
make lint          # gofmt + go vet + staticcheck (staticcheck is advisory locally)
make lint-python   # ruff check + ruff format --check over the SDKs, scripts and examples

Refreshing the hash-pinned CI requirements

CI never names a third-party Python package on a pip install line: every one it installs directly comes from a hash-pinned lock under requirements/ (pip install --require-hashes -r requirements/<name>.txt), and the repo's own packages install afterwards with pip install --no-deps -e. That does not cover build isolation: pip install -e and python -m build still fetch the build backend (setuptools) into an isolated environment without hashes. Each .txt is generated from the .in beside it, whose header names the exact command. To change a dependency (or pick up new releases), edit the .in and regenerate with uv:

uv pip compile --python-version 3.12 --generate-hashes --universal \
  -o requirements/ci-langgraph.txt requirements/ci-langgraph.in

(same for ci-pytest-plugin and ci-publish-tooling). --universal keeps the markers valid for both CI interpreters (setup-python 3.12 and ubuntu's system python3). When an SDK's pyproject.toml changes its dependencies or test extras, update the matching .in in the same PR — CI installs the editable package with --no-deps, so a dependency missing from the lock fails the job instead of being fetched unpinned. The Dockerfile's base images are digest-pinned (image:tag@sha256:...) and Dependabot's docker ecosystem bumps them weekly.

Test tiers

There are eight, and they cost very different amounts of time. Run the tier that matches what you touched — don't run torture on a docs typo, and don't skip it on a capture-engine change.

go test ./... -race is hermetic and needs only the sqlite3 CLI — that's the bar for "does this even work," and CI runs it on every PR. make test runs the same suite without -race if you want a faster local loop, but -race is what actually gets checked before merge, so run it before you push.

make test-torture is not optional cosmetics: it's the suite that proves the capture engine detects every divergence instead of silently absorbing it (random kill -9 of the writer plus graceful capturer shutdown/restart, verified with sqlite3 .dump equivalence after every round). The harness does not SIGKILL the capturer; docs/testing.md is the canonical statement of that boundary. If your change touches internal/capture or a flush path in internal/session, a PR without a torture run attached will get sent back for one, not tests we'll take on faith.

make test-s3 needs a real provider or a local RustFS — the in-process fake used by the unit tests proves nothing about a real bucket's conditional-write behavior, which is what the whole compare-and-swap safety story rests on.

Local CI

.github/workflows/ci.yml runs five jobs on every PR: test (ubuntu, gofmt + go vet + go test ./... -race), cow-paths (macOS), metrics-lint (promtool over the exposition format), sdks (both SDK suites plus the publish dry-run), and s3-conformance (RustFS in Docker). Prose-only changes (docs/**, site/**, **.md) skip ci.yml entirely by design; .github/workflows/pages.yml builds the docs site on those PRs instead, so a broken intra-doc link still fails the PR.

cow-paths is the one macOS job: since v0.2.12 checkouts clone from a by-chain cache entry and at-rest checkpoints diff against a reflinked shadow, and those fast paths only run on a filesystem that can clone, which ubuntu's ext4 cannot — so that job runs just the packages holding those paths on APFS, and proves a clone works before trusting the result. The expensive tiers live in .github/workflows/nightly.yml: torture daily, fuzz daily, linux-reflink daily (the copy-on-write tests on a loopback XFS filesystem with reflinks, so Linux FICLONE is covered and not only APFS), macos-test (the full go test ./... on macOS) weekly, and real-provider-conformance against AWS S3 daily (on main, when the NIGHTLY_S3 repository variable is on). docs/testing.md explains what each of those proves.

make ci-local mirrors the test, s3-conformance and sdks jobs locally through four targets, in minutes instead of waiting on a runner (cow-paths needs a real APFS runner and metrics-lint a pinned promtool; see "What it does NOT replicate" below):

make ci-local runs those four in sequence and prints a pass/fail-per-job summary table with timings; a single job is runnable alone too. See scripts/ci-local.sh's header comment for the job-by-job detail.

What it does NOT replicate — CI remains the merge gate, this is pre-merge signal:

• GHA runner image drift. ci.yml's macOS leg broke without a version bump on our side because macos-latest rolled to a new image ("macos-26") that had quietly dropped the sqlite3 CLI from PATH — see ci.yml's "Install sqlite3 CLI (macos)" step comment. ci-local-host runs on whatever your machine's own Homebrew/PATH state already is; it cannot catch a runner image regression like that, only a real macos-latest run can.
• Runner pip/HOME layout. The sdks job's pytest-plugin step installs into a venv instead of pip install --break-system-packages onto system Python specifically because, on the actual (non-root, no-sudo) GHA runner, that flag falls back to a user-site install under $HOME — and pytest's pytester fixture repoints HOME for its subprocess runs, making pytest invisible to them (ValueError: Pytest terminal summary report not found). ci-local-sdks uses a venv from the start, so it can't reproduce the runner-layout failure mode it was written in response to — see ci.yml's sdks job comment for the full incident.
• Actions-specific behavior — actions/checkout, actions/setup-go, caching, GITHUB_PATH, etc. — none of it runs locally; ci-local calls the same underlying commands (go vet, go test, make test-s3, ...) directly.

A green ci-local is strong local evidence, not a substitute for the real GitHub Actions run — push and let CI have the final word.

Code style

• Match the patterns already in the file you're editing over anything you'd reach for on a green field. This codebase leans on long doc comments that explain why (races found, invariants relied on, what breaks if you change the order of two calls) — see internal/capture/engine.go for the norm. Terse code with a paragraph above it explaining the non-obvious part beats clever code with no comment.
• Errors are wrapped with context, not just passed through: fmt.Errorf("rebase: checkpoint TRUNCATE: %w", err) — prefix with the operation, wrap with %w so callers can still errors.Is/errors.As.
• No new dependencies without a good reason — the dependency list is deliberately short (AWS SDK, mattn/go-sqlite3, superfly/ltx). Pulling in something new for a small feature is a smell; ask in the issue first.
• go vet ./... and gofmt clean before you push.

PR expectations

• Tests required for behavioral changes. A PR that changes behavior with no test coverage will get asked for one before review continues.
• Torture run required (and mentioned in the PR description — the template has a checkbox) for anything touching capture or flush paths, per the table above.
• Docs updated in the same PR when behavior changes — README, ROADMAP, or code comments, whichever describes what you touched.
• Conventional commits: feat:, fix:, test:, docs:, chore:, etc. Look at git log for the norm in this repo.
• DCO sign-off required. Every commit needs a Signed-off-by trailer — use git commit -s. This project uses the Developer Certificate of Origin instead of a CLA: signing off is your statement that you wrote the patch or otherwise have the right to submit it under the project's license (Apache-2.0), not a copyright assignment to anyone. Unsigned commits will get bounced back for a rebase with -s, not silently merged.

Release process

There are two independent release tracks — the offshoot binary and the two SDKs — with two independent versioning schemes, on purpose: the SDKs are thin wrappers over a stable daemon wire protocol and can move on their own cadence, while the binary's v0.2.x tags govern the on-disk storage format. The two schemes already read differently (binary at v0.2.x, SDKs at 0.1.0) — that divergence is the design, not drift.

Binary releases

Tag v0.2.<n> (vX.Y.Z generally) and push it. .github/workflows/release.yml builds Linux/macOS binaries for both architectures, creates a GitHub release, and publishes a ghcr.io Docker image. workflow_dispatch builds the same artifacts under a dev-<short-sha> name without tagging, for a smoke check before tagging.

Before tagging, bump every surface that states the number, in one PR:

1. CHANGELOG.md — a ## [X.Y.Z] - YYYY-MM-DD header above the entries
2. Formula/offshoot.rb's url + sha256
3. server.json's "version" (the MCP registry manifest)
4. docs/ci-recipes.md's OFFSHOOT_VERSION: vX.Y.Z pins
5. the version chip in site/index.html's header

scripts/check-version-surfaces.sh vX.Y.Z fails loudly, naming each surface that disagrees with the tag, so run it before pushing the tag; release.yml runs it too before signing anything. The Formula is the one surface bumped after the tag (its sha256 is of the tarball GitHub serves for the tag), so it is checked separately once that commit lands: scripts/check-version-surfaces.sh --with-formula vX.Y.Z.

SDK releases

Both SDKs publish together, from one tag, at one version number — sdk-v<version> (e.g. sdk-v0.1.0), never sdk-py-v.../sdk-ts-v... separately. This is the simplest scheme for a two-SDK repo where both clients track the same daemon wire protocol and ship from the same PR/review cadence; splitting them into independently-versioned tags would mean two release processes to keep straight for no benefit either SDK currently needs. Revisit only if one SDK ever needs to ship a fix the other genuinely doesn't.

sdk/VERSION is the single source of truth for the SDK version number. Both manifests spell the version out literally (PyPI and npm both require a real version string in their own manifest — neither supports pulling it from an external file at publish time without extra tooling this repo doesn't carry), so a release bump is three edits kept in lockstep by hand:

1. sdk/VERSION
2. sdk/python/pyproject.toml's [project].version
3. sdk/typescript/package.json's version

python3 scripts/check_sdk_versions.py (also make check-sdk-versions) fails loudly, listing every mismatch, if any of the three disagree — this runs as the first step of make dry-run-python-sdk and again in .github/workflows/publish.yml's check job before either publish job does anything. Tagging sdk-v<version> is a fourth place the same number has to agree; scripts/check_sdk_tag_version.py <tag> (run automatically on a tag push) catches a tag cut against a stale sdk/VERSION.

Publishing gate

.github/workflows/publish.yml triggers on sdk-v* tags and workflow_dispatch, and always builds + validates both SDKs' real publish artifacts (sdist/wheel + twine check + install-test for PyPI; npm pack

• install-test for npm). It only uploads them when the PUBLISH_ENABLED repository variable (Settings > Secrets and variables > Actions > Variables) is exactly "true" — unset or anything else runs the same job in dry-run mode (build + check + install-test, no upload). The same build-and-check tier also runs on every PR via make dry-run-sdks in ci.yml's sdks job, so a manifest mistake is caught long before a release tag exists, not discovered the first time PUBLISH_ENABLED flips on.

Turning it on requires, at minimum:

• The offshoot-db name claimed on PyPI, with Trusted Publishing (OIDC) configured for this repo + publish.yml + the pypi job — no PyPI secret is stored in this repo; pypa/gh-action-pypi-publish authenticates via the job's id-token: write permission alone.
• The @offshoot-db scope claimed on npm, and registry auth configured: today that means an automation token in the NPM_TOKEN repository secret (used as NODE_AUTH_TOKEN for npm publish --provenance). npm has since shipped OIDC-based Trusted Publishing for GitHub Actions, which would remove the need for a stored token the same way PyPI's does — adopting it means confirming the npm CLI version on the runner supports it and configuring @offshoot-db/client as a Trusted Publisher on npmjs.com for this exact repo + workflow, then deleting the NODE_AUTH_TOKEN env var from the npm job. Not done yet; NPM_TOKEN is the working path today.
• PUBLISH_ENABLED=true set as a repository variable.
• No org transfer needed — the manifests already match the real repo. sdk/python/pyproject.toml's [project.urls], sdk/typescript/package.json's repository/homepage/bugs, and server.json's repository.url all point at github.com/sricola/offshoot — the repo's actual remote today (git remote -v). npm publish --provenance independently verifies that package.json's repository.url matches the repo the publish is actually running from (it builds the provenance attestation from the real GitHub OIDC claims), so publishing from sricola/offshoot with the manifest already pointing at sricola/offshoot passes that check with nothing further to do here. Only the PyPI/npm name-claims and PUBLISH_ENABLED above are still outstanding; the package registry names themselves stay offshoot-db (PyPI) and @offshoot-db (npm) — those are unrelated to the repo's GitHub identity and are not being renamed.

Claiming both names is a manual, one-time, user-side action tracked as a deliberately-out-of-band item in ROADMAP.md and docs/status.md — not something a PR against this repo can do on its own.

PUBLISH_ENABLED=true alone is not enough to publish anything. A real upload additionally requires the workflow run itself to be an actual sdk-v* tag push — workflow_dispatch (a manual "Run workflow" click) is always dry-run, unconditionally, even with the gate on. Note that a ref check alone (startsWith(github.ref, 'refs/tags/sdk-v')) is NOT sufficient to establish that: GitHub's "Use workflow from" picker lets a maintainer dispatch a workflow_dispatch run against a tag ref, which makes github.ref match refs/tags/sdk-v* too, even though nothing was pushed. Both the pypi and npm jobs' actual upload steps therefore check github.event_name == 'push' as well (startsWith(github.ref, 'refs/tags/sdk-v') && github.event_name == 'push'), not just the ref pattern and not just the check job's resolved output, so the guard survives both a dispatch-against-a-tag-ref run and a bug in that job's own gating logic.

Optional extra gate: GitHub Environments. publish.yml's pypi and npm jobs each target a GitHub Environment (pypi-publish, npm-publish respectively). Both exist with no protection rules the first time the workflow runs, until a maintainer adds one — configuring required reviewers on either (repo Settings > Environments > pypi-publish / npm-publish > required reviewers) is a one-click way to require a human approval on top of the tag+PUBLISH_ENABLED gate before that job's steps run at all. Like the two name claims and the PUBLISH_ENABLED flip, this is a manual, user-side action this repo's PRs can't do on their own — worth doing before the first real release, not required for the pipeline to be correct without it.

What's most welcome right now

• Docs. The gap between what's implemented and what the design spec promises is real at 0.x, and closing that gap in writing — an honest implemented/deferred matrix, a CLI reference, fixing stale examples — is some of the highest-leverage work available and doesn't need prior coordination.
• Provider conformance runs. make test-s3 against AWS S3 and a report of pass/fail (see the provider table in README.md) is directly useful and low-risk to contribute.
• CLI ergonomics. Better error messages, --help text, flag consistency, output formatting — open a PR directly, no issue needed first, unless it changes a flag's meaning or removes one.

What needs an issue first

Open an issue (or comment on an existing one) before writing code for:

• Storage format — anything touching the LTX segment layout, snapshot format, ref schema, or layout versioning. This is the hardest thing to change once something has written data with it.
• Capture engine — internal/capture's lock dance, rebase/takeover logic, or the Sink contract. The invariants here (no duplicate Apply, every divergence detected not absorbed) are load-bearing and easy to violate in a way that only shows up under torture.
• Fencing — epoch bumps, lease semantics, ref CAS. This is what keeps two writers from corrupting a branch; a change here needs a design discussion, not just a green test run.

For everything else: if you're not sure, open an issue and ask — that's cheaper than a rejected PR.