Skip to content
Documentation
Testing

Testing

This guide describes libfossil’s test strategy for contributors adding tests or investigating failures. For the module layout referenced throughout, see architecture.md.

Overview

libfossil’s test suite has three tiers:

  1. Unit tests — standard go test ./... coverage for individual packages under the root module and internal/....
  2. DST simulation tests — deterministic simulation scenarios under ./dst/... that drive multiple repos through a virtual clock with fault injection and invariant checks.
  3. Dual-driver matrix — the whole suite runs against two SQLite drivers (modernc and ncruces) to catch driver-specific behavior.

Running tests

The Makefile wraps the common targets:

  • make test — runs go test ./... -count=1 -timeout=120s with the default modernc driver.
  • make test-drivers — runs the full suite twice: go test ./... -count=1 -timeout=120s followed by go test -tags test_ncruces ./... -count=1 -timeout=120s.
  • make test-otel — runs the out-of-workspace OTel submodule with cd observer/otel && GOWORK=off go test ./... -count=1.
  • make test-all — chains test-drivers, test-otel, and go build ./cmd/libfossil/. This is the target CI uses and what you should run before opening a PR.

make setup-hooks points core.hooksPath at .githooks/. The installed pre-commit hook runs both drivers with -short, go vet, the OTel submodule, and the CLI build in roughly 45 seconds. Skip with git commit --no-verify only in emergencies.

Driver matrix

libfossil abstracts SQLite behind db/ and two driver submodules. Both must pass identical tests:

  • modernc (default) — pure-Go port of SQLite. Used by every build that doesn’t set a tag. Registered via internal/testdriver/modernc.go under //go:build !test_ncruces && !test_mattn.
  • ncruces (wasm-capable) — selected with -tags test_ncruces, suitable for WASM targets. Registered via internal/testdriver/ncruces.go under //go:build test_ncruces.

GitHub Actions runs test and test-ncruces as parallel jobs (see .github/workflows/test.yml). The ncruces job excludes cmd/libfossil because the shipped binary only needs one driver built in; DST and unit tests are executed under both. For the canonical invocation, use make test-drivers rather than hand-rolling tag flags.

DST (Deterministic Simulation Testing)

DST is inspired by FoundationDB and TigerBeetle: run many repos in a single process under a simulated clock, inject faults, and replay a failing seed deterministically. The harness lives in dst/simulator.go and dst/invariants.go.

Seeds and reproducibility

dst.SimConfig.Seed drives every random source used by the simulator: the event queue, network fault decisions, and a SeededBuggify PRNG that gates fault-injection sites. Any failure on seed N reproduces exactly on seed N.

dst/scenario_test.go exposes three flags driving TestDST:

go test ./dst -run TestDST -seed=42 -level=hostile -steps=10000
  • -seed=<int64> — seed passed to SimConfig.Seed (0 uses test-specific defaults).
  • -level={normal,adversarial,hostile} — picks a severity struct setting DropRate and Buggify. normal disables faults; hostile uses 20% drop and BUGGIFY.
  • -steps=<int> — caps the number of events processed.

Seed-parameterized scenarios (e.g. TestCloneDSTSeedSweep in dst/clone_test.go) loop over seeds via t.Run(fmt.Sprintf("seed_%d", seed), ...). To reproduce a single sub-seed: go test ./dst -run TestCloneDSTSeedSweep/seed_7 -v.

Invariants

dst/invariants.go defines per-node and cross-node checks. Simulator.CheckSafety runs the per-node set after every SafetyCheckInterval steps; convergence checks run once the sim has quiesced:

  • Safety (anytime): CheckBlobIntegrity (UUID matches hash of expanded content), CheckDeltaChains (delta.srcid resolves to a blob), CheckNoOrphanPhantoms, CheckUVIntegrity, CheckTagxrefIntegrity, CheckTableSyncIntegrity.
  • Convergence (after fault-free sync): CheckConvergence (leaves match master blob-for-blob), CheckSubsetOf, CheckUVConvergence, CheckTableSyncConvergence, CheckTombstoneConvergence.

Failures return a *InvariantError{Invariant, NodeID, Detail} so the failing invariant, the offending node, and the seed are all in the log line.

BUGGIFY

BUGGIFY marks code points where faults can be injected — truncated batches, dropped cards, retries, corrupted nonces, byte-flipped content. In production the check is a single boolean read; under DST the seeded PRNG decides whether to fire.

Two styles appear in the tree:

  1. Process-global, used outside the sync path. simio.Buggify(probability) consults a process-global state toggled by simio.EnableBuggify(seed) / DisableBuggify() (see simio/buggify.go). Example from internal/content/content.go:

    // BUGGIFY: flip a byte in expanded content to exercise UUID-mismatch detection.
if simio.Buggify(0.01) && len(content) > 0 {
    corrupted := make([]byte, len(content))
    copy(corrupted, content)
    corrupted[0] ^= 0xFF
    return corrupted, nil
}

  2. Scoped, passed through session options. BuggifyChecker is an interface (Check(site string, probability float64) bool) defined in internal/sync/session.go and re-exported from the root as libfossil.BuggifyChecker. Sync code threads a checker through so each site has a stable name:

    // BUGGIFY: 5% chance drop the last gimme card.
if len(gimmes) > 1 && cs.opts.Buggify != nil &&
    cs.opts.Buggify.Check("clone.buildRequest.dropGimme", 0.05) {
    gimmes = gimmes[:len(gimmes)-1]
}

dst.SeededBuggify in dst/simulator.go implements BuggifyChecker with a seeded *rand.Rand. Simulator.New enables both styles when SimConfig.Buggify is true, wiring independent PRNG streams into the client, the server-side MockFossil, and the global simio state.

Writing new tests

  • Prefer table-driven tests in unit code; include success and failure rows.
  • If you touch sync, merge, clone, or checkin, add a DST scenario under dst/ and run it across a seed range (see TestCloneDSTSeedSweep for the pattern).
  • Mark any new fault-sensitive code point with BUGGIFY. Use the scoped BuggifyChecker style when the site lives inside the sync session; reserve simio.Buggify for code that runs outside an explicit session scope.
  • Assert invariants. If the new behavior warrants one, add it to dst/invariants.go and wire it into CheckSafety or a convergence helper.
  • Confirm tests pass under both drivers (make test-drivers) before requesting review.
Extension PointsProduction patterns