# External comparator and declared-evaluator harness

`mutationsieve external-validate` is the executable counterpart to the public
validation protocol. It can bind precomputed or newly generated callsets whose
operator-declared labels are Mutect2, Strelka2, LoFreq, DeepSomatic, or Lancet2,
then run one declared evaluator command for each callset. Labels are retained
as declarations; they do not prove the identity or behavior of an executable.
The harness does not download software or data, select tool parameters,
interpret evaluator outputs, or confer scientific or clinical validity.

The contract boundary is intentional:

- the plan is strict JSON: duplicate fields, unknown fields, non-finite
  numbers, malformed hashes, and string-form commands are rejected;
- every input, executable, launcher, model, JAR, or other tool artifact is a
  path-bound regular file pinned by SHA-256 (the backing filesystem may still
  be remote);
- commands are JSON argv arrays and run with `shell=False`, no stdin, a small
  deterministic environment, bounded time, bounded combined logs, an isolated
  empty working directory, and best-effort grouped-descendant cleanup on
  completion, timeout, or log overflow. POSIX descendants can escape a process
  group with `setsid`; Windows uses a Job Object when available, but neither
  mechanism is an OS filesystem/network sandbox;
- URL, traversal, response-file, shell-like, attached-option, and unpinned
  path argv tokens are rejected; file arguments must be complete-token
  placeholders;
- declared executable/tool-artifact bytes are hash-checked and the exact
  expected version-command output must match before any analysis runs. This is
  evidence about those bytes and output, not proof that the declared caller
  label identifies the actual biological engine or behavior;
- truth, reference, query callability, confident regions, optional assay
  targets, assembly, and exact reference contig lengths are explicit bindings;
- caller `analysis_inputs` must exactly match its `${input:...}` argv tokens.
  Direct truth/confident/effective bindings and exact-SHA-256 aliases are
  excluded from caller argv and precomputed query callsets. Reformatted,
  transformed, or semantically equivalent copies are not detectable by this
  rule. Scientific independence therefore requires an externally controlled
  lockbox and operator separation that this harness does not enforce;
- the harness independently parses the BEDs and proves that `effective_bed` is
  exactly the merged interval-set intersection of the confident, callable,
  and optional assay-target BEDs;
- every expected output is declared, hashed, and size-bounded. Protected
  artifacts are rehashed at before/after checkpoints, and the inventory within
  the configured output root is exact. Missing or undeclared entries, symlinks,
  Windows reparse points, or a changed artifact detected at a checkpoint fail
  the run. These checks do not enforce immutability, detect a transient change
  restored between checkpoints, audit writes outside the output root, or rule
  out later writes by a detached process;
- `external-validation.manifest.json` is written after the declared payload and
  returned only after a final checkpoint verification. Verify it again when
  consuming the result; its mere presence cannot rule out a later mutation by
  an escaped process or another actor.

Hashes detect byte substitution at the recorded checkpoints; they do not make
an executable trustworthy or immutable. Run only reviewed tools and plans.
File-level pinning also does not close over
transitive dynamic libraries, Python packages, JVM contents, drivers, kernels,
or other runtime dependencies unless they are explicitly included as tool
artifacts. The harness makes no intentional socket, HTTP, or FTP client calls
and rejects URL argv tokens. However, accepted paths can be UNC paths or
network-mounted filesystems, and invoked programs can discover files or make
network connections. Preventing either requires external OS/container
filesystem and network isolation. The truth-access rule is only an argv and
exact-digest exclusion, not lockbox independence or an OS sandbox.

## Three explicit modes

One mode flag is required, so inspecting a plan can never accidentally execute
third-party code.

```bash
# Syntax and protocol only. Files and tools need not be present. Exit 3.
mutationsieve external-validate comparison-plan.json \
  --out external-protocol --protocol-only

# Verify local hashes, exact BED intersection, and expected version output. Exit 3.
mutationsieve external-validate comparison-plan.json \
  --out external-dry-run --dry-run

# Explicitly execute callers and one declared evaluation per callset. Exit 0
# only after the declared output-root checkpoint contract and manifest verify.
mutationsieve external-validate comparison-plan.json \
  --out external-results --execute
```

Protocol-only and dry-run artifacts deliberately have
`declared_execution_contract_pass: false`. A successful execute result uses
status `DECLARED_EXECUTION_CONTRACT_SATISFIED_WITH_LIMITATIONS` and
`declared_execution_contract_pass: true`. This means only that declared
commands completed and artifacts satisfied the output-root checkpoint
contract. It does not mean the run was isolated, independently blinded,
scientifically valid, or immutable. Every mode reports
`scientific_pass_fail: "NOT_COMPUTED"`,
`scientific_independence_established: false`,
`os_sandbox_established: false`, and
`clinical_validation_established: false`. Read the machine-readable
`limitations`, `truth_separation_scope`, `process_isolation_scope`,
`completion_contract`, `network_scope`, and `tool_identity_scope` fields before
using the evidence.

The output directory must be absent or empty. This prevents a stale manifest
or stale callset from being mistaken for the current run.

## Plan shape

The following abbreviated example imports a precomputed callset labeled
Mutect2 and runs a hash-pinned evaluator launcher. Every `sha256` value must be
replaced with the actual lowercase digest; relative paths resolve from the plan
file's directory. `reference_contigs` must exactly equal the bound FASTA.

```json
{
  "schema": "mutationsieve.external-validation-plan.v1",
  "run_id": "hg002-lockbox-001",
  "assembly": "GRCh38",
  "reference_contigs": {"chr1": 248956422, "chr2": 242193529},
  "analysis_policy": {
    "truth_access": "caller_argv_excludes_truth_and_exact_digest_aliases",
    "independence": "external_lockbox_and_operator_independence_required_not_enforced",
    "callability_mode": "shared_effective_bed_all_callsets",
    "callset_selection": "operator_prespecified_not_interpreted_by_harness"
  },
  "inputs": {
    "reference": {"path": "inputs/GRCh38.fa", "sha256": "<64 hex>"},
    "truth": {"path": "inputs/HG002.truth.vcf.gz", "sha256": "<64 hex>"},
    "confident": {"path": "inputs/HG002.confident.bed", "sha256": "<64 hex>"},
    "callable": {"path": "inputs/query.callable.bed", "sha256": "<64 hex>"},
    "targets": {"path": "inputs/assay-targets.bed", "sha256": "<64 hex>"},
    "effective": {"path": "inputs/evaluation-intersection.bed", "sha256": "<64 hex>"},
    "mutect2_calls": {"path": "inputs/mutect2.vcf.gz", "sha256": "<64 hex>"}
  },
  "bindings": {
    "reference_fasta": "reference",
    "truth_vcf": "truth",
    "confident_bed": "confident",
    "callable_bed": "callable",
    "effective_bed": "effective",
    "assay_target_bed": "targets"
  },
  "comparators": [
    {
      "id": "mutect2-frozen",
      "declared_engine": "mutect2",
      "mode": "precomputed",
      "callset_input": "mutect2_calls",
      "producer": {
        "name": "GATK Mutect2",
        "version": "4.x.y.z",
        "command_sha256": "<64 hex>"
      }
    }
  ],
  "evaluation": {
    "declared_evaluator": "hap.py vX.Y.Z",
    "policy": {
      "region_policy": "shared_effective_bed",
      "output_interpretation": "raw_declared_outputs_only",
      "pass_fail_policy": "not_computed_by_harness"
    },
    "tool": {
      "executable": {"path": "/pinned/python", "sha256": "<64 hex>"},
      "artifacts": {
        "driver": {"path": "tools/hap.py", "sha256": "<64 hex>"}
      },
      "version_argv": ["${tool:driver}", "--version"],
      "expected_version": "hap.py vX.Y.Z"
    },
    "argv": [
      "${tool:driver}",
      "${binding:truth_vcf}",
      "${comparator:callset}",
      "-r",
      "${binding:reference_fasta}",
      "-f",
      "${binding:effective_bed}",
      "-o",
      "${evaluation:prefix}"
    ],
    "output_prefix": "evaluation/${comparator:id}",
    "outputs": {
      "summary": "evaluation/${comparator:id}.summary.csv",
      "extended": "evaluation/${comparator:id}.extended.csv"
    }
  }
}
```

The example's angle-bracket hash placeholders are explanatory and are not
valid plan values. Compute each digest from the exact frozen file.

For a precomputed callset, `producer.version` and `producer.command_sha256` are
operator-supplied provenance; the callset bytes themselves are independently
verified. A recommended command digest is SHA-256 over the UTF-8 canonical JSON
argv array, not a shell command string.

## Executing a caller

Set comparator `mode` to `execute`, replace `callset_input` and `producer` with
`tool`, `argv`, and `outputs`, and bind all paths through placeholders:

```json
{
  "id": "lofreq-frozen",
  "declared_engine": "lofreq",
  "mode": "execute",
  "analysis_inputs": ["query_bam"],
  "tool": {
    "executable": {"path": "/opt/lofreq/bin/lofreq", "sha256": "<64 hex>"},
    "artifacts": {},
    "version_argv": ["version"],
    "expected_version": "version: 2.x.y"
  },
  "argv": [
    "call-parallel",
    "--ref",
    "${binding:reference_fasta}",
    "-o",
    "${output:callset}",
    "${input:query_bam}"
  ],
  "outputs": {"callset": "calls/lofreq-frozen.vcf"}
}
```

Supported complete-token placeholders are:

| Context | Placeholders |
|---|---|
| caller | declared `${input:<id>}` entries; non-restricted `${binding:<role>}` entries; `${tool:<artifact>}`, `${output:<role>}`, `${comparator:id}` |
| evaluator | `${input:<id>}`, `${binding:<role>}`, `${tool:<artifact>}`, `${comparator:id}`, `${comparator:callset}`, `${evaluation:prefix}` |

Placeholders must occupy a complete argv token. This prevents quoting and
concatenation ambiguity. Multi-stage callers such as Strelka2 or deployments
that require a Java/JAR launch should use a reviewed, hash-pinned launcher as a
declared tool artifact; the launcher is then invoked through a pinned absolute
interpreter executable. Declare every file it produces. No wildcard output
acceptance is available.

## Interpretation and remaining validation work

The harness records raw declared evaluator outputs instead of silently choosing
metrics or collapsing variant classes. It does not infer that a launcher is
hap.py, vcfeval, or any other engine from a label or version string. Prespecify
SNV and indel size-bin endpoints, representation-matching policy, no-call
handling, and acceptance thresholds
before opening a lockbox. Keep development, calibration, and final lockbox
samples disjoint, store the final truth where caller operators and caller
runtimes cannot access it, and have an independent evaluation operator open it
only after callsets are frozen. The plan's independence declaration records
that requirement; it does not prove the arrangement occurred. GIAB establishes
performance only within the selected truth and confident regions;
assay-specific negatives, dilution series, lots, runs, operators, sites, and
physical reference materials remain necessary for LoB, LoD, repeatability,
reproducibility, and transportability claims.

See `PUBLIC_VALIDATION_PROTOCOL.md`, `VALIDATION_PLAN.md`, and
`ALGORITHM_REVIEW.md` for the broader scientific design and primary-method
references.