# MutationSieve analytical validation plan

The frozen software benchmark is only the start of validation. A future
assay-specific study should predeclare and execute the following.

## Cohorts

- Independent error-model fit normals and untouched threshold-calibration
  normals, disjoint by donor and physical library.
- Development positives spanning all substitution classes, MNVs, insertions,
  deletions, complex alleles, repeats, homopolymers, pseudogenes, and panel
  edges.
- Truth-locked dilution replicates across intended VAF and depth.
- Negative/background specimens representing intended tissue, FFPE, cfDNA,
  extraction, chemistry, lot, and instrument conditions.
- Separate same-assay verification, within-assay stress, changed-assay
  transport, and sealed lockbox cohorts.

## Endpoints

- Exact normalized allele match by SNV/insertion/deletion/size/context.
- Positive percentage agreement and precision with exact confidence intervals.
- False-positive burden per sample and callable megabase.
- Callability/no-call rate with intention-to-test denominators.
- AF/genotype error, repeatability, reproducibility, robustness, interference,
  contamination, and specimen stability.
- LoB and LoD95 from an appropriate replicate design and monotone passing tail,
  not the lowest one-off detected dilution.
- Calibration/Brier only if an independently trained probability model is
  deployed; otherwise report the score as rank.

## Comparators

Use identical regions, reference, preprocessing, truth normalization, and
folds for a transparent pileup baseline and pinned established callers such as
GATK HaplotypeCaller/Mutect2, Strelka2, LoFreq, DeepSomatic, and Lancet2 as
appropriate to germline/somatic mode. Lancet2 is prioritized for difficult
indels and DeepSomatic for a current learned baseline, while retaining the
other callers because their error and haplotype models differ. A published
ranking on another assay is not a universal winner. Comparator disagreement
is not truth; all discordances require adjudication.

The executable mechanics are defined in `EXTERNAL_VALIDATION.md`. The
`external-validate` command requires content-pinned inputs and declared tool
files, proves
the confident/callable/target BED intersection, excludes direct and exact-digest
truth/evaluation-region aliases from caller argv, attempts grouped-descendant
cleanup, runs shell-free argv commands, and writes an exact output-root
inventory/hash manifest last. This is not OS isolation: detached processes,
writes outside the output root, transient change-and-restore between hash
checkpoints, transformed truth aliases, and network access are not ruled out.
An externally controlled lockbox and independent evaluation operator remain
required. Tool labels plus executable hashes/version output do not prove engine
identity, and the harness does not interpret evaluator output. Providing it
does not change the current comparator or GIAB evidence status from `NOT_RUN`.

## Acceptance and claims

Freeze code, schema, reference, targets, UMI/duplicate policy, PoN, thresholds,
endpoints, and confirmation SOP before opening the lockbox. Retain failed
loci/samples in denominators. Orthogonally confirm every research referral.
Stop with a documented no-go if prespecified gates fail. Simulation and in
silico spike-ins may supplement physical validation but cannot replace it.
