# MutationSieve public analytical-validation protocol

Protocol state: prospective template, not yet executed. Populate all bracketed
fields, freeze the protocol, and content-identify the software/resources before
opening any lockbox. Generated results must be published separately; do not
replace this protocol with favorable excerpts.

## 1. Objective and claims boundary

The objective is to measure the analytical performance of one frozen
MutationSieve deployment for one declared intended use: assay, specimen type,
library preparation, platform, aligner/preprocessing chain, target territory,
reference assembly, variant classes/lengths, allele-fraction range, depth
range, and germline/somatic/mosaic decision rule.

The study must answer whether the complete workflow, including no-call and QC
behavior, meets prespecified gates. It must not infer clinical validity or
utility from analytical agreement. Until this protocol is completed and
independently approved, automatic patient reporting remains disabled.

The following claims require separate evidence and must not be pooled:

- germline heterozygous/homozygous SNVs and small indels;
- tumor-normal somatic SNVs and small indels;
- tumor-only somatic analysis;
- low-fraction mosaic or cfDNA analysis;
- ordinary fragment counting versus a named UMI/duplex protocol; and
- each materially changed assay, chemistry, instrument, aligner, target panel,
  or reference assembly.

## 2. Governance and prospective freeze

Before evaluation, record and SHA-256 identify:

1. MutationSieve source revision, wheel/archive, schemas, test record, and
   release verifier;
2. reference FASTA/assembly, target BED, compact bundle, contig policy, and
   callable-region construction;
3. aligner, duplicate marking, UMI/duplex consensus preprocessing, BAM/CRAM
   backend, and all command/container digests;
4. complete configuration, candidate/rule thresholds, reportability switch,
   PoN, matched-normal policy, and decision rule;
5. truth VCF/BED, normalization and comparison tools, confidence level,
   endpoint formulas, strata, acceptance gates, and multiplicity policy;
6. sample identifiers, physical-library identities, donor relationships,
   exclusions, randomization, blinding, and partition assignment; and
7. confirmation method, discordance-adjudication SOP, authorized roles, and
   deviation/change-control process.

The analysis plan and lockbox digest must be signed or otherwise independently
time-stamped. Development and calibration may change the frozen candidate only
through a new versioned protocol. Lockbox data may be evaluated once. Failure
does not authorize retuning and reopening the same lockbox.

## 3. Non-overlapping evidence partitions

Assign each donor and physical library to exactly one primary partition.
Related derivatives remain in the same partition unless a prespecified
repeatability experiment explicitly models their dependence.

| Partition | Required role | Prohibited use |
|---|---|---|
| PoN fit | Independent assay-matched negative libraries used only to fit empirical background. | Threshold calibration or performance reporting. |
| Calibration | Independent truth-bearing positives/negatives used to choose thresholds and estimate model calibration if a model exists. | Training features or final claims. |
| Development | Method debugging and prespecified stress exploration. | Final lockbox claims. |
| Lockbox | Representative, blinded positives, negatives, and no-call challenges; opened once after freeze. | Any fitting, threshold choice, or case exclusion based on result. |
| Stress | Boundary conditions within the intended assay: depth, damage, contamination, GC, repeats, insert size, batch, and storage. | Substitution for ordinary intended-use material. |
| Changed assay | Materially different chemistry/platform/panel/preprocessing used to test transport. | Combining with the primary assay unless transport gates pass. |

Synthetic and in-silico derivatives from one host are correlated
pseudoreplicates. The effective independent unit is normally donor/physical
library, not the number of planted alleles or downsampled files. The statistical
analysis must cluster or stratify accordingly.

## 4. Materials

### 4.1 Public germline benchmark

Use the current, explicitly pinned NIST GIAB HG002 small-variant benchmark only
after verifying the truth VCF and confident-region BED digests and their exact
assembly. NIST describes the GIAB resources at
https://www.nist.gov/programs-projects/genome-bottle and its benchmark-set
methodology at
https://www.nist.gov/publications/open-resource-accurately-benchmarking-small-variant-and-reference-calls.

Required GIAB steps:

1. Acquire reads/alignment, truth VCF, confident-region BED, reference FASTA,
   and published checksums from authoritative repositories. Record URLs,
   release names, dates, sizes, and hashes. MutationSieve's `fetch-public`
   command defaults to a protocol record and performs no download unless an
   explicit HTTPS URL, destination, and expected SHA-256 are supplied.
2. Confirm assembly/contig identity. Do not liftover silently and do not mix a
   truth release with a different confident-region version.
3. Restrict evaluation to the intersection of GIAB confident regions, the
   declared assay target territory, and regions assessable by the complete
   workflow. Preserve truth outside callable depth as intention-to-test
   no-calls; also report the conventional confident/callable subset separately.
4. Normalize truth and calls with one pinned, validated representation tool,
   then use a pinned haplotype-aware comparison engine such as hap.py/vcfeval
   for the primary comparison. Retain one-to-one exact `CHROM/POS/REF/ALT`
   matching as a secondary representation diagnostic and report unresolved
   complex disagreements explicitly.
5. Report by `SNV`, `MNV`, `INS`, `DEL`, and `COMPLEX`, indel length,
   homopolymer/repeat/low-complexity context, target-edge distance, genotype,
   depth, and callable state.

GIAB germline performance does not validate low-VAF somatic/mosaic calling,
FFPE/cfDNA behavior, an assay-specific PoN, or tumor-normal logic.

### 4.2 Physical and clinical material

Use characterized reference materials and representative residual clinical
specimens under appropriate ethics/privacy controls. Include independently
confirmed positives and negatives for each reportable class. Truth must be
derived without MutationSieve and without a simple majority vote of comparator
callers. Suitable orthogonal methods depend on allele/context and may include
validated ddPCR, digital PCR, Sanger at adequate VAF, an independently prepared
library/platform, or a qualified reference-material certificate.

The negative set must represent intended specimen types, extraction methods,
library lots, instruments, operators, sites, batches, damage states, and common
interferences. Public samples without orthogonal assessment are technical
backgrounds, not verified negatives.

### 4.3 Dilution and limit materials

Construct physical dilution series independently by variant class and relevant
context, using multiple positive donors/materials, independently prepared
replicates, and negative matrix. Span concentrations above and below the
anticipated boundary and cover the intended depth range. In-silico spike-ins
may diagnose software behavior but may not replace physical LoB/LoD material.

### 4.4 PoN and matched normals

PoN libraries must be independent of calibration, development, and lockbox,
assay/chemistry/platform/preprocessing matched, and sufficiently numerous for
the prespecified empirical resolution. Build by discovery union followed by
fixed-union recount. Review recurrent loci and missing-depth patterns without
deleting inconvenient normal observations. Report the finite empirical
resolution and its compatibility identity.

Matched-normal studies must predeclare tumor-normal pairing, minimum normal
depth, allowed alternate evidence, contamination checks, and sample-swap
controls. Tumor-only and tumor-normal performance are separate endpoints.

## 5. Coverage matrix and sample-size plan

Before acquiring/opening lockbox material, produce a matrix crossing the
following factors and declare which cells are reportable, exploratory, or out
of scope:

- variant class: all six substitution changes, MNV, insertion, deletion,
  simple complex;
- indel length buckets through the configured maximum;
- VAF/genotype and expected support near every decision boundary;
- total depth and independent analysis-unit depth;
- homopolymer length, tandem repeat, entropy/low complexity, GC, segmental
  duplication/pseudogene, and mapping ambiguity;
- target interior versus capture/compact-reference edge;
- forward/reverse balance, fragment starts, insert size, read position, and
  local-haplotype support;
- specimen type, damage state (including FFPE where applicable), contamination,
  storage, extraction, library lot, instrument, run, operator, and site;
- matched-normal depth/contamination; and
- UMI tag completeness, family size, consensus, collision, and duplex state
  for each claimed molecular protocol.

Determine sample size from the prespecified acceptance boundary and desired
confidence bound, not from what material is convenient. Report both independent
specimen/library counts and allele-level observations. If multiple events are
planted into one library, use clustered confidence methods or a design-based
analysis in addition to allele-level descriptive intervals.

AMP/CAP guidance supports documenting validation performance separately by
variant type and intended use
(https://pmc.ncbi.nlm.nih.gov/articles/PMC6941185/). FDA's SEQC2 program is a
useful model for multi-site and cross-platform reproducibility design
(https://www.fda.gov/science-research/bioinformatics-tools/microarraysequencing-quality-control-maqcseqc).
Neither source supplies universal numeric gates for this assay; the laboratory
and governing authority must set them prospectively.

## 6. Comparator study

Run at least a transparent pileup baseline and pinned established tools chosen
for the declared mode. Candidate comparators include GATK Mutect2 or
HaplotypeCaller, Strelka2, and LoFreq. Primary method sources are:

- Strelka2: https://www.nature.com/articles/s41592-018-0051-x
- Mutect2 official documentation:
  https://gatk.broadinstitute.org/hc/en-us/articles/360036463432
- Mutect2 technical manuscript: https://www.biorxiv.org/content/10.1101/861054v1
- LoFreq: https://pmc.ncbi.nlm.nih.gov/articles/PMC3526318/

For every tool, pin image/version/digest, reference, intervals, sample pairing,
PoN/germline resources, preprocessing, and emitted decision threshold. Evaluate
the identical sample set, confident/callable mask, normalized truth, and
one-to-one matcher. Publish unfiltered and selected calls where licensing and
privacy permit. A comparator disagreement is not truth; adjudicate every
discordance under the frozen SOP. Do not optimize each tool on the lockbox.

The current MutationSieve comparator register is deliberately `NOT_RUN`.
Replace that status only with a content-identified execution record and result.
The strict executable plan and artifact contract is implemented by
`mutationsieve external-validate`; see `EXTERNAL_VALIDATION.md`. Protocol-only
or dry-run output is preparation evidence and must not replace `NOT_RUN`. Even
a successful execute record establishes only declared command completion and
checkpointed inventory within its output root. It does not establish an OS
sandbox, prevent network/UNC access, prove engine identity, detect transformed
truth aliases or transient restore attacks, or verify independent lockbox
custody. Supply those controls externally and retain their evidence alongside
the execution record.

## 7. Prespecified endpoints

### 7.1 Primary accuracy accounting

Use a pinned haplotype-aware comparison as the primary public-benchmark matcher,
with record-exact normalized matching reported separately. The primary positive
agreement denominator is every truth allele in the intention-to-test cohort. A truth
allele in a low-depth, ambiguous, failed-QC, or unassessed interval remains a
miss/no-call in the primary denominator. Also report a callable-only analysis,
but never substitute it for the primary result.

For every class and prespecified stratum, report:

- positive percent agreement (PPA) on intention-to-test truth;
- PPA among callable truth;
- precision/positive predictive agreement, with every extra selected call in
  the denominator;
- callability and specimen no-call rates;
- false positives per specimen and per callable megabase;
- false negatives, false positives, truth found only as rejected candidates,
  and their evidence/reasons; and
- two-sided exact Clopper-Pearson intervals for simple binomial proportions,
  plus cluster-aware intervals when observations share donors/libraries.

Do not label assumed-negative specificity, prevalence-enriched precision, or
target-level observations as clinical specificity/PPV. Specificity requires a
truth-qualified negative unit and a prespecified denominator.

### 7.2 Analytical range and quantitative behavior

Report measured versus expected AF/support bias and error by class, VAF,
depth, and context. Genotype agreement may be reported only where genotype is
an implemented, frozen output; the current research VCF deliberately leaves
genotype missing. A rank may be evaluated for discrimination, but Brier score,
reliability, and probability thresholds apply only to an independently fitted
and calibrated probability model.

### 7.3 LoB and LoD95

Estimate LoB from sufficient independently prepared negative replicates across
representative conditions. Estimate detection probability over physical
dilutions using a prespecified model and confidence criterion. LoD95 is the
lowest concentration at which the fitted lower confidence bound or other
predeclared rule meets 95% detection **and the passing tail remains monotone**.
A lowest one-off detection, pooled pseudo-replicates, or a single downsample is
not LoD95. Report LoD separately by variant class and any materially different
length/context group.

### 7.4 Precision, robustness, and interference

Measure within-run, between-run, between-operator, between-lot, between-
instrument, and, if claimed, between-site agreement using the same expected
alleles and independently prepared replicates. Consistently missing an allele
is not successful repeatability. Require agreement on exact allele and decision
state, not merely any nearby candidate.

Challenge contamination/tumor-normal cross-contamination, DNA input, damage,
fragment length, PCR/duplicate burden, UMI completeness, index hopping/sample
mix-up, common polymorphisms, mapping ambiguity, storage stability, and depth
extremes. Predeclare no-call expectations and safety stops.

UMI/duplex claims require protocol-specific evidence. General background on
error-corrected and duplex sequencing is available at
https://pmc.ncbi.nlm.nih.gov/articles/PMC6423013/ and
https://pmc.ncbi.nlm.nih.gov/articles/PMC6735726/; these papers do not validate
MutationSieve's grouping for a particular laboratory protocol.

## 8. Acceptance gates

Populate this table before lockbox opening. Use numeric gates appropriate to
the intended use and governing standards; blank gates mean the study is not
ready to run.

| Endpoint/stratum | Point-estimate gate | Confidence-bound gate | Maximum no-call/FP burden | Status |
|---|---:|---:|---:|---|
| Overall intention-to-test PPA | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| SNV PPA | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| MNV PPA | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| Insertion PPA by length/context | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| Deletion PPA by length/context | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| Negative-specimen agreement / FP burden | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| Specimen and target callability | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| LoB and LoD95 by claimed class | `[predeclare]` | `[predeclare]` | n/a | `UNSET` |
| Repeatability/reproducibility | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |
| Stress and changed-assay transport | `[predeclare]` | `[predeclare]` | `[predeclare]` | `UNSET` |

Failure of any primary gate produces a no-go or a prospectively defined scope
restriction. It must not be converted to success by post-hoc exclusion,
threshold changes, pooling strata, switching to callable-only denominators, or
calling `REVIEW` a reportable positive without having frozen that rule.

## 9. Execution sequence

### Stage A - software and release verification

1. Verify the release archive/checksum manifest in a clean environment.
2. Record Python/OS/dependency versions and optional BAM/CRAM backend.
3. Run the complete tests and store stdout/stderr, exit code, duration, source
   identity, and test inventory.
4. Run deterministic benchmark and validation commands; verify every sample
   manifest and `benchmark.manifest.json`.
5. Record each synthetic hypothesis as `PASSED`, `FAILED`, or
   `NOT_EVALUABLE`. Software failures stop later stages until a new frozen
   release is issued.

This stage verifies specified software behavior only.

### Stage B - public GIAB and independent in-silico checks

1. Execute the pinned GIAB plan and publish the exact evaluation mask.
2. Run the SpikeForge control or another independently implemented mutation
   engine; disclose shared reference/baseline inputs.
3. Run ordinary, edge, repeat, long-indel, multiallelic, damage, contamination,
   low-depth, and changed-platform stress fixtures.
4. Preserve all misses, false positives, rejected truth, and no-calls.

### Stage C - comparator study

Run the pinned comparator matrix on identical data and masks. Publish command,
version/digest, resource identities, raw metrics, discordance list, and
adjudication outcome.

### Stage D - physical analytical development

Execute PoN, negatives, positives, dilution, precision, robustness,
interference, and stability experiments. Freeze thresholds/resources only on
fit/calibration/development partitions.

### Stage E - blinded lockbox

Confirm the lockbox digest and blind before analysis. Run once with the frozen
candidate. Include failed-QC/no-call specimens in intention-to-test accounting.
Apply prespecified gates mechanically and issue `GO`, `NO_GO`, or a prospectively
allowed restricted-scope decision.

### Stage F - changed-assay and ongoing monitoring

Any material change triggers impact assessment and the prespecified partial or
full revalidation. Monitor PoN drift, no-call rate, call burden, contamination,
batch/platform shifts, and confirmation discordance without silently changing
thresholds.

## 10. Required public artifacts

Publish a privacy-reviewed bundle containing, at minimum:

- frozen protocol, deviations, acceptance table, and decision;
- source/release/version/environment/checksum identities;
- sample/partition manifest with non-identifying IDs and dependency groups;
- reference/target/truth/confident-region/PoN/config identities;
- commands and pinned tool/container versions;
- per-sample callability and QC states;
- complete selected-call and rejected-candidate audits;
- exact-match assignments, every miss, every false positive, truth found only
  among rejects, and no-call reasons;
- metrics and confidence intervals by class and prespecified stratum;
- LoB/LoD, precision, robustness, interference, comparator, and transport
  records, including failures and unevaluable cells;
- immutable failed-hypothesis/adverse-event register;
- artifact manifest written last and an independent verifier; and
- a plain-language report that states limitations and explains how to use the
  technical files without inflating claims.

Dynamic values belong in generated `benchmark.json`, `validation.json`,
assay-validation results, and their manifests. The static protocol must never
invent or cache favorable result numbers.

## 11. Failure and adverse-result rules

Preserve failed samples and loci in denominators unless a prospectively defined
pre-analytical exclusion can be applied without seeing the caller result.
Record exclusions both before and after application. Preserve failed
hypotheses, comparator failures, resource incompatibilities, and deviations.
Do not delete stale/failed models without recording why they were rejected.

If an endpoint is underpowered, label it `NOT_EVALUABLE`, not passed. If GIAB,
a comparator, or physical study was not executed, retain `NOT_RUN`. If a gate
fails, issue `FAILED` and a no-go/scope restriction. A new method or threshold
requires a new version and an untouched lockbox.

## 12. Claim-promotion checklist

The phrase "analytically validated" is permitted only after all applicable
items are independently reviewed and accepted:

- intended use and reportable range are explicit;
- physical positives and verified negatives are representative and adequate;
- PoN/matched-normal/UMI policies are assay-specific and frozen;
- every variant class and difficult-context stratum meets prespecified gates;
- no-call, FP, LoB, LoD95, precision, robustness, interference, and stability
  gates pass;
- fair comparator and GIAB results are complete where applicable;
- the blinded lockbox passes without retuning;
- changed-assay transport is either passed or explicitly excluded;
- all adverse evidence remains public in the validation bundle; and
- the governing laboratory/quality authority approves the bounded claim.

Clinical validity, clinical utility, pathogenicity, oncogenicity, and treatment
selection require additional evidence outside this software protocol.
