Scientific software-verification report · 1 August 2026
Small-variant software that passed its control gates—not a validated assay.
MutationSieve detects and audits point mutations, adjacent substitutions, insertions, deletions, and bounded complex alleles in targeted paired-end DNA data. This page separates what the deterministic software controls establish from the biological and clinical work that has not been done.
01 · Plain-language summary
What was built, what passed, and what remains unknown
The pipeline converts aligned DNA reads into a complete ledger of small sequence-change candidates. Every candidate—including rejected candidates—retains its evidence and decision reasons. The software-control suite passed, but the test material was deliberately constructed for software reachability and cannot estimate performance on real specimens.
What passed
All 13 prespecified deterministic software-control gates passed. The caller recovered 28 of 29 conditionally constructed control alleles and all 28 callable controls; the remaining allele was an intentional off-target no-call challenge. The test suite recorded 340 passed and 3 skipped direct test cases plus 3 passed subtests, with zero failures and zero errors.
How the controls were made
Twenty-four of 25 positive alleles made by MutationSieve's aligned-read simulator required disclosed forced top-up to reach the prespecified three-primary-unit floor; the remaining allele reached its floor through ordinary selection. Three additional alleles came from SpikeForge's separately implemented BAM edit-output path. The baseline alignment context remained shared, so this was not an independent experiment. This design tests software reachability, not natural sampling.
What was not run
External callers, GIAB or another public truth benchmark, FASTQ-to-pinned-production-aligner spike-ins, and physical reference or patient specimens were NOT RUN. There is no LoB, LoD95, sensitivity, specificity, precision, reproducibility, or clinical-validity result.
What the output means
Every selected control was retained as REVIEW. Automatic PASS and HIGH reportable tiers remained disabled and absent. A recovered REVIEW candidate is detectable for human review; it is not automatically reportable.
02 · Questions and answers
The controls answered narrow engineering questions
Did all software-control gates pass?
Yes. Thirteen of thirteen prespecified gates passed in this frozen run, including class presence, no-call accounting, control realization, matched-normal handling, UMI handling, changed-platform fail-closed behavior, and SpikeForge integrity checks.
Were the callable planted alleles recovered?
Yes. Twenty-eight of 28 callable constructed controls were selected for REVIEW. Including the intentional off-target no-call challenge, the intention-to-test control recovery was 28/29 (96.6%; exact 95% interval 82.2%–99.9%).
Does that measure sensitivity?
No. Twenty-four of the 25 MutationSieve-simulator positive alleles needed forced top-up to reach a minimum evidence floor. Recovery therefore demonstrates software reachability after control construction; it cannot estimate sensitivity or LoD.
Did the small deterministic negative produce selected calls?
No selected false positives were observed across 64,000 callable bases in one tiny deterministic negative control. This is not specificity, a limit of blank, or a general false-positive-rate estimate.
Were no-calls kept visible?
Yes. The intentional off-target truth remained in the denominator, and all eight evaluated sample outputs remained partial no-calls because the compact fixtures do not cover the full target. No-call territory was not rewritten as negative.
Is it ready for automatic or clinical reporting?
No. Every selected call is REVIEW, zero PASS/HIGH calls were emitted, reportable tiers are disabled, and the analytical and clinical studies needed for patient use were not performed.
03 · Scope
One small-variant workflow, with the sibling projects' safeguards
MutationSieve combines the conservative engineering requirements developed in FusionSieve and LGASieve without mixing biologically different evidence models.
It is scoped to targeted DNA small variants: SNVs, adjacent substitutions or MNVs, insertions, deletions, and bounded complex replacements. Fusion calls, exon copy-number changes, large structural variants, repeat expansions, pathogenicity interpretation, and treatment recommendations stay outside this caller.
The shared safeguards include strict configuration, exact reference identity, complete adverse-evidence retention, fail-closed resources, explicit matched-normal and panel-of-normals compatibility, no-call denominators, frozen hypotheses, deterministic artifact manifests, and separate detection versus reportability states.
04 · Methods
A transparent two-pass candidate caller
-
Freeze the searchable territory
The builder binds the exact compact FASTA, target BED, reference/index identity, configuration, input hashes, and source fingerprint. Low depth or missing target coverage becomes an explicit no-call.
-
Discover permissively
High-quality mismatches and CIGAR junctions propose SNVs, MNVs, insertions, deletions, and bounded complex alleles. REF/ALT is verified, minimized, and repeat-aware left-normalized.
-
Recount independent evidence
Overlapping mates are reconciled into paired fragments. REF, the focal ALT, and conflicting observations share one denominator. Explicit UMI families become primary only under the configured tag and consensus contract; coordinate families remain a labeled shadow metric.
-
Challenge local haplotypes
Bounded Smith–Waterman scoring compares local reference and alternate haplotypes and helps reconcile repeat-shifted CIGAR events. This is not graph assembly, so alleles never proposed during discovery can still be missed.
-
Use compatible background evidence
The matched normal is always opened and checked when configured. Panel-of-normals evidence is accepted only when platform, reference, target, counting-unit, and UMI provenance match.
-
Keep statistics class-appropriate
SNV/MNV evidence uses a bounded exact Poisson-binomial tail over primary-unit quality. Indels do not borrow base quality as an indel-error probability and require compatible empirical background evidence.
-
Separate detection from reportability
A transparent score ranks hypotheses; it is not a calibrated probability. This release forces selected candidates to REVIEW because the model, panel, and auto-reporting readiness gates are not satisfied.
What was borrowed from leading public methods
The design review selected inspectable principles from complementary callers. No third-party caller source was copied or vendored, and no publication result was treated as inherited validation.
| Method family | Useful principle | Boundary in MutationSieve |
|---|---|---|
| Mutect2 | Local assembly and likelihood-based haplotype evaluation | MutationSieve locally rescores explicit REF/ALT haplotypes but does not implement de-novo assembly graphs or PairHMM genotyping. |
| LoFreq | Quality-aware non-identical Bernoulli trials and Poisson-binomial tails | Used for substitution evidence with conservative quality handling; base qualities are never reused as indel-error probabilities. |
| Strelka2 | Empirical indel background and matched-normal evidence | Compatible background evidence is required; fitted indel mixtures and contamination models are not reproduced. |
| Lancet2 | Joint tumor-normal local assembly, partial-order alignment, and read realignment for difficult indels | Lancet2 is the priority difficult-indel comparator, but its authors' four-cell-line result is not assumed to rank universally. It was NOT RUN here. |
| Octopus | A unified, inspectable haplotype contract for SNVs, indels, and complex alleles | MutationSieve uses one normalized evidence contract but no Bayesian posterior or genotype model. |
| DeepSomatic | Technology-specific learned models and broad somatic benchmark evaluation | DeepSomatic is the priority learned comparator. No pretrained model was imported and the comparator was NOT RUN. |
| fgbio consensus | Real molecular families need an explicit consensus model and realignment | UMI provenance is audited; coordinate families are not mislabeled as molecules. |
| BAMSurgeon | Spike-ins should preserve empirical context and avoid incompatible edits | The local fixtures add strict accounting and a separately implemented SpikeForge edit path, but remain hand-constructed aligned-BAM controls with shared baseline context. |
| GA4GH / hap.py | Benchmark normalized outcomes inside exact confident regions | The built-in equivalence helper is bounded and explicitly not a general haplotype-graph comparator. |
A WES study found prespecified voting ensembles could outperform its best individual caller, while a recent patient-derived cfDNA benchmark found rankings changed with depth and decision task. Those findings support a diverse, assay-specific comparator plan—not a universal “best caller” claim.
05 · Frozen results
The controls passed; the evidence boundary did not move
Prespecified software-control decision
13 / 13 PASSED
This is a software-regression result only. Analytical validation, synthetic-read alignment validation, and clinical validation remain NOT PERFORMED.
Software tests
340 passed
346 total JUnit cases: 343 direct cases (340 passed and 3 skipped) plus 3 passed subtests; zero failures, zero errors; recorded time 85.968 seconds.
Constructed control recovery
96.6%
28/29 intention-to-test constructed controls; exact 95% interval 82.2%–99.9%. This is not sensitivity.
Callable control recovery
28 / 28
All callable controls were selected. The one missed control was deliberately outside the callable target and stayed in the primary denominator.
Selected calls matching controls
28 / 28
Exact 95% interval 87.7%–100.0%. This is a constructed-control match rate, not clinical PPV or assay precision.
Control construction
24 / 25
Twenty-four simulator-positive alleles used 67 forced units; eight ordinary floor contributions and 75 total floor-prioritized units are disclosed. SpikeForge supplied three independent edit-output alleles.
Tiny deterministic negative
0 selected
Zero selected false positives across 64,000 callable bases in one control only. This does not establish specificity or LoB.
Constructed-control recovery by class
These counts are normalized software-control matches, not clinical sensitivity estimates.
Gate register
| Gate group | Status | Observed | Interpretation |
|---|---|---|---|
| Variant-class presence | 5 PASSED | SNV, MNV, INS, DEL, COMPLEX represented | Each declared class reached REVIEW at least once. |
| Constructed-control recovery | PASSED | 28/29 overall; 28/28 callable | The sole miss was the prespecified off-target no-call challenge. |
| Tiny negative control | PASSED | 0 selected false positives; 64,000 callable bases | A software regression check, not specificity. |
| Matched normal, UMI, no-call, platform | 4 PASSED | Expected compatibility and fail-closed outcomes | Engineering contracts behaved as prespecified. |
| SpikeForge integrity and structure | 2 PASSED | 3/3 independently edited alleles realized | Independent edit-output control; still aligned-BAM based. |
| Control realization | PASSED | 28/28 designed alleles realized in every included run | Conditional top-up is disclosed separately and prevents a sensitivity interpretation. |
All eight evaluated sample outputs were partial no-calls. All selected calls were REVIEW. Reportable tiers were false/disabled, with zero PASS and zero HIGH calls.
06 · Conclusions
A releaseable software-control snapshot, not a validated test
- The implementation is coherent enough for technical review. All frozen software-control gates and integrity checks passed, and the complete evidence bundle is reproducible and hash-bound.
- The 28/29 result has a deliberately narrow name. It is conditionally constructed control recovery. Because 24 of 25 simulator positives needed forced top-up, it cannot be relabeled sensitivity, LoD, or natural sampling performance.
- Detection remains separate from reporting. Every selected allele stayed REVIEW. The software did not emit PASS or HIGH and makes no automatic-reportability claim.
- Biological performance remains unknown. Real FASTQ alignment, public truth, external callers, physical standards, and patient-specimen studies were not run.
07 · Limitations
What this evidence cannot support
Not synthetic-read alignment validation
The regression fixtures contain hand-constructed aligned BAM records. Coupled fields such as CIGAR, MAPQ, flags, mates, NM/MD, AS/XS, and SA were not jointly generated by a pinned production aligner against the intended full reference.
Conditional top-up changes interpretation
Twenty-four of 25 positive simulator alleles used 67 forced primary-unit observations to meet a three-unit floor. The fixtures test control reachability and caller handling, not probability of observing an allele at a nominal VAF.
No external caller execution
Mutect2, Strelka2, LoFreq, DeepSomatic, and Lancet2 were NOT RUN against this evidence. The online algorithm review guides a future comparison; it is not a performance ranking.
No public truth benchmark
GIAB HG002 or another public truth set, confident regions, and a pinned hap.py/vcfeval-style comparison were NOT RUN. Internal bounded equivalence is not a substitute.
No physical material
No reference material, dilution series, FFPE specimen, cfDNA specimen, patient sample, replicate operator, site, lot, instrument, or longitudinal stability study was run.
No analytical claims
There is no sensitivity, specificity, PPV, NPV, LoB, LoD95, precision, reproducibility, robustness, carryover, interference, or transportability estimate.
No clinical claims
The software is not clinically validated and is not for patient reporting, diagnosis, treatment selection, or exclusion of disease.
Partial no-calls remain
All eight evaluated outputs are partial no-calls because compact fixtures leave parts of the target uncovered. An absent selected candidate is not a clinical negative.
08 · Required next work
What must happen before broader claims
- Create aligned synthetic evidence correctly. Start from paired FASTQ spike-ins and run the hash-pinned historical or intended production aligner against the full intended reference. Preserve pairing, deterministic de-clustering, read groups, exact indexes, commands, and output hashes.
- Run a pinned public benchmark. Use an appropriate GIAB truth release and confident regions, with a pinned hap.py or vcfeval-style evaluator and public discordance adjudication.
- Execute diverse comparators. Run Mutect2 and Strelka2 as established assembly/model baselines, LoFreq as a quality-aware low-frequency baseline, Lancet2 for difficult indels, and DeepSomatic as the learned comparator under one prespecified policy.
- Use physical assay material. Estimate LoB and LoD95 with independent replicates, then study class, context, VAF, depth, specimen type, preparation, lot, instrument, operator, site, and software-version effects.
- Keep REVIEW separate. Calibrate automatic reporting only after the model, panel of normals, duplicate/UMI provenance, and governance gates are satisfied. Never promote REVIEW merely to improve apparent success.
09 · Software and evidence
Download the inspectable software-control snapshot
The public archive contains sanitized derivatives of the deterministic benchmark, validation JSON, JUnit evidence, algorithm and external-validation protocols, release notice, wheel, and source distribution. It is software-verification evidence, not an analytical or clinical validation bundle.
Software-verification evidence
Includes per-sample outputs, truth and control-generation ledgers, benchmark manifests, public redaction/evidence manifests, and the exact packaged software.
SHA-256b67eb669f469ecce1ff80b5a6b7440f7718c91e265e99252c008c13054ec83d7
Python wheel
Installable MutationSieve 0.1.0.dev1 package.
Download wheelSHA-2569f6a13dec11785f1e27d0581627b48f33fae865ea57b474379e20de114303a03
Source distribution
Source, tests, documentation, and package metadata.
Download sourceSHA-25657caa716c5476d4a97f597691295ce313a3f1b6a103239a0a90328f5626f249e
Direct machine-readable evidence
9342b3baa63fc9269a577acda704079f83370568af6a52a36a3bee4dd650544e
Frozen benchmark JSONf360677d69e69b2b19a3e6144b6cddf7ee6dbf376b0bacd4b1a00fd535de373c
Benchmark manifesthashes and sizes for the complete benchmark tree
Public JUnit record0ae7d65db532781333289aa303b9dd1e90b0c5ec58c7a3b2c275ac2dff52859f
Control-generation accountingordinary selection and conditional top-up kept separate
Release noticesoftware-control GO; analytical/clinical NO-GO boundary
Algorithm reviewprimary-source method and comparator decisions
External-validation protocolhow future caller runs must be pinned and isolated
Public redaction manifestoriginal-to-public hash mapping
Public evidence manifesthashes of public derivatives
Independent review sequence
- Verify the outer SHA-256 list, then verify the ZIP's internal
PUBLIC-EVIDENCE-SHA256SUMS.txt. - Read
software-verification-manifest.json,RELEASE_NOTICE.md, andvalidation/validation.jsonbefore interpreting the metrics. - Inspect
benchmark/control-generation-accounting.jsonand per-run truth ledgers; do not merge ordinary and forced units into a sensitivity denominator. - Confirm reportable tiers are false, selected calls are REVIEW only, all comparator rows remain NOT_RUN, and every evaluated output retains its partial no-call state.
10 · Privacy and publication
Synthetic content, sanitized operational identifiers
The public build contains deterministic chrSynthetic control data, synthetic sample labels, aggregate metrics, source code, and software provenance. It contains no patient data, patient identifiers, clinical interpretation, credentials, private keys, environment-secret files, local workspace path, workstation hostname, analytics beacon, or external runtime dependency. CSS and all downloads are same-origin; there are no external runtime scripts, fonts, images, or APIs.
Public-derivative rule: the builder replaced 111 workspace-root occurrence(s) and 1 JUnit hostname attribute(s) across 19 transformed file(s). Frozen local originals were not modified. public-redaction-manifest.json records each transformation and both hashes; public-evidence-manifest.json governs the public derivatives.
11 · Primary sources
Methods reviewed before freezing the design
- GATK. Mutect2: local assembly, likelihood models, and somatic filtering.
- Kim S, et al. Strelka2: fast and accurate germline and somatic variant calling. Nature Methods. 2018.
- Wilm A, et al. LoFreq: quality-aware low-frequency variant calling. Nucleic Acids Research. 2012.
- Wood DE, et al. Lancet2: joint multi-sample local assembly, partial-order alignment, and read realignment. 2026. Its four-cell-line indel result is benchmark-specific, not a universal ranking.
- Cooke DP, et al. Octopus: unified haplotype-aware variant calling. Nature Biotechnology. 2021.
- Belyaeva A, et al. DeepSomatic: deep-learning somatic small-variant calling across technologies. Nature Biotechnology. 2025.
- Whole-exome individual-caller and prespecified voting-ensemble benchmark. Its ensemble finding is triangulation evidence, not ground truth.
- Patient-derived cfDNA caller benchmark across depth and use case. It illustrates why caller ranking must be assay- and task-specific.
- Fulcrum Genomics. fgbio molecular consensus documentation.
- Ewing AD, et al. BAMSurgeon: in silico tumor genome variant simulation. Bioinformatics. 2015.
- Krusche P, et al. GA4GH best practices for benchmarking germline small variants. Nature Biotechnology. 2019.