Skip to content

Historical USPTO-50k Stress Test (Frozen, v0.15.5)

This entire page is a frozen historical artifact, not a live or current benchmark. Every number below — including the "Corrected Baseline" section — was measured once, against one specific commit (e20dc8c, RENKIN v0.15.5, 2026-07-22), and has not been re-run since. "Corrected" describes the rule set at that commit, not RENKIN's present state; treat every figure on this page as a snapshot of what RENKIN did on one day, not as its current performance.

Looking for current, matched-condition comparison data? See the Open-Source Retrosynthesis Comparison guide instead: a 500-target, paired-bootstrap, exact-McNemar-tested comparison against AiZynthFinder under both a shared stock and each tool's own native stock, kept current. This page is not kept current and should not be used for that purpose.

⚠️ Notice (2026-07-22): historical 78.0% (single-pass) / 95.9% (cascade) / 81.8% (ChEMBL OOD) figures on this page are invalidated and have NOT been re-measured. They were measured before fixing four retrosynthesis-rule/validator bugs that inflated solved counts with chemically-invalid or falsely-corroborated routes (full history below). Only the "Corrected baseline" section below reflects the rule set as it stood at the frozen commit — everywhere else on this page still shows the old, invalidated numbers for historical continuity, each marked accordingly. Do not cite any figure on this page, marked or unmarked, as current RENKIN performance.

Corrected baseline — USPTO-50k Stage 1 (single-pass), commit e20dc8c, 2026-07-22. Search-to-stock rate (raw_solved_rate) 20.09% (986/4,907) → atom-balance-filtered rate (atom_balanced_solved_rate) 15.41% (756/4,907) → current-validator-confirmed rate (provenance_validated_solved_rate) 0.88% (43/4,907). These three are a nested series over the same 4,907 targets (each a stricter subset of the previous), not independent measurements, and none is an experimentally-verified synthesis success rate or a human-chemist-reviewed route-accuracy figure.

What 0.88% actually is, precisely: the fraction of targets with a complete stock route where every step passes the coarse atom-balance check AND is positively confirmed by its own originating rule's current validator. This is not a measured chemical-accuracy rate, and it is not a proven lower bound on true correctness — establishing a lower bound would require knowing the validator has no false positives, which hasn't been shown (1 of the 44 validated routes fails atom-balance; a diagnostic sample shows 14/864 steps are Valid-but-imbalanced). Separately, Invalid verdicts are not proven errors either: 72.2% of steps in a diagnostic n=300 sample are Invalid but atom-balanced, and an unknown fraction of those may be validator false negatives (canonicalization/tautomer/regiochemistry edge cases) rather than genuine rule or route errors — the split has not been measured (suzuki_retro steps are 0% invalid vs. cn_aliphatic_cleavage at 97.6%, a spread worth investigating further but not yet attributed to either cause). Full methodology, config hashes, and per-rule breakdown: tasks/phase31_final_remeasurement_run.md.

Fix history: aryl_carboxylation_retro ester-overmatch, fixed (PR #26). Three more atom-loss rules — aryl_chloride_retro, aryl_iodide_retro, aryl_fluoride_snAr_retro — found and removed (PR #31, "31.11": each deleted a halogen atom with no tracked reagent). Forward validator, which had accepted a step as Valid if any rule's SMIRKS coincidentally reproduced the target rather than only the rule the step actually used, bound to each step's own originating rule (PR #33, "31.12"). All merged to master before this re-measurement, in that order, each individually CI-verified. Cascade (95.9%) and ChEMBL OOD (81.8%) have not been re-run against the corrected rule set and remain invalidated historical figures — see their own sections below. Phase 31 corrected-baseline publication is complete; validator fidelity analysis (separating real rule errors from validator false negatives) remains an explicit, open follow-up.

USPTO-50k Test Set

USPTO-50k is primarily used as a single-step retrosynthesis benchmark (see "Comparison: Single-Step Top-1 Models" below for that use). This page repurposes a frozen, 4,907-row target corpus derived from USPTO-50k as a route-to-stock stress test for RENKIN's multi-step search — it is not a canonical multi-step benchmark like PaRoutes (which RENKIN also supports directly, via renkin-bench --input-format paroutes — see the README).

There is also a known, disclosed provenance gap in this corpus: data/uspto50k_test.smi's header claims "5007 reactions," but the file has 4,907 data rows, and no record in this repository traces the exact upstream Hugging Face revision this file was derived from. See the Open-Source Retrosynthesis Comparison guide's "Known gaps" section for the full disclosure — not repeated here to avoid two independently-drifting copies of the same caveat.

What "solved" means: A target is solved if at least one complete retrosynthetic route is found where every leaf precursor is in the building block set (402 unique compounds loaded from data/building_blocks.smi for the corrected-baseline run below — see that section for how this differs from the file's raw line count). This is not a check against ground-truth reactants from the USPTO dataset.

Corrected Baseline (commit e20dc8c, 2026-07-22) — depth=5, beam=100, 5,000 extracted templates, 28 handcrafted rules

Public label Internal metric Value Denominator
Search-to-stock rate raw_solved_rate 20.09% (986/4,907) all 4,907 targets
Atom-balance-filtered rate atom_balanced_solved_rate 15.41% (756/4,907) all 4,907 targets — subset of search-to-stock
Current-validator-confirmed rate provenance_validated_solved_rate 0.88% (43/4,907) all 4,907 targets — subset of atom-balance-filtered; see notice above for what this figure is and isn't
depth=0 direct stock hit — 0.04% (2/4,907) all 4,907 targets
latency, all targets — p50 7.3s / p95 28.2s / p99 51.2s includes unsolved targets run to the search budget
latency, solved only — p50 1.0s / p95 9.4s / p99 15.6s

These three rates are a nested series over the same 4,907 targets, not independent numbers, and none is a measured or bounded chemical-correctness rate on its own — read the notice above before citing any of them in isolation.

Building blocks: 402 unique compounds actually loaded by ChemEnv::load("data/building_blocks.smi") (ChemEnv::bb_count()) — i.e. after parsing, canonicalization, and de-duplication, not the file's raw line count (449 non-comment lines → 3 fail to parse → 44 duplicate after canonicalization → 402 unique). Full config/hashes/per-rule breakdown: tasks/phase31_final_remeasurement_run.md.

Historical Results (v0.15.5, pre-fix) — depth=5, beam=100, ~5,000 extracted templates

Config Solved Success Rate Avg Time Hardware
depth=5, beam=100, ~5,000 templates + Phase A 3,826 / 4,907 78.0% ≈2,800 ms/mol Apple M-series, 8 threads

Status: invalidated historical measurement, pre-31.11/31.12 — see notice above. Kept for continuity only.

Progress History (Table A — RENKIN internal)

Phase Solved Rate Notes
31 rules only, depth=3 366 / 4,907 7.5% handcrafted rules only
+ 191 extracted templates, depth=3 1,363 / 4,907 27.8% rdchiral top-300
+ depth=5 1,909 / 4,907 38.9% depth increase
+ top-500 templates, depth=5 2,315 / 4,907 47.2% 314 rules total
+ beam=100 2,688 / 4,907 54.8% beam search
+ Phase A frequency weighting 3,540 / 4,907 72.1% step_cost bonus for high-freq templates
+ ~5,000 templates (v0.15.5) 3,826 / 4,907 78.0% pre-fix measurement, invalidated
Cascade: Stage 2 (depth=7, beam=300, unsolved only) 4,705 / 4,907 95.9% 2026-06-29 ✅ (pre-fix, invalidated)

Status: invalidated historical measurement — see notice above.

Comparison: Single-Step Top-1 Models (different metric)

⚠️ Different metric. These measure single-step top-1 prediction accuracy (does the model's top-1 prediction match the known reaction?), not multi-step planning success rate. Direct comparison with RENKIN's multi-step figures above is not valid.

System Single-Step Top-1 Source
LocalRetro 53.4% Chen et al., ACS Cent. Sci. 2021

RENKIN's configuration

RENKIN's 20.09% search-to-stock rate above uses only 402 building blocks and ~5,000 templates. RENKIN's strength is portability: Pure Rust, zero C/C++ dependencies, WASM + Python + CLI from one binary.

What RENKIN solves well

⚠️ Not a measured accuracy claim. The list below describes which transformation families RENKIN has an explicit hand-crafted or graph-based rule for — it is a statement about rule coverage, not a re-measured per-class accuracy figure (no such figure exists against the corrected rule set; the historical 78.0%/95.9% figures this section previously implied are invalidated, see the notice at the top of this page).

RENKIN contains explicit rules for these transformation families:

  • Esters → carboxylic acid + alcohol
  • Amides → acid + amine (graph-based cleavage)
  • Biaryls → aryl halide + boronic acid (Suzuki, graph-based)
  • Boc / Cbz protecting group removal (graph-based)
  • Diaryl sulfones → arylsulfonyl chloride + arene (graph-based)
  • Sulfonamides → sulfonyl chloride + amine

Out-of-Distribution (OOD) Evaluation

⚠️ Not re-measured against the corrected rule set (pre-31.11/31.12, invalidated). ChEMBL OOD has not been re-run since the fix history in the notice above; treat both rows below as historical only.

To check whether RENKIN's accuracy is specific to the USPTO-50k domain, we evaluated on 500 FDA-approved drugs from ChEMBL (Phase 4, MW 150–700, no salts, 2026-06-25).

Dataset Solved Success Rate Notes
USPTO-50k test set (in-domain) 3,826 / 4,907 78.0% (pre-fix) templates from USPTO train set
ChEMBL approved drugs (OOD) 409 / 500 81.8% (pre-fix) real FDA-approved drugs

The +3.8 pp difference on approved drugs is consistent with the hypothesis that the rule set covers common pharmaceutical transformations. However, this result should be interpreted cautiously: both datasets are small-molecule organic chemistry, so the OOD gap is limited. Unsolved molecules in both datasets share the same profile: nitrogen-rich heterocycles (+17 pp) and fluorinated compounds (+11 pp).

Failure Taxonomy (2026-06-29, 500-mol sample)

⚠️ Not re-measured against the corrected rule set (pre-31.11/31.12, invalidated). Kept for methodology reference only.

renkin-bench --failure-taxonomy classifies unsolved targets by cause:

Cause Count % of unsolved Description
beam_limit_hit 111 / 112 99.1% beam pruned promising nodes
max_depth_reached 111 / 112 99.1% route depth > 5 required
stock_near_miss 111 / 112 99.1% BB found in frontier but no complete route
no_template_match 1 / 112 0.9% fewer than 3 templates matched

Key finding: Template and building block coverage is not the bottleneck. Nearly all unsolved targets hit the search budget limit (beam/depth). Cascade search (Stage 2: depth=7, beam=300 on unsolved only) resolved 879/1,081 (81.3%) of previously unsolved targets, lifting the overall rate from 78.0% to 95.9%.

Improving the success rate

⚠️ Point 1 below cites the invalidated failure taxonomy above (pre-31.11/31.12, not re-measured against the corrected rule set). The general levers still apply, but "primary lever" is not re-confirmed evidence.

  1. Cascade search — re-run unsolved targets with higher beam/depth (--depth 7 --beam-width 300). The (invalidated, pre-fix) failure taxonomy above indicated this was the primary lever.
  2. Expand the building block database — supply eMolecules, ZINC, or your internal stock via --building-blocks
  3. Add more templates — extract additional templates from the full USPTO training set (--templates data/templates_extracted_5000.smi)

Running the benchmark

# Build
cargo build --release

# Full benchmark — 50 chunks × 100 mol, resumable
bash scripts/run_benchmark_chunks.sh \
    data/uspto50k_test.smi \
    data/templates_extracted_5000.smi \
    data/bench_chunks \
    5 100

# Failure taxonomy on unsolved
./target/release/renkin-bench \
    --input data/uspto50k_test.smi \
    --depth 5 --beam-width 100 \
    --templates data/templates_extracted_5000.smi \
    --failure-taxonomy \
    > bench_result.json

# Aggregate chunks
python3 -c "
import json, glob
files = sorted(glob.glob('data/bench_chunks/chunk_*.json'))
total = solved = 0
for f in files:
    d = json.load(open(f))
    total += d['total']; solved += d['solved']
print(f'{solved}/{total} = {solved/total:.1%}')
"