Skip to content

Benchmark-Grounded Hero Images

NERB hero images should be plotted benchmark artifacts, not abstract generated art. The goal is to make the value proposition visible through real measurements: train/test entity-bank quality, construction performance, scale behavior, and the autoresearch objective.

Committed assets live under examples/artifacts/hero-images/:

  • enron-quality-performance.png: Enron split size, exact-span NER precision/recall/F1, confusion counts, and construction/extraction measurements.
  • scale-100k-entities.png: measured synthetic scale run from 1,000 to 100,000 entities, names, and patterns with a fixed capped scan workload.
  • autoresearch-objective.png: the intended keep/discard scoring shape with held-out F1 as the primary metric and compile/size/path checks as gates.
  • hero_measurements.json: aggregate measurement data used to render the images. It contains no raw Enron text, addresses, names, or extracted strings.

Generate or refresh the assets with:

uv run --with matplotlib==3.10.9 python examples/generate_benchmark_hero_images.py \
  --enron-artifact-dir .nerb/enron-benchmark/issue-89-candidate \
  --quality-documents 1000 \
  --autoresearch-results-jsonl .nerb/autoresearch/f1-results.jsonl

The script expects private Enron benchmark artifacts under .nerb/: train.jsonl, test.jsonl, bank.json, and benchmark.json. It also expects an autoresearch result row generated by scripts/nerb_autoresearch.py. Those private files stay ignored. Only the aggregate measurement JSON and rendered plots should be committed.

Measurement Notes

enron-quality-performance.png currently uses the pinned Enron benchmark artifacts with:

  • 165,180 selected records.
  • 132,196 training records and 32,984 held-out test records.
  • 5,500 active names and 5,500 active patterns across the two Enron email entity classes.
  • Exact-span held-out quality measured on the first 1,000 prepared test documents.

scale-100k-entities.png uses deterministic compact JSON banks with one active fixed-width, word-bounded literal pattern per entity. The target scan document is capped at 2,000 generated tokens so the plot shows compile, cache, and source-size behavior across 1,000 to 100,000 entities without turning the hero generator into a long-running stress test. The cap changes needed for the 100,000-entity measurement are part of the source: native bank source bytes now allow 64 MiB, and the Rust entity-count limit now allows 100,000 entities while the existing 100,000-pattern cap remains in force.

Usage Guidance

Use these images as splashy benchmark panels in README, docs, PRs, or future site surfaces. Do not crop away the axes, metric names, or scale labels; those are the point of the image.

When the figures are refreshed, keep the matching hero_measurements.json in the same commit. If benchmark source data, quality-document count, scale cases, or limits change, update this doc and the figure titles rather than relying on adjacent prose to correct stale images.