Skip to content

NERB Autoresearch Harness

The autoresearch harness turns the Enron construction benchmark into a bounded optimization loop. It follows the Karpathy-style pattern: keep data prep and evaluation fixed, let an agent edit a small experiment surface, extract one scalar score, append a result row, and keep or discard the candidate based on measured evidence.

This is an experiment runner, not a merge gate bypass. A kept experiment still needs a focused PR, independent review, green CI, and a Review Record before it can merge.

Fixed Evaluator

Use the Enron benchmark from docs/enron-benchmark.md as the evaluator. The candidate command should run scripts/enron_bank_build_benchmark.py with the same data source, split, seed, sample settings, benchmark settings, and stored baseline benchmark.json.

The candidate benchmark command should include baseline gates:

uv run python scripts/enron_bank_build_benchmark.py \
  --input-jsonl tests/data/enron_sample.jsonl \
  --output-dir .nerb/enron-benchmark/autoresearch-candidate \
  --sample-fraction 1.0 \
  --test-fraction 0.35 \
  --seed fixture-seed \
  --created-at 2026-06-09T00:00:00Z \
  --min-address-count 1 \
  --min-domain-count 1 \
  --benchmark-documents 5 \
  --quality-documents 5 \
  --benchmark-iterations 1 \
  --baseline-benchmark-json .nerb/enron-benchmark/autoresearch-baseline/benchmark.json \
  --max-cold-compile-seconds-ratio 1.05 \
  --max-warm-cached-compile-seconds-ratio 1.10 \
  --min-target-bytes-per-second-ratio 0.95

For real-corpus runs, use the pinned Hugging Face command in docs/enron-benchmark.md; keep raw and cleaned artifacts under ignored .nerb/ paths.

Editable Surface

By default, the harness allows construction-related source edits in:

  • src/nerb/enron_bank_builder.py
  • src/nerb/bank.py
  • src/nerb/engine.py
  • src/nerb/engines.py
  • src/nerb/records.py
  • rust/Cargo.lock
  • rust/Cargo.toml
  • rust/src/bank.rs
  • rust/src/engine.rs
  • rust/src/flags.rs
  • rust/src/formats.rs
  • rust/src/ids.rs
  • rust/src/lib.rs
  • rust/src/match_buffer.rs

It freezes evaluator and large-source guidance files:

  • scripts/enron_bank_build_benchmark.py
  • src/nerb/benchmarks.py
  • src/nerb/enron_benchmark.py
  • tests/nerb/test_enron_benchmark.py
  • tests/data/enron_sample.jsonl
  • docs/enron-benchmark.md
  • .agents/skills/nerb-large-source-bank-building

Pass repeated --editable-path or --frozen-path values when an issue deliberately changes the boundary, including other Rust source files. Any custom values replace that default list, so pass the full intended boundary. If an experiment touches a frozen file or a file outside the editable surface, the result is logged and discarded.

Scoring And Decisions

The primary scalar score is quality.test.f1; higher is better. The Enron evaluator computes exact-span NER precision, recall, and F1 against prepared train/test documents. Compile time, throughput, size, and path checks remain gates and context; they are not the reward.

A candidate is kept only when all of these are true:

  • the candidate command exits successfully within the timeout
  • no frozen or out-of-surface files changed relative to the resolved --checkpoint-ref SHA
  • the candidate benchmark JSON has configured gates and gate.passed == true
  • the candidate evaluator fingerprint independently matches the baseline dataset, split artifacts, sampling, and benchmark tier sizes
  • evaluator, held-out quality, and configured performance gates pass
  • canonical and extractable JSON byte sizes stay within configured ratios
  • the primary held-out F1 score improves over the current-best baseline by at least --min-improvement-ratio

Crashes, timeouts, evaluator fingerprint mismatches, held-out F1/precision/recall regressions, size ceiling failures, and insufficient score improvements against the current best are logged as discard.

Running One Experiment

First create a current-best benchmark JSON under .nerb/. Before the agent edits anything, capture the current previous-best commit:

CHECKPOINT=$(git rev-parse HEAD)

Then let the agent make one bounded, uncommitted change on an experiment branch. Score it with:

uv run python scripts/nerb_autoresearch.py \
  --baseline-benchmark-json .nerb/enron-benchmark/autoresearch-baseline/benchmark.json \
  --candidate-benchmark-json .nerb/enron-benchmark/autoresearch-candidate/benchmark.json \
  --results-jsonl .nerb/autoresearch/results.jsonl \
  --description "try construction optimization idea" \
  --checkpoint-ref "$CHECKPOINT" \
  --timeout-seconds 1800 \
  --min-improvement-ratio 0.01 \
  --promote-kept-benchmark \
  --candidate-command uv run python scripts/enron_bank_build_benchmark.py \
    --input-jsonl tests/data/enron_sample.jsonl \
    --output-dir .nerb/enron-benchmark/autoresearch-candidate \
    --sample-fraction 1.0 \
    --test-fraction 0.35 \
    --seed fixture-seed \
    --created-at 2026-06-09T00:00:00Z \
    --min-address-count 1 \
    --min-domain-count 1 \
    --benchmark-documents 5 \
    --quality-documents 5 \
    --benchmark-iterations 1 \
    --baseline-benchmark-json .nerb/enron-benchmark/autoresearch-baseline/benchmark.json \
    --max-cold-compile-seconds-ratio 1.05 \
    --max-warm-cached-compile-seconds-ratio 1.10 \
    --min-target-bytes-per-second-ratio 0.95

Put --candidate-command last; all remaining arguments belong to the evaluator command. The executable requires --candidate-command so a normal keep/discard decision is tied to a fresh evaluator run. Passing --checkpoint-ref HEAD is safe only when HEAD is still the previous-best commit. If a candidate was already committed, pass the prior SHA instead so path gating and discard cleanup compare against the correct baseline.

By default the harness is dry-run safe: it logs the keep/discard decision but does not mutate git state or benchmark artifacts. Pass --promote-kept-benchmark when the baseline path is an ignored current-best artifact and a kept candidate should become the next comparison target. This copies the candidate benchmark.json over the baseline benchmark.json only after scoring succeeds, so the next run must beat the best kept candidate rather than the original starting point.

To make non-improving or failed experiments reset to the previous best commit, also pass --apply-git-decision. This can run git reset --hard <resolved-checkpoint-sha> plus git clean -fd for the changed experiment paths on discard. The checkpoint ref is resolved once before the candidate command runs, so a command that moves HEAD cannot change the comparison or cleanup target. Use it only on an experiment branch with result logs and benchmark artifacts under ignored .nerb/ paths.

Result Log

Each result is one compact JSON object per line. The schema version is nerb.autoresearch_result.v1. Rows include:

  • commit, checkpoint ref, changed paths, editable paths, frozen paths, and path-gate result
  • candidate benchmark output freshness fingerprints when a candidate command is required
  • evaluator baseline/candidate paths, bank hashes, and artifact hashes
  • process command, exit code, timeout flag, elapsed seconds, and stdout/stderr tails
  • independently recomputed evaluator fingerprints, primary score, timing metrics, gate status, and memory/size metadata
  • decision value and reason
  • optional git action and current-best benchmark promotion applied by the harness

See examples/artifacts/autoresearch/results.jsonl for a redacted fixture-shaped row.

Using The Bank-Building Skill

The large-source skill at .agents/skills/nerb-large-source-bank-building/SKILL.md remains the guide for corpus profiling, taxonomy design, candidate mining, curation, privacy, and eval integrity. The autoresearch harness should be used after the evaluator is frozen: it measures construction changes and keeps the loop honest, but it should not decide which entity classes matter or silently change train/test data.

When a kept experiment is ready, open a normal PR with the result row, exact commands, candidate/baseline benchmark hashes, and a short explanation of why the result is not an evaluator artifact.