NERB Schema Reference¶

This document describes the public JSON-compatible contracts for NERB banks, extraction records, replacement databases, anonymization responses, eval refs, YAML detector configs, and shared diagnostic objects. The runtime source of truth is the code in src/nerb/schema.py, src/nerb/extraction.py, src/nerb/replacements_schema.py, src/nerb/deanonymization.py, src/nerb/reports.py, and src/nerb/evals.py.

Shared Rules¶

IDs use this pattern:

^[a-z][a-z0-9_]{0,79}$

IDs apply to bank IDs, entity IDs, name IDs, pattern IDs, and batch document_id values. Status values are draft, active, inactive, and deprecated. Regex flag values are ASCII, IGNORECASE, MULTILINE, DOTALL, and VERBOSE. Unicode normalization values are none, NFC, and NFKC.

metadata fields are JSON objects whose values must be JSON-compatible. Metadata above 16 KiB produces a warning; metadata above 1 MiB is an error. Descriptions are capped at 2,000 characters. Pattern values are capped at 10,000 characters.

eval_refs can appear on a bank, entity, name, or pattern. Each value is a non-empty string. Local eval execution requires relative paths under the eval base path; absolute paths, parent traversal outside the base path, remote URIs, non-regular files, and invalid UTF-8 are rejected.

JSON Bank¶

A JSON bank is the main agent and service format. It stores authoring metadata plus extractable literal and regex patterns in one validated object.

Top-Level Bank¶

Additional top-level properties are rejected.

Entity¶

Each entity is stored under entities.<entity_id>.

Additional entity properties are rejected.

Name¶

Each name is stored under entities.<entity_id>.names.<name_id>.

Additional name properties are rejected.

Pattern¶

Each pattern is stored under entities.<entity_id>.names.<name_id>.patterns.<pattern_id>.

Literal patterns also require:

Regex patterns also require:

Literal patterns must not include regex_flags. Regex patterns must not include case_sensitive, normalize_whitespace, left_boundary, or right_boundary. Additional pattern properties are rejected.

Minimal Complete Bank¶

{
  "schema_version": "nerb.bank.v1",
  "id": "company_entities",
  "name": "Company Entities",
  "description": "Companies to recognize in internal documents.",
  "version": "2026.06.24",
  "status": "active",
  "created_at": "2026-06-24T00:00:00Z",
  "updated_at": "2026-06-24T00:00:00Z",
  "unicode_normalization": "none",
  "default_regex_flags": ["IGNORECASE"],
  "entities": {
    "company": {
      "description": "Organizations.",
      "status": "active",
      "regex_flags": [],
      "names": {
        "acme_corp": {
          "canonical": "Acme Corp",
          "description": "Primary account.",
          "status": "active",
          "patterns": {
            "primary": {
              "kind": "literal",
              "value": "Acme Corp",
              "description": "Exact company alias.",
              "status": "active",
              "priority": 100,
              "case_sensitive": false,
              "normalize_whitespace": true,
              "left_boundary": "word",
              "right_boundary": "word",
              "metadata": {}
            }
          },
          "metadata": {}
        }
      },
      "metadata": {}
    }
  },
  "metadata": {}
}

Extraction Records¶

Direct Bank scans and config-backed CLI/MCP extraction return Rust-backed records:

JSON-bank extraction enriches each record with source IDs and captures:

Example JSON-bank record:

{
  "entity": "company",
  "canonical_name": "Acme Corp",
  "surface_name": "Acme Corp",
  "string": "Acme Corp",
  "start": 13,
  "end": 22,
  "offset_unit": "byte",
  "entity_id": "company",
  "name_id": "acme_corp",
  "pattern_id": "primary",
  "pattern_kind": "literal",
  "captures": {}
}

Direct Bank records are sorted deterministically by start offset, end offset, entity, canonical name, surface name, and matched string. JSON-bank records are sorted by start offset, end offset, entity_id, name_id, pattern_id, and matched string. Batch flat records add a leading document_id field.

Extraction Responses¶

Single-document JSON-bank extraction returns:

Batch extraction returns:

Batch input documents must be objects with a valid document_id or id, and exactly one of text or file_path. Default limits are 100 documents, 10 MiB per document, and 25 MiB combined text.

Report Responses¶

extract_report returns extraction records plus overlap resolution and context:

An overlap object has id, policy, span, records, resolved_record, and dropped_records. The current public report policy is priority. Lower numeric pattern priority wins; ties prefer longer matches, earlier starts, stable IDs, and matched string ordering.

Report batch responses mirror batch extraction and add flat resolved_records, flat overlaps, and per-document report objects.

Replacement Database¶

A replacement database is a local JSON object with schema version nerb.replacements.v1. It stores default replacement policy, optional per-entity policy, replacement candidate sets, and assignments. When store_originals is true, the database contains sensitive originals and must be treated as sensitive local state.

Top-Level Replacement DB¶

Additional top-level properties are rejected.

Replacement Policy¶

defaults must contain the base policy fields. replacement_set_id is optional for redaction policies, but required in the effective policy when replacement_mode is pseudonym. entities.<entity_id> may contain any non-empty subset of the same fields.

Assignment scopes:

• name: uses JSON-bank entity_id and name_id; aliases share one assignment and de-anonymization restores the canonical value.
• canonical: uses entity plus normalized canonical_name; useful for YAML detector configs.
• surface: uses entity plus normalized matched string; useful when exact source-surface restoration matters.

Replacement Sets¶

Each replacement set contains description, reuse, candidates, and optional metadata. Candidates are objects with id, non-empty value, and metadata. Pseudonym mode allocates from the configured candidate set and rejects exhausted or ambiguous candidate pools.

Assignments¶

Assignment keys use this opaque format:

<entity>|<scope>|sha256:<64 lowercase hex>

Assignment rows contain assignment_key, entity_id, identity, replacement, redaction, timestamps, use_count, and metadata. original is present only when the effective policy stores originals. Fingerprints, assignment keys, source IDs, originals, replacement values, and hashes are linkable or sensitive. Default CLI response metadata redacts them, while the transformed text contains replacement values by design. Default Python and MCP anonymization response metadata include replacement values because those values are also present in the transformed text; they still redact originals, raw assignment keys, fingerprints, and hashes.

Anonymization Responses¶

anonymize_text, anonymize_file, anonymize_config_text, and anonymize_config_file return nerb.anonymize_response.v1 payloads.

Default Python and MCP applied_replacements entries include opaque assignment_ref, entity, mode, original_span, replacement_span, and replacement. CLI output strips replacement unless --include-sensitive-metadata is set. include_originals adds original strings. include_sensitive_metadata adds raw assignment keys, source record IDs, DB data, hashes, and file paths where available.

Config-backed anonymization uses Rust Bank.scan_text() records from YAML detector configs. It does not run JSON-bank report resolution. Because config records do not include name_id, use assignment_scope: "canonical" or assignment_scope: "surface" for config-backed replacement DBs.

De-Anonymization Responses¶

deanonymize_text and deanonymize_file return nerb.deanonymize_response.v1 payloads.

Redaction tokens are restored by default. Pseudonym restoration requires restore_pseudonyms=true or --restore-pseudonyms and emits a warning because it is exact string replacement. If store_originals was false or a replacement maps to multiple originals, de-anonymization returns diagnostics instead of guessing.

Eval JSONL¶

Eval references are UTF-8 JSONL files. Blank lines are ignored. Each non-empty line must be one JSON object with one of these type values: positive, negative, or provenance.

Positive records require exact expected matches:

Positive match objects require string, start, and end. They may also include entity, entity_id, name, name_id, pattern_id, pattern_kind, and captures. start and end are non-negative byte offsets, end >= start, and string must equal the UTF-8 text slice at [start:end]. When both entity and entity_id are present, they must agree.

Capture objects are keyed by capture name and contain only string, start, and end.

Negative records assert that scoped extraction should return no records:

Provenance records are counted but do not affect pass/fail:

Example eval ref:

{"type":"positive","text":"Send this to Acme Corp.","matches":[{"string":"Acme Corp","start":13,"end":22}],"metadata":{"case":"fixture_positive"}}
{"type":"negative","text":"The acme of performance was impressive.","reason":"Common-word false positive guard.","metadata":{}}
{"type":"provenance","source_type":"crm","observed_at":"2026-06-03T00:00:00Z","evidence":"Fixture eval record for Acme Corp.","metadata":{}}

Eval Responses¶

eval_bank returns:

Diagnostic Objects¶

Diagnostics are JSON objects with stable core fields:

YAML Detector Config¶

YAML detector configs are compact maps used by config-backed CLI and MCP extraction:

ARTIST:
  _flags: IGNORECASE
  Pink Floyd: 'Pink\sFloyd'
  The Who: '[Tt]he\sWho'

GENRE:
  _flags: [IGNORECASE, MULTILINE]
  Rock: '(?:progressive\s)?rock'

The top level must be a mapping of non-empty entity names to mappings. Each entity mapping must contain at least one detector pattern. _flags is reserved for regex flags and may be a flag string or an array of flag strings. All other keys are non-empty detector names and all pattern values must be regex strings.

Empty top-level configs are valid for config authoring, but every non-empty entity must contain at least one detector pattern. Config extraction compiles through the same Rust-backed Bank scanner and returns the direct record schema above.