CONFIDE-Bench — Which Layer Earns Its Compute?

Structured direct IDs (email, phone, policy ID) and names/orgs/locations reach 0.9–1.0 from regex + Natasha. Dates are now caught too — a numeric-date regex rule was added after the benchmark exposed the gap.

Quasi-identifiers needing world-knowledge — a drug name, an occupation, a spelled-out age — are invisible to pattern and NER layers; the LLM is the only layer that lifts their mention-recall.

Caveat: this is mention recall. At entity level (all mentions masked), medication and profession stay at 0 even with qwen — a higher bar.

Russian runs only the local-first stack — Natasha + regex + qwen2.5:3b. Three detectors are English-only by design, so any combo containing them has no RU bar: Presidio (its RU is spaCy-NER-dependent and weak — left unscored rather than misrepresented), Philter (an English clinical-notes rule set), and the OpenAI Privacy Filter (an English token-classifier). This is a scope decision, not a measured RU failure.

RU: the proposed default natasha+regex+ollama ★ reaches 88% coverage recall. The OPF-on-RU row is omitted until its detector cache is regenerated for the current 30-document corpus.

EN-synth: OPF is the name/address backbone (English’s Natasha). Default opf+regex+ollama ★; opf+regex edges it on F2.

RU-real (JayGuard): on external real Russian text the local stack reaches strong coverage — a real-text anchor for the otherwise-synthetic RU corpus (PERSON/LOCATION only).

16 snippets, 20 gold forms: patronymics, transliteration, diminutives, VK/Telegram handles, SNILS/INN/passport IDs, abbreviated addresses, and code-switching.

Regex catches the structured IDs and handles; Natasha + qwen2.5:3b recover patronymics, diminutives and code-switching. The one residual leak is “Sergey Volkov” — a Latin-transliterated Russian name: Natasha is Cyrillic-only, regex has no name rule, and qwen missed it. This is the argument for adding an English/Latin NER (OPF) when transliteration is expected.

Direct (name, phone, email, policy): masked at 0.81 entity recall in the default stack.

Quasi (age, profession, city, employer, medication, date): the LLM helps but the ceiling stays low — these are the attributes that, combined, re-identify a person.

Method — following the re-identification / inference-attack literature (Staab et al.; RAT-Bench; Tau-Eval). An entity survives if any one of its mentions is left unmasked.

A local 3B qwen attacker recovered 2 of 10 tested attributes from the redacted text (e.g. the medication, because its entity-recall is 0). A weak attacker is a lower bound — the inference-attack literature (Staab et al.) reports much higher re-identification rates for frontier models.

Implication: redacting direct identifiers is table stakes; quasi-identifier survival is a useful gate to check before sending a session to cloud analysis.

Method — top-k inference attack with a fixed, declared budget (qwen2.5:3b, temp 0.4, top-3 guesses/attribute, redacted text only) + downstream task preservation (re-run cognitive-distortion extraction on redacted vs. original). Aligned with Staab et al. / RAT-Bench (privacy) and Tau-Eval (task-sensitive utility).

The same masking that lowers attacker success can erase clinical content. Here it does not: the default stack keeps ~91% of distortion signal and 99.5% of non-PII text while a weak attacker recovers nothing top-3.

Caveat: this attacker is a lower bound — quasi-identifiers still survive in text (medication entity-recall is 0), so a frontier attacker would score higher. Residual risk stays MEDIUM for client B.

Coverage F2 (orange) asks only “did we mask the span at all”. Type micro-F1 (green) demands the right label too — what a redaction policy actually needs.

On EN-synth, Presidio edges the stack on coverage F2 (a broad DATE_TIME recognizer) but its type-F1 is far lower; Philter is high-coverage yet emits almost everything as untyped OTHER, so its type-F1 is unusable.

Takeaway: a generic system is not a therapy-tuned one; the only coverage a baseline adds is relative/colloquial dates.

WP29 (Art-29 WP 05/2014) re-identification triad — identifiability decomposes into singling out (residual quasi surface via a caveated population-fraction estimator — NOT corpus k-anonymity, N is tiny), linkability (pairwise session-linking ROC AUC 0.46 — the area under the ROC curve: 1.0 = perfectly linkable, 0.50 = no better than chance, which is the safe direction), and inference (attribute-recovery attack recovers 20%). HIPAA coverage is a Safe-Harbor-inspired checklist, not a legal determination (AGE is N/A; structured IDs collapsed).

RED = any in-scope direct identifier leaks at entity level (one unmasked mention is a key). AMBER = special-category residual, nonzero inference, or linkability above chance. GREEN = all clear.

What leaks: not whole names but specific variants — inflected/possessive/patronymic forms (Артёмом, Натальин, Денису), lowercase surnames, vocatives, Latin transliteration (Timur), and name/common-word collisions (Вера, Роман). Mention-level recall hides this; the strict TAB entity bar (one miss ⇒ unprotected) surfaces it.

The singling-out estimate is illustrative (a caveated population-fraction method, not corpus k-anonymity) — it is not a guarantee of non-identifiability.