Technical · 2026-11-04 · 12 min

The 11 MB PII model that survived our test

Inkfish scans a PDF on the Mac and removes private data before the file ever leaves the machine. Picking the local model behind that scan took longer than expected, mostly because every candidate looked great on its own Hugging Face model card and none of those cards resembled the PDFs our users actually open.

An on-device PII detection model is a compact sequence-tagging or token-classification network bundled inside a local application to identify personal data such as names, street addresses, national IDs, and financial accounts without network calls or cloud APIs. In a desktop PDF redaction pipeline, it works alongside deterministic regex and checksum validators: rules lock down structured numbers, while the local model catches unstructured person names, organizations, and multi-line addresses.

Our real inputs mix three awkward shapes: consulting and lease agreements with names buried in legal prose, financial tables with bare phone numbers in a narrow column, and scanned pages where Apple Vision OCR hands back German street addresses one line at a time. I could not find a public benchmark that tested all three together, so I built a test runner and ran the candidates myself.

How we scored “found”: two metrics that ignore label names

Every model invents its own label taxonomy. Scoring strict F1 by label name mostly measures how closely a model’s schema matches the benchmark’s schema. For PDF redaction, the practical question is simpler: did the characters that needed to be covered actually get covered?

I used two label-agnostic metrics across every run:

  1. Entity pass rate counts a target entity as passed only when every non-whitespace character inside it is covered by the model’s output spans. Leaving half a surname or the last two digits of an apartment number exposed counts as a miss. Ignoring internal whitespace is essential; otherwise, an unmasked space between a first and last name would fail a visually complete redaction.
  2. Character miss rate measures the percentage of ground-truth PII characters that remained visible on the page. This is the fairer score for models that emit fragmented spans. If a model splits a 16-digit card number or a hyphenated name into two adjacent masked pieces, the characters are still hidden.

I ran those two metrics against three distinct datasets:

Test setSample sizeWhat it tests
Handwritten trilingual cases20 documents, 85 entitiesNatural prose in English, Chinese, and German, plus table rows and clean negatives
ai4privacy/pii-masking-300k200 German + 200 French rowsPublic labeled corpus packed with dense, multi-field template strings
Inkfish rule fixtures43 positives, 12 negativesThe exact positive strings and broken-checksum counterexamples written for Inkfish’s 43 built-in rules

The third set carried the most weight. I wrote the first set and sampled the second, so both reflect my blind spots to some degree. The 43 positive fixtures and 12 negative counterexamples existed in the repository long before we evaluated any neural model.

Four open-source candidates, and where the ranking flipped

Running all four candidates on the same Mac produced a table where the winner depends entirely on which column you read first:

ModelDisk sizeEntity pass rate (prose)Character miss rate (dense text)German pass rateLatency per call
OpenAI Privacy Filter2.6 GB88%6.4%82%230 ms
AstrLink Guard79 MB73%40.3%61%5 ms
newmean-pii medium11 MB94%23.7%92%15 ms
GLiNER2-PII1.2 GB97% (43 rule fixtures only)Not run on same batch96%255 ms

Look only at the third column and the 1.2 GB GLiNER2-PII or the 11 MB newmean-pii medium wins. Move one column to the right and the leaderboard turns upside down. Three specific findings from this run drove our final architecture:

First, the ranking flips when the text density changes. On our handwritten trilingual prose, newmean-pii medium missed only 1.3% of PII characters, beating the 2.6 GB OpenAI Privacy Filter (5.9% missed). On the dense, multi-field template rows from ai4privacy, newmean’s character miss rate jumped to 23.7%, while Privacy Filter stayed flat at 6.4%. No single model won across both distributions.

Second, the 11 MB model’s blind spots are narrow and predictable. When I broke the ai4privacy misses down by entity category, newmean missed only 0.5% to 5% of characters across national IDs, passports, driver’s licenses, and US Social Security numbers, compared with 57% to 87% missed by the 79 MB AstrLink Guard. What dragged newmean’s overall average down were three categories: passwords (86% missed), standalone city names (65% missed), and postal codes (35% missed). My initial 20 handwritten documents had zero passwords, zero bare dates, and no standalone municipality names, which is why my first draft of this test overrated the model.

Third, token classifiers cannot compute checksums. Our 12 negative fixtures included a Chinese resident ID with an invalid 18th check character, a European IBAN that fails mod-97, and a credit card number that fails the Luhn check. Every model we tested masked them anyway.

Negative fixtures (12 broken-checksum inputs)AstrLink Guardnewmean-pii mediumOpenAI Privacy Filter
False positives (masked invalid numbers)3109

Tweaking a confidence slider will not fix this. A transformer sees character shapes and surrounding words; it does not run modulo arithmetic inside an attention head. Any redaction rule that depends on a check digit has to stay in code.

Why we shipped the 11 MB newmean-pii medium

We chose newmean-pii medium (roughly 11.3 MB of quantized weights plus a 0.8 MB tokenizer, 11.55 MiB on disk). Four practical reasons tipped the scale:

  • It passed 40 of Inkfish’s 43 built-in rule fixtures (93%), while the other two full-run candidates managed 32 (74%) and 15 (35%).
  • It reached a 92% pass rate on German, which Inkfish requires for DACH contracts and leases. In our follow-up engineering check on full trilingual contracts, AstrLink Guard missed multiple multi-line German street addresses even in FP32.
  • Its footprint fits a native Mac utility. The local Swift package wraps a pure C inference core with zero external runtime dependencies (no ONNX Runtime or Python bridge). After stripping its network download code, it adds 369 KB to the compiled binary, peaks near 60 MB of RAM in CLI benchmarks, and reloads in under a second when cached. Asking someone to download a 1.2 GB or 2.6 GB model just to redact a two-page consulting agreement was never realistic.
  • Its 23.7% miss rate on dense template dumps is a known trade-off we can design around. If Inkfish fed raw, unsegmented page dumps exclusively to a neural net, 23.7% would be a dealbreaker. Because Inkfish already runs deterministic regional rule packs for structured IDs, tax numbers, and postal codes, the local model’s job is to catch unstructured person names, street addresses, and organizations inside clauses and table cells.

Repositories, base models, and Apache-2.0 license audit

Before bundling weights inside a Mac App Store binary, I read the actual LICENSE and NOTICE files in each repository rather than trusting the license badge in the Hugging Face sidebar:

ModelRepositoryLicenseBase model & tokenizer
newmean-piinewmean-ai/pii-mediumApache-2.0mMiniLMv2-L6-H384 (MIT), XLM-R tokenizer (MIT)
OpenAI Privacy Filteropenai/privacy-filterApache-2.0Custom MoE, 1.5B total / 50M active parameters
AstrLink GuardQuantumNous/astrlink-guardApache-2.0Multilingual-MiniLM-L12-H384
GLiNER2-PIIfastino/gliner2-privacy-filter-PII-multiApache-2.0mDeBERTa-v3-base

Reading newmean’s NOTICE file surfaced two details that matter for commercial shipping:

First, its public evaluation dataset (newmean-ai/pii-dataset) is licensed under CC BY-NC-4.0 (non-commercial). You cannot copy that dataset into a commercial repository as a regression suite. The SDK code and the four model weight tiers (nano, small, medium, large) are explicitly licensed under Apache-2.0 and are unaffected.

Second, its training corpus was synthetic text generated with Claude models and labeled by fine-tuned mDeBERTa-v3-base and XLM-RoBERTa-large teacher models, with no third-party personal data corpus included. Under Apache-2.0 Section 4(d), shipping the weights requires reproducing the NOTICE file alongside the upstream MIT notices for Microsoft’s mMiniLMv2 and Facebook’s XLM-R tokenizer in the app’s acknowledgments screen. The notice’s request for a "Powered by newmean.ai" link is explicitly marked not a license condition. This section is engineering context, not legal advice.

Three engineering fixes required after integration

Picking the weights was only half the work. Wiring the model into the scan-and-review pipeline described in the Inkfish user guide exposed three integration gaps right away:

  1. Upstream NewmeanPII includes a PIIModelStore that calls URLSession to download weights when no local path is supplied, and its token spans arrive fragmented. Because Inkfish ships with a strict zero-network sandbox entitlement, we removed the download and remote-cache code entirely so PIIRedactor requires a bundled directory inside the app bundle. We feed page text in 1,024 UTF-16 code-unit chunks with a 128-unit overlap and map utf16Range offsets back to PDF page coordinates. Because the model often splits a bank account number into six pieces or a name into two, we merge overlapping same-label spans within strict line and column bounds so the review sidebar stays readable.
  2. The model outputs broad buckets (ID_NUMBER for any passport, SSN, or national ID; FINANCIAL for any card, IBAN, or routing number), and its C core picks labels from raw token logits without producing a calibrated entity probability. Instead of inventing a fake “90% confidence” badge, Inkfish displays calibrated percentages only for rule-based findings (pre-ticked at 85% by default) and labels model-unique findings with their source (Local model suggestion).
  3. Because a token classifier will happily mask an IBAN with a broken checksum or confuse a contract signing date with a date of birth, Inkfish runs its deterministic regional rule packs and Apple NaturalLanguage pass first, then fuses the local model’s suggestions afterward. Structured identifiers still require a matching rule and checksum validator; the model is never allowed to override a failed checksum or re-enable a rule pack the user turned off.

What I got wrong in the first draft

My first pass at this benchmark used only the 20 handwritten documents and concluded that the 11 MB model beat the 2.6 GB model across the board. Adding the 400 independent ai4privacy rows reversed that claim: on dense multi-field text, the small model misses nearly four times as many characters.

Two mistakes caused that early false victory: 85 entities are far too few to separate models within a few percentage points, and when I wrote test cases by hand, I subconsciously wrote clean, grammatical sentences while leaving out passwords, bare dates, and isolated city names.

The rule we use now is straightforward: any claim that one model beats another requires at least 200 independent samples, a per-category breakdown, and deliberate negative counterexamples. How Inkfish turns these detections into permanent PDF content-stream removal and communicates progress through its orange squid mascot is covered in Why Inkfish Uses a Squid for PDF Redaction and our Mac PDF redaction guide.

Common questions

Does Inkfish download the PII model over the internet on first launch?

No. The newmean-pii medium weights and tokenizer (about 11.55 MiB) are bundled directly inside the Inkfish app on the Mac App Store. All network-download code was removed from the local Swift package, and Inkfish ships without any network client entitlement in its macOS sandbox profile.

Why can’t a local neural PII model replace regex and checksum rules?

Token-classification models recognize visual character patterns and surrounding phrasing, not mathematical check digits. In our 12 negative test fixtures with broken Luhn card digits, invalid IBAN mod-97 remainders, and wrong ID check characters, all tested models still flagged the invalid numbers as PII. Neural models also group identifiers into coarse buckets (ID_NUMBER, FINANCIAL) without knowing whether a string is a US SSN or a German tax ID. Deterministic rules and checksums must validate structured numbers, while the neural model supplements them on unstructured names and addresses.

Which languages does the bundled PII model support, and is it commercially licensed?

In Inkfish’s local detection pipeline, the bundled model supports English, Chinese, and German (reaching a 92% pass rate on our German test set), paired with Apple Vision on-device OCR for scanned PDF pages. Both the newmean-pii SDK and model weights are licensed under Apache-2.0, which permits commercial redistribution provided the NOTICE file and upstream MIT notices (mMiniLMv2 and XLM-R tokenizer) are included. Note that the upstream newmean-ai/pii-dataset evaluation corpus is CC BY-NC-4.0 and cannot be used commercially.

Inkfish app icon

More notes