Train an SLM for PII Redaction
Redaction is a question-answering task whose output is a JSON blob, and its errors are not symmetric. A missed email address is a disclosure; a wrongly redacted product name is an inconvenience. Build the seed data, the rubric, and the review process around that asymmetry.
A missed identifier costs more than a false positive
Under-redaction and over-redaction are both wrong, and they are not equally wrong. One leaks personal data into a log store, an analytics warehouse, or a training set. The other removes a word from a support transcript.
That imbalance should be visible in three places, and usually is not:
- Seed data. Weight it toward the identifiers your model is likely to miss, not the ones it will obviously catch.
- The evaluation rubric. A metric that averages both error directions lets recall failures hide behind precision.
- The deployment. If the model is a gate on data leaving a boundary, the fallback on uncertainty is to redact, not to pass through.
Over-redaction has a real cost too — redacting “I” as a person name or every company name destroys the operational signal that made the data worth keeping. The policy is where you draw that line explicitly.
Step 1: Write the policy as a redact list and a keep list
Two lists, both explicit. The distil-PII policy is a worked example: it redacts PERSON, EMAIL, PHONE, ADDRESS, SSN, ID, UUID, CREDIT_CARD, IBAN, GENDER, AGE, RACE, and MARITAL_STATUS to fixed tokens, and keeps card last-4 when only last-4 is present, order and ticket numbers, device serials, company and product names, and bare city or country mentions.
The keep list is the half people skip, and it is what stops the model redacting everything that looks vaguely like an identifier. Some entities are partial: a credit card becomes [CARD_LAST4:####] rather than disappearing, because the last four digits are what makes the record usable downstream.
Then pin the output schema in task_description: a single JSON object with redacted_text (the input with minimal in-place replacements) and entities (an array of {value, replacement_token, reason}). Nothing else — no preamble, no explanation.
Step 2: Seed the obfuscated cases
Twenty curated examples were enough for the published models, but they were the right twenty. Clean inputs teach the model nothing it will not generalise anyway.
| Seed category | Why it earns its place |
|---|---|
Obfuscated emails (jane (at) example (dot) org) |
Regex misses these; models trained only on clean emails do too |
| Partial numbers (“ending 9021”, “•••• 9021”) | Tests the keep rule, not the redact rule |
| Full card and IBAN numbers | Tests partial replacement with last-4 preserved |
| Self-declared attributes (“I am female”, “29 y/o”) | Easily missed; not shaped like an identifier |
| Operational IDs alongside personal ones | The discrimination the keep list encodes |
| Text with no PII at all | Prevents phantom entities in the output |
Each example is a plain question-answering line: the instruction and text in the user turn, the exact JSON blob in the assistant turn. Turn on JSON-only generation in the config so the teacher cannot fabricate malformed targets:
base:
task: question-answering
student_model_name: Llama-3.2-1B-Instruct
teacher_model_name: zai.glm-5
synthgen:
teacher_temperature: 0.6
output_is_json: trueoutput_is_json is documented in the config reference and only applies to QA tasks. It is cheap insurance against schema drift, the most common base-model failure on this task.
Step 3: Write a rubric that fails under-redaction
llm_as_a_judge_instructions in job_description.json decides what “correct” means, so make it binary and unforgiving. The published rubric fails a prediction unless all of these hold: the output is JSON and nothing else; it parses with redacted_text (string) and entities (array); every entity has value, replacement_token, and reason; redacted_text equals the reference exactly; and the set of (value, replacement_token) pairs equals the reference set, ignoring order and reason.
Set equality is the load-bearing clause. It fails a prediction that misses one entity and also one that invents one, with no partial credit — which is what you want when the question is whether the text is safe to release.
Step 4: Upload, evaluate, train
distil model create pii-redactor
# Output: Model created with ID: <model-id>
distil model upload-data <model-id> --data ./data
distil model run-teacher-evaluation <model-id>
distil model teacher-evaluation <model-id>
distil model run-training <model-id>
distil model training <model-id>
distil model download-training-predictions <model-id>Base models are not close to usable here, and the size of the gap is the point. From the distil-PII release, scored on the rubric above with DeepSeek 3.1 as judge:
| Model | Parameters | Base | Fine-tuned |
|---|---|---|---|
| DeepSeek 3.1 (reference, untrained) | 685B | 0.84 ± 0.03 | — |
| Llama 3.2 3B | 3B | 0.03 ± 0.02 | 0.82 ± 0.03 |
| Llama 3.2 1B | 1B | 0.00 ± 0.00 | 0.81 ± 0.02 |
| Gemma 3 270M | 270M | 0.07 ± 0.05 | 0.73 ± 0.07 |
| SmolLM2 135M | 135M | 0.17 ± 0.07 | 0.25 ± 0.05 |
The 1B model lands within one standard deviation of a 685B reference. The last row is the honest caveat: SmolLM2 135M resisted training on this task and reached 0.25, so this is not a task where the smallest student works — see SmolLM2 135M, the smallest useful student.
Step 5: Read the failures by entity type, not in aggregate
An aggregate judge score tells you nothing about which identifier leaks. Group failures by the entity type involved and split each group into missed and invented. A model that misses one entity type consistently needs four more seeds of that type, not a bigger student.
The residual failure modes to look for are the ones the base models exhibited: schema drift, hallucinated entities, under-redaction on obfuscated inputs, and over-redaction of pronouns and organisation names.
Step 6: Deploy where the data already is
distil model deploy local <model-id>
distil model invoke <model-id>Redaction is nearly always deployed inside the trust boundary — the point of the exercise is that raw text never leaves. deploy local serves an OpenAI-compatible endpoint; local deployment covers vLLM for higher throughput.
Where a redaction SLM still under-redacts
Quasi-identifiers. A model trained on a token list catches emails and card numbers; it does not catch a combination of job title, employer, and city that identifies exactly one person. Re-identification risk is not an entity-extraction problem and no redaction model solves it.
Two more limits: the model sees one text at a time, so cross-document linkage is invisible to it, and any new identifier type — a new customer reference format, a new national ID — is a policy change and therefore a retraining trigger. For the task type see what is question answering as a training task; for the general failure conditions see when does distillation fail.