Train an SLM for Content Moderation

Moderation is a classification task, and the model will reproduce your policy including the parts of it nobody has written down. The work is converting an enforcement policy into labels with explicit edge-case rules, then keeping the model inside a review path.

Moderation is a policy problem before it is a model problem

Every moderation policy has a documented core and an undocumented margin. The core — explicit threats, sexual content involving minors, doxxing — is unambiguous and easy to label. The margin is where all the volume is: sarcasm, reclaimed slurs, quoted abuse, satire, medical language, threats phrased as jokes.

Your reviewers already adjudicate the margin every day. They do it from precedent, team norms, and Slack threads. A model cannot read those, so the first task is extraction: for each recurring edge case, write down the rule your reviewers actually apply.

This has a second benefit. Writing the rules down surfaces the ones your reviewers disagree about, and those are exactly the cases where a model trained on their labels will be unstable.

Step 1: Turn the policy into labels with written adjudication rules

Decide the label set first. Two shapes work; a third does not.

Label shape	Works?	Notes
Binary allow/remove	Yes	Simplest; loses the reason for removal
One label per policy violation type	Yes	Gives reviewers routing as well as a verdict
Severity score 1–5	No	Not a classification task; annotators will not agree on adjacent scores
Multiple labels per item	No	The task assigns exactly one class

Then write each class with its edge cases stated:

{
  "task_description": "Assign each user-submitted post to exactly one moderation outcome.",
  "classes_description": {
    "harassment_targeted": "Abuse aimed at a specific identifiable person. Includes abuse quoted approvingly. Excludes abuse quoted in order to report or criticise it.",
    "allow": "No policy violation. Includes profanity not directed at a person, dark humour with no target, and clinical or medical descriptions.",
    "needs_human_review": "The post may violate policy but the decision depends on context this system does not have: ongoing disputes, in-group language, or an unclear target."
  }
}

needs_human_review is not an admission of weakness. It is the class that makes the rest of the taxonomy honest, and it gives the model somewhere to put the cases your reviewers escalate too.

Step 2: Seed from overturned appeals, not clean examples

Clean examples teach the model nothing. Appeals that were overturned are records of a decision your own system got wrong, adjudicated by a human — they are the highest-value training data you own.

Build the seed set roughly as: a third clear violations, a third clear allows, and a third drawn from overturned appeals and reviewer-escalated cases. Keep label strings identical to the classes_description keys. Add an unstructured.jsonl of unlabelled real posts so the teacher generates in your platform’s register rather than in generic English. The classification data preparation guide has the file spec.

Step 3: Configure the run

distil model create content-moderation
# Output: Model created with ID: <model-id>

distil model upload-data <model-id> --data ./data
distil model upload-status <model-id>

base:
  task: classification
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5
synthgen:
  teacher_temperature: 0.6
  num_negative_exemplars_per_generation: 4
  mutation_topics:
    - "sarcasm and irony where the literal reading is misleading"
    - "abuse quoted in order to report it"
    - "reclaimed or in-group language"
    - "threats phrased as jokes or hypotheticals"

mutation_topics is doing the important work. Without it the teacher generates whatever is easiest, which for moderation means obvious violations you already handle. Naming the hard categories forces coverage of the margin. The config reference documents the rest.

Step 4: Evaluate the teacher, then train

distil model run-teacher-evaluation <model-id>
distil model teacher-evaluation <model-id>
distil model download-teacher-evaluation-predictions <model-id>

distil model run-training <model-id>
distil model training <model-id>

Read the teacher’s errors before training. A teacher that misreads your edge cases will generate thousands of examples that encode the misreading, and the student will learn it faithfully — see when does distillation fail.

Step 5: Split the errors by direction, not by count

Accuracy is the wrong headline for moderation because the two error directions have different costs and different owners. Over-removal generates appeals and user churn; under-removal generates harm and, in some jurisdictions, liability. Report them separately, per class.

The AI slop detector is a useful cautionary example of what a clean headline hides. A fine-tuned Gemma 3 270M matched its 120B teacher at 100% on the held-out set from one Kaggle dataset, then in wider spot checks scored ~92% on Reddit comments and 88% on formal emails, where formal human writing triggers false positives. On a graded test suite it scored 100% on easy cases and 83% on the hard ones. Your evaluation set must contain the hard cases in the proportion your queue actually sees, or the number you report is not about production.

Step 6: Deploy behind the review queue

distil model deploy local <model-id>
distil model invoke <model-id>

Route needs_human_review and any low-confidence verdict to people. Log every automated decision with the model version so an appeal can be traced to the model that made it, and keep overturned appeals flowing back into the seed set — retraining on your own reversals is the loop that makes the system improve.

What a moderation model must never decide alone

Irreversible actions. Account termination, legal referral, and law-enforcement escalation need a human in the path regardless of the model’s score, because a classifier has no notion of the cost of being wrong.

Two more limits. The model knows only the text you send it, so anything decided by account history or an ongoing dispute belongs in code, not in the taxonomy. And policies change: when yours does, the model does not, so treat a policy revision as a retraining trigger. For the task type itself see what is text classification, and for taxonomy mechanics see support ticket triage.