Overview

Whether you are looking to classify text, answer questions, interact with internal tools, or solve other language tasks, our step-by-step workflow will take you from initial concept to production-ready model. Let’s dive in!

Authentication

First, authenticate with the distil labs platform:

CLI

distil login

API

import json
import requests

# See Account and Authentication for distil_bearer_token() implementation
auth_header = {"Authorization": f"Bearer {distil_bearer_token()}"}

Step 1: Create a model

Register a new model to track your experiment:

CLI

# Returns your model ID
distil model create my-model-name

API

import json
import requests
from pprint import pprint

response = requests.post(
    "https://api.distillabs.ai/models",
    data=json.dumps({"name": "my-model-name"}),
    headers={"Content-Type": "application/json", **auth_header},
)
pprint(response.json())
model_id = response.json()["id"]
print(f"Created model with ID={model_id}")

You can list all your models with:

CLI

distil model list

API

response = requests.get(
    "https://api.distillabs.ai/models",
    headers=auth_header,
)
pprint(response.json())

Step 2: Task selection and data preparation

Begin by identifying the specific task you want your model to perform. Different tasks require different approaches to data preparation and model configuration.

Learn more about task selection →

Prepare your data and configuration files according to your chosen task type. A training job requires the following files in a directory:

File	Format	Required	Description
job_description.json	JSON	Yes	Task objectives and configuration
train.csv	CSV or JSONL	Yes	20+ labeled (question, answer) pairs
test.csv	CSV or JSONL	Yes	Held-out evaluation set
config.yaml	YAML	Yes	Training hyperparameters
unstructured.csv	CSV or JSONL	No	Text documents relating to your problem domain which we may use for synthetic data generation

Learn more about data preparation →

Upload your data to the model (from your local ./data directory):

CLI

distil model upload-data <model-id> --data ./data

API

data = {
    "job_description": {"type": "json", "content": open("data/job_description.json").read()},
    "train_data": {"type": "csv", "content": open("data/train.csv").read()},
    "test_data": {"type": "csv", "content": open("data/test.csv").read()},
    "unstructured_data": {"type": "csv", "content": open("data/unstructured.csv").read()},
    "config": {"type": "yaml", "content": open("data/config.yaml").read()},
}
response = requests.post(
    f"https://api.distillabs.ai/models/{model_id}/uploads",
    data=json.dumps(data),
    headers={"Content-Type": "application/json", **auth_header},
)
upload_id = response.json()["id"]
print(f"Upload successful. ID: {upload_id}")

Step 3: Teacher evaluation

Before training your specialized small model, validate whether a large language model can accurately solve your task with the provided examples. If the teacher model can solve the task, the student model will be able to learn from it effectively. Learn about teacher evaluation →

CLI

# Start teacher evaluation
distil model run-teacher-evaluation <model-id>

# Check status
distil model teacher-evaluation <model-id>

API

import time

data = {"upload_id": upload_id}
response = requests.post(
    f"https://api.distillabs.ai/models/{model_id}/teacher-evaluations",
    data=json.dumps(data),
    headers={"Content-Type": "application/json", **auth_header},
)
eval_job_id = response.json()["id"]
print(f"Started teacher evaluation with ID: {eval_job_id}")

# Poll for completion
running = True
while running:
    response = requests.get(
        f"https://api.distillabs.ai/teacher-evaluations/{eval_job_id}/status",
        headers=auth_header
    )
    status = response.json()["status"]
    if status != "JOB_RUNNING":
        running = False
    print(f"Evaluation status: {status}")
    time.sleep(10)

print(f"Results: {response.json()}")

Step 4: Model training

Once your teacher evaluation shows satisfactory results, train your specialized small language model using knowledge distillation.

Understand the model training process →

CLI

# Start training
distil model run-training <model-id>

# Check status
distil model training <model-id>

API

data = {"upload_id": upload_id}
response = requests.post(
    f"https://api.distillabs.ai/models/{model_id}/training",
    data=json.dumps(data),
    headers={"Content-Type": "application/json", **auth_header},
)
slm_training_job_id = response.json()["id"]
print(f"Training started with ID: {slm_training_job_id}")

# Check status
response = requests.get(
    f"https://api.distillabs.ai/trainings/{slm_training_job_id}/status",
    headers=auth_header
)
pprint(response.json())

# Get evaluation results when complete
response = requests.get(
    f"https://api.distillabs.ai/trainings/{slm_training_job_id}/evaluation-results",
    headers=auth_header
)
print(f"Evaluation results: {response.json()}")

Step 5: Download your model

Once training is complete, download your model:

CLI

distil model download <model-id>

API

response = requests.get(
    f"https://api.distillabs.ai/trainings/{slm_training_job_id}/model",
    headers=auth_header
)
print(f"Model download URL: {response.json()}")

Step 6: Model deployment

Deploy your trained model locally or using distil labs inference for immediate integration with your applications.

Explore deployment options →

Local deployment

Use the distil CLI with llama-cpp as the inference backend:

distil model deploy local <model-id>

Once running, get a ready-to-run invocation script with distil model invoke:

distil model invoke <model-id>

This outputs a command using uv that you can copy and run directly:

uv run $PATH_TO_CLIENT --question "Your question here"

For question answering models that require context, use the --context flag:

uv run $PATH_TO_CLIENT --question "Your question here" --context "Your context here"

Remote deployment

Alternatively, deploy your model on distil-managed remote infrastructure using the inference playground:

distil model deploy remote <model-id>

The CLI will provision your deployment and display the endpoint URL, API key, and a client script you can use to query your model.

Once deployed, you can also use distil model invoke to get a ready-to-run invocation script for your remote deployment:

distil model invoke <model-id>

Next steps

You’ve successfully trained and deployed a specialized small language model! For more details, explore:

• Tutorials for complete end-to-end examples
• Deployment options for production deployment