Fine-tuning your own model

Fine-tuning does not make a model smarter. It makes one model consistently good at one narrow job. That is the whole value, and most disappointment with fine-tuning comes from expecting the other thing.

I have one use for it: a small model that writes like me, sitting behind a frontier model that does the reasoning. This note is built around that, with a practical guide by Rahul (@sairahul1) and the current tooling underneath.

Rahul on fine-tuning as an engineering skill

The post, with a full step-by-step guide linked from it

https://x.com/sairahul1/status/2100882424343265527

What I actually want

Not a smarter model. A voice model: something small that writes like me, taking its facts and its argument from a frontier model that already did the thinking.

It's the same shape as the decision layer in Jev: the expensive model thinks, a small specialised model handles the part that has to be fast, cheap and consistent.

Why fine-tuning rather than another prompt: my voice rules already exist as prompt text, in SOUL.md, in my anti-AI-writing guidelines, in the style files. They work. They also cost context on every single call, drift over a long session, and have to be re-read every time. Tone, sentence length, line breaks, the words I never use: that is behaviour, which is exactly what training bakes in and prompting has to keep re-explaining.

The rule for the small model is that it must not think. It receives the facts and changes the wording, never the meaning.

The closest public precedent I found runs the same split: a frontier model reasons, then a Qwen 2.5 3B fine-tuned on 75,329 personal messages rewrites the output in the author's voice.

I trained an LLM on 75k of my own messages

A working reasoner-plus-rewriter setup, with the dataset work described

https://dev.to/niclydon/i-trained-an-llm-on-75k-of-my-own-messages-so-it-would-stop-writing-like-a-chatbot-1dpi

The dataset is already on disk

⚠️ WARNING: the note corpus is no longer all mine. 99 of the 1,284 notes carry the NicAI signature, so a model wrote them. Train on those and the voice model learns to imitate an imitation. Filter on the signature before building the set.

The email pairs are the valuable part. An AI draft on one side, what I actually sent on the other, teaches the transform rather than just the target. That is the supervision style transfer wants, and it's the same data that produced the "narrated opener, justification tail, closing offer" findings in my writing guidelines.

What the evidence supports

• Hundreds of paired examples move the voice. 1,000+ is the safer target for a general set.
• LoRA rank 16, 1 to 3 epochs is the usual starting point for style. Style is a small change to the weights, so big ranks are the wrong instinct here.
• Evaluate blind. Hold out prompts, generate from the base model and the tuned model, compare unlabelled. There is no dependable automatic metric for "sounds like me", so the judge is a person who knows the voice.

How this fails

• The rewriter washes out the facts. It keeps the tone and quietly drops a number or a caveat. Check names and figures after the rewrite, not before.
• Style markers turn into tics. Too many epochs on too few examples and every output reaches for the same 3 constructions.
• Two models, two failure points, plus latency, for text a well-prompted frontier model nearly produces already.
• The baseline is not the generic model. It's Claude with SOUL.md and the guidelines loaded. That is what the fine-tune has to beat, and it's a higher bar than it sounds.

Pick the right tool first

Four techniques get confused because they all "customise the model". They change different things:

The order matters. Fine-tuning a model to know your data is the classic mistake: weights are a bad database, and every update means retraining. Retrieval handles facts. Training handles behaviour.

⚠️ WARNING: only start fine-tuning after a prompt-only baseline fails at something specific you can name and measure. Without that baseline you can't tell whether training helped.

For business people

The case for it is narrow and real. A small model trained on a few hundred of your own examples can beat a frontier model that's been prompted generically, on that one task, while costing a fraction per call and answering in a fraction of the time. Rahul's framing: "A 1.5B model doing this reliably is more valuable than a 70B model doing it inconsistently."

What it's good at: tone and house style, output format, domain vocabulary, following your instructions without being reminded, and classification or routing at volume.

What it can't do: know today's facts, learn your changing product catalogue, or fix a task you can't define. Those need retrieval, or better prompts.

What it costs. The compute is cheap and everyone quotes it: Rahul prices a first training run at about $1.70 (4 hours of a $0.40/hour GPU plus storage), then adds the warning that matters. Budget 5 to 10 times your first estimate, because the spend is in the reruns: rebuilt datasets, different ranks, new base models, evaluation passes.

The real cost is the examples. 200 to 500 reviewed, correct, consistent examples is the entry ticket, and producing them is human work. A rough ladder: v0 is prompt-only, v1 lands at 200 reviewed examples, v2 at 1,000, v3 adds retrieval on top.

Where it goes wrong commercially: you train on a task that keeps changing, you have no evaluation set, so nobody can say whether the new model is better, or you skip the question of whether you had the right to train on that data. Public availability is not training permission.

For technical people

The shape of the job

1. Build a prompt-only baseline and a fixed evaluation set. Nothing else counts as a reference.
2. Pick a small base model, 0.5B to 3B. Qwen2.5-1.5B-Instruct is a sane default. Move up only when evaluation says you must.
3. Settle data rights per source before downloading anything. Record licence, permission and provenance.
4. Build examples as the task you actually want: input state in, finished output out. Not "page in, text out".
5. Split by entity, never by row. If a company or customer appears in both train and validation, the model memorises it and your numbers lie. 80/10/10 works.
6. Train a LoRA or QLoRA adapter.
7. Evaluate against the prompt-only baseline on the held-out set. Once you look at the test set, it's spent.
8. Serve the adapter, then monitor quality, not just uptime.

Data format

Both MLX and the Hugging Face stack read JSONL. Chat format is the one to use:

{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

MLX also accepts tools, completions ({"prompt": ..., "completion": ...}) and plain text rows. The data directory holds train.jsonl, optional valid.jsonl, and test.jsonl for the test pass.

Training locally on Apple Silicon

This is the part that changed the calculus for me. MLX fine-tunes on the Mac's unified memory, no cloud GPU involved. My Mac Studio (M1 Ultra, 128GB) sits well above what a small-model LoRA needs.

pip install mlx-lm

# LoRA fine-tune. Point --model at a 4-bit model and you get QLoRA.
mlx_lm.lora \
  --model mlx-community/Qwen2.5-1.5B-Instruct-4bit \
  --train \
  --data ./data \
  --iters 600 \
  --batch-size 4 \
  --adapter-path ./adapters/v1

# Evaluate on test.jsonl
mlx_lm.lora --model <base> --adapter-path ./adapters/v1 --data ./data --test

# Try it
mlx_lm.generate --model <base> --adapter-path ./adapters/v1 --prompt "..."

# Merge the adapter back into the weights for serving
mlx_lm.fuse --model <base> --adapter-path ./adapters/v1

--fine-tune-type takes lora (default), dora or full. Full means updating every weight, which you rarely want: reserve it for a genuine distribution shift, like a new language or a domain the base model never saw.

What training actually produces

Not one file. Training writes an adapter directory, and it's small:

adapters/
  adapters.safetensors            # the trained weights, tens of MB
  0000100_adapters.safetensors    # checkpoints, written as it goes
  adapter_config.json             # rank, scale, which layers

An adapter is not a model. It's a thin set of extra weights that only means anything beside the exact base model it was trained against, so the 2 always travel together. Fusing merges them:

mlx_lm.fuse --model mlx-community/Qwen2.5-3B-Instruct-4bit

That loads adapters/ by default and writes fused_model/: an ordinary model directory with safetensors shards, config.json and the tokenizer files. Now it's a model like any other.

⚠️ WARNING: MLX's --export-gguf only covers Mistral, Mixtral and Llama-style models, in fp16. A Qwen fine-tune will not export that way. For Qwen, either import the fused directory into Ollama as it is, or convert it with llama.cpp separately.

Running it locally

Two routes, and the first one needs no conversion at all.

MLX server. Same stack the training ran in:

# serve the fused model
mlx_lm.server --model ./fused_model

# or keep the base model and apply the adapter per request
mlx_lm.server --model mlx-community/Qwen2.5-3B-Instruct-4bit

It listens on localhost:8080 and speaks the OpenAI shape at /v1/chat/completions. The request can name an adapter path, relative to the directory the server started in, so several voices can share one loaded base model:

curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Rewrite this in my voice: ..."}],
       "adapters": "adapters", "temperature": 0.7}'

Ollama. Worth it when the model should be available to everything else on the machine, next to the models already there:

cd fused_model
printf 'FROM .\n' > Modelfile
ollama create nic-voice
ollama run nic-voice "Rewrite this in my voice: ..."

FROM takes either a safetensors directory or a .gguf file. Ollama does not quantise a GGUF during import, so shrink it beforehand if size matters.

Wiring it into the NicAI workflow

The local model is an HTTP endpoint, so it fits the existing pattern: one small script in ~/ai/_shared/, called like every other tool.

#!/usr/bin/env python3
"""Rewrite a draft in Nic's voice using the local model."""

import argparse
import json
import sys
import urllib.request

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "nic-voice"
SYSTEM = "Rewrite the user's text in Nic's voice. Keep every fact, name and number. Change wording only."


def rewrite(draft: str, model: str = MODEL) -> str:
    """Send a draft to the local voice model and return the rewrite."""
    payload = json.dumps({
        "model": model,
        "stream": False,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": draft},
        ],
    }).encode("utf-8")

    request = urllib.request.Request(
        OLLAMA_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        return json.load(response)["message"]["content"]


def main() -> None:
    parser = argparse.ArgumentParser(description="Rewrite a draft in Nic's voice.")
    parser.add_argument("--file", help="Draft file, else read stdin")
    parser.add_argument("--model", default=MODEL)
    args = parser.parse_args()

    if args.file:
        with open(args.file, encoding="utf-8") as handle:
            draft = handle.read()
    else:
        draft = sys.stdin.read()

    print(rewrite(draft, args.model))


if __name__ == "__main__":
    main()

The chain then runs in one direction, with each part doing only what it's good at:

1. Claude researches, reasons and drafts with the facts in place.
2. voice.py sends that draft to the local model and returns the rewrite. Milliseconds, no API cost, nothing leaves the machine.
3. Claude checks the rewrite against the draft: numbers, names and caveats must survive intact. This step is not optional, because washing out a fact is the rewriter's known failure.
4. Me. The last read before anything is sent.

Ollama is already running here on port 11434, so route 2 costs one ollama create once the model exists. The MLX route wins while iterating, because swapping "adapters" per request tests a new voice without fusing or restarting anything.

The cloud path

For bigger bases or faster iteration, the same job runs on rented GPUs. Unsloth and Axolotl wrap the training loop, and TRL's SFTTrainer is the reference implementation underneath. Rent by the second on RunPod or Modal. Colab's free T4 is fine for a first experiment as long as you checkpoint every 50 steps, because the session can drop at any moment.

Serving: vLLM serves LoRA adapters at throughput, and a fused model runs in Ollama locally. Never expose a model server straight to the internet: auth, rate limiting and queueing belong in front of it.

Hosted fine-tuning, if you'd rather not run any of it

OpenAI supports supervised fine-tuning, vision fine-tuning, DPO (you supply a good and a bad answer) and reinforcement fine-tuning (you grade the output and the training reinforces the reasoning that got there). Together and Google both run LoRA fine-tuning on open and hosted models.

Claude is the exception worth knowing: the Messages API has no fine-tuning endpoint. Customisation there means prompting, prompt caching, tools and structured outputs. If your plan depends on training Claude itself, the plan needs changing.

The mistakes that make the result worthless

• Splitting by row instead of by entity. The model memorises names and scores beautifully on data it has effectively seen.
• Treating falling training loss as progress. Lower loss with more epochs often means memorisation. Judge on held-out data only.
• Auto-correcting the source. If OCR turns $12.5M into $12.SM, send it to a human. A model trained on repaired numbers produces fluent, confident, wrong figures.
• Peeking at the test set. Once you tune against it, it's a training signal, not an evaluation.
• Collecting user data silently for the next run. Explicit opt-in, and strip names, emails, phone numbers and commercial terms before anything is stored.
• No rollback path. Version the dataset, the adapter and the base model together, or you can't undo a bad release.

The plan for the voice model

Ordered so the boring part comes first, because that is the part that decides whether any of it works.

1. Build the pair set. Pull sent mail, match each message to the draft it came from, and target 300 to 500 pairs. Draft in, what I sent out. This is the job, and it's most of the work.
2. Filter the note corpus. Signature present means a model wrote it. Keep only the 1,185 that are mine, and drop the terse list-style notes that carry no voice.
3. Freeze a test set before training anything. 30 pairs, untouched, held for the blind comparison. Once I tune against them they stop being an evaluation.
4. Set the real baseline. Claude with SOUL.md and the writing guidelines, generating the same 30. That is the bar.
5. Train locally. A 1.5B to 3B instruct base, LoRA rank 16, 2 epochs, mlx_lm.lora on the Mac Studio. Qwen 2.5 3B is the documented precedent for this exact job.
6. Judge blind. Base, baseline and tuned outputs side by side with the labels off. If I can't pick mine out, the fine-tune hasn't earned its place in the chain.
7. Wire it as the last stage only. Frontier model drafts with the facts, the small model does the voice pass, I review before anything leaves.

The honest position today: steps 1 to 3 don't exist yet, so nothing else matters. The data work is the project, and the training is an afternoon at the end of it.

Further Reading

MLX LM: fine-tuning with LoRA or QLoRA

The authoritative CLI reference for training on Apple Silicon

https://github.com/ml-explore/mlx-lm

Unsloth docs

Faster, lower-memory fine-tuning on rented or local GPUs

https://docs.unsloth.ai/

TRL SFTTrainer

The reference supervised fine-tuning implementation

https://huggingface.co/docs/trl/en/sft_trainer

OpenAI model optimization

Hosted SFT, DPO and reinforcement fine-tuning

https://developers.openai.com/api/docs/guides/model-optimization