Our QA fine-tune, aligned with Direct Preference Optimization on 500 AI-judged preference triplets. It learns to prefer the better answer directly from preferences — no reward model.
The QA-SFT checkpoint, further trained with DPO against a frozen reference copy of itself. Every preference triplet (prompt, chosen, rejected) was written by Gemini and verified by a blind LLM judge that the chosen answer is genuinely better.
750 steps, ~4 min on an L4. Judged correctness 0.279 [0.245–0.313] on the blind-judge harness — well above the 125M models but far short of Gemma, as expected at 500M scale. At this scale RLAIF beat DPO (0.443 vs 0.279).
| Class | LlamaForCausalLM |
| Layers | 24 |
| Hidden size | 1,280 |
| Attention | 20 heads · dim 64 · full MHA |
| Feed-forward | SwiGLU · inner 3,456 |
| Positional | RoPE · θ 10,000 |
| Norm | RMSNorm · ε 1e-5 |
| Context | 1,024 tokens |
| Vocabulary | 32,768 |
| Embeddings | tied input/output |
| Init from | slm-500m QA-SFT |
| Method | DPO (full fine-tune) |
| Trainable params | 517.8M — all |
| Training data | 500 preference pairs |
| Training tokens | 325K (chosen + rejected) |
| Optimizer steps | 750 |
our cost begins at fine-tuning — the base is imported.
| Stage | Detail | Cost |
|---|---|---|
| QA supervised fine-tune | the checkpoint DPO starts from | $0.93 |
| DPO alignment | direct preference optimization on L4 | $0.08 |
| Evaluation (shared) | 13 versions on 500 held-out questions, this model's share | $1.48 |
| Total | $2.49 |
Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.