Chat Dataset Forge — ChatML / Alpaca

A small CLI to convert, validate and inspect instruction-tuning datasets between the two dominant formats: Alpaca (instruction/input/output) and ChatML (role-tagged multi-turn messages).

Bad formatting is the #1 silent killer of fine-tunes: a missing <|im_end|>, a system prompt in the wrong place, or training on the prompt tokens can each quietly wreck a run. This repo makes the format layer explicit and testable.

The two formats

Alpaca — flat, single-turn, born with Stanford Alpaca:

{
  "instruction": "Translate to French.",
  "input": "Good morning",
  "output": "Bonjour"
}

ChatML — role-tagged turns, introduced by OpenAI, now the template of Qwen, and (with variations) most chat models:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Translate to French: Good morning<|im_end|>
<|im_start|>assistant
Bonjour<|im_end|>

In datasets form, ChatML is stored as a messages list — the template string is rendered by the tokenizer's chat_template, never hand-built.

	Alpaca	ChatML
Multi-turn	no	yes
System prompt	fixed preamble	first message
Loss masking	mask up to ### Response:	mask everything except assistant turns
Used by	Alpaca, early LLaMA fine-tunes	Qwen, and most chat models via chat templates

Quickstart

pip install -r requirements.txt

# Alpaca -> ChatML messages (JSONL in/out)
python forge.py convert --from alpaca --to chatml data/alpaca.jsonl data/chatml.jsonl

# ChatML -> Alpaca (single-turn conversations only; multi-turn rows are skipped with a warning)
python forge.py convert --from chatml --to alpaca data/chatml.jsonl data/alpaca.jsonl

# Validate a file: schema, empty fields, role alternation, token stats
python forge.py validate --format chatml data/chatml.jsonl

# Preview how a tokenizer's chat template renders your rows
python forge.py preview data/chatml.jsonl --tokenizer Qwen/Qwen2.5-3B-Instruct

What validate checks

• required keys present and non-empty per row
• ChatML: roles alternate user/assistant (after an optional system turn), last turn is assistant
• Alpaca: output non-empty, no ChatML tags leaked inside fields
• length stats: p50/p95/max tokens per row so you can pick max_seq_length from data, not folklore

Design rules baked in

1. Never hand-concatenate ChatML tags at training time. Store messages, let tokenizer.apply_chat_template render — templates differ subtly between models.
2. Convert losslessly or loudly. ChatML→Alpaca refuses multi-turn rows instead of flattening them into garbage.
3. Validate before every run. It's one command and it catches the class of bug you cannot see in the loss curve.

License

MIT