Knowledge distillation trains a student language model to match a teacher's predictions on the training text. We ask whether the student also benefits from the teacher's predictions on text that has been deliberately perturbed. For an early part of pretraining we perturb the inputs and train the student to match the teacher on the perturbed text, and we spend the rest of the same token budget on ordinary distillation. Teacher and student usually have embeddings of different sizes, so we perturb the one-hot token vectors, which each model then maps through its own embedding table. We test three perturbations: additive token noise, blending two documents, and blending neighbouring tokens.
When pretraining from scratch, the method improves on standard distillation at both scales we tried, by 0.025 nats at 135M parameters and by 0.020 nats at 0.5B (three seeds). At 135M this is 1.45 times the gain that distillation itself provides over next-token training. The gain depends on the teacher seeing the perturbed text: without a teacher it largely disappears, and perturbing only the student's input, as pretraining augmentation usually does, recovers about a third of it, while softer targets and a weaker early teacher do not reproduce it. When 262M unique tokens are repeated for 16 epochs, the gain persists against a baseline with tuned weight decay. The perturbed students also quantize at least as well with GPTQ and AWQ, and at 0.5B they gain about one point of average zero-shot accuracy over seven tasks.
