Input Perturbation as Augmentation for Pretraining Distillation
Distilling on perturbed inputs for part of pretraining improves small language models at 135M and 0.5B, and the gain comes from the teacher's predictions on the perturbed text.
Writing
Research notes on how models generalize, how learning systems improve themselves, and which parts of discovery we might automate.
Index01 published notes