On-Policy Distillation Works Better Without the Teacher
信息来源:DEV Community·
内容摘要
A new Purdue paper analyzed teacher supervision in on-policy distillation. The teacher scores are mostly noise, and a fixed negative penalty beats a full teacher model on AIME24.