Paper accepted for MLSP, together with Jakob Lindqvist, Amanda Olmin and Lennart Svensson! We develop a general framework for ensemble distillation that, instead of distilling the ensemble into a single predictive model, retain the distribution over the ensemble members. This provides a more complete description of the predictive uncertainty, making the model more robust and enables epistemic uncertainty quantification. Preprint is available here.