Did We Practice what We Preached?

Abstract

Reproducibility of statistical simulations is crucial but proved being a challenge in its own right. Recently, lack of reproducibility of important simulation studies stimulated developments of reporting guidelines and specific protocols trying to improve on this situation. The problem, of course, is not new and issues regarding reproducibility of numerical results, for example statistical analyses or simulations, have long been known. Documented lack of progress regarding reproducibility in the past decade naturally leads to the question if problem awareness was not powerful enough to lead to improved reproducibility. As a benchmark case, I tried to reproduce a simulation study in a so far unpublished manuscript by Fritz Leisch and myself. The results show that, time and again, the devil is in the details and much self-discipline and extensive record-keeping and documentation are mission critical.