APSIM (Agricultural Production system SIMulator) Next Generation has in part been developed to accelerate the collation of test sets for crop model improvements. However, as far as we know, there is a lack of systemic testing of the quality of simulation configurations in such test sets. In this paper we suggest using all observations (i.e. measured datasets) available to scrutinize the test set and guide subsequent model calibration. We describe a simple but robust approach for scrutinizing simulation configuration. An exemplar crop model – potato (Solanum tuberosum L.) is used to identify the main sources of uncertainty during simulation configuration to accelerate crop model improvements. 426 experiments (44 cultivars from 55 locations across 19 countries) were run using APSIM. Model inputs and outputs were plotted using an automatic script. Based on these plots, we conducted a detailed interrogation of simulation configuration and sources of uncertainty, looking for systematic effects related to location, cultivar and crop management parameters. Sources of uncertainty were mainly associated with model input configuration (crop management>soil>climate) and inconsistencies in measured data. This study highlights the importance of high levels of rigor for configuring test sets and thorough consideration of the model performance so that subsequent model improvement can be effectively targeted.