How to properly set up neural network training for stable accuracy and loss

I have a DenseNet121 implemented in Pytorch for image classification. For now, the training set-up is pretty straightforward:

• the data is loaded. An important characteristic here is that the validation data is fixed from the outset and can never change. The rest of the data I split into training and testing. None of the data sets overlap!
• for every epoch iterate through the training data loader, calculating loss, optimizing etc.
• every 100 batches evaluate the loss using the validation data loader
• at the end of an epoch compare the current state’s loss on validation data with the state that previously had the best validation loss (for the first epoch just compare this with a random high number like 1e5) and save the current state if it better or keep the older state.
• after all epochs are finished save the state with the lowest validation error as the best model
• the best model is then applied to test data. Accuracies are calculated and ROC curves drawn

I was wondering how to extend my set-up to make it a proper statistical experiment set-up, with the goal of getting stable accuracy and loss results ie I want to be able to say that my model gives consistently the same results. I was thinking along the lines of running the testing step say 10 times and averaging the error? Or do you see some deficiency in the training that I could improve to improve stability?

Thanks in advance