Random seed in Machine learning model comparison

I agree that sometimes the authors are not that clear about some details. If a dataset is already splitted into train/test/validation, then there is not much to do (supposing the dataset is well made). Given a dataset not already splitted then you have (at least) two ways to test your model:

• fix a random seed, check that the split is not biased and run several trainings. Even with the same split, the same model can converge to different local minima, because of the stochastic descent;
• don't fix a random seed and run several trainings (but store the seed where you keep your experiment data!).

If your findings are not coherent with the literature, and you are sure there aren't bugs in the code, then you should ask specific questions or write to the authors. Depending on the model, you might try some standard validation techniques.

The best way to avoid your issue is using a K-Fold Cross Validation. Having a 0.1 difference in accuracy between 2 seeds is a lot, so I'd suggest you doing cross validation, with a random splits (you can, for example, shuffle your data before entering cross-validation loop). Running 2 times your Cross-Validation Loop, shuffling data before, could give you a bit of a change (because of the shuffle, cross validation sets won't be exactly the same), but I highly doubt it'll be as high as 0.1 Then you could compare your model, since you're using a stable evaluation process.