Causes of inconsistent results with neural network?

Data: It depends on how distinguishable your classes are. Like cats vs dogs is quite easy, while disgust expression vs angry expression is quite hard.

Epochs: Over-fit your model and do early stopping. To handle over-fitting, increase regularization or decrease the complexity of model by decreasing the number of hidden neurons / layers. See what's best for you given the time and performance.

Learning rate: Start with the largest learning rate that allows the model to learn. From there, experiment by dividing that learning rate by 10 or 2, observe the results, see which learning rate allows your model to converge faster and leads to higher accuracy.

There are still a lot of things to consider to achieve the maximum performance such as learning rate decay, gradient descent optimization algorithms, weight initializations, etc.

More observations = more productivity. Be mindful of the time and speed in choosing the best hyperparameters.

To know more, I suggest reading this by Michael Nielsen - Neural Networks and Deep Learning

Before briefing about the other things, I would like to suggest you something good. It is always better to set a seed number in machine learning and deep learning problems to get a reproducible result and by that I mean that every time when you try to run your model anywhere, it can produce the same best result which you got on your machine while experimenting. So, you should always set numpy.random.seed(*anynumber*).

Now how to compare your result to your expected result? There are a number of parameters that you can look upon during training your model. Some of these are:

• Validation accuracy: It is the one of the most important parameter to look upon for evaluating your model's performance. This gives you a fair idea if your model is overfitting or not. More the validation accuracy, the better is the model.
• Validation loss: If your model is really good, then validation loss must decrease, at least initially until your best epoch hasn't been obtained. Lower the validation loss, better the model.
• Optimizers: There are a number of optimizers like 'rmsprop', 'sgd', etc. which can be used for different types of problems but not all give the same performance. You should grid-search for a number of optimizers for your model if you are not specific about which is best for your model.
• Number of training epochs: Depending upon the size of dataset, no. of training epochs differ but if you are not giving enough training to your model, then your model performance will suffer.
• Training data: If you think your model is fine, but you are not achieving expected results, then you can try to train your model on a large dataset. NN require large dataset for achieving awesome performance.