Physical modelling with neural networks - single output + stack ensemble vs multi-output

Question

We are trying to replace an existing physical model (8 inputs/7 outputs) with artificial neural networks. The physics behind the existing model is mainly thermodynamics of humid air for air conditioning, with some turbomachinery involved, which yields most likely complex functions between inputs and outputs.

One approach was already done:
single output neural networks (10 NN with same # hidden layers but different parameters like batch size, # epochs, optimizer, etc). Then some sort of stacking ensembled was used: every predictions were  used as new inputs for a single NN to predict a final value.

The accuracy is pretty well, however there are some test datapoints where the absolute errors are high enough to be worried about the predicted value (this could be used for example for air conditioning control strategies, so a bad prediction would result in uncontrolled system).

In order to improve accuracy some colleagues were suggesting to keep it simpler and perform just a multi-output regression with a single neural network.

From the mathematical point of view, I have following questions:

1) would a single output NNs stacking ensembled outperformed the multi-output single NN?
2) is the way of stack ensembling using NN a good approach? I saw some different techniques like arithmetic averaging the inputs

Thanks for your time!
Regards

D.W. · Answer

My guess: I agree with your colleagues.  I see no reason to do anything other than a single neural network with multiple outputs.  If necessary, increase the capacity of that single neural network until you see no further improvement.

An stacking ensemble where you have a few neural nets whose inputs are fed as input into another neural network is itself equivalent to one bigger neural net.  I don't see any reason to expect the stacking approach to train more effectively or to be better for some other reason.

However, this is ultimately an empirical field.  The only way to find out what will work best is to try different approaches and see.  There's not a huge amount of theory that enables us to make predictions about what approaches will be most effective.

You might consider whether you can obtain more training data, particularly in the regimes of particular interest.  Often better training data sets offer higher gains than modifying the neural network architecture.

Physical modelling with neural networks - single output + stack ensemble vs multi-output

One Answer

Add your own answers!

Ask a Question