Sampling uncertainty of posterior probability distribution

Question

I'm working on a problem with 3 possible outcomes and a bunch of features. I have a regression model that outputs probabilities for each category and I'd like to extend these probabilities to probability distributions. So instead of getting an output of [0.2,0.3,0.5], I'd get a probability density function or at least a quantification of the uncertainty of the prediction. I've looked into some models that give these distributions but haven't found any way to use it on a multiple output problem. Is there any way to do this?

Match Maker EE · Answer

Interesting problem - which is most often overlooked in data science and machine learning. The output probabilities $bf{y}$ are indeed estimates of the underlying (true) posterior probabilities (your $[0.2,0.3,0.5]$). Sampling a different training set (from your presupposed 'oracle'), will yield a slightly different set of output probabilities, when the identical input feature vector $bf{x}$ is presented to the classifier.
The distributions of $hat{P}(bf{y} mid bf{x},bf{theta})$ - they have been studied for linear and quadratic discriminant analysis ($theta$ is the parameter vector of the discriminant classifier).
And yes, also the sufficient parameters of these distributions of $hat{P}(bf{y} mid bf{x},bf{theta})$ have been derived. Specifically the variance of each posterior probability has been derived. A mathematically sound description (with the relevant references to papers in the statistical literature), can be found in Chapter 11 in the book: Discriminant analysis and statistical pattern recognition by G.J. McLachlan, Wiley (2004).

Sampling uncertainty of posterior probability distribution

One Answer

Add your own answers!

Ask a Question