Quantify whether a set of binary segmentation models (experts) have diversity on a fixed dataset?

Question

I have a set of models for binary segmentation task (M=10), and a set of images (N=1000). I also can collect the prediction map for all these N=1000 images, resulting in 10,000  grayscale maps.  Now, I want to quantitatively check whether these models have strong diversity in their predictions or not.
Diversity holds when the rank of the models in terms of their performance (e.g. Jaccard Index or Dice similarity) differs for each single query image.
What do you suggest? We also can reduce the set of models to N=3 to simplify the problem.

MachineEpsilon · Answer

If you are interested in measure how heterogeneous the ranks are, then you can look at rank correlations statistics like $rho$ or $tau$. Otherwise, you could think of the problem as being an inter-rater reliability problem and ask the "dual" question of how much homogeneity there is between your models. This means you could look at statistics like intraclass correlation or models like ANOVA to assess the diversity.

Answered by MachineEpsilon on December 30, 2020

Quantify whether a set of binary segmentation models (experts) have diversity on a fixed dataset?

One Answer

Add your own answers!

Ask a Question