How does the construction of a decision tree differ for different optimization metrics?

Question

I understand how a decision tree is constructed (in the ID3 algorithm) using criterion like entropy, gini index, variance reduction. But the formulae for these criteria do not care about optimization metrics like accuracy, recall, AUC, kappa, f1-score and others.

Packages on R and Python allow me to optimize for such metrics when I construct a decision tree. What do they do differently for each of these metrics? Where does the change happen?

Is there a pattern to how these changes are done for different classification/regression algorithms?

Noah Weber · Answer

Splitting is the same your are optimising for the parameters.

In other words, gini, entropy etc logic will stay the same but the breadth, width, number of features etc will be different when you optimise it with different loss functions.

Accuracy, precision, F-scores etc. are evaluation metrics computed from binary outcomes and binary predictions. They are NOT (but there are modification) for them to be a loss function.
For model training, you need a function that compares a continuous score (your model output) with a binary outcome - like cross-entropy. Ideally, this is calibrated such that it is minimised if the predicted mean matches the population mean (given covariates)

How does the construction of a decision tree differ for different optimization metrics?

One Answer

Add your own answers!

Ask a Question