TransWikia.com

Solving the dual problem of boosting using column generation

Data Science Asked by Code Pope on November 8, 2020

In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem.
So lets say we have want to solve the following primal linear problem based on hinge loss:
$$min_{w,xi} Sigma_{m=1}^Mw_m + CSigma_{i=1}^Nxi_i$$
$$ y_iSigma_{m=1}^MH_{im}w_m+xi_igeq 1 text{ ,} iin[N]$$
$$w,xi geq 0,$$

where $H_{im}:=phi(x_i),iin[N],min[M]$ and $C>0$ is a weighting factor.
Now the dual is:
$$max_z mathbb{1}^Tz$$
$$Sigma_{i=1}^N y_i H_{im} z_i leq 1 text{ }, min[M]$$
$$0leq zleq Cmathbb{1}$$

Now in our book, it stated:

The base learniners $phi(x)$ can be obtained dynamically, by solving the dual problem with optimal solution $z^*inmathbb{R}_+^N$. For a given subset of base functions, we need to check whether there exists another function $phi_m(x)$ such that the corresponding dual constraint is violated, i.e.,
$$Sigma_i=1^N y_iH_{im}z_i^* >1$$
Thus the left hand side has to be maximized, which can be handled by calling a base learner with weights $z^*$ trying to maximize the weighted number of points that are correctly classified. If the value is larger than 1, $phi_m(x)$ is added to the set of base learners ant the process is repeated. Note that this process will terminate, since there are only finitely many vectors $H_{. m}in{-1,+1}^N$.

For me it is totally unclear why we are checking for functions $phi_m(x)$ which violate the constraint. I mean we want to solve the dual problem and if so the constraints forces us to use functions which classify the points incorrectly. Then we had $yH_m<0$ and the constraint would be active and the dual problem could be solved. I don’t know how the problem is going to be solved if we only use functions which violate the constraint

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP