Data Science Asked by Code Pope on November 8, 2020
In our book there is boosting algorithm using column generation method (Dantzig-Wolfe decomposition) to solve the dual problem.
So lets say we have want to solve the following primal linear problem based on hinge loss:
$$min_{w,xi} Sigma_{m=1}^Mw_m + CSigma_{i=1}^Nxi_i$$
$$ y_iSigma_{m=1}^MH_{im}w_m+xi_igeq 1 text{ ,} iin[N]$$
$$w,xi geq 0,$$
where $H_{im}:=phi(x_i),iin[N],min[M]$ and $C>0$ is a weighting factor.
Now the dual is:
$$max_z mathbb{1}^Tz$$
$$Sigma_{i=1}^N y_i H_{im} z_i leq 1 text{ }, min[M]$$
$$0leq zleq Cmathbb{1}$$
Now in our book, it stated:
The base learniners $phi(x)$ can be obtained dynamically, by solving the dual problem with optimal solution $z^*inmathbb{R}_+^N$. For a given subset of base functions, we need to check whether there exists another function $phi_m(x)$ such that the corresponding dual constraint is violated, i.e.,
$$Sigma_i=1^N y_iH_{im}z_i^* >1$$
Thus the left hand side has to be maximized, which can be handled by calling a base learner with weights $z^*$ trying to maximize the weighted number of points that are correctly classified. If the value is larger than 1, $phi_m(x)$ is added to the set of base learners ant the process is repeated. Note that this process will terminate, since there are only finitely many vectors $H_{. m}in{-1,+1}^N$.
For me it is totally unclear why we are checking for functions $phi_m(x)$ which violate the constraint. I mean we want to solve the dual problem and if so the constraints forces us to use functions which classify the points incorrectly. Then we had $yH_m<0$ and the constraint would be active and the dual problem could be solved. I don’t know how the problem is going to be solved if we only use functions which violate the constraint
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP