Error in Built Up area Classification using Landsat 8 in google earth engine ( SVM, RF and Cart)

I am using Earth Engine to Classify an area using supervised classification with a combination of reflectance bands and indices. The built-up area gets mixed with fallow land (bare soil) and I have tried many different combinations and somehow I am unable to resolve it.

I have used built-up index, tillage index, ndvi, mndwi and evi if different combinations. I have referred to many journal papers and still unable to resolve it.
I have tried increasing and decreasing the training points too.

Could anyone please help me with this.

Here is the code for the classification. https://code.earthengine.google.com/20bc92c707702e1fdb32bd5325ff19af

This image shows the results where the cart and random forest classification mix with other classes and whereas SVM results fail to classify the built-up areas properly with same training points.

It would be great if the code is improved to get confusion matrix (kappa and overall accuracy).