Merging sparse and dense data in machine learning to improve the performance

Try PCA only on sparse features, and combine PCA output with dense features.

So you'll get dense set of (original) features + dense set of features (which were originally sparse).

+1 for the question. Please update us with the results.

In addition to some of the suggestions above, I would recommend using a two-step modeling approach.

1. Use the sparse features first and develop the best model.
2. Calculate the predicted probability from that model.
3. Feed that probability estimate into the second model (as an input feature), which would incorporate the dense features. In other words, use all dense features and the probability estimate for building the second model.
4. The final classification will then be based on the second model.

This seems like a job for Principal Component Analysis. In Scikit is PCA implemented well and it helped me many times.

PCA, in a certain way, combines your features. By limiting the number of components, you fetch your model with noise-less data (in the best case). Because your model is as good as your data are.

Consider below a simple example.

from sklearn.pipeline import Pipeline
pipe_rf = Pipeline([('pca', PCA(n_components=80)),
                    ('clf',RandomForestClassifier(n_estimators=100))])
pipe_rf.fit(X_train_s,y_train_s)

pred = pipe_rf.predict(X_test)

Why I picked 80? When I plot cumulative variance, I got this below, which tells me that with ~80 components, I reach almost all the variance.

So I would say give it a try, use it in your models. It should help.

The best way to combine features is through ensemble methods. Basically there are three different methods: bagging, boosting and stacking. You can either use Adabbost augmented with feature selection (in this consider both sparse and dense features) or stacking based (random feature - random subspace) I prefer the second option you can train a set of base learners ( decisions. Trees) by using random subsets and random feature ( keep training base learners until you cover the whole set of features) The next step is to test the Training set to generate the meta data. Use this meta data to train a meta classifier. The meta classifier will figure out which feature is more important and what kind of relationship should be utilized

The variable groups may be multicollinear or the conversion between sparse and dense might go wrong. Have you thought about using a voting classifier/ ensemble classification? http://scikit-learn.org/stable/modules/ensemble.html That way you could deal with both above problems.