Logic Check: Building a SKLearn Pipeline

I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set.

Background:

I have a sparse, high-dimensional data-set (370×1000) with a continuous variable as the target. At present, I have been running a random forest regression on all the features with a 90/10 split, followed by parameter tuning via grid search on the training set, followed by 5-fold cross validation on the optimized model (with the entire dataset).

Problems with this approach:

As I understand the situation, there are a number of things I am doing that might be harming the model and introducing undesired bias. Specifically, my concerns are:

1. As I am tuning parameters only on the initial train/test split, I am not accounting for other split combinations that arise during the K-Fold CV. Might optimal settings for fold 1 be different for fold 2? Intuitively I would assume so.

2. I am not doing any feature selection that might remove otherwise redundant features and shorten my feature-space (I know RF is generally quite good with high-dimensional spaces but I would still like to try). Some suggestions I have read have included removing features with very low variance. But I find myself in the same conundrum as above: if I remove low variance features from the original training set I am not accounting for other combinations during K-fold. Comparatively, if I remove all low-var features prior to splitting the data I am surely leaking info between the train/test states.

3. An alternative approach I have seen is recursive feature selection (with CV as per SKLearn) – this looks promising, as I think it means that it will partition the data-set in folds and conduct RFE on each fold, presumably giving me an averaged score of the best number of features to keep.

My possible solution:

I have been doing some reading around Pipelines in SKLearn and think that might be the way to go. My understanding is that an advantage of a pipeline is that i can stack transforms together, preserving the individual folds and allowing me to address the problems I have detailed above. What I am considering, and what I would appreciate sense-checking from anyone with more experience, is the following:

1. As the data-set is small, I would not split the data-set in the conventional train/test split manner, but would use K-Fold across the whole data-set.

2. Run a RF (using default params) with K-Fold to get a baseline level of performance.

3. Create a pipeline whereby I (3.1) create folds, (3.2) and then within each fold find the optimal number of features to keep, (3.3) tune hyper-parameters for that fold, and finally (3.4) predict the values in the test fold.

As you might be able to tell I am struggling to get to grips with what order things should go in, and whether step 3 is actually what a Pipeline does. If someone can provide pointers/recommendations/corrections it would be appreciated.

from sklearn.ensemble             import RandomForestRegressor 
from sklearn.feature_selection    import VarianceThreshold
from sklearn.model_selection      import GridSearchCV
from sklearn.pipeline             import Pipeline

regressor = Pipeline([('vt', VarianceThreshold(threshold=0.0)),
                      ('rf', RandomForestRegressor())])

gsvc = GridSearchCV(regressor, {'vt__threshold':    [0, .1, .5, .7, 1],
                                'rf__n_estimators': [10, 100, 1_000],
                                'rf__max_depth':    [2, 10, 100]})