How to grid search feature selection and neural network hyperparameters in the same grid?

Question

I'm using the GridSearchCV () class from scikit to perform hyperparameter optimization in a sequential neural network. I've built a pipeline to also find the best number of features by putting a feature selector inside the pipeline. The problem is how to define the input_shape, since this depends on the k parameter from the feature selector. Is it possible to set the value of classifier__input_shape to be the same value (at all times) of feature_selector__feature__selector_k?
I've provided the correspondent piece of code below.
def create_model (learn_rate = 0.01, dropout_rate = 0.0, weight_constraint = 0, input_shape):
  model = Sequential ()
  model.add (Dense (units = 64, activation = 'relu',
                   input_shape = (input_shape, )))
  model.add (Dropout (dropout_rate))
  model.add (Dense (32, activation = 'relu'))
  model.add (Dense (1, activation = 'sigmoid'))
  model.compile (loss = 'binary_crossentropy',
                optimizer = Adam (lr = learn_rate),
                metrics = ['accuracy'])#, metrics.CategoricalAccuracy ()])
  return model

standard_scaler_features = remaining_features
my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

clf = KerasClassifier (build_fn = create_model, verbose = 2)
clf = Pipeline (steps = [('scaler', my_scaler),
                         ('feature_selector', feature_selector_transformer),
                         ('classifier', clf)],
                verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
              'feature_selector__feature_selector__k' : [7, 9, 15],
              'classifier__input_shape' : [7, 9, 15],
              'classifier__epochs' : [2, 3, 4]}
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = STATE)
grid = GridSearchCV (estimator = clf, param_grid = param_grid, scoring = 'f1',
                     verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit (X_train_df, y_train_df)

And the error:
ValueError: Input 0 of layer sequential_9 is incompatible with the layer: expected axis -1 of input shape to have value 9 but received input with shape [None, 7]

etiennedm · Answer

I see two solutions:

either you pass a list of dictionnaries to param_grid avoiding irrelevant combinations
or you use a single variable in your pipeline for feature_selector__feature__selector_k and classifier__input_shape

First solution:  you can generate the right list of combinations using something close to this:
param_grid = [
    {
        'feature_selector__feature_selector__score_func' : [f_classif],
        'feature_selector__feature_selector__k' : [k],
        'classifier__input_shape' : [k],
        'classifier__dropout_rate' : [0.0, 0.5]
    } 
    for k in [7, 9, 15] 
             ]

Second solution, you can use a specific class that create your model when fitting based on the shape of X. Here is a code sample:
class MyKerasClf():
    def predict(self, X):
        y_pred_nn = self.clf.predict(X)
        return np.array(y_pred_nn).flatten()
    
    def create_model(self, learn_rate = 0.01, weight_constraint = 0 ):
        model = Sequential ()
        model.add (Dense (units = 64, activation = 'relu',
                       input_shape = (self.input_shape, )))
        model.add (Dropout (self.dropout_rate))
        model.add (Dense (32, activation = 'relu'))
        model.add (Dense (1, activation = 'sigmoid'))
        model.compile (loss = 'binary_crossentropy',
                    optimizer = Adam (lr = learn_rate),
                    metrics = ['accuracy'])
        return model
        
    def fit(self, X, y, **kwargs):
        self.input_shape = X.shape[1]
        self.clf = KerasClassifier(build_fn = self.create_model, verbose = 2)
        self.clf.fit(X, y, **kwargs)
    
    def set_params(self, **params):
        if 'dropout_rate' in params:
            self.dropout_rate = params['dropout_rate']
        else:
            self.dropout_rate = 0.0

Then you can use the class in your pipeline
X, y = make_classification(n_features=50, n_redundant=0, n_informative=2,
                           random_state=42, n_clusters_per_class=1)

my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

# Create a specific clf
my_clf = MyKerasClf( )
pip_clf = Pipeline (steps = [('scaler', my_scaler),
                         ('feature_selector', feature_selector_transformer),
                         ('classifier', my_clf)],
                verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
              'feature_selector__feature_selector__k' : [7, 15],
              'classifier__dropout_rate' : [0.0, 0.5]
             }
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = 42)
grid = GridSearchCV (estimator = pip_clf, param_grid = param_grid, scoring = 'f1',
                     verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit(X, y)

Note: I although added the dropout to be tested in the gridsearch as an example.

How to grid search feature selection and neural network hyperparameters in the same grid?

One Answer

Add your own answers!

Ask a Question