Random Forests Feature Selection on Time Series Data

Some EDA might be needed to create new features for each time-series item. You might want to mine for patterns and have random forest reduce the overfitting. Exactly how mining is done depends on the nature of the problem, which might indicate for things like:

• interesting time periods,
• events that happen at a time,
• time lag between different series,
• dynamical systems,
• latent variables,
• scedasticity

Breiman's landmark paper on random forest gives some theoretical guarantees that random forest works well when individual classifiers are good and the correlation between these individuals are low. This can also be a heuristic to prune features.

If you want to preserve and utilize the 2D structure, use something like a Convolutional Neural Network. Feature selection can be done using L1 regularization. Otherwise you will have to do feature engineering outside the classifier.

This 2D structure with one axis being time is quite similar to spectrograms used in audio, where CNNs are frequently applied. So check out literature on Acoustic Event Recognition and Acoustic Scene Classification for more details.

From your question I'm not sure I fully understand the nature of your data, but since you mention ScikitLearn and Random Forests and feature selection, you may be interested in the Python implementations of the Boruta all-relevant feature selection method. Boruta is based on Random Forests, automatically selects important features and is based on SciKit methods. If you can shoehorn your data into this framework, it might answer your needs.

Check out the example here. Just modify to suit your specific data-set.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# matplotlib inline

df = pd.read_csv("https://rodeo-tutorials.s3.amazonaws.com/data/credit-data-trainingset.csv")
df.head()

from sklearn.ensemble import RandomForestClassifier

features = np.array(['revolving_utilization_of_unsecured_lines',
                     'age', 'number_of_time30-59_days_past_due_not_worse',
                     'debt_ratio', 'monthly_income','number_of_open_credit_lines_and_loans', 
                     'number_of_times90_days_late', 'number_real_estate_loans_or_lines',
                     'number_of_time60-89_days_past_due_not_worse', 'number_of_dependents'])
clf = RandomForestClassifier()
clf.fit(df[features], df['serious_dlqin2yrs'])

# from the calculated importances, order them from most to least important
# and make a barplot so we can visualize what is/isn't important
importances = clf.feature_importances_
sorted_idx = np.argsort(importances)


padding = np.arange(len(features)) + 0.5
plt.barh(padding, importances[sorted_idx], align='center')
plt.yticks(padding, features[sorted_idx])
plt.xlabel("Relative Importance")
plt.title("Variable Importance")
plt.show()

All details are available here.