When to preprocess data for neural network

Experience with many different datasets containing discrete or continuous input variables have shown me that normalization makes training much easier. The randomly selected set of initial weights is most often selected from a uniform (+/-) distribution. Hence, normalized variables favor a smooth and unbiased start of the learning process.

A practical approach is to normalize based on the data distribution in the training set. The training and test sets should be obtained from the same distribution. I use a random generator the choose which case goes to the training set, or whether it is assigned to the test set - all before any training takes place. You can then apply the normalization constants to the test set. As these two sets are interchangeable, there is no issue with using the normalization constants calculated from the training set, for normalizing the test set as well.

My simple scheme for continuous input variables, was to divide each data point by its maximal value in the training set

$ x_{norm}(i) = x_{data}(i) / max(x_{data}(i)) $

Discrete input variables need not be normalized, when they are either $0$ or $1$.

There are published papers in the literature that compare normalization schemes and the properties of the resulting neural networks.

Preprocessing - in general - is an open field.

Within image processing, some colleagues have initialized the NN-weights of the first hidden layer with the coefficients of different linear filters. See for an overview for example Egmont-Petersen et al. 2002, Image processing with neural networks - a review, Pattern Recognition.