Data Science Asked by Izo on September 5, 2021
I have a data set that contains the same Time series "Sensor readings" for different days and I want to make a deep learning model to predict these values. What I did was I splatted the data into Time series data according to the day, then I normalized it separately (min-max) (the readings have different ranges, for example, the max value for the first day is 100 but the max for second is 48) but I’m really confused now do I need to normalize it using the max/min of the all days or what I did was right?
If you know strict bounds on the sensor output, that would be better than normalizing by the min/max of the dataset. Even if the bounds are not necessarily strict, but simply reasonable, that would suffice. For example, if there are no theoretical bounds on a temperature sensor, you might reasonably impose strict bounds given prior knowledge about its environment (e.g. if a temp sensor was placed in NY, you might assume strict bounds as -30C to 50C)
If you were to normalize by the min/max of the training data, what do you expect to happen if the deployed model encounters a value outside this range? If, for example, your training set had min 5 and max 30, how would you normalize an input of -10? It would be much more intuitive and reliable to shift up to a nonnegative domain, (i.e. subtract your strict minima), then scale to a value in [0,1] via the strict range.
Also, depending on your neuronal activation functions, consider centering your data in addition to scaling it.
Answered by Benji Albert on September 5, 2021
You should apply and normalize using the total min/max including all the historical data in your dataset. Your model expects the same normalization within each feature across all measurements in that feature. For example
sensor_1_day_1 -> 0, 1, 2, 2, 3 sensor_1_day_2 -> 0, .1, .3, .4, .1
normalize sensor_1 for both days with [min,max] of [0,3] and normalize
sensor_1_day_1_norm -> 0/3, 1/3, 2/3, 2/3, 3/3 sensor_1_day_2_norm -> 0/3, .1/3, .3/3, .4/3, .1/3
Don't forget to de-normalize the predictions (multiply by 3 in this example) as they will also be normalized. Side note: If you apply a different normalization to each day, and have to de-normalize each day differently, this would be very complex to remember and handle.
Answered by Donald S on September 5, 2021
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP