In Time Series forecast, should Scaling be done on both train and test features combined ( test is 1 new data point)?

Let say I have a Time series, I’m using sliding/expanding window method to split to train and test data: train would be all the data I have until day x and test is day x+1.

To avoid Data leakage I’m re-fitting the scaler for each day with the data I have until that day.

My question is: Why not fit the scaler with the data using all the training points and the Features from the next day ( Test Data)?

It wont be data leakage as I already have the test features available to me at the day of the forecast..

Thanks