Does correlation between residuals of ARIMA models for two time series tell anything about how they fluctuate together?

Yes, you could look at the correlation between the two residual series, but you should also see if it is really significant. A correlation of 0.2 might well not be, without details (like length of the series) we cannot say.

Better, you should look at the cross-correlation function between the residual series, for instance Cross-correlation significance in R or Do we need to detrend when do Cross-Correlation between two time series?. The correlation you reported probably is the correlation at the same time points, but there could be a lead/lag relationship between the series, and that could show up in the crosscorrelations.

Can I say that random fluctuations of one series "explain" about 4% of random fluctuations of another series?

Maybe not, there could be some common cause behind both, or one could cause the other but from correlations you cannot say which way. Crosscorrelations could help here, if there is either a lead or a lag, maybe the series changing first is driving the other, but also a common cause could act faster on one of the series than on the other ... there are many possibilities, and only correlations do not say enough. Look into multivariate timeseries models like VAR, maybe ...

And, finally:

Both series are strongly periodic and stationary. I fit, let's say, ARIMA(2, 0, 0) to both of them and both models are pretty good

This does NOT exclude stationarity! the periodicity could be dynamic, and AR(2) models does have such quasiperiodic behaviour for some parameter values, and are used precisely to model periodic behaviour. See the answers to Forms of non-stationary process

You have stated "both series are strongly periodic and stationary". I guess by "periodic" you meant seasonal.

If a series is seasonal, then it can not be stationary.

Your series and (probably) your residuals are therefore non-stationary and so you can not rely on the correlation coefficient at all.