What is the formula for ripple voltage in integrating circuit?

I guess you know how to calculate the output with a sinusoidal input wave. There are several techcnics for that, but the easiest one is probably using phasors.

And thanks to Fourier series you can write your periodic rectangular wave input as a sum of sinusoidal waves. Let's take for example a square wave $V_{in}(t)$ with periodicity T:

$ V_{in}(t) = left{ begin{array}{ll} û & mbox{if } 0 leq t < T/2 -û & mbox{if } -T/2 < t < 0 end{array} right.$ $>$ $>$ with $>$ $V_{in}(t+T) = V_{in}(t)$

Since it is an odd function, the real Fourier series will only have sinus terms since it is odd too. So we can calculate the coefficients $b_n$ given by:

$b_n= frac{1}{T}int_{-T/2}^{T/2}{V_{in}(t)sin(nwt)} dt >$ which satisfy $>$ $V_{in}(t)=sum_{n=1}^{infty} b_n sin(nwt)$

Solving this integral would give you $b_n = left{ begin{array}{ll} frac{4 û}{n pi} & mbox{if $n$ is odd } 0 & mbox{if $n$ is even } end{array} right.$ which implies

$>$ $V_{in}(t)=frac{4 û}{pi}big[ ,sin(wt)+ frac{1}{3}sin(3wt) + frac{1}{5}sin(5wt) >+ >... big] ,$ with $ w = 2 pi /T$

Then since the RC circuit is linear you can say that the effect of the square wave is equal to the sum of the effects of all these sinus waves, which I assume you know how to compute. Your answer will therefore be an infinit sum, but as you can see the coefficients decrease with a factor of $frac{1}{n}$ so it is a good approximation to only consider the few first elements of this sum.