Divergence theorem: compute triple integral over a paraboloid between two planes

If you are applying divergence theorem, you will have to close the surface with disks in planes $z=1$ and $z = x+y+3$. You will need to separately calculate flux through the disks and subtract from the flux through the closed surface.

Now good thing is that within the paraboloid domain, plane $z = x + y + 3$ is throughout above plane $z=1$. Also note that the intersection of the given circular paraboloid and the given plane can be seen in $XY$ plane as -

$x^2+y^2 = x + y + 3 implies (x-0.5)^2 + (y-0.5)^2 = 3.5$

So at the minimum, it makes sense to use cylindrical coordinates with

$x = 0.5 + r cos theta, y = 0.5 + r sintheta, 0 leq r leq sqrt{3.5}$

$1 leq z leq 4 + 0.5 costheta + 0.5 sintheta, 0 leq theta leq 2pi$

So the integral to find flux becomes

$displaystyle int_0^{2pi} int_0^{sqrt {3.5}} int_1^{4+rcostheta+rsintheta} r(3+ 2rcostheta) dz dr dtheta$

Now please find flux through disks at $z=1$ and $z=x+y+3$ and subtract from the above. That should give you the flux through paraboloid surface.

---

Just as a side note: if you want to simplify the limits of integration further, you can in fact rotate the paraboloid by $frac{pi}{4}$. There is no change to Jacobian just like in the first case. It will simplify the equation of the plane $z = x + y + 3$.

$x = 0.5 + r cos (frac{pi}{4} + theta), y = 0.5 + r sin(frac{pi}{4} + theta), 0 leq r leq sqrt{3.5}$

Equation of plane becomes $z = x+y+3 = 4 + r cos(frac{pi}{4} + theta) + r sin(frac{pi}{4} + theta) = 4 + r sqrt2 costheta$.