Conformal transformation vs diffeomorphisms

I'm a mathematician, not a physicist, so I learned all of these ideas with different notation, but I think I understand what might be confusing you.

Conformal transformations are indeed a special kind of diffeomorphism, and a rotation (say in the plane with the usual metric) is indeed conformal, so the two formulas you listed had better agree in this case.

But in fact, if your manifold is $mathbb{R}^2$, your metric is the usual one ($g_{munu}$ is the identity matrix at every $x$), and your coordinate change is a rotation, the second formula you listed will show you that the metric looks unchanged in the new coordinates. (This is not a coincidence: preserving this metric is exactly the property that makes rotations special in the first place!) That is, there is no conflict between the two formulas here, it's just that seeing it involves a bit of computation.

Working this out is a very good exercise and I don't think you'd gain much from me typing it all out here. A hint that might help you get oriented is that, since rotations are linear in the coordinate system we've chosen, the Jacobian matrix at every point is the same as the matrix for the rotation itself.

I deleted my previous answer because I was very confused when I wrote it. I realized that I just wanted to stress MBolin's answer.

From a differential geometric point of view, an active transformation of space(time) is a diffeomorphism $phi:Mrightarrow M$. Given a curve $gamma$ with initial velocity $X=dot{gamma}(0)$ at $gamma(0)$, we can define the new transformed curve $phicircgamma$ which has velocity $phi_{*,gamma(0)}X$ at the transformed initial point $phi(gamma(0))$. This is the push-forward of the vector $X$. The conformal condition is then just that the angles between two vectors and their push-forwards coincide. In other words, that there is a function $Lambda:Mrightarrowmathbb{R}$ such that for any two vectors $X$ and $Y$ at $p$ we have $$g_{phi(p)}(phi_{*,p}X,phi_{*,p}Y)=Lambda(p) g_p(X,Y).$$ The left hand side can be thought of as the pull-back of the metric so this condition is at times written as $phi^*g=Lambda g$, but this is just a more condensed version of the statement above .

In a coordinate system $x$ defined on a chart containing both $p$ and its transformed point $phi(p)$, we can write the equation above as $$Lambda g_{munu}(x)=frac{partial x^alphacircphi}{partial x^mu}frac{partial x^betacircphi}{partial x^nu}g_{alphabeta}(x)circphi.$$ In here I have used the physicists notation of writing $g_{munu}(x)$ for the composition $g_{munu}circ x$. In here my convention is that $$g_{munu}(x(p))=g_pleft(left(frac{partial}{partial x^mu}right)_p,left(frac{partial}{partial x^nu}right)_pright)$$

There is a parallel (although local) discussion on can make from the passive point of view. In here one is instead replacing the coordinates $x$ by a new system $x':=xcircphi^{-1}$ (thinking of active vs. passive rotations clarifies why this is the correct coordinates to choose). Then the trick for relating these two is to note that $$frac{partial x^alphacircphi}{partial x^mu}=partial_mu(x^alphacircphicirc x^{-1})circ x=partial_mu(x^alphacirc {x'}^{-1})circ x'circphi=frac{partial x^alpha}{partial x'^mu}circphi.$$ Then the conformal condition becomes $$Lambda g_{munu}(x)=left(frac{partial x^alpha}{partial x'^mu}frac{partial x^beta}{partial x'^nu}g_{alphabeta}(x)right)circphi.$$ We recognize that the term in parenthesis is just the metric in the new coordinates $g'_{munu}(x')$. Then we have $g'_{munu}(x')circphi=g'_{munu}circ x'circphi=g'_{munu}circ xequiv g'_{munu}(x)$. We thus recover the statement as in Zee's book that the conformal condition is $$g'_{munu}(x)=Lambda g_{munu}(x).$$