How do the symmetries of a system constrain gauge fields?

The relation of magnetic field and the vector potential is independent of the coordinate system. My first attempt I got the same result, however, soon I realized that the coordinates also have to undergo the same rotation. So if one has the following vector potential

$$ mathbf{A}(x,y,z) = left( begin{array}{c} 0, & Bx, & 0 end{array}right)^T $$

so rotating it by the well-known rotation matrix gives: $$mathbf{A'}(x,y,z)=left(begin{array}{c} Bx sintheta, & Bx costheta, & 0 end{array}right)^T $$

For the curl computation, the curl has to be computed in the rotated coordinates. The rotated coordinates are:

$$ x' = costheta x + sintheta y quad text{and} quad y' = -sintheta x + costheta y$$

Actually we need the inverse relation as we want to express the old coordinates by the new ones (fortunately we need it only for x):

$$ x = costheta x' - sintheta y' $$

We now express the rotated vector potential by the new coordinates:

$$ A'(x',y',z') = ( begin{array}{c} B(sintheta costheta x' - sin^2 theta y'), & B(cos^2theta x' - sinthetacostheta y'), & 0 end{array} )^T$$

If we take now the curl of $mathbf{A}$ we get the desired result:

$$ mathbf{B'}=nabla' times A'(x',y',z') = left( begin{array}{c} 0, & 0, & B(cos^2theta + sin^2theta )end{array} right)^T$$

So the raised question has nothing to do with gauge invariance, it is all about rotational invariance.

Following Frederic Thomas's answer pointing out the error in my example of not accounting for rotation of the coordinates, I am now able to answer the question of the constraint on the gauge field:

Under the symmetry transformation $x to x'$, $A(x) to A'(x')$ may be expressed $A(x') + nabla lambda(x')$.

Using Frederic Thomas's expression for $A'(x')$ we can see that

$$A'(x') = B(sin theta cos theta , x' - sin^2 theta y', cos^2 theta x' - sin theta cos theta , y', 0)^T = B(0, x', 0)^T + nabla lambda(x')$$

where

$$lambda(x') = sin^2 theta , x' y' - sin theta cos theta (x'^2 + y'^2).$$