Do roots of polynomial with coefficients in a CM field lie in a CM field?

This is something that I have been thinking about for a while now, not sure if it is standard (or even true at all) or not:

Let $K/ mathbb Q$ be a CM number field, that is, it is closed under complex conjugation, which induces an automorphism that commutes with the other $mathbb Q$-embeddings of $K$ into the field of complex numbers. Let $P(z) in K[z]$ be irreducible and monic. Will the splitting field of $P$ over $K$ be CM? If not, does there exist a CM field containing the roots of $P$?

Can we say something if $K/mathbb Q$ is Galois and CM (so complex conjugation commutes with the rest of the Galois group)?

Partial progress(?):

If $alpha_1, cdots , alpha_d$ denote the roots of $P(z):=sum_{j=0}^d a_jz^j in K[z]$ lying in some splitting field $E/K$, then Vieta’s relaions tell us that $a_j = (-1)^{n-j}e_j(alpha_1, cdots , alpha_d)$ where $e_i$ denotes the $i$-th symmetric function. Thus for any $sigma in text{Gal}(E/mathbb Q)$, we have
$$(-1)^{n-j} e_j(overline{sigmaalpha_1}, cdots ,overline{sigmaalpha_d}) = overline{sigma a_j} = sigma({overline{a_j}}) = (-1)^{n-j} e_j(sigma(overlinealpha_1), cdots , sigma(overlinealpha_d))$$
Since the tuples ${overline{sigmaalpha_1}, cdots ,overline{sigmaalpha_d}}$ and ${sigma(overlinealpha_1), cdots , sigma(overlinealpha_d)}$ have equal values of elementary symmetric functions, they must be roots of the same polynomial and hence must be equal upto permutation. What I want however is that $(overline{sigmaalpha_1}, cdots ,overline{sigmaalpha_d}) = (sigma(overlinealpha_1), cdots , sigma(overlinealpha_d))$, the equality holding as ordered tuples.

Maybe I am missing something really simple but the observation that I just obtained also makes me fear that a counterexample might also be constructible (no pun intended) by just enforcing these conditions, although I haven’t been able to construct the same. If none of these hold, can we say something along these lines under some (hopefully minor) additional conditions? I would really appreciate any help, thanks.

Edit: So I’ve already asked this in a comment, but I feel like it is more appropriate a an edit. As Prof. KConrad has pointed out, there is a very simple counterexample. That leads me to ask the following:
Is there anything we can say concerning the relationship between $sigmaoverlinealpha$ and $overline{sigmaalpha}$ for some $alpha$ which is a root of a monic irreducible polynomial with coefficients in a CM field $K$ (where $sigma$ is some $mathbb Q$-automorphism of a finite Galois extension over $K$ containing the conjugates of $alpha$)? What about when we impose conditions on $sigma$? Or on $alpha$ (like say, $alpha$ has to be an algebraic integer)? I might be naive to expect this, in which case I would appreciate some examples suggesting very erratic behaviour.

Edit 2: I suppose a more precise reformulation of the question asked in the above edit might be the following: what we can say about the size (absolute value) of the element $|(sigmatau – tausigma)alpha|$ where $alpha$ and $sigma$ are as above and $tau$ is complex conjugation.