Why the result of two equivalent TikZ math calculations is different?

By default, pgfmath does trigonometric calculations using degrees, so rather than cos(deg(30)) you should use cos(30). With this correction, your MWE produces:

For completeness, I have also used pgfmathresult to explicitly show the numerical results of these calculations. Here is the updated code:

documentclass[12pt, margin=3mm]{standalone}
usepackage{tikz}
usepackage{tikz-3dplot}
%usetikzlibrary{calc}
usetikzlibrary{arrows.meta}
begin{document}

cos: pgfmathparse{0.8*cos(30)} pgfmathresult,
sqrt: pgfmathparse{0.4*sqrt(3)} pgfmathresult

    begin{tikzpicture}[scale=10]
    coordinate (O) at (0,0,0);

    tdplotsetcoord{a}{0.4*sqrt(3)}{0}{0}
    draw[-stealth,color=blue] (O) -- (a);

    tdplotsetcoord{b}{0.8*cos(30)}{0}{0}
    draw[-stealth,color=red, dashed] (O) -- (b);
    end{tikzpicture}
end{document}