If $A$ is a cofibrant commutative dg-algebra over a commutative ring of characteristic $0$, then its underlying chain complex is cofibrant

Fernando's answer tells you how to prove the statement directly. Alternatively, if you want a reference, this is proven in Corollary 3.6 of my PhD thesis paper (published in JPAA) Model Structures on Commutative Monoids in General Model Categories. I introduce an axiom that a monoidal model category $M$ can satisfy, the "strong commutative monoid axiom," which guarantees that:

1. Commutative monoids in $M$ inherit a model structure transferred from $M$ along the forgetful functor $U$, meaning that a morphism $f$ is a weak equivalence or fibration if and only if $U(f)$ is in $M$, and
2. $U$ preserves cofibrations with cofibrant source.

Note that (1) implies immediately that $U$ preserves fibrant objects. Then, in Section 5.1, I verify that the example you mention does satisfy this axiom. Furthermore, the initial CDGA is cofibrant, so a corollary of (2) is that $U$ takes cofibrant CDGAs to cofibrant chain complexes (sometimes said "$U$ preserves cofibrant objects").

Disclaimer: In Section 5.1 of that paper, I only state the result for when $R$ is a commutative $mathbb{Q}$-algebra. In a later paper, I make the observation that everything works when $R$ has characteristic zero. Search Homotopical Adjoint Lifting Theorem for "characteristic" to see. This paper was joint with Donald Yau and published in Applied Categorical Structures.

Cofibrant CDGAs are retracts of cellular ones. A cellular cofibrant CDGA is a free commutative graded algebra on a (possibly transfinite) sequence of generators $x_1,x_2,dots$ such that $d(x_i)$ only depends on previous generators. A linear basis is given by monomials $x_{i_1}^{n_{i_1}}cdots x_{i_r}^{n_{i_r}}$ such that $i_j<i_{j+1}$ and $n_{i_j}=1$ if $|x_{i_j}|$ is odd. This follows from $mathbb{Q}subset R$. You can put a kind of lexicographic order in these monoimials in such a way that the differential of each one only depends on previous monomials. This is cofibrant as a complex in the projective model structure by virtue of the well-known set of generating cofibrations, i.e. it is a cellular complex.