An essential exercise involving spin composition

Let us first fix clearer notation. We call the spin operators on the $i$-th particle $S_{ix}$ (and so on) and their eigenvalues $s_{ix}$ (and so on). $S_i^2 = S_{ix}^2 + S_{iy}^2 + S_{iz}^2$ is the Casimir operator of the rotation algebra $mathfrak{su}(2)$ has a single eigenvalue $s_i(s_i + 1)$ on irreducible representations of the rotation algebra. For the "total" operators and eigenvalues on the combined two-particle space we just drop the $i$ subscript. In this notation your question asks you to find the two-particle state with $s = 1$ and $s_y = 0$ in terms of eigenstates of $S_{z}$.

1. You have correctly determined that a general eigenstate of $S_y$ has the form $alvert +,-rangle + blvert -,+rangle$ in terms of eigenstates of the $S_{iy}$. But you also need it to be an eigenstate of $S^2$ with the eigenvalue $1(1+1) = 2$.

2. Since the eigenstates of each of the $S_{ix},S_{iy},S_{iz}$ form a basis for the one-particle space, the eigenstates of one of these operators can be expressed as a linear combination of the eigenstates of another of these operators. The notation $lvert +,-rangle$ is just shorthand for $lvert +rangle otimes lvert -rangle$, so by linearity of the tensor product the expressions from the one-particle space straightforwardly extend to the two-particle space - just plug them in and multiply out.

(Brief introduction about spin for a $1/2$ particle)

If for a particle with spin $s=1/2$ the set of commuting operators $left{S_y,boldsymbol{S}^2right}$ is chosen as basis, then one may label the states according to the $pm 1/2$ eigenstates of $S_y$ (I will use the following notation instead of your simplified one because it will be useful when addressing your second point of the question)

$$S_y|S_y;pmrangle=pmfrac{1}{2}|S_y;pmrangletag{1}label{sy}$$

with total angular momentum

$$boldsymbol{S}^2|S_y;pmrangle=frac{3}{4}|S_y;pmrangletag{2}label{s2}$$

One may introduce raising and lowering operators as $S_pm=S_xpm mathrm{i}S_z$ whose actions yield

$$S_+|S_y;+rangle=0,quad S_+|S_y;-rangle=|S_y;+rangletag{3}label{s+}$$

along with

$$S_-|S_y;-rangle=0,quad S_-|S_y;+rangle=|S_y;-rangletag{4}label{s-}$$

(Moving to a system of two particles of spin $1/2$)

The total spin along $Oy$, whose eigenvalue you seek to be null, that it $s_y^mathrm{tot}=0$ (notice I denoted the eigenvalue with small letter in order to not cause any confusion) would be given by

$$S_y^mathrm{tot}=S_y^1otimesmathsf{I}_2+mathsf{I}_1otimes S_y^2tag{5}label{sytot}$$

This may seem just spin with extra steps but I always find it useful when I know on which space the operators act (the identity was denoted by $mathsf{I}$).

There aren't many possibilities. Direct computations show that states of the type $|S_y;+rangle|S_y;+rangle$ would yield $s_y^{tot}=1$, states $|S_y;-rangle|S_y;-rangle$ give $s_y^{tot}=-1$. Nevertheless, the states $|S_y;+rangle|S_y;-rangle$ and $|S_y;-rangle|S_y;+rangle$ result in, via direct computation using eqref{sytot}, the eigenvalue $s_y^{tot}=0$. Therefore, a state with $s_y^{tot}=0$ must be a linear combination of the type

$$|s_y^{tot}=0rangle=a|S_y;+rangle|S_y;-rangle+b|S_y;-rangle|S_y;+rangletag{6}label{state}$$

where the coefficients must of course satisfy $a^2+b^2=1$ as a normalization condition. This is a result which you also obtained but I just wanted to go through it with my notation.

On the other hand, the total spin of a composite system may be expanded as

$$boldsymbol{S}^2=(boldsymbol{S}^1)^2otimesmathsf{I}_2+mathsf{I}_1otimes(boldsymbol{S}^2)^2+2boldsymbol{S}^1otimesboldsymbol{S}^2tag{7}label{s2tot}$$

It is useful to express the term $boldsymbol{S}^1boldsymbol{S}^2$ (I'm dropping the direct product term since too much rigurosity is slowing my typesetting) as

$$boldsymbol{S}^1boldsymbol{S}^2=underbrace{S_x^1S_x^2+S_z^1S_z^2}_{displaystyle dfrac{1}{2}(S_+^1S_-^2+S_-^1S_+^2)}+S_y^1S_y^2tag{8}label{s1s2}$$

Now one may proceed to act with the total spin from eqref{s2tot}, along with the term from eqref{s1s2}, on the proposed state eqref{state}. The computation is not difficult, it just uses repeatedly the action of the operators from eqref{sy}, eqref{s2}, eqref{s+} and eqref{s-}. There might be a shorter more elegant way to derive it than with raising and lowering operators. Nevertheless, the final result reads

$$boldsymbol{S}^2|s_y^{tot}=0rangle=(a+b)|S_y;+rangle|S_y;-rangle+(a+b)|S_y;-rangle|S_y;+rangle $$

One is interested in states having $s^mathrm{tot}=1$ which imply

$$boldsymbol{S}^2|s_y^{tot}=0rangle=underbrace{s^mathrm{tot}(s^mathrm{tot}+1)}_{displaystyle 2}|s_y^{tot}=0ranglestackrel{eqref{state}}=2a|S_y;+rangle|S_y;-rangle+2b|S_y;-rangle|S_y;+rangle$$

Therefore $a=b$ and along with $a^2+b^2=1$ and assuming $a,binmathbb{R}$, the state characterized by $s_y^{tot}=0$ and $s^{tot}=1$ is

$$|s_y^{tot}=0, s^{tot}=1rangle=frac{1}{sqrt{2}}(|S_y;+rangle|S_y;-rangle+|S_y;-rangle|S_y;+rangle)tag{9}label{finalstate}$$

Now that we have the state, the following question was how to express it in the basis (for each particle) labeled by the eigenvalues of $left{S_z,boldsymbol{S}^2right}$ (at least this is how I understood it, correct me if I am mistaken).

This would somehow assume that you now do a measurement of spin along another axis, which should erase the information measured before. It is then fair to assume that it is equally probable to obtain projections of spin along $Oz$ of $pm 1/2$, irrespective of what you obtained before along $Oy$, thus

$$|langle S_y;+|S_z,+rangle|^2=|langle S_y;-|S_z,+rangle|^2=frac{1}{2}$$

Consequently, one may expand

$$|S_z,+rangle=underbrace{langle S_y;+|S_z;+rangle}_{displaystyledfrac{1}{sqrt{2}}}|S_y;+rangle+underbrace{langle S_y;-|S_z;+rangle}_{displaystyledfrac{1}{sqrt{2}}}|S_y;-rangle$$

and similarly (but with a change in sign which will be explained immediately)

$$|S_z,-rangle=underbrace{langle S_y;+|S_z;-rangle}_{displaystyledfrac{1}{sqrt{2}}}|S_y;+rangle+underbrace{langle S_y;-|S_z;-rangle}_{displaystyle-dfrac{1}{sqrt{2}}}|S_y;-rangle$$

where the change in sign is chosen in order to assure

$$|langle S_z;+|S_z,-rangle|=0$$

Therfore, you have your $left{|S_y;pmrangleright}$ basis expressed in terms of $left{|S_z;pmrangleright}$ as

$$|S_y;+rangle=frac{1}{2}(|S_z;+rangle)+|S_z;-rangle),quad|S_y;-rangle=frac{1}{2}(|S_z;+rangle)-|S_z;-rangle)$$

which allow you to immediately express your state from eqref{finalstate} in this basis now labeled by the eigenstates of $S_z^1,S_z^2$.