How to calculate the overlap of the orthogonal state?

TL;DR: These inner products are equal to the amplitudes and therefore the squares of their magnitudes sum to one. By Pythagoras theorem $sin^2theta + cos^2theta = 1$, so if one of the amplitudes is $costheta$ then the magnitude of the other must be $|sintheta|$.

---

Since ${|psi_1rangle, |psi_1^perprangle}$ is a basis, we can expand $|psi_2rangle$ as

$$ |psi_2rangle = alpha |psi_1rangle + beta |psi_1^perprangletag1 $$

where $|alpha|^2 + |beta|^2=1$. Moreover, since the basis is orthonormal, we can compute $alpha$ and $beta$ in terms of inner products

$$ alpha = langle psi_1|psi_2rangle beta = langle psi_1^perp|psi_2rangle $$

as is easy to check by taking the inner product of $(1)$ with the elements of the dual basis. Now, from $|alpha|^2 + |beta|^2=1$ we see that

$$ |langle psi_1^perp|psi_2rangle| = |beta| = sqrt{1 - |alpha|^2} = sqrt{1 - |langle psi_1|psi_2rangle|^2} = sqrt{1 - cos^2theta} = |sintheta| $$

but $theta in (0, pi)$ so $|langle psi_1^perp|psi_2rangle|=sintheta$.