Finding the Expectation value for a wave function

The operator $|5ranglelangle2|$ is exactly what it looks like. If you apply it to any state $|phirangle$ you get:

begin{equation} (|5ranglelangle2|)|phirangle=langle2|phirangle|5rangle end{equation} That is, it gives you the ket $|5rangle$ multiplied by the projection of the state $|phirangle$ on $|2rangle$.

In general, remember that an operator is defined by how it acts on the states of your vector space.

Can you now evaluate the expectation value?

The sequence is the following $$ langlephi|Q|phirangle=langlephi|5ranglelangle 2|phirangle. $$ The number 5 and 2 refer to $n=5$ and $n=2$ occupation values and so, to the states $|5rangle$ and $|2rangle$ respectively. Please, note that $$ langle m| nrangle=delta_{mn} $$ with $m,n=1,2,ldots$ and $delta_{mn}=1$ for $m=n$ and = otherwise. Then, using all this, one has $$ langlephi|Q|phirangle=frac{6}{pi^2}sum_{m=1}^inftysum_{n=1}^infty C_mC_nlangle m|5ranglelangle 2|nrangle. $$ Could you go on from here?

So first of all, your state seems to be well-defined and normalised, as, $$ leftlangle phileft|phirightrangle right.=1 $$ on account of, $$ zeta(2)=sum_{n=1}^{infty}frac{1}{n^{2}}=frac{1}{1^{2}}+frac{1}{2^{2}}+frac{1}{3^{2}}+frac{1}{4^{2}}+cdots=frac{pi^{2}}{6} $$ https://en.wikipedia.org/wiki/Basel_problem

But you have a problem: $left|5rightrangle leftlangle 2right|$ is not a self-adjoing operator, so it's not a valid observable. A self-adjoint extension of it would be: $$ frac{1}{2}left(left|5rightrangle leftlangle 2right|+left|2rightrangle leftlangle 5right|right) $$ Once you do that, it's a simple matter of "sandwiching" the operator with the state in the <bra|ket> way and using orthogonality. Other way to interpret $left|5rightrangle leftlangle 2right|$ is as a transition amplitude of going from $left|5rightrangle $ to $leftlangle 2right|$. It's the words "expected value" that do not sit well with the particular operator you've proposed. I hope that was helpful, and I didn't make any idiotic mistake, to which I seem to be prone lately.