About the behavior of the position and momentum operators

Here's a more abstract derivation of the action of the momentum operator on position eigenstates. To start, we note that momentum is the generator of spacial translation in classical mechanics. Therefore, it stands to reason that we should identify momentum in quantum mechanics as the generator of spacial translations. Let $U(x)$ be defined so that $$U(x)|x'rangle=|x'+xrangle.$$ Then $U(x)$ is the translation operator (I'm working in one dimension for simplicity). We require that $U(0)=1$ (translation by zero doesn't change anything). We also want $U(x)$ to be unitary ($U^dagger=U^{-1}$) so that it doesn't mess up normalization of the vectors it acts on. These requirements means that, for small $epsilon$, we can expand (think Taylor expansion) $U(x)$ as $$U(epsilon)=U(0)+epsilon Kequiv 1-frac{iepsilon}{hbar}P,$$ where $K$ is anti-Hermitian (and so $P$ is Hermitian). The anti-Hermitian requirement on $K$ comes from the fact that $U(epsilon)$ needs to be unitary to order $epsilon$. Then we define $P=ihbar K$ (the factor of $i$ makes $P$ Hermitian, and the factor of $hbar$ is necessary for dimensional analysis if we identify $P$ as momentum). $P$ is the generator of spacial translations. Any finite translation can be performed by simply performing a bunch of infinitesimal ones using $P$. Then we have $$left(1-frac{iepsilon}{hbar}Pright)|xrangle=|x+epsilonrangleimplies P|xrangle=ihbarfrac{|{x+epsilon}rangle-|xrangle}{epsilon}.$$ Taking the limit as $epsilonxrightarrow{}0$, we get $$P|xrangle=ihbarfrac{partial}{partial x}|xrangle.$$

The key results to remember are begin{align} langle xvert prangle &= frac{1}{sqrt{2pihbar}}e^{i p x/hbar}, \ hat plangle xvert prangle &=-ihbarfrac{d}{dx}langle xvert prangle , ,\ psi(x)&=langle xvertpsirangle, ,\ psi(p)&=langle pvertpsirangle. end{align} Thus, begin{align} hat x langle pvert xrangle : =langle pvert hat xvert xrangle = x frac{1}{sqrt{2pihbar}}e^{-i p x/hbar}=+ihbar frac{d}{dp}langle hat pvert xrangle end{align} and thus begin{align} hat xpsi(p)&=langle pverthat xvertpsirangle, ,\ &=int dx langle pvert hat xvert xrangle langle xvert psirangle, ,\ &= int dx ihbarfrac{d}{dp} langle pvert xranglelangle xvert psirangle, ,\ &= ihbarfrac{d}{dp}int dx langle pvert xranglelangle xvert psirangle = ihbarfrac{d}{dp}psi(p), . end{align} The other cases are done in a similar way, remembering that $langle pvert xrangle = langle xvert prangle^*$. Note that derivatives must act on functions, not on kets.

The change of sign in the derivative is similar to the change in sign of $x$ and $p$ when one makes a canonical transformation $xto P,pto -Q$.

You should be able to confirm all the given expressions, but not your derived ones, (7), (8), by the formal representations, (setting $hbar=1$, i.e. nondimensionalizing, to avoid predictable mistakes; think of it as absorbing $hbar$ into all derivative operators, or else its inverse square root into x and p), $$ bbox[yellow,5px]{hat p = int! dp~ |prangle plangle p|= int! dx~ |xrangle (-ipartial_x) langle x| \ hat x = int! dp~ |prangle ipartial_p langle p|= int! dx~ |xrangle x langle x| } ~ . $$ That should allow you to catch your sign errors in your last two equations, (7) and (8).

Your curiosity objective, (7) and (8) with their signs corrected, $$hat p|xrangle= ipartial_x |xrangle, quad hat x |prangle= -ipartial_p|prangle,$$ is not as useful as operators acting on bras, as you ought to find out. Try complex conjugating (2) and (3).

Do you now at last see $$ langle phi| hat p| psi rangle= -i int ! dx ~ langle phi| xrangle partial_x langle x| psi rangle\ = -i int ! dx ~ phi(x)^* partial_x psi(x) = i int ! dx ~ partial_x phi(x)^* ~ psi(x) ~~? $$