What is the inverse of the cyclic shift operator?

I have given the cyclic shift operator $X(x) in mathbf{R}^{d times d}$, the generalized pauli operator, for a qudit system. This operator is defined by the action it takes on an arbitary ket vector $left| j rangle right. in mathbf{R}^d$ with: $$X(x) left| j rangle right. = left| x oplus j rangle right..$$ Thereby $oplus$ describes the cyclic addition operator defined by: $$x oplus j = (x + j) text{mod}(d).$$ Now I’m supposed to show that the inverse of this operator equals $X(-x)$. My approach is to look at how this operator acts on an arbitary ket vector. This should be something like: $$X(x) left| j rangle right. = left| x oplus j rangle right. langle j left| j rangle right. = left| x oplus j rangle right..$$
So the inverse of this operator should act as:
$$ X^{-1}(x) left| j rangle right. = (left| x oplus j rangle right. langle j left| right.)^{-1} left| j rangle right. = left| j rangle right. langle x oplus j left| right. jrangle.$$
But at this point I have no clue how to show that this expression equals $X(-x)$.

Edit:

With the correction by By Symmetry I’ve tried to invert the operator directly. So my new approach is:

$$X(x)=sum_j left| right. x oplus j rangle langle j left. right|.$$

So the inverse operator is given by:

begin{align}
X^{-1}(x)
&= left( sum_j left| right. x oplus j rangle langle j left. right| right)^{-1}
&= sum_j left| right. j rangle langle x oplus j left. right|
&= sum_j X^{-1}(-x) X(-x) left| right. j rangle langle x oplus j left. right|
&= sum_j X^{-1}(-x) left| right. -x oplus j rangle langle x oplus j left. right|
&= sum_j left| right. -x oplus j rangle X(-x) langle x oplus j left. right|
&= sum_j left| right. -x oplus j rangle langle x-x oplus j left. right|
&= sum_j left| right. -x oplus j rangle langle j left. right| = X(-x).
end{align}

I hope I got this right.