What's the difference between a linear operator and a tensor?

Matrices, operators and tensors may often look similar, but they mean somewhat different things:

• matrices are essentially 2-dimensional tables with numbers. Matrices used in physics are usually supplied with indices and have mathematical operations defined for them, which makes them somewhat more like tensors, but not quite yet. For example, any linear system of equations can be written as $$ mathbf{A}mathbf{x}=mathbf{b}, $$ where $mathbf{A}$ is a matrix, whereas $mathbf{x},mathbf{b}$ are vectors. Yet, they are not necessarily tensors.
• tensors are usually not limited to being two-dimensional, and possess transformation properties in respect to the coordinate system, in which they are defined. Note that the use of word vector is ambiguous - it can be first rank tensor and a simple column-matrix.
• operators are often represented by matrices/tensors, but do not have to be. Momentum operator in quantum mechanics is a simple example: $$hat{p}_x=-ihbarpartial_x$$. Its action on functions is unambiguous, and one can define all its properties (including, e.g., hermicity) without resorting to matrix representation.

A key difference between the linear operators of QM and the tensors of GR is the structure of the underlying vector spaces. The vector spaces on which QM operators act are complex Hilbert spaces, typically with infinite dimensions (although simple examples such as a particle with spin $pm 1$ may be finite dimensional). GR tensors on the other hand act on vectors and vector fields in spacetime which has just four real dimensions.

If $V$ is a finite dimensional vector space, $hom(V)$ is the space of linear operators on $V$ (i.e. linear maps from $V$ to itself), then $$ hom(V)cong Votimes V^ast, $$ where $V^ast$ is the dual space of $V$ and $cong$ denotes canonical (i.e. basis-independent) isomorphism.

If $T^{r,s}(V)=V^{otimes r}otimes V^{astotimes s}$ denotes the space obtained by taking $r$-fold tensor products of $V$ and $s$-fold tensor product of $V^ast$ (so-called type $(r,s)$ tensors), and define the mixed tensor algebra of $V$ as $$ T^{bullet,bullet}(V)=bigoplus_{r,sinmathbb Z^2}T^{r,s}(V), $$ this is a $mathbb Z^2$-graded unital associative algebra whose elements (especially pure grade elements) are often referred to as tensors over $V$.

Thus we can say that the linear operators over $V$ are a special class of tensors, namely the $(1,1)$ grade subspace of $T^{bullet,bullet}(V)$.

---

OP however asked about quantum mechanics, which usually involves infinite dimensional vector spaces. As always, infinite dimensional spaces have subtleties about them.

It is also possible to define tensor products of infinite dimensional vector spaces, but there are various different classes of infinite dimensional vector spaces, and the tensor products within these different categories are not the same.

To give an example relevant to quantum mechanics, if $mathcal H$ and $mathcal K$ are both (separable) Hilbert spaces with orthonormal Schauder bases $(left| h_nrightrangle)_{n=1,...,infty}$ and $(left| k_mrightrangle)_{m=1,...,infty}$, then (omitting the rigorous definition), the Hilbert space tensor product $mathcal Hotimesmathcal K$ is defined in such a way that the system $$ (left| h_nrightrangleotimesleft| k_mrightrangle)_{m,n=1,...,infty} $$ is an orthonormal Schauder basis of the Hilbert space $mathcal Hotimesmathcal K$.

For simplicity consider real Hilbert spaces because I don't want to deal with complex conjugations and antilinear things. A Hilbert space is always isomorphic to its topological dual ($mathcal Hcongmathcal H^sharp$), and I will make this identification.

One may thus try to identify $$ mathcal {H}otimesmathcal {H}^sharpcongmathcal Hotimesmathcal H$$ with the space $hom(mathcal H)$ of linear operators on $mathcal H$.

However since $mathcal Hotimesmathcal H$ is a Hilbert space, if we write $$ A=sum_{n,m}a_{nm}left| h_nrightrangle otimes left| k_mrightrangle,quad B=sum_{n,m}b_{nm}left| h_nrightrangle otimes left| k_mrightrangle, $$ the induced inner product is finite and is $$ langle A,Brangle=sum_{n,m}a_{nm}b_{nm}<infty. $$

Thus elements of $mathcal Hotimesmathcal H$ are Hilbert-Schmidt operators, but not all relevant operators in QM are Hilbert-Schmidt.

Thus we see that $hom(mathcal H)$ is not equivalent with the Hilbert space tensor product $mathcal Hotimesmathcal H$.