Why do we call translations and time evolutions "operators" while they are not (linear) operators?

I have been reading Sakurai’s Modern Quantum Mechanics and I am confused by this:

A translation operator $mathscr T(delta x)$ is defined as $mathscr T(delta x) |x rangle = | x + delta x rangle$ and it is used to derive the momentum operator $hat p$ by the expression $mathscr T(delta x) = exp(- mathrm i hat p delta x/hbar)$. And similarly to derive Hamiltionian $hat H$ the time evolution operator $mathscr U(t, t_0)$ is introduced.

Question:

Notice that

1. $mathscr T(delta x) | 0 rangle = | delta xrangle neq | 0 rangle$;
2. $mathscr T(delta x) |lambda x rangle = |lambda x + delta x rangle neq lambdamathscr T(delta x) |x rangle$ ($lambda in mathbb C$);
   …

We know that the translations are not linear operators, but we still call them operators, and treat them as operators. We even find them Hermitian adjoints ($mathscr T(delta x)^dagger mathscr T(delta x) simeq hat 1$, or "the infinitesimal translations are unitary"), which doesn’t make any sense to me.

The problem also occurs in discussing the time evolution operators and Sakurai wrote: $mathscr U(t, t_0)^dagger mathscr U(t, t_0) = hat 1$, where the time evolution operators are not infinitesimal.

So when we call $mathscr T(delta x)$ and $mathscr U(t, t_0)$ "operators" what do we want to say? If they are not operators in $mathrm{End} (mathbb H)$ (endomorphisms, i.e. linear operators), where do they belong to? Are they affine mappings in the affine group (I know little about this group)? And, how do we treat the Hermitian adjoints of them?

I would like a mathematically rigorous answer instead of an intuitive explaination, thank you!