Why is there a factor of $hbar$ in the physicists' $mathfrak{su}(2)$ Lie algebra?

I think the physical context of your question is spin angular momentum of elementary particles (for example: an electron) and rotations of this particle in space.

Let's begin with spin angular momentum $bf{S}$ (a vector with 3 components $S_1$, $S_2$, $S_3$). Physical experiments showed that the measured spin component $S_j$ of the particle is either $+frac{1}{2}hbar$ or $-frac{1}{2}hbar$. You see, Planck's constant $hbar$ here (and its extremely small numerical value $1.055 cdot 10^{-34}rm{ kg m^2/s} $) comes from physical experiments.

Hence the observable $S_j$ is represented by the self-adjoint operator $$S_j=frac{1}{2}hbarsigma_j.$$ The two eigenvectors of this operator are the measured values.

On the other hand you have rotations of the particle in space. These rotations are represented by unitary elements of the Lie group $SU(2)$. $$U = e^{frac{i}{2}(theta_1sigma_1+theta_2sigma_2+theta_3sigma_3)}$$ where $theta_1$, $theta_2$, $theta_3$ are arbitrary real parameters encoding the axis and angle of the rotation.

These Lie group elements are obviously generated by the 3 self-adjoint generators $$X_j=frac{1}{2}sigma_j$$ which spawn the Lie algebra $mathfrak{su}(2)$. The definition is chosen without the $i$ so that the $X_j$ will be self-adjoint.

You see, the observable $S_j$ and the generator $X_j$ are nearly the same, only different by the factor $hbar$: $S_j=hbar X_j$.

_{By the way: This is essentially the contents of Noether's theorem, saying every symmetry transformation (here generated by $X_j$) corresponds to a conserved physical observable (here $S_j$).}

Now we physicists are lazy people. And therefore we use $S_j$ for both purposes ($S_j$ as the spin observable and $S_j/hbar$ as the rotation generator). Thus we can write $$U = e^{i(theta_1 S_1+theta_2 S_2+theta_3 S_3)/hbar}$$

$hbar=frac{h}{2pi}$ is proportional to the Planck constant $h$ and has the dimension of angular momentum, so the observables consist of Pauli matrices multiplied by a $hbar$ correspond to the angular momentum, which is not just real numbers, but have direct physical meaning.
Sometimes physicists would ignore the dimension of an obervable and only concerned about the algbra, then they just take $hbar=1$ or similarly the elementary charge $e=1$ for simplicity, but at the final step the dimension must be recovered to get the value of observable.

the reason is that the real world has dimensions. Spin in physics originates from the irreducible representations of the Lorentz group, which is $SO(3,1)$. The Loretnz group relates to transformations of coordinate systems. These transformations correspond to real space-time quantities, and therefore come with dimensions attached to them. Rotations come with dimensions of angular momentum, which is the same dimensions of $hbar$. So naturally the physical quantity that correspond to rotations, here the spin, should have the same dimensions.

It turns out that the universe has scales in it, and phenomena behave differently when examined at different scales. For example - physics at low velocities is different than at high velocities. When we say "high" velocity or "low" velocity we ask ourselves "with respect to what?" and the answer the universe give us is the speed of light $c$. So we measure velocities relative to that quantity, if th velocity is much smaller than $c$, the physics is classical (can be approximated by Newton's mechanics) and if it comparable to $c$, it is relativistic. Similarly, in angular momentum we measure large and small with respect to $hbar$. If things are of the scale of $hbar$ they are quantum, and if they are much larger than $hbar$ they are classical (that is - can be approximated by classical mechanics).