What is the "secret " behind canonical quantization?

Intuition isn't something that one receives as a gift - it needs to be developed through experience. As it turns out, quantum mechanics is very different from classical physics, so your experience with the latter does not translate into much useful intuition for the former.

---

In the Hamiltonian formulation of classical mechanics, the state of a system is represented by a point in phase space, and observable quantities can be thought of as $mathbb R$-valued continuous functions of the phase space variables (e.g. position, momentum, etc). Experiments such as Stern-Gerlach demonstrated that this perspective is insufficient.

In the SG experiment, one finds that the spin angular momentum observable is quantized, with precisely two possible measurement outcomes. This is impossible in the classical picture - a continuous function cannot map the whole phase space$^dagger$ to two distinct numbers. Furthermore, the measurement of one observable can affect the measurement of another in a way which cannot be accounted for by the modeling of physical observables as simple functions.

From this, we are obliged to seek a different model. Classical measurement outcomes take the form of connected intervals of $mathbb R$. Quantum measurements can yield such results, but they can also result in discrete values (as per SG, the measurement of atomic spectral lines, etc) and disconnected intervals (see e.g. band structure in solids). These possibilities can be accounted for by modeling observables with self-adjoint operators on some Hilbert space, with the possible measurement outcomes being given by the spectrum of the corresponding operator. This is the POV adopted by the standard formulation of quantum mechanics.

---

Having adopted this viewpoint, there is still no obvious way to decide which operators represent which observables. The canonical quantization procedure is ultimately a (physically-motivated) guess. Experiments such as the double-slit experiment suggest the existence of a spatially-varying wave function which can give rise to interference effects. The Born interpretation of this wave function is a spatial probability amplitude $psi$ such that $int_a^b |psi(x)|^2 dx$ yields the probability of measuring a particle to lie in the interval $[a,b]$.

From here, we can define the action of the position observable relatively naturally - its action on a wave function is simply multiplication by $x$. This yields the correct spectrum of possible position measurements, and its "expected value" is simply the mean of the spatial probability distribution.

The definition of the momentum operator is a bit trickier, but it can be motivated by examining the algebraic structure of observables which is present in classical Hamiltonian mechanics. The momentum observable is the infinitesimal generator of spatial translations - imposing the same structure on the quantum theory yields the definition of the momentum operator in terms of a differential operator on $psi(x)$.

---

As stated before, however, canonical quantization (as well as any other quantization procedure) is ultimately a guess. Measurements of a system give clues as to the nature of the physical observables of interest, which in turn give clues to the Hilbert space on which they are built. We then construct the relevant model, make predictions, compare with further experiments, and evaluate whether our model is sufficient to accurately predict how the system will behave.

---

$^dagger$ This would only be possible if the phase space were disconnected, consisting of two distinct pieces corresponding to the different possible values of the spin angular momentum. However, rotational invariance rules this out, and the non-commutation of spin measurements along different axes provides a further nail in the coffin of this idea.

It is very strange that quantum mechanics, which is supposed to be more fundamental, is constructed using the classical theory. The logic is a bit backwards, but there's a good reason why it is done this way. The canonical quantization ensures that the quantum theory approaches the appropriate classical limit.

There have been some attempts to describe quantum mechanics in a purely quantum way, but it always amounts to just stating the spectrum of states the theory has. Not super illuminating if you ask me.

For instance, there are attempts at formulating quantum field theory using solely the S-matrix, which describes the probabilities of particles scattering at various energies and angles. But defining a theory amounts to stating what those probabilities are. There's no equation that one can solve that will give you those probabilities (unless we use canonical quantization). There's also an inherent issue with the S-matrix formulation, since it cannot properly account for massless particles.

It is useful to have the spectrum of states in a quantum theory determined from a finite set of equations, instead of listing an infinite number of states. This is why canonical quantization is so widely used.

Quantizing a classical field is usually the pedagogical easiest way to introduce quantum mechanics. However, it really feels like a magic trick...On the other hand, it is possible to derive QM without introducing any classical field. The key of doing this is to use the path integral formulation of QM.

In classical mechanics, one can derive the Euler-Lagrange equation or the Lagrange equation of motion from a variational principle, i.e., by minimizing the classical action. Analogously, in the path integral formulation one minimizes the quantum action in order to derive the quantum field equations. This is done without the need to define an intermediate classical field.

An important point is that the path integral formulation is equivalent to the canonical quantization approach. However, the former feels like a more natural way to introduce QM, at least conceptually.

There is however a little elephant in the room in this approach: The path integral itself is not well defined mathematically, i.e., there is no broadly-accepted and well-defined way to rigorously define the path integral from a mathematical point of view. But physicists don't care :D

TL,DR
I suggest you to look to the path integral formulation of QM https://en.wikipedia.org/wiki/Path_integral_formulation

Indeed,

canonical quantization works just when it works.

It is in my view wrong and dangerous to think that this is the way to construct quantum theories even if it sometimes works: it produced astonishing results as the theoretical explanation of the hydrogen spectrum.

However, after all the world is quantum and classical physics is an approximation: the quantization procedures go along the wrong direction! There are in fact several no-go results against a naive validity of such procedures cumulatively known as Groenewold -Van Hove's theorem.

However, the question remains: why does that weird relation between Poisson brackets and commutators exist?

In fact, this relation motivates the naive quantization procedures.

In my view, the deepest answer relies upon the existence of some symmetry groups in common with classical and quantum theory.

These groups $G$ of transformations are Lie groups and they are therefore characterized by their so called Lie algebras $mathfrak{g}$, which are vectors spaces equipped with a commutator structure $[a,b] in mathfrak{g}$ if $a,bin mathfrak{g}$. We can think of $ain mathfrak{g}$ as the generator of a one-parameter subgroup of $G$ usally denoted by $mathbb{R} ni t mapsto exp(ta) in G$. If $a_1, ldots, a_n in mathfrak{g}$ form a vector basis, it must hold $$[a_i,a_j] = sum_k C^k_{ij}a_ktag{1}:,$$ for some real constants $C_k^{ij}$. These constants (almost) completely determine $G$. For instance, if $G=SO(3)$ the group of 3D rotations, the one-parameters subgroups are rotations around fixed axes and it is always possible to choose $C_k^{ij}= epsilon_{ijk}$ (the so-called Ricci symbol).

In classical physics, one represents the theory in the Hamiltonian formulation. States are points of a $2n$ smooth dimensional manifold $F$ called the space of phases, with prefereed classes of coordinates, said canonical, denoted by $q^1,ldots, q^n, p_1,ldots, p_n$.

If $G$ is a symmetry group of the system, then there is a faithful representation $G ni g mapsto tau_g$ of it in terms of (canonical) transformations $tau_g : F to F$ which move the classical states according to the transformation $g$. The representation $G ni g mapsto tau_g$ admits an infinitesimal description in terms of infinitesimal canonical transformations strictly analogous to the infinitesimal description of $G$ in terms of its Lie algebra $mathfrak{g}$. In this case the corresponding of the Lie algebra is a linear space of smooth functions, $A in C^infty(F, mathbb{R})$ representing classical observables, and the Poission bracket ${A,B} in C^infty(F, mathbb{R})$.

An (actually central) isomorphism takes place between the Lie algebra $(mathfrak{g}, [:,:])$ and the similar Lie algebra $(C^infty(F, mathbb{R}), {::})$ made of physical quantities where the commutator ${::})$ is just the famous Poisson bracket.

If $a_kin mathfrak{g}$ corresponds to $A_kin C^infty(F, mathbb{R})$ and (1) is valid for $G$, then $${A_i,A_j} = sum_k C^k_{ij}A_k + c_{ij}1 tag{2}$$ where the further constants $c_{ij}$, called central charges, depend on the representation. $$a mapsto Atag{2'}$$ defines a (projective or central) isomorphism of Lie algebras.

When passing to the quantum description, if $G$ is still a symmetry group a similar mathematical structure exists. Here, the space of (pure) states is a complex Hilbert space $H$ and the (pure) states are normalized vectors $psiin H$ up to phases.

If $G$ is a symmetry group there is a (projective/central) unitary representation $G ni g mapsto U_g$ in terms of unitary operators $U_g : Hto H$. The one-parameter subgroups of $G$ are now represented by unitary groups of exponental form (I will systematically ignore a factor $1/hbar$ in front of the exponent) $$mathbb{R} ni t mapsto e^{-it hat{A}}:,$$ where $hat{A}$ is a (uniquely determined) selfadjoint operator.

Again, if (1) is valid and $hat{A}_k$ corresponds to $a_kin mathfrak{g}$, we have that $$[-ihat{A}_i,-ihat{A}_j]= -isum_k C^k_{ij}hat{A}_k -i c'_{ij}I tag{3}$$ where $[:,:]$ is the commutator of operators. In other words $$a mapsto -ihat{A} tag{3'}$$ defines a (projective) isomorphism of Lie algebras.

I stress that the isomorphisms (2') and (3') independently exist and they are just due to the assumption that $G$ is a symmetry group of the system and the nature of the representation theory machinery.

Using these two isomprphisms, we can construct a third isomorphism (assuming $c_{ij}=c'_{ij}$) that interpolates between the classical and the quantum realm.

In this way, if $A in C^infty(F, mathbb{R})$ corresponds to $hat{A} : H to H$ (actually one should restrict to a suitable dense domain), then $${A,B} quad mbox{corresponds to} quad i[hat{A},hat{B}]tag{4}$$ when comparing (2) and (3). (I again ignored a factor $hbar$ since I have assumed $hbar=1$ in the exponential expression of the one-parameter unitary groups.)

It is now clear that (4) is the reason of the correspondence principle of canonical quantization when the same symmetry group exists both in classical and in quantum physics.

In non relativistic physics, the relevant symmetry group is the Galileo group. This plays a crucial role both in classical and in non-relativistic quantum physics.

So we must have a (central) representation of its Lie algebra both in classical Hamiltonian and in Quantum physics.

Relying upon the above discussion, we conclude that the isomorphism relating the isomorphic classical and quantum representations of the Galileo group -- the map associating classical quantities to corresponding operators preserving the commutation relations -- includes the so called canonical quantisation procedure

Let us illustrate this fact in details. The Lie algebra $mathfrak{g}$ includes a generator $p$ which, in classical Hamiltonian theory, describes the momentum (generator of the subgroups of translations) and another generator $k$ (generator of the subgroup of classical boost) corresponding to the position up to a constant corresponding to the mass of the system $m$.

Let us focus on the three levels.

Geometrically $$[k,p]=0:.$$ In the Hamiltonian formulation, a central charge show up $${k,p}= m 1$$ so that, defining $x:= k/m$, we have $${x,p}= 1:.$$ In Quantum physics, in view of the discussion above, we should find for the corresponding generators/observables $$[-ihat{K},-ihat{P}]= -im hat{I}$$ hence, defining $hat{X}:= frac{1}{m}hat{K}$, $$[hat{X},hat{P}]= i hat{I}$$

This correspondence, which preserves the commutation relation, can be next extended from the initial few observables describing the Lie algebra to a larger algebra of observables said the universal enveloping algebra. It is constructed out of the Lie algebra of the Galileo group. It includes for instance polynomials of observables.

Summing up: there are some fundamental symmetry groups in common with classical and quantum physics. These groups are the building blocks used to construct the theory, since they are deeply connected to basic notions as the concept of reference frame and basic physical principles as the relativity principle. The existence of these groups creates a link between classical and quantum physics. This link passes through the commutator structure of (projective) representations of the said group which is (projective) isomorphic to the Lie algebra of the symmetry group. Quantization procedures just reflect this fundamental relationship. Next the two theories evolve along disjoint directions and, for instance, in quantum theory, further symmetry groups arise with no classical corresponding.

Instead, we were given a recipe how to quantize a classical theory, which is based on the rule of transforming all quantities to operators, and that Poisson bracket is transformed to a commutator. For me it seems like a big secret remains out there, it just hard to me to believe that this is the way our world behaves without further intuitive explanations.

You were given these recipes because they were discovered first, on our planet, and describe the situation optimally, and people can work out predictions of physical phenomena easiest this way. What you, and I in school, and most people, at first, are really complaining about is really two different things:

1. Weird new concepts: Probabilistic predictions, uncertainty, interference, discrete energy spectra...

2. The Hilbert space formulation, linear algebra, wave functions, weird differential equations describing them, solution techniques, paradoxes, linear algebraic stunts; along with a rough "path" to it starting from classical mechanics, as you outline.

Not much to say about the first: it is a fact of nature, the world behaves this way, intuitively or not, and, surprisingly, it was figured out right a century ago, by a blessed generation of intellectual heroes in our field. The second part, developed together with the first, is not ineluctable, however.

On another planet, far-far away, it could have turned out very different, and be replaced by an alternate formalism and path: phase-space quantization, eschewing Hilbert space and commutators, operators, etc... It "extends" classical mechanics by "correcting" Poisson brackets to Moyal Brackets, which add extra $hbar$-dependent pieces to them, associatively. (On our sad planet, this was only discovered in the 1940s, two decades after the Hilbert space formulation. The formulation is still technically demanding, so the Hilbert space formulation is still the mainstream, justifiably, but for the cri-de-coeur 's you voice...)

So all kinds of cultural difficulties involving newfangled operators for observables, and commutators never arise to add to the culture shock.

Ultimately, of course, the big enchilada is 1. Even classical phase-space function observables exhibit non commutativity, as they are usually composed by a special star-product operation, and probabilities flow and leak in manners radically different to classical phase-space flows, and the uncertainty principle rises even more magical and astounding than in the Hilbert-space formulation. But that is another story. Of course, all you hope to compute and predict is expectation values of observables. This is the heart of 1.

So, can you quantize unambiguously this way? Of course not. Quantization is a mystery. (Weyl, the godfather of this formulation, thought he'd found the true and only way to quantization, along this path, in 1927. Wrongly.) There are several different ways to consistently quantize many classical systems, and none is better than the rest, but depends on your specific physical system described. Some pick one path, others the other. (But they all have the same classical limit.)

The issue is that is that the fundamental problem in understanding quantum mechanics is, from a strictly logical point of view, posed backwards. We start with an understanding of classical physics, and want to discover quantum physics. But you cannot derive a more fundamental theory from a less fundamental theory. On the other hand, it is possible to derive classical physics from quantum mechanics, if only one formulates it correctly. But to do that, we first have to have a correct formulation of quantum mechanics.

Historically, canonical quantisation was important, because it enabled Dirac (who introduced it) to establish a correct mathematical formulation of quantum mechanics. Logically it is not so important, because the logical argument works in the opposite direction.

Dirac, and von Neumann, did give us another way to approach the problem, based on the Dirac–von Neumann axioms. From a mathematical point of view, these axioms are more satisfying, and they enable us to derive the canonical quantization relations (from the properties of Hilbert space) instead of imposing them. This changes the question, which becomes "why should we use Hilbert space?" The question was actually answered by von Neumann, but one thing von Neumann was not good at was explaining mathematics to mortals. The book is almost unreadable, and further attempts a elucidation "quantum logic" are not much better.

I wrote my published paper The Hilbert space of conditional clauses precisely to clarify what the mathematical structure of quantum mechanics means, and I hope it may give you more intuitive understanding. I have expanded on this, and filled in necessary detail, in my books (see profile).