What is a 'kinematic endpoint' in experimental particle physics?

First, one can make a short detour to the usual beta-decay: the plot below shows the allowed range of energies of electrons. The maximal possible energy, $E_{max}$, is driven by the difference in mass of neutron and proton (or rather of the original and final isotopes, if we consider nuclei): This point $E_{max}$ is often referred to as kinematic endpoint.

Your example differs by the fact that instead of nuclei, mesons are considered. The $B to X_c ell nu$ decay is a Cabibbo-allowed transition of a B meson to a charm meson (usually D or D*). The Q-value of this decay is driven by $m_B-m_D sim 3 GeV$. The $B to X_u ell nu$ decay is a Cabibbo-suppressed transition of a B meson to a light meson (pions, rho, etc). The Q-value of this decay is driven by $m_B-m_pi sim 5 GeV$, which is significantly larger than that in $B to X_c ell nu$ transitions. Thus, from the lepton energy point of view, the kinematic endpoint is indeed lower in $B to X_c ell nu$ decays.

In experiment, when studying such semileptonic decays, they can't reconstruct the neutrino. The notation $X_c$ and $X_u$ implies they are interested in inclusive measurement of all charm (light) hadrons. How could one do such a measurement? By looking at the lepton energy (or momentum) distribution.

One should remember that $B to X_c ell nu$ is Cabibbo-allowed (read, huge) compared to the suppressed $B to X_u ell nu$ transitions. The two can be difficult to separate because charm hadrons decay themselves to light hadrons. But if you plot the distribution of lepton energies coming from B candidates, in a proper frame, you will (I am over-simplifying now) see a two-fold structure: the huge bump coming from the $B to X_c ell nu$ and the smaller tail coming from $B to X_u ell nu$. The fact that the latter goes to higher energies (or, as you quote, has higher kinematic endpoint) can be used to separate the two types of decays and measure such quantities as CKM matrix element $V_{ub}$. To do that, one needs to know the theoretical distribution of lepton energies in $B to X_u ell nu$, and count how many of them are above the charm endpoint, but that is another story.

Addition: Sometimes the "kinematic endpoint" can also refer to the invariant mass of two particles rather than the energy of one particle. Say, the maximal invariant mass of $(ell nu)$ in your example is $m(B)-m(X_c)$.