Collapse of the wave function in non-discrete systems

It depends on the type of measurement you perform. The idealized case is described by the PVM (projector valued measure) uniquely associated to the observable viewed as a selfadjoint operator through the spectral theorem. The PVM is a collection of orthogonal projectors $P_E$, where $E$ is a Borel set of the real axis, typically a finite interval, defined in the practice by the precision of the instrument. This collection of projectors satisfy some mathematical properties similar to those of a positive measure.

If the initial state is represented by $psi$ and the outcome is $E$, the post-measurement state is always described by the vector $P_Epsi neq 0$ up to normalisation.

Here $||P_Epsi||^2$ is the probability to obtain the outcome $E$ when the initial state is represented by the normalized vector $psi$.

All that is nothing but the Luders-von Neumann postulate.

If the spectrum is continuous, single points $E={lambda }$ have automatically zero projector $P_E=0$, so that "non-normalizable vectors" cannot be produced this way.

For instance, for the position operator, if the position measurement produced the outcome $E=[a,b]$, the corresponding projector is $$(P_E psi)(x) := chi_E(x) psi(x)$$ where the function $chi_E$ is zero outside $E$ and $1$ in $E$.(Notice that if $E$ is a single point, the associated projector is zero since single points have zero Lebesgue measure.)

It is worth stressing that this description is valid also when the spectrum is a point spectrum. In that case single points (eigenvalues) have non-zero projectors: the ones onto eigenspaces.

A more realistic description is provided by a POVM (positive operator valued measure) and its decomposition (it is not unique) in terms of Kraus operators, but also this description does not give rise to non-normalized state vectors.