QGIS mean error - RMSE - QGIS georeferencer

Looking at the source code for qgsgeorefmainwindow.cpp, function QgsGeoreferencerMainWindow::calculateMeanError in particular, the calculated "mean error" is the sum of the squares of the residuals divided by the number of points minus the minimum points required to specify the transform, i.e. adjusted for the degrees of freedom.

I haven't redone your manual calculation point by point, but a poly 2 transformation is specified by a minimum 6 points, and 4.48^2*15 approximately equals 5.79^2*(15-6), i.e. your sum of squared residuals is about 301 and the difference is the denominator.

The rationale here is that the total (squared) error should be considered not as spread over all (enabled) georeferencing points, but only over the excess points that overspecify the distribution and contribute to "error". There is of course no actual differentiation between "first" points that define the transform and "excess" points that have errors, but to have the RMSE scale properly as you add or subtract points, or be otherwise comparable between different data fitting, you need to make the adjustment.