What is the origin of leaving group ability?

For example, during an SN2 attack:

1. In the previous organic chemistry book that I am self-studying with, it says the fluorine atom is a better leaving group (become fluoride ion). This is because HF (having a conjugate base of F-) has a pKa of 3.1 compared to CH4 (having a conjugate base of CH3-) – pKa = 50. Therefore, F- is more stable when it leaves the molecule and remains in the solution, making a better leaving group.

2. In this lecture that I am now self-studying, at 17:50, I learn the choice of the leaving group of SN2 in a very different view. F is a better leaving group is because antibonding sigma R-F has lower orbital energy compared to antibonding sigma R-CH3, so much lower activation energy is required to break the bond and as a leaving group.

From my understanding, the downward trend of antibonding energies from C-C to C-F is because the corresponding bonding sigma energies have an upward trend. As the atomic radius decreases from F to C in the second period, the bond length of C-X (X = C / N / O / F) becomes shorter, hence bond energy increase. (with C-N bond somehow as a weird exception for me, what caused that?)

In the lecture, the professor says about the leaving group abilities (which I think is about pKa) is totally ‘fictitious’ and the antibonding orbital energy is the reason behind.

What is the true reason for F in C-F a better leaving group or CH3 in C-CH3? (thermodynamic or kinetic) I am extremely confused.

Also, if there is anything/concepts that I say it wrong in my deductions above, please kindly let me know. I really want the right concept. Thank you.