Is a cheque made out directly to the bank susceptible to fraud?

I would expect the mortgage account is in your name, and I would expect that the cheque should be made out to the name on the account that it is being deposited into - otherwise (exactly as you say), if it is just made out to the bank's name, it could be deposited into any account at that bank.

I can't speak to any specific protections that might otherwise be in place in the UK (I'm in Australia), but I was always instructed that my name (the account name) should be used on cheques, NOT the bank's name, otherwise it's effectively a cheque anyone can use.

I assume your mortgage and savings accounts are with different banks? Otherwise online banking would normally allow you to transfer between your own accounts with no risk of making a mistake. Otherwise, I would try giving internet banking a try, but maybe with a smaller amount first if you're concerned. You should be able to set up a payee with the details, and once you're happy it worked correctly for the smaller amount, just re-use those details to transfer larger amounts. I'd consider this safer than mailing cheques, but if you're still uneasy, I would at least try to deposit the cheque in person, and I would be very careful about mailing any cheque made out to the bank's name.

Why You Have to Pay This Way

At a slight tangent to the main question, it may be helpful to know why you are asked to pay off your mortgage this way.

UK bank accounts have a six-digit Sort Code (xx-xx-xx) and a usually-eight-digit Account Number. Standardization of this format started in the late 1950s to help automate cheque clearing, but is now also at the heart of the UK's electronic payment systems, BACS and Faster Payments. (Since 2018, all three are now under the pay.uk umbrella authority. Note: their website is https://www.wearepay.uk: "pay-dot-uk" appears to be an unrelated website to do with cryptocurrencies.)

Historically, Building Societies were outside the main banking system: they tended to be local organisations that people dealt with in-person using a Passbook. The building society would have accounts with one or more "high street" banks, in which the society's "collective funds" would be held, but individual members would only have a Roll Number by which the society would track each member's funds. From that Wikipedia page:

Because most building societies were not direct members of the UK clearing system, it was common for them to use a roll number to identify accounts rather than to allocate a six-digit sort-code and eight-digit account number to the BACS standards.

More recently, building societies have tended to obtain sort-code and account number allocations within the clearing system, and hence the use of roll numbers has diminished. When using BACS, one needs to enter roll numbers for the reference field and the building society's generic sort code and account number would be entered in the standard BACS fields.

Although, as noted above, building societies have moved towards "real" accounts (at least for "everyday" accounts), some still use roll numbers for mortgage accounts. As the OP was told, cheques destined for someone's mortgage will need to be made payable to the building society itself, with sufficient details on the back for the society to allocate the funds to the correct person.

Back On-Topic: Is This Susceptible to Fraud?

Historically, I would have said the risk was very low. To process the payment, someone in the building society would have looked on the back to enter/check the details of the account the cheque was destined for. To "divert" such a cheque, the instigator would need to cross-out the original details and write in their own. One would hope that the society's employee, on seeing an alteration, would not blindly enter the new details without seeking further confirmation.

Nowadays, if such cheques are processed automatically, there is probably an increased risk, although even here one would hope the ICR process would be programmed to detect alterations¹ and send the cheque for manual verification.

Limiting the risk are (a) the perpetrator would have to have a mortgage with the same building society, and (b) the ability to easily trace where the money went should mean all but the stupidest would not make the attempt.

---

¹ ICR (intelligent character recognition) is the more advanced form of OCR (Optical...) for handwriting – as the details on the back of a cheque are likely to be. If the original details were merely "crossed through", then ICR should still detect that "there is something there" (albeit with a probably-reduced accuracy of what they say). Similarly, fully blocked-out areas ("█████████████████") should be detectable. I would consider it essential that any ICR system for cheques rejects any item with more than one set of details, or that contains blocked-out areas.