Protecting firmware .bin from reverse engineering

In order to prevent reverse-engineering of their Management Engine, Intel used:

• An obscure CPU architecture (not their own)
• an exotic compression algorithm with hardware-backed dictionary
• a less-known OS framework

Ultimately, they failed (well, almost). But they kept secret the internals of the ME for ~10 years.

@Marcus is completely correct about using encryption, and how the key has to be stored somewhere safe and possibly used in conjunction with secure boot. As far as tools go, there are many symmetric and asymmetric encryption schemes that may be suitable for your use case.

However, you can't protect 100% from someone obtaining your firmware. You can only increase the difficulty of reverse engineering such that the effort required outweighs the potential reward of obtaining it. Of course, this will depend on the type of adversary; a casual user may give up immediately after seeing that it is encrypted, but a professional outfit may spend much more time and money until access is gained.

So, you can tailor your defenses to what level of threat you anticipate and balance that with what you are able/willing to spend. If you only want to protect against a casual user, it may be sufficient to only encrypt the firmware in transit, and have the firmware decrypted upon receipt using a key stored in firmware. This method requires very little investment on your part, but it also may fall over pretty quickly for someone who is more determined.

Of course, you can step up to using secure boot with a hardware security module, using the keys to decrypt firmware only on boot, but as mentioned, this is an architectural decision and will possibly incur more time/cost.

The problem is, even if you store the firmware encrypted in the best way possible and make the keys extremely hard to recover from the hardware, you also have to do everything else right. All it takes is one vulnerability in your software for a user to get a shell on your system, from where they could dump the firmware. Or, maybe there are debug ports (JTAG/SWD/UART) left on the board itself that could allow access with a little hardware hacking.

The point is, once your system is physically in someone else's hands, it's a losing battle from there. So maybe it's time to ask; what is it that you are really worried about being reverse engineered? If it's for security purposes, consider searching for and fixing vulnerabilities internally instead of hoping to hide them with security through obscurity. If you are trying to protect proprietary code, you could protect it through legal means instead of hoping nobody recovers it.

What you want is impossible; there's no "tool" that can do that.

When you encrypt your file, and then only someone in ownership of the key can get the contents – that's the point of encryption. Since you still want to use it, you, however, need to supply that key to the device that will use your image. Can't include the key anywhere unencryptedly in a file, otherwise everyone will trivially be able to decrypt your image.

The only way encrypted images work is with hardware support for securely hardware-stored keys from the boot away. That's not a "tool", that's an architectural choice with quite big consequences for your overall boot process.

You'll probably want to look into what "Secureboot" is.