How does RSA signature verification work?

How does RSA signature verification work?

In unencrypted communication between A and B RSA signatures (without utilizing hash functions) are simply the message $m$ encrypted with sender's private key $d$ (not $e$ as in usual encrypted communication): $c equiv m^d pmod n$ which enables the receiver to verify a message by 'decrypting' it with public key $e$ and comparing the result to unencrypted message: $m equiv c^e pmod n$.

This procedure is supposed to ensure integrity in communication as the receiver (if the message matches the 'decrypted' signature) can be sure the messages is valid as only the sender is capable of computing signatures 'decryptable' using encryption exponent $e$.

Why is it common practice to create a hash of the message and sign that instead of signing the message directly?

RSA signatures can be forged or spoofed in certain scenarios which is shown in the following demonstration:

• A and B are communicating on an unencrypted channel and attacker E is able to capture their messages $m$. However, they send RSA-signatures $c$ along with every package and therefore forged messages sent by E will be filtered.
• Assume A sends messages $m_1$ and $m_2$ with signatures $c_{1,2} equiv m_{1,2}^d pmod n$ and E captures them.
• E is now capable of generating the valid signature for a message $m_x$ for which $m_x = m_1 cdot m_2$ must be true. This is feasible without private key $d$ because signature $c_x$ of $m_x$ can be calculated by multiplying signatures of $m_1$ and $m_2$: $$m_1^dm_2^d equiv (m_1 cdot m_2)^d equiv m_x^d equiv c_x pmod n$$. This is possible due to homomorphism of RSA / module: $$(m_1 cdot m_2) bmod n = ((m_1 bmod n cdot m_2 bmod n) bmod n)$$ Simply put homomorphism means $f(xy) = f(x) cdot f(y)$.
• Therefore E can send a message to B which will pass integrity and authenticity test possibly causing errors or leading to confusion. However, E is not able to change contents of $m_x$ as $m_x = c_x^e$ must remain true.

Concluding RSA encrypted messages as signatures can be insufficient depending on the scenario, thus hash functions are commonly used in digital signature generation and additionally @poncho's answer is of relevance too.

Wikipedia articles on homomorphic encryption and module homomorphism dive into detail of this aspect of RSA encryption:

https://en.wikipedia.org/wiki/Homomorphic_encryption

https://en.wikipedia.org/wiki/Module_homomorphism

The important part and this is where I really started scratching my head: How can the recipient verify that I own the private key if the public key seems to be enough to recreate the signature?

You can use public key to "encrypt" (or "decrypt" which is same in "textbook" RSA) the signature and get hashed message. If the hashed message equals hashed message, then you verified the message being correctly signed.

You cannot use public key and message to recreate a signature that can pass the above verification though.

P.S. For "textbook" RSA, I mean https://www.cs.cornell.edu/courses/cs5430/2015sp/notes/rsa_sign_vs_dec.php

Why is it common practice to create a hash of the message and sign that instead of signing the message directly?

Well, the RSA operation can't handle messages longer than the modulus size. That means that if you have a 2048 bit RSA key, you would be unable to directly sign any messages longer than 256 bytes long (and even that would have problems, because of lack of padding).

In contrast, a cryptographical hash can take an arbitrarily long message, and 'compress' it into a short string, in such a way that we cannot find two messages that hash to the same value. Hence, signing the hash is just as good as signing the original message; without the length restrictions we would have if we didn't use a hash.

The important part and this is where I really started scratching my head: How can the recipient verify that I own the private key if the public key seems to be enough to recreate the signature?

What made you think that the public key is enough to recreate the signature? It is sufficient to verify a signature that you're given, but it is not sufficient to generate new ones (or so we hope; if that's not true, the signature scheme is broken).

If you're using RSA, the signature verification process is (effectively) checking whether:

$S^e = operatorname{Pad}(operatorname{Hash}(M))pmod N$

Definitions: $S$ is the signature; $M$ is the message; $e$ and $N$ are the public exponent and modulus from the public key; $pmod N$ means that equality is checked modulo $N$; $operatorname{Pad}$ is the padding function; and $operatorname{Hash}$ is the hashing function. Note I say "effectively" because sometimes the padding method is nondetermanistic; that makes this check slightly different, but not in a way that matters for this discussion.

Now, if we were trying to forge a signature for a message $M'$ (with only the public key), we could certainly compute $P' = operatorname{Pad}(operatorname{Hash}(M'))$; however, then we'd need to find a value $S'$ with:

$S'^e = P' pmod N$

and, if $N$ is an RSA modulus, we don't know how to do that.

The holder of the private key can do this, because he has a value $d$ with the property that:

$(x^e)^d = x pmod N$

for all $x$. That means that:

$(P')^d = (S'^e)^d = S' pmod N$

is the signature.

Now, if we have only the public key, we don't know $d$; getting that value is equivalent to factoring $N$, and we can't do that. The holder of the private key knows $d$, because he knows the factorization of $N$.