Using different hashing function with ECDSA

I will first address the issues of the diagram;

Encryption part

Although the encryption is mentioned as optional

1. there is no mention of how the AES key is generated. The common method is the Diffie-Hellman Key Exchange and the Elliptic Curve version of it is preferred ECDH.

2. there is no mention of the mode of operation. CBC, CTR, GCM,... etc.

3. There is no mention of the nonce/IV generation. The nonce generation is crucial;

   • In the CBC it must be unpredictable
   • In the CTR and GCM the (IV,key) pair should be never use more than once.

ECDSA part;

1. The naming is completely wrong. We use signature and verification and those processes are completely different than encryption and decryption, even in the RSA. In short RSA decryption is not signature.

2. The verification part is wrong. Only the signature part is sent, the signature and message must both exist during the ECDSA verification. They tend to make it symmetric but it is wrong.

Questions

I'm wondering must the ECDSA pair with the SHA Hashing function?

They didn't show which curve is used there. If a curve like P521 or Goldilocks is used the one may need XOFs like SHAKE128-512 series, or use SHA256 as CTR mode to generate more than one block to support all of the curve needs.

Alternatively, what if using the AES-GCM to generate both CIPHER & TAG? The GF generated TAG later go through the ECDSA to be encrypted with the private key.

The AES-GCM doesn't produce a digital signature, they can provide confidentiality, integrity, and authentication only if correctly used. AES-GCM can not provide non-repudiation. For digital signatures, you need a digital signature algorithm.

Or will anything go wrong with the scheme I describe above?

If there is no need for a digital signature, then it can be fine. The AES-GCM has a better alternative; AES-GCM-SIV is a nonce misuse resistant scheme. This can only leak the same message is sent again if the same message is sent again with the same (key,IV) pair is.

Or will anything go wrong with the scheme I describe above?

Is the AES key public? If it is, then it is easy to generate two messages that have the same tag (and hence would act like a collision).

If the AES key secret to both the signer and the verifier? If it is, well, why bother with ECDSA at all; if you have a shared secret key, you can use a Message Authentication Code (e.g. HMAC, CMAC, GMAC) which is far more efficient than ECDSA (or any other public key signature operation).