How do transactions know how to behave based on the genesis block?

The genesis blocks of sender and miner do not need to be identical per se, as long as the transaction is valid, the address has enough ether, the nonce is one more than the last outgoing transaction, the chainId matches, and there is enough gas allocated for execution.

Unless the transaction fits the rules of one genesis block and not the other, it will work. The transaction itself does not reference the genesis block. The reason for the chainId and nonce is expressly to discourage these kinds of situations from being used maliciously in replay attacks. However, if you are able to coordinate getting all of these variables to match in the first place, you are not an attacker and have nothing to gain.

Also, note that if the sender and miner had different genesis blocks, they would be on different blockchains entirely. The miner would check if the sender has enough balance based off its known state, not off the sender's known state. Such transactions will be valid if they just so happen to match.

first you do not mine transaction but blocks that are block of transactions.

I believe from your question that there is confusion about three distinct concepts:

1. the network behavior: that is initialized in a Genesis block.
2. the creation of a new block (through proof-of-work, proof-of-authority, proof-of-stack) usually simply called mining. The current ethereum proof of work is described here https://github.com/ethereum/wiki/wiki/Ethash
3. the consensus algorithm that is about accepting (or not) the newly mined block being broadcasted through the network. Every node participates in the consensus although not every node is mining.

The Genesis Block defines rules for initializing the network itself. For example looking at

https://github.com/ethereum/go-ethereum/wiki/Private-network

we see parameters like the following:

• GasLimit (maximum amount of gas used in a block)
• difficulty (mining difficutly to enforce a more or less constant mining period of new blocks)

and here is how the geth client parses and knows about those parameters: https://github.com/ethereum/go-ethereum/blob/master/params/config.go