Gifting The Same Asset Back & Forth - 2x Exemption Used? Or $0 Exemption Used? (U.S.)

I have a question about U.S. estate/gift tax law.

Imagine one person gifts an asset – i.e. a $1 million house – to another, & files a gift tax return in the year in which it was gifted. Then several years later, the recipient wants to gift the same house back to its original owner. Would this second transfer "negate" the original gift (for gift tax purposes), by virtue of the fact that it’s the same asset – so in the end, neither party have used any of their lifetime gift exemption (currently ~$11 million)? Or would both parties have used up $1m of their lifetime gift exemption (even though the original asset is back with its original owner)?