counterfactual if-clause and gerund

It appears that counterfactual if-clauses can modify gerunds, as the following demonstrates:

A: What would you do if you were a billionaire?

B: I would buy a caslte. Buying a medieval building if I were megarich sounds so great.

Does the if-clause modify the gerund phrase alone or the whole sentence? Do you have any arguments for your stand?