How does paying off the mortgage work if I demolish a home and rebuild another home on the property?

1. The existing mortgage is for that house on that land.
2. It's (almost certainly) an "owner-occupied" mortgage.

Thus, you'll have to talk to the bank. They'll want you to refinance the property (which pays off the original loan) with a new "commercial" mortgage (probably at a higher interest rate).

A mortgage is a secured loan, like a car loan.

A secured loan has collateral - the thing of value that backs up the loan. When the collateral stops having value, the loan principal is immediately due in full. Normally, insurance takes care of that.

In practice, if they catch you, things could turn ugly for you. They could immediately call the note. Now you have to rush around for a construction mortgage that will both pay off the note and fund the construction. They don't owe you any cooperation; they could foreclose at the worst possible time, and then go after you for their net losses, since you deliberately destroyed the collateral without telling them.

The correct path is to draw up your plans and then get a new construction mortgage, which pays off the old mortgage and finances demolition and construction. The lender will perform due diligence and make sure that your plan makes sense. If it doesn't, they won't issue the loan.

By the way... there are severe limits to how much DIY work you can do in a rental unit. Expect to have to hire all work done.

However... sometimes this can work. It's not unheard of for properties to be worth more with the house gone. That happens when desirable land with a house that's not a good fit for what the neighborhood has become. You still can't "just do it", but the lenders may be OK with the change.