Can option writers "buy to close" if holders refuse to sell?

You don't necessarily buy your option back from the original buyer. You'll buy from someone else on the open market (or possibly a "market maker") and the clearing house will connect the new seller with the original buyer, so you are no longer part of the transaction.

So whether the original buyer wants to sell or not is irrelevant.

If the buyer exercises the call/put before expiry, then you do get exercised and must sell/buy your shares. But it is very rare that it is optimal to exercise an option before expiry, so early exercise may be to your benefit since you get out of a position that you have a larger unrealized loss on.

On expiration day, any time that the underlying is higher than the strike price of the call or lower than the strike price of the put then it is ITM. When that happens, it is possible that the option owner exercises intraday.

As an aside, American options are exercised early if:

• They are ITM and the bid trades at a discount to the intrinsic value and
• There is a pending dividend which offers an arbitrage.

To avoid this possible early assignment, you need to understand and recognize these situations.

It's improbable that you and the buyer are the counterparties when you BTC your short option. New contracts are created, contracts cease to exist (BTC + STC) and existing contracts change hands many, many times during their lifespan.

The market maker is always there, offering you the ability to buy or sell. You may not like his prices and you may have issues if you have a huge position but he is obligated to make a market and trade.