Can my sibling double his portion of our inherited property by buying us out and then selling?

You’re thinking about it wrong. To make the maths easier, assume the estate has just $600K in cash and the $300K property. If you each inherit equally, you each end up with $200K and a third of the property. If one sibling buys out the others, then he ends up with the $300K property and no cash, and the others each have $300K in cash. So you all end up with something worth $300K.

Imagine if there was no money in the inheritance, just the house. However your brother has his own $200k account with money he saved from his day job. He starts with $200k, inherits $100k worth of house, total $300k. He pays you and your third sibling $100k each, now he owns an entire house worth $300k but his bank account is $0, total is still $300k. He sells he house and now he has $0 worth of house but $300k in his bank account, total is still $300k.

Your scenario is the same except that the bank account was funded with the inheritance instead of his day job.

Ignore the inheritance, and for a moment ignore the fact that your brother would be on both sides of the sale.

You and your two siblings each own an equal third of a $300,000 property ($100,000 each). A buyer gives you $300,000 in exchange for the property. You now do not have the house, but you each have a third of the $300,000 payment ($100,000 each). So you each are now -1/3 house, +$100k. The buyer has -$300k and +1 house.

Now consider the buyer being one of the three sellers. As the buyer, he is -$300k and +1 house. As a seller, he is +100k and -1/3 house. Together, he is -$200k and +2/3 house (having paid $200k to you and the other sibling in exchange for your 2/3s of the house). From his starting position ($0k and 1/3 house), this leaves him with 1 house and -$200k. If he then sells the house to someone else for $300k, we add in +$300k and -1 house, bringing him to $100k and no house, the same as the other two siblings.

---

You say he theoretically gets 100k of his share from the property (yes, it’s still IN the property) and after buying us out he will be theoretically negative 100k. (100k-200k= -100k), but this isn't right. Either he pays each of you $100k for your shares (pays $200k and receives $0), or he pays $300k for the whole house (pays $300k and receives $100k). If he only pays $200k, and you and your other sibling each get $100k, there is no additional $100k for him to receive.

This is very simple. The 3 of you need to split a $2M estate (or whatever the TOTAL value of everything is). So that's about $670k each. If one of you wants the house, then they can take the house worth $300k, plus another $370k on top of that. The other two just get $670k in cash. And that's it.

Just divide up the total value (including all property and cash) and divide by three. Then each of you can take whatever you all agree on, as long as it adds up to the total value divided by three.

You missed the initial balance in your calculation when he sells the house:
Here it is in long form.

Networth before: 0.

Networth after he buys out siblings: -100k.
100k - 100k (Bankloan1 to sibling A) - 100k (Bankloan2 to sibling B) = -100k

Network after he sells the house: 0
-100k + 300k (= 200k) - 100k (payback bankloan1) - 100k (payback bankloan2) = 0

※ You can substitute the bankloan with part of the cash inheritance, but it doesn't change the net result.

You start with $2 million total inheritance. That means $1.7 million in cash, and $300k in the house. I think this is the part that you're missing - the house is part of the total inheritance, not anything on its own.

You each get $567k in cash, plus a third share in the house.

Then your brother buys you and your other sibling out. That's $100k for you, and $100k for your other sibling.

Your brother now has $367k in cash, plus a house worth $300k. You and your other sibling now have $667k in cash. Everyone's square, and no-one's lost out.

Of course if your brother has done a side deal with a property developer and immediately sells it for $500k, then he'd be a weasel. Clearly the house would have been worth more than $300k at the time it was valued, which legally would give you a potential claim against both your brother and the agency which valued the property. (Although realistically the only people who'd win there are the lawyers. You don't need the extra cash, and you'd be better cutting your losses and cutting a toxic person.)

But if he lives in it for a while and then sells it for $500k a few years down the road, that was his choice for how to invest the inheritance, and either of the rest of your could have done the same with a different property with your chunk of cash. (See the Parable of the Talents, for example.) Equally the roof could fall in tomorrow and he has to spend $50k fixing it up, and that's on his dime too.

Others have shown how to do the math correctly. I'll try to show where, exactly, you had it wrong.

Lets break up the equations in your sentence:

he theoretically gets 100k of his share from the property (yes, it’s still IN the property) and after buying us out he will be theoretically negative 100k. (100k-200k= -100k)

But now he has another 2/3 of the house. So, balance 1 (B1) is:

B1 = 1/3 house (held initially) - 200k (buyout) + 2/3 house (just bought)

If you consider 1/3 house = 100k, (as you did) and sum it with the -200k, the real balance will be:

B1 = 100k (held initially) - 200k (buyout) + 2/3 house (just bought)
B1 = -100k + 2/3 house                  (Equation 1)

Or, alternatively, if you sum the house fractions:

B1 = -200k + 1 house                    (Equation 2)

See, both B1 above are equivalent. Either he is only $100k in debt, but has 2/3 of a house, or he had a full house and a $200k debt.

You have chosen the first, but omitted the share of the house he now have, which is only 2/3, because you have considered his original 1/3 as part of the buyout payment.

But say he immediately sells the house for 300k.

So, he gave up a whole house, and got back for it 300k:

B2 = B1 - (1 house) + 300k

Replacing B1 in this new equation:

B2 = (-100k + 2/3 house) - 1 house + 300k
B2 = 200k - 1/3 house
B2 = 100k

See? He only got $100k, as expected.

Now the mistake you made is that, after the buyout, you considered he owned a full house, like in Equation 2, but had only a -100k debt, like in Equation 1. Both can't be true, because you can't use the 100k he initially had in the house to offset the 200k he spent buying the other two thirds, and still have the original one third. In your calculations, the original third was spent.

Your thinking is way off. If he buys you out at 100k each, he started with 100k worth of the house and now owns a 300k house and is out 200k cash for no change. If he sells the house he is now +100k cash as are the two of you. This is the appropriate accounting if the house is worth 300k.

The point that there are costs involved in selling a house is important, but on another scale. An appraisal assumes you have a willing buyer and willing seller. The alternative to one of you buying the others out is to sell the house, netting 300k less the costs of sale. If the costs of sale are 10% or 30k, you would only get 90k from the sale of the house. There is a negotiation to be held that should be between 90k and 100k. Is it important to keep the house in the family? If so, maybe you settle lower in the interval or even lower than 90k.