capital gains tax on house sale

The capital gain is 900k-500k, because they get no benefit from any "primary residency" that transpired more than 5 years before the sale.

Determine whether you meet the residence requirement. If you owned the home and used it as your residence for at least 24 months of the previous 5 years, you meet the residence requirement. The 24 months of residence can fall anywhere within the 5-year period, and it doesn't have to be a single block of time.

https://www.irs.gov/pub/irs-pdf/p523.pdf

Imagine that a married couple buys a house on January 1st 2001 for 500K and lives in it for 5 years. On January 1st 2006 they rent out the house. The market value of the house is 700K at that point. The house is rented out for several years and on January 1st 2011 the couple sells the house for 900K.

Is the capital gain 900K - 500K or 900K - 700K (the house value at the time it stopped to be their primary residence)?

I am not looking at precise calculations - just trying to understand if the calculations goes back all the way to the initial purchase price or just back to when they converted it to a rental property.

The answer is not so simple.

On January 1st 2006, the homeowners put the property in service as a rental. At that time they set the basis of the property. They then each year depreciated the property when they figured their income tax each year.

Of course when they calculated the value for depreciation, that is the house only, not the land.

from IRS Pub 527 Residential Rental Property

Separating cost of land and buildings.

If you buy buildings and your cost includes the cost of the land on which they stand, you must divide the cost between the land and the buildings to figure the basis for depreciation of the buildings. The part of the cost that you allocate to each asset is the ratio of the fair market value of that asset to the fair market value of the whole property at the time you buy it.

If you aren’t certain of the fair market values of the land and the buildings, you can divide the cost between them based on their assessed values for real estate tax purposes.

Example

You buy a house and land for $200,000. The purchase contract doesn’t specify how much of the purchase price is for the house and how much is for the land. The latest real estate tax assessment on the property was based on an assessed value of $160,000, of which $136,000 was for the house and $24,000 was for the land. You can allocate 85% ($136,000 ÷ $160,000) of the purchase price to the house and 15% ($24,000 ÷ $160,000) of the purchase price to the land. Your basis in the house is $170,000 (85% of $200,000) and your basis in the land is $30,000 (15% of $200,000).

Of course there is the complication that the house was not a rental for the first 5 years of ownership.

also from IRS Pub 527:

Basis of Property Changed to Rental Use

When you change property you held for personal use to rental use (for example, you rent your former home), the basis for depreciation will be the lesser of the fair market value or adjusted basis on the date of conversion.

Fair market value.

This is the price at which the property would change hands between a willing buyer and a willing seller, neither having to buy or sell, and both having reasonable knowledge of all the relevant facts. Sales of similar property, on or about the same date, may be helpful in figuring the fair market value of the property.

Figuring the basis.

The basis for depreciation is the lesser of:

• The fair market value of the property on the date you changed it to rental use; or

• Your adjusted basis on the date of the change—that is, your original cost or other basis of the property, plus the cost of permanent additions or improvements since you acquired it, minus deductions for any casualty or theft losses claimed on earlier years' income tax returns and other decreases to basis. For other increases and decreases to basis, see Adjusted Basis in chapter 2.

Example.

You originally built a house for $140,000 on a lot that cost you $14,000, which you used as your home for many years. Before changing the property to rental use this year, you added $28,000 of permanent improvements to the house and claimed a $3,500 casualty loss deduction for damage to the house. Part of the improvements qualified for a $500 residential energy credit, which you claimed on a prior year tax return. Because land isn’t depreciable, you can only include the cost of the house when figuring the basis for depreciation.

The adjusted basis of the house at the time of the change in its use was $164,000 ($140,000 + $28,000 − $3,500 − $500).

On the date of the change in use, your property had a fair market value of $168,000, of which $21,000 was for the land and $147,000 was for the house.

The basis for depreciation on the house is the fair market value on the date of the change ($147,000) because it is less than your adjusted basis ($164,000).

Which then brings you to the fun calculation when the house is sold, and the IRS wants to recapture the depreciation.

So knowing that the house entire property was worth $700K isn't enough. There will have to be a split of the value between house and land at the time of purchase and the time it became a rental property. Then the house is depreciated for 5 years. All this assumes that there were no other events that changed the basis, or have to be depreciated on their own.

Note: Pub 527 has changed along with tax law, so it is possible that items in the publication don't apply to the question because of recent tax law changes.

The capital gain would be $900k - ($500k - depreciation that was taken or should have been taken), assuming no improvements were done for simplicity. From https://www.irs.gov/publications/p544#en_US_2020_publink100072284:

Property Changed to Business or Rental Use

You can deduct a loss on the sale of property you acquired for use as your home but changed to business or rental property and used as business or rental property at the time of sale. However, if the adjusted basis of the property at the time of the change was more than its fair market value, the loss you can deduct is limited.

Figure the loss you can deduct as follows.

1. Use the lesser of the property's adjusted basis or fair market value at the time of the change.
2. Add to (1) the cost of any improvements and other increases to basis since the change.
3. Subtract from (2) depreciation and any other decreases to basis since the change.
4. Subtract the amount you realized on the sale from the result in (3). If the amount you realized is more than the result in (3), treat this result as zero. The result in (4) is the loss you can deduct.

Gain. If you have a gain on the sale, you must generally recognize the full amount of the gain. You figure the gain by subtracting your adjusted basis from your amount realized, as described earlier.

This also matches the basis for depreciation ($500k), although as noted in another answer, that must be allocated between land and buildings, and only the latter can be depreciated.

Various websites say the same. The Tax Adviser:

If a residence converted to rental property is later sold at a gain, the basis in the converted property is the original cost or other basis plus amounts paid for capital improvements, less any depreciation taken.

Merriman:

Now that we have investigated potential capital gains tax exclusions and issues like depreciation recapture that is recognized first on your rental, we’ll break down how to determine your adjusted cost basis for calculating gains on the sale of your property. Your adjusted basis is typically the original purchase price of the home, plus improvements made, minus depreciation on the property.

The Section 121 capital gains exclusion can still be used for a primary residence converted to a rental property assuming all the conditions are met, but they are not here as it has been more than 5 years since it was the owner's primary residence. So the owner will realize the entire gain.