What is the formula for rebalancing a portfolio?

You have to adjust some of the numbers in your question.

I have a portfolio of two assets. I've invested X% in asset A and Y% in asset B.

For example, say X = 50, Y = 50, and A grows by 10 %. Then my new portfolio is 60 % in A, and 50 % in B, meaning I've gained an extra 10 %, which I then split equally so that my rebalanced portfolio is 55 % in A, 55 % in B.

Lets assume that when specified in $'s each investment is worth $5,000 with a total value of $10,000.

Now if A grows by 10% that could mean that the split is now $5,500 for A and still $5,000 for B. So A is now 52.38% of the total value and B is now 47.62%.

What you do from here is that when you want to re-balance, you either sell some of A or you add more to B the next time you put more money into your investment account.

If you want to sell A then you need to sell ($5,500 - $5,000)/2 or $250 of A and then buy $250 of B. The formula is (Value of 1 - Value of 2)/2. Of course that works if there are two investments and you want them to be equal to each other.

But lets say you want them to be in a 70% /30% split. In this case the value of A grew by $1,000 and B didn't change. So A is now $8,000 of $11,000 or 72.727% and B is 27.273%. A now makes up too large a portion by a couple of percentage points.

To calculate how much to sell of A just figure out in $'s how much over the value is. So (72.727% - 70%) * $11,000. That equals 2.727%*$11,000 which is $300. So you sell $300 of A and buy $300 of B.