Formula like Kelly criterion for choosing how much of an investment to liquidate?

There is a simple answer: Look at your stocks and decide whether you would buy them now if you did not own them yet.

If your answer is "why should I pay this crazy price?", it is probably time to sell. If it still looks like a good investment, keep it.

I don't think the Kelly criterion is something most investors should consider for portfolio allocation. The counterpart is the efficient market hypothesis. I don't know how efficient markets actually are, but they are probably efficient enough that most investors can't apply the Kelly criterion in a meaningful way!

Instead, I would take a look at guidelines for portfolio rebalancing. Rebalancing less frequently lets your winners keep winning for awhile. You should find good advice here and via Google searches about how often you should rebalance.

In addition to rebalancing, you could also decide to change your target portfolio allocation, and this is akin to Manziel's answer.

If you divide the market into asset classes and study the long-term returns and covariances of those classes, then a Kelly-like formula will tell you the optimal fractions of your portfolio that belong in each of those classes.

For example, you could decide that you want to invest in a mix of S&P500 stocks and intermediate term government bonds. (Hopefully through low cost ETFs.) Your Kelly analysis will tell you something like: "The optimal investment is 70% stocks and 30% bonds."

Great, now you have a strategy. You might also notice that the optimal peak is broad. It doesn't matter significantly if your ratios slip out of tune by 5% either way.

So now invest your capital accordingly, and revisit it every six months, or even annually, and "retune" your portfolio. Sell and buy enough of each class to move back to your optimal split.

You will be selling off a part of the winners when necessary, just as you felt you should. You will always be nearly optimally invested.

To expand on @Menziel's answer, you have to consider opportunity cost, taxes, and your personal wants and needs.

1. Opportunity cost is what Menziel was leading towards. If you have a fixed amount of money today, would you rather invest in the company you already own or a different company/asset that you think will do better. That said the world is uncertain, so your judgement on which company does better will be fuzzy. Particularly there's a chance the asset you have will do better than any other asset you know, so you have to incorporate that thesis in whether you decide to sell and how much.
2. Taxes are the thing that Menziel disregarded in his answer. However they have a disproportionally large effect the more the asset that you own has increased since you bought it. For a concrete example imagine you held bitcoin since it was worth pennies and you decide to sell now and pay nearly 20% of the value in capital gains taxes. If you have $100 now then $20 goes to taxes and $80 goes to a second investment. If you invest in a different asset and it doubles, you'll have in the end $144 ($160 - $16 new taxes), however if you held on to the bitcoins and bitcoin doubled, you'd have $160 ($200 - $40 taxes). The tax "penalty" means the competing investment opportunity needs to be 20% better.
3. Finally there are personal wants and needs. If you want a fancy car now and you have the money for it from investments, then obviously you could sell stocks and buy the car. Or worse case you or your spouse gets cancer so you have to liquidate to cover treatment... This point is obviously personal preference or circumstances.

To reiterate and expand on gaefan's answer. You set a target portfolio allocation. Periodically you examine the values of the allocations. Cull the high values (take profit!) and distribute the gains to lowest value allocations. This allows you to sell when high and buy low, without trying to time the market. You simply, on a pre-planned take take profits and buy-in to undervalued stocks.

For example you might invest in 3 stocks. And you allocate them evenly 33% 33% 33%. When you compare their values 3 months later you see that they are now 50% 30% 20%. You would then sell off enough of the 1st stock to reduce its total value to 33% of your total investment and then use the cash proceeds to buy into the 2nd and 3rd stocks to get them up to 33% of your total investment.

The Kelly Criterion implies you have some sort of estimate on these probabilities when in reality nobody knows this. There would be a ton of variables involved in estimating the probabilities of equities being at certain values and at the end of the day they would never be definitive numbers.

You should ideally be searching for some sort of probability agnostic allocation formula which I'm not sure even exists (or at least to the extent that it can be mathematically proven as being the best possible action like Kelly's derivation makes clear). The efficacy frontier is probably your best bet as someone mentioned.

If you're using options at all and want to maintain certain a predetermined risk profile, you can look into what's called "beta weighting".