Why aren't most suspension bikes extremely long travel?

Put simply, a master of a craft will be better at it than a "jack of all trades".

A dedicated cross country race bike will be better for XC racing than an All Mountain bike. The same would be true of the dedicated downhill race bike. An All Mountain bike strikes a compromise in the middle, making it a good choice for someone who wants to do a bit of everything.

It's a bit more complex than that though, e.g. the Specialized Enduro makes different compromises to their Stumpjumper Evo. One is closer to the DH end of the spectrum, while the other is more of an XC/Trail machine. Cherouvim's answer points out some of the compromises (geometry, inefficiency, weight, flex, cost). Others include things like wheel size (XC racers might want a 29er, DH racers a 26er, and Enduro riders — who do a bit of both — might go for 27.5) and axle size (9mm QR is lighter but more flexible than 20mm thru). Which compromise you choose depends on what you want to ride!

The "do everything" bike you describe would be complex and compromised, but it does exist:

• Bionicon specialise in such bikes. Their "Edison" model has 160mm travel in the fork. At the flick of a switch, the fork adjusts down to 80mm and the frame changes its geometry (by lengthening the shock). It's complex (it needs very special suspension to enable the adjustment) and compromised (the shock and dual crown fork are significantly heavier than dedicated XC suspension, and you have to carry the extra weight of the adjustment equipment too).

• The Scott Genius LT offers 170mm travel in a lightweight carbon frame, along with an adjustable fork and shock. It doesn't change geometry, but offers three suspension modes (I think it limits travel). It's still too much of a compromise for a dedicated XC racer, who wants their bike to be even lighter, will probably never need all 170mm travel and probably wants 29" wheels and a front mech. It's also not far pushed enough in the other direction for the dedicated downhill racer. It's a compromise.

Additionally, many forks offer a travel adjust option (e.g. Rockshox Dual Position and Fox TALAS). They tend to be big, stiff and relatively heavy forks which work best in their longer travel mode, but can be lowered in order to aid climbing up really steep ascents. However, they're far too heavy for XC racers. If you made them light enough for XC racing, they'd be too flexible for the more gravity-focussed riders. Additionally, the adjustable spring can have an effect of the suspension's sensitivity. They're a compromise.

By the way, features such as Fox's CTD/ProPedal and Cane Creek's Climb Switch don't usually limit suspension travel. They firm up the shock's Compression (and in some cases Rebound) damping. This is to improve the pedalling efficiency of a longer-travel suspension design, in order to help with climbing, or riding on flatter, smoother terrain. They can still use all of the travel if they need to. The climb setting is a compromise.

Those compromises aren't necessarily a bad thing though: They enable me to ride my 150mm travel 30lb bike up and down some of the craziest terrain you can imagine. It'll probalby never win an XC or DH race, but it's still a fantastic bike.