How can I tell if a rim is bent or if the wheel is just out of true?

The best advice here is to take your wheel to an accomplished wheel builder who should be able to true your wheel if it's possible or rebuild it if it needs it.

Truing wheels is one of those things that's more Art than Science and once it starts going wrong, it just gets worse.

http://sheldonbrown.com/tooltips/truing.html

Well, sometimes the rim is visibly bent, especially if there are creases in the sides. But if not visibly bent, and you don't have an experienced eye, then you have to attempt to true the wheel and see if it trues up well.

Some minor dents in a rim can be repaired with the proper equipment, but others require rim replacement.

Beyond that you need to read a book.

(It should be noted that if the wheel is not particularly expensive then this is a good opportunity to learn wheel truing. If you're reasonably careful (and use the right spoke wrench) you won't screw it up much worse, and, unless you true it, you're going to have to take it to the shop anyway. So a good time to give wheel truing a try.)

And in response to @WTHarper:

It helps to know how the wheel got cocked up in the first place. If it was hit by a car then the odds of a bent rim are pretty high. If it "just happened" -- the wheel went out of alignment on its own -- then a bent rim is very unlikely (and a broken spoke is likely). If it happened in a "soft" fall -- eg, the wheel slid out from under you on loose gravel -- then a bent rim is possible but unlikely. It's more likely that some spokes were just stretched out. If it happened in a hard fall or when hitting an obstruction then a bent rim is more likely but far from certain.

(And I should have mentioned checking for broken spokes: First check that all spokes are tight, and if any are loose check whether they are in fact broken. It would be very unusual for a wheel to not "taco" slightly when a spoke breaks, and replacing the spoke should get it back in true with relatively little effort.)

The cause of your question should probably determine if the rim is bent or just needs "adjusting" by a qualified wheel builder. If contact with a vehicle or other solid object is the cause of the question, then there's a good chance the wheel is actually bent and will likely need to be entirely re-built or replaced ( replacing may be more cost effective ). If you just discovered the wheel seems to be "wobbling" in a certain spot or is rubbing the frame or forks in a certain area, the problem is likely a wheel rim that needs trueing by spoke adjustment or spoke replacement in a certain area of the wheel rim. In either case, in the interest of safety and rim longevity, I'd suggest having a knowledgeable wheel builder at a bicycle shop take a look at the wheel. That's better than worrying about a possible accident caused by a bad wheel.

If you remove the tire and loosen all spokes completely, or better remove them as well, and put the rim on a flat surface, it should become clear if the rim is bent or not. Obviously this requires some work which may be unnecessary if the wheel simply needs truing, so I'd try that first. If it fails and you are thinking of replacing or repairing the rim, it will need to be removed anyway.

First: don't try truing a wheel you rely on without experience!

If spoke tension is high on the side of the rim deviation, then you can improve the true by adjusting spoke tension. If spoke tension is low on the side of the deviation or the tensions are roughly even, then the rim is bent and trying to fix it with tension will make the tension balance worse.

If you can hear tones pretty well, you can check tension by plucking spokes. Either pluck at the cross (in which case be aware you're conflating the tension of two spokes) or damp the spokes at the cross and pluck one spoke rimward of the cross.

On undished wheels, you can compare the tensions of spokes on the two sides (i.e., coming from different flanges of the hub) directly. If the wheel is dished (as with geared rear or most disc-brake wheels), you'll want to pay attention to the relative tension with respect to same-side spokes, and compare that between spokes on both sides of the wheel near the deviation.

For example, if I have a road rear wheel with a rim deviation to the left (non-drive), I know the right-flange (drive) spokes are at higher tension than the left, so I can't compare tones directly. I'll pluck some left-flange spokes near the deviation, and then some other left-flange spokes around the wheel, and do likewise for right-flange spokes. If left-flange spokes at the rim deviation are tighter than average, and right-flange spokes are looser than average, then I can true with a spoke wrench. If it's the opposite, or if the sides are the same, then it's a judgement call (regarding whether truing will be able to correct the deviation while maintaining acceptably even tension).

If there are no obvious signs of damage, then the wheel needs to be true in order to tell.

That's because unbalanced spoke tensions can cause the rim to wobble or be flatter in some areas than others.

Other answers say that a bent rim will have higher tensions on some spokes than others. But that's the case only if:

1. it's true
2. it has no tire irregularities, such as an uneven tube, or tire protector inside

Higher tensions can be required even on a non-bent rim once the tube/tire is installed, in the second case, unless you can perfectly balance the tire and what's inside.