I drove my car with the dipstick left out, did I do any damage?

Mechanically, its hard to say without actually seeing the engine internals, however as a precaution you can get a oil change done, this will remove (or nearly remove) all the debris which might have gotten in, you could also use an Engine Flush as well, if you are unsure of how to use an Engine flush properly then take a can to your mechanic with new oil and filter, and have him/her do it.

I see no cause for concern here.

Even if debris makes its way into the oil sump and contaminates the engine oil, your oil filter should be able to capture it and remove it from the oil to prevent extended wear and tear on any bearings.

The only way this might have damaged your engine is if oil flowed out the tube and left the engine dry. I'd suggest since no oil light came on, suggesting a lack of oil, you never had this issue. (There would be a large oil slick inside the engine bay as well.) Some vehicles have the tube stuck down into the oil. If there is excess pressure in the crankcase, oil can flow from the dipstick tube if the dipstick isn't securely fastened in the tube; the pressure will cause oil to flow out of the tube.

The tube itself is so small, even if some dirt got into the tube, the amount would be very minimal. There just isn't enough of a cross section for any amount of dirt to travel into. As has been stated, there is no real need for concern here.

For short durations it's unlikely to cause problems. As others have suggested an oil and oil filter change will remove most contamination, if any got in. It's likely not worth bothering unless the vehicle was driven in very dusty, dirty, or wet conditions

However if the dipstick is left out for very long periods of time there is more of a risk of contamination. You should also check that the dipstick tube is clean... In some cases insects may nest in them, water might enter, and larger debris such as small pebbles, sand, etc may have found its way in.

Again, even in this worst case, an oil and filter change will likely mitigate any problems. If there is water in the oil it should be changed frequently until it's no longer present. If there's reason to believe large contaminants made it into the engine then it's best to remove and clean out the oil pan.