What is a "pony soldier"?

It's not a common idiom. There is an old movie called "Pony Soldier", based on the Royal Canadian Mounted Police, but it's not clear how that reference would relate, unless Biden was hinting that the guy was Canadian.

The voter in question was a young women with short cropped hair. The "idiom" was insulting and suggestive on several levels. It is not idiomatic if the phrase was never in common usage by any group at any time period. It is more likely a "senior moment".

Dog faced pony soldier is referencing an Indian scout (dog faced) pony soldier (U.S. Cavalry). It is not just any Cavalry recruit, but instead an Indian helping the Cavalry.

Apparently Biden was mixing his metaphors. There are several movies in question, the 1949 John Wayne film “She Wore a Yellow Ribbon," the phrase “dog-faced soldiers” occurs. While the term “pony soldiers” occurs in the 1953 John Wayne movie “Hondo” and the 1952 Tyrone Power western “Pony Soldier.” This basically backs up Russ Tinder's answer. In the Power movie, a Pony Soldier is a Royal Canadian Mountie. Essentially, it appears to be a homespun way to jokingly call someone a liar concocted by Biden from memories of the movies mentioned above.