Which type of material should I use to fill in gaps below the trim in unfinished stairs to basement

If you just want to paint the stairs, floor paint, perhaps with a traction admixture to keep them from being slippery. Still going to look like painted plywood.

If you are not going to rip out the poor trim job, might as well just paint the exposed wood and call it a day, given what a mess that appears to be. Nothing short of adding a new sheet of drywall over the whole wall is going to make that upper section look "normal" and if you are not doing that, filling the lower part feels pointless. But you could fill it with drywall or plywood (or plain wood) as you prefer. Wood will take a bit more "toe damage" than drywall, though the paint will get scuffed in either case if toes or other things knock into that area.