Is wearing earplugs at a startup odd?

You could also wear overear headphones but underneath that earplugs.
You look normal and the headphones dim the sound a bit too.

There are headphones that act as earplugs, if you're worried you'd look out of place or people wouldn't notice you have earplugs in then this might be a solution. They are more obvious than real earplugs and people would just assume you're listening to music so you wouldn't look out of place.

https://www.plugfones.com/

Start ups in software often try all sorts of weird things. Seeing someone wearing ear plugs in a noisy environment may well be seen as sanity.

When the start up start getting customers, the other departments will start to take the work a bit more seriously.

Just ask. Otherwise you'll never know if you're the only person affected by noise. In our office, we had a Sonos system playing music (we could all affect the playlists). I can't remember the last time it was used, because I know that my choice in music won't match the other people, and vice-versa. We also had a foosball table in the office - but some people spoke up and said that it was distracting, so now it has an office all of it's own.

If you ask, you may find other people have the same issue. If not, then a good set of sound-cancelling headphones is worth the investment.

Just say you find the speaker distracting. Generally, people won't inquire any further. Coming to work to do work is not out of the ordinary so they'll see the speaker as the problem before you.

You can also wear noise cancelling headphones rather than earplugs as if someone wants your attention they will make themselves visible rather than audible and this applies to those wearing all headphones so you won't stick out if that is your concern.

As a personal addition, I work at a tech startup and the guy across from me wears earplugs in an open office that's generally silent but for the sound of clicking keys and no-one has ever asked him why. People generally don't pry into that sort of thing in my experience, especially as there are many medical conditions that necessitate earplugs (tinnitus being an example I suffer from) and that is considered too personal for most workplaces.