Use magic weapons

It may be difficult to acquire one of these weapons, and until you do you'll need to rely on Slings (which are not nearly as good as bows).

Two magic weapons can make your build work well:

• The most direct option is probably the two-birds sling, which is a rare magic weapon from Mythic Odyssies of Theros which gives you a +1 to hit and damage, and also doubles the attacks you make each round (as long as you have two targets). Which is a big deal. Like, it practically doubles your damage output as a Dex-based fighter. It also doesn't require attunement.

• An even better option, if you could get enough of them, which you can't, or find a way to reliably retrieve them, which you maybe can, is the uncommon magic weapon Storm Boomerang from Princes of the Apocalypse which has way better range than a sling and does 1d4 bludgeoning and 3d4 thunder damage on a hit and forces the target to make a DC 10 Constitution save not to be stunned for a turn. The problem is that after each hit it turns into a 1d4 bludgeoning ranged weapon that is admittedly better than a sling but still not exactly great. It also only returns for free on a miss so even as 'better than a sling' you need to either be picking it up after each hit with an object interaction or just carrying around a whole pile of them. If you can somehow get a whole pile of them, that would obviously be great, but even one will be a big deal, and it does only take an hour to recharge if you can get easy portable access to an elemental air node (which you probably can at high levels).

Multiclass into Artificer

The official Artificer class (from Eberron: Rising from the Last War) has an option for using Intellegience for weapon attacks, which of course would work for bludgeoning weapons. This is a class with magic, but spells aren't actually needed to make use of this solution.
(Thanks for @Someone_Evil via comments for pointing this out.)

Battle Smith

This official subclass from the same book as the Artificer class has the following feature (p. 61):

Battle Ready

When you reach 3rd level, your combat training and your experiments with magic have paid off in two ways:

• You gain proficiency with martial weapons.
• When you attack with a magic weapon, you can use your Intelligence modifier, instead of Strength or Dexterity modifier, for the attack and damage rolls.

This does require you to own a magic weapon, but that weapon can be a martial bludgeoning weapon, like a maul.

---

Honourable mention: Armorer

This subclass from Unearthed Arcana also allows you to use your Intelligence for attacks (although the damage type isn't bludgeoning damage, hence why it's only an honourable mention), using this feature:

Armor Model

At 3rd level, you can customise your power armor. When you do so, choose one of the following armor models: guardian or infiltrator. The model you choose gives you special benefits while you wear it.

Each model includes a special weapon. When you attack with that weapon, you can use your Intelligence modifier, instead of Strength or Dexterity modifier, for the attack and damage rolls.

[...]

Thunder Gauntlets. [from the Guardian armor subtype] Your armored fists each count as simple melee weapons, and each deals 1d8 thunder damage on a hit. [...]

[...]

Lightning Launcher. [from the Infiltrator armor subtype] A gemlike node appears on one of your armored fists or on the chest (your choice). It counts as a simple ranged weapon [...] and it deals 1d6 lightning damage on a hit. Once on each of your turns when you hit a creature with it, you can deal an extra 1d6 lightning damage to that target.

A wood elf can take the feat Wood Elf Magic (XGtE, p. 75). In addition to letting you learn the longstrider and pass without trace spells, you learn one druid cantrip of your choice. Wisdom is your spellcasting ability for all 3 spells.

With this feat, you could choose the druid cantrip shillelagh, which you cast on a club or quarterstaff, letting you use that spellcasting ability (Wisdom) for attacks with that weapon. Both clubs and quarterstaffs do bludgeoning damage.

However, a build requiring an extra feat (in addition to Elven Accuracy and Crusher) takes a while to come online. A faster way to get shillelagh would be to multi-class druid 1 / ranger X.

As a pure fighter you can use a sling

A sling as ranged weapon uses Dex by default. It is probably far from optimal, however.

Otherwise you'll have to multiclass

A hexblade can add the Charisma mod to a weapon attack with their chosen weapon. This does work for martial weapons such as flails or warhammers. For this you need one level of warlock.

Multiclassing monk (one level) gives you the martial arts feature letting you use Dex for simple one-handed weapons (monk weapons), including a club, light hammer, or quarterstaff. Using three levels of monk, you can take the path of the Kensei, and choose fails or warhammers as Kensei weapons, making them monk weapons.

There are not even many spell options

Using shilellagh on a quarterstaff would be possible with one level of druid. As would be using a spell attack, dealing bludgeoning damage. You could use magic stone or wrath of nature but both don't seem to be useful options if you want to play Fighter, so you are not really missing out on anything when excluding spells.

Ask your GM

If you want to have a special built, especially if it is supposed to be interesting rather than optimized, you could ask you GM for help. They might provide you with a finesse bludgeoning weapon (even if this is somewhat anti-thematic). Concerning ranged weapons, an improved sling would be possible (1d6 damage, as suggested in the comments). Otherwise, they could introduce arrows with blunt / spherical tips. I'm unsure how realistic this is, but it would allow to deal bludgeoning damage with bows.

They could make a magic item that fits the bill. Concerning existing magic items, the storm boomerang (suggested by Medix2) from Princes of the Apocalypse published adventure would fit.

On the ranged side, there's really only the sling. You could theoretically go whole-hog on ranged Fighter, taking the Archery style to bolster your attack rolls, and just accepting that your damage isn't going to be fantastic.

There are three decent options for melee combat:

Learn martial arts

If you take at least one level of Monk, you'll gain the Martial Arts class feature, which allows you to use your Dexterity modifier when fighting with unarmed strikes and "monk weapons". Monk weapons are all the simple melee weapons that aren't two-handed or heavy, plus short swords. The mace, quarterstaff, light hammer, and club all fit your needs; personally I'd go for the quarterstaff because you can use it in two hands to get a 1d8 damage die. (While your Martial Arts damage die starts at 1d4, it's optional; you can choose to use the weapon's standard damage instead.)

I believe your unarmed strike would also work for your purposes (dealing 1d4 damage due to Martial Arts), should you happen to be caught without your trusty stick.

This will also allow you to throw hammers and make it work with your plan. While light hammers normally use strength when thrown, they are valid monk weapons, so you could use Dexterity when you hurl them at people.

This seems like the best option to me; it has a lot of flexibility and some good side benefits, and it doesn't involve using magic, as you requested. One down-side is that you can't wear armor or use a shield with this method. You'd be focusing on Dex anyway and can add Wis to it via your Unarmored Defense, so with decent stats, you will probably have a respectable AC, but probably not as high as a traditional sword-and-board fighter. You won't be able to use most of the Fighter styles, though. Your best bet is probably Great Weapon Fighting if you're going with a staff, or Dueling if you're using a smaller weapon.

Learn magic

The second easy way to deal bludgeoning damage using wis/cha/dex would be to learn the shillelagh druid spell, which is magic, but it's not too costly. You could take a one-level dip into Druid to pick up the cantrip, or grab the Magic Initiate feat and avoid the hassle of multiclassing. (Although, given you're already pulling in two feats for this, you may not want to spend a third!) That will allow you to use a club or quarterstaff to fight using Wisdom. This also makes your weapon deal 1d8 damage even if you use it one-handed, which is nice.

Note that the spell is cast as a bonus action, so you don't have to give up an Attack action to activate it. As a fighter, you won't have a lot of competition for what to do with your bonus action anyway, unless you pick a couple of very specific Battlemaster maneuvers.

Multiclassing into Druid does put some limits on your armor choices, though, so you should consider the AC loss due to not wearing metal armor.

Strike a bargain

The third way is also technically magical in nature, but does not involve spellcasting as such. If you multiclass into Warlock and choose the Hexblade patron, then you can wield a bludgeoning weapon that uses your Charisma for attack rolls. The Hexblade comes with martial proficiency, so you could use the warhammer or flail in addition to the simple bludgeons. With those weapons, you can go with a shield and hit for d8, or go two-handed and get the d10 from your warhammer.

Unlike the other options, this method doesn't impact your AC, which is nice, but it does come along with some spellcasting ability that may conflict with your design intent.