How do you know how much of an effect on the image will be when using a strangely shaped aperture like star-shaped or heart shaped?

The only way to know the effect of aperture shape is make a picture, then look at it.

With practice, you may have enough experience to make accurate predictions before making the picture, but “try it and see” experiments will always be necessary.

Experiment is part of photography and most experiments fail. The aperture shape will always have some effect on the picture. Usually it will not be the effect you want.

Fortunately, unsuccessful pictures are not crimes and for better or worse few people will care one way or the other.

Making pictures to find out is almost always better than guessing while sitting in a chair. Even with film, making pictures doesn’t cost much and ends whatever worry kept you from it.