In what ways is a set's inversion different than its prime form?

The difference between the minor triad and the major triad is a good case study. People are often surprised when they hear that they have the same interval vector, since they "sound so different".

The commonplace that minor and major triads, which have the same prime form, sound very different, can be put into question. I'm pretty sure that it takes quite a while for music students to reliably tell apart minor and major triads by ear. However, if we compare, say, the minor triad and a three-note cluster (comprised of C, C#, D in some inversion or transposition, Forte number 3-1), I am pretty certain that people with no musical background at all can learn to tell them apart pretty much immediately. So yes, the major and minor triad are different, but by far not as different as many of the other possible three-note chords/chordioids.

The reason that minor and major triads do not sound all that different is that they have the same interval vector, etc. etc.. Now to get to your question: what feature makes that they nonetheless sound different? It is a good question, and actually a bit mysterious to me. The best answer I can offer is in the way the overtone series of the three tones interact. On most instruments, one cannot hear very much of the overtone series. So it's important not to overemphasize the contribution of overtones to the overall sound of chords.

In trying to work this out I added the first five harmonic overtones to each of the three chord notes of a major and a minor triad. The resulting pitch collections are not inversions of each other and have a different interval vector, so this could be a good answer to your question.

The first five overtones are three different octaves, a fifth, and a major third. So on the level of the pitch class, we only need to add the fifth and major third to all three of the chord notes.

For C major (C, E, G), we get (C, D, E, G, Ab, B).

For C minor (C, Eb, G), we get (C, D, Eb, E, G, Bb, B)

As mentioned, these are not inversions of each other. (They do not even have the same cardinality.)

In other words, although the fundamentals of the major and minor triad have the same interval vector, as soon as you start to consider overtones, some divergences start to occur, which could explain the difference in sound.

As a sidenote: This train of thought would predict that major and minor chords made with pure sine waves would be harder to tell apart than the same chords made with more harmonic content. (Perhaps interesting to test this empirically?)

Finally, perhaps a simpler answer is that minor and major chords are typically embedded differently in harmonic progressions. And they also have different melodic implications. Perhaps that is the simpler explanation.

It depends on the set. For diatonic subsets the differences are essential, and underpin much of the Western tradition. e.g. (037) is both the minor triad and it’s inversion the major triad. Similarly (0258) as prime can be voiced as a half diminished seventh or a minor triad with added sixth, or as the ‘Tristan-chord’; all of which sound and function completely differently to each other, and differently again from the inversion, which we know as the dominant seventh. With more complex sets it’s the opposite problem. By ear you’d be hard pushed to tell if you were listening to 6-z4, 6-5 or 6-z6, however they were spaced, let alone whether they were in prime or inversion.

Prime form is just a canonical way of specifying all the pc-sets that are T- and I-related. It's the most "compact" way to express a set: in normal form and transposed to start at 0.

For example, see "Set Class and Prime Form" on OpenMusicTheory.com.

In order for a pitch-class set to be transpositionally or inversionally related to some other pitch class set, they must share the same collection of intervals....All pitch-class sets that are transpositionally and inversionally related belong to the same set class, and they are represented by the same prime form. (italics original)