Although I have no concrete evidence, I highly doubt that. Why would clothes make loud noises, unless they had way too many beads and metallic things stuck onto them? I doubt it would be the fabrics that were loud. Reading this again, I am beginning to understand why you would ask this question, but I'm pretty sure it started off as a personification or something like that and gained popularity. Don't make your analysis based on my answer though - I have no evidence!

Having read the comment above by user121863 regarding the Italian phrase un abito chaissòso meaning "a loud garment" I followed the link in the comment

Although I've never studied Italian I can sometimes make some sense of it and I noticed two things: firstly chiassòso seems to also mean noisy or, perhaps, busy as it can be applied to streets and squares which we would call noisy or busy rather than loud; secondly I can see (having looked up quadro online) that Italians also talk about paintings as being chiassòso.

We also talk about paintings and patterns being busy. I'm becoming convinced that "a loud garment" and "a busy pattern" come from Italian phrases using chiassòso, possibly via the Grand Tour.

The idea of “loud” applied to clothes refers essentially to colors and patterns of colors so the original sense is probably to be found in that respect rather than in garments and their supposed noise. Strong noticeable colous are loud in the sense that demand attention and cannot go unnoticed. The noise they make is figurative.

One early usage instance of loud referring to colors dates back to 1861, but its use is most likely earlier: A Manual of Artistic Colouring :

So in painting should the “loud colours be united to the quiet” by interposition of the more positive or more broken.

An early usage of “loud clothes” is from 1867 from The College Courant:

There were old men and young men; men in “loud” clothes, old clothes and every kind of clothes.

The same figurative sense is present in other languages such as Italian “colori chiassosi” or French couleurs criardes but I couldn’t find evidence of earlier usages.

I found the 1849 hit and in this context it would make no sense for it to mean audible as it talks about “a very loud pattern” (emphasis in original).

Similarly, I found a hit from 1852 for “some noisy pattern”. Again, it’s pretty clear that there’s no connection to audibility.

But, notably, I found this definition for “loud” from 1882:

Flashy; showy: applied to dress or manner; as, a loud pattern; he is decidedly loud. [Colloq.]—SYN. Noisy, boisterous, vociferous, clamorous, emphatic, positive, vehement, flashy, showy.

What I found significant here is that it specifically calls out that loud can be used to describe “manner”. It’s quite possible that this is where the association came from. An even earlier dictionary from 1841 has the definition “Emphatical; impressive; as, a loud call to avoid danger”, the connection being that the literal volume is meant to draw attention, much like someone or something that is showy.

The question more or less accidentally stumbles onto something known as synesthesia. Years ago, I read of mathematical savants who innately transpose complex math problems into colors, which, in terms of matching up problems with solutions, are much easier to work with than similarly complex equations. It was thought that this happens because particular areas of the brain have physically closer crossroads and connections. In my opinion, that also all of this stuff comes together on some warped level, everything must come together on some basic level.

"Chromesthesia or sound-to-color synesthesia is a type of synesthesia in which sound involuntarily evokes an experience of color, shape, and movement. Individuals with sound-color synesthesia are consciously aware of their synesthetic color associations/perceptions in daily life."

'Unraveling the Genetics Behind Why Some People “See” Sound and “Hear” Color'

https://www.smithsonianmag.com/smart-news/researchers-begin-unravel-how-some-people-see-sound-and-hear-color-180968374/

"Why Can Some People 'Hear' Colors?"

https://www.livescience.com/61930-synesthesia-hear-colors-genes.html

A video about how also (visible) light is, in a sense, a sound.

https://www.youtube.com/watch?v=o3HVF6C14HU

***** Late edit: I guess that people can hear, etc, words, too. That question here about what can be made of only the twenty-six letters, I think, is fertile ground for looking at also these other senses, but on a basic level. But, ultimately, the persons with the "hammers" are the ones who rule society. Brut force over finesse and resilience.

********** Edit in response to the comment: "The Greek philosopher and mathematician Pythagoras, who himself displayed signs of the remarkable trait, produced the first known description of synesthesia around 500 B.C. But experts believe the genetic basis for synesthesia, which tends to run in families, has been present throughout human history."

"Although synesthesia was long thought to be a rare condition, new research reveals that it's not that uncommon. Around 4 percent of the population is naturally synesthetic all of the time, and many people experience synesthesia temporally when they use psychedelic drugs."

"No, synesthesia is not a disease. In fact, several researchers have shown that synesthetes can perform better on certain tests of memory and intelligence. Synesthetes as a group are not mentally ill. They test negative on scales that check for schizophrenia, psychosis, delusions, and other disorders."

"Lexical-Gustatory Synesthesia is the rarest form of synesthesia. An individual with this type of synesthesia can taste certain words or sounds."

Furthermore, I imagine that persons with very sensitive hearing can hear regular clothing in motion. For sure, we can all hear someone walking in thick snow pants. As well, it may be that such sounds actually are so common that we have learned to just tune it out. How about the sound of zippers, snap buttons, and, putting cloths on and off? And, shoes hitting the pavement?

In the mid 19th century, taffeta was ‘loud’

The taffeta and silk used in dresses in the 1830s could have inspired the expression “loud clothes”. This is only a conjecture of mine but it is based on numerous references and eye-witness accounts who either admired or complained of the rustling sounds these heavy gowns made.

The noise a dress made was a sign of wealth and status.

Around the 1830s a type of plain woven silk called “Gros de Naples”, meaning “thick of Naples”, was a popular choice for dresses in that period (circled in the illustration above). The fabric was expensive, resistant, and had the added bonus of creating a pleasant rustling sound as the owner walked into a room.

“…: a very stout kind of gros de naples is also worn, which gives notice of its approach by its loud rustling” (1829)

In the late 19th and early 20th centuries, silk was also artificially ‘weighted’. Weighting involved chemically bonding metal salts–iron or tin–to the fabric to make the silk rustle and gave an impression of quality and luxury.

One dressmaker and expert in historical costumes, Cathy Hay, reads the following extract in this YouTube video

“Almost all the skirts were lined with stiff white taffeta […] That rustled whenever the wearer moved.” Which was one way of showing your wealth and status if your dresses were ‘loud’, which is why some of the linings were infused with tin because it made them rustle.

But not everyone approved, in a novel published in 1856

Dear countrywomen, in spite of the time and money you waste upon your toilet, believe me, you are the worst dressed angels in Europe! I hear your gros de naples rustle with indignation at the suggestion. I see your brows knit, and your fair shoulders shrugged to your ears. But if any man save myself dare tell you a word of truth, he would join with me in replying to these peevish demonstrations, –“be as angry as you please; but so it is!”

In 1870, one author advised their readers

Strive for a simple, modest elegance. Above all, avoid fabrics that rustle or make noise. “Rustling silks are especially annoying in church, as the least movement of the wearer causes them to make a noise sufficient to make inaudible for a moment the voice of the preacher.”

The Art of Dressing Well, pg 55

That taffeta was renowned for being “loud“ is also documented in a magazine article called
The Art of Noise Making (1929)

  

“A heroine, for example, can't faint in a taffeta silk dress, because taffeta silk is altogether too noisy”

Perhaps the ‘loud‘ rustling silks anticipated ‘loud’ colours by a mere decade or two at the most; the earliest instance of “loud colours” I discovered was written in 1850 by Ida Laura Pfeiffer. However, note the adjective, loud, is in italics.

“Especial reverence is shown to the twenty-four gods of Compassions and to Kwangfootse, a demi-god of war. Of the first, some have four, six, or even eight arms. All the gods, Buddha not excepted, are of wood gilt, and mostly painted in loud colours.”

(work in progress)

---

References

https://en.wikipedia.org/wiki/Victorian_fashion
https://en.wikipedia.org/wiki/Crinoline
http://www.katetattersall.com/victorian-fashion-terms-a-m/