Is it correct to use a tilde (~) in a paper as punctuation meaning “sort of identical to”?

I think using a tilde as proposed would be a bit ugly, and potentially confusing.

In engineering or IT contexts, I have often seen ~ before numbers to indicate approximately equal but I’ve never before words. This ties up with Wikipedia’s view:

This symbol (in English) informally means "approximately", "about", or "around", such as "~30 minutes before", meaning "approximately 30 minutes before". It can mean "similar to", including "of the same order of magnitude as", such as: "x ~ y" meaning that x and y are of the same order of magnitude.
- wikipedia

I don't know the context or why you think this information needs to be included here in that manner, but in general I feel rather strongly that people use parentheses to include unnecessary details far too often and that if the information is important enough to be included, then the writer ought to make the effort to write it into the sentence properly. If it is not important enough to make that much of an effort, then it's not necessary to the sentence, is it.

So, use one or the other of these:

"Another category of common sounds found across many languages are the palato-alveolar consonants."

"Another category of common sounds found across many languages are the palato-alveolar consonants, which are similar to sibilants but have important distinctions."

I'd recommend rephrasing without the symbol. The symbol should be reserved for indicating approximate quantities.

What you are trying to convey is that in languages which distinguish a palato-alveolar from an alveolar place of articulation, the distinction is always in sibilant fricatives and affricates. Note that palato-alveolar is an articulatory term, fricative is a physico-acoustical term, and sibilant is an auditory term. So palato-alveolar and sibilant are names from different sets of terminology. Sibilant can also refer to alveolar consonants, so just use a more precise auditory-based term to get equivalence:

...palateo-alveolar (hushing sibilant) consonants