Use of the pronoun "which"

Although you didn't ask specifically about which as opposed to that, your question about the ambiguity of a clause's noun phrase referent also relates to the clause's restrictiveness, if I'm reading it correctly.

(Side note on that vs. which: The most commonly followed distinction between the use of that and which relates to whether the clause they precede is restrictive or non-restrictive. The Perdue Online Writing Lab defines a restrictive clause as one in which "the information in the clause is necessary to understand the preceding noun [phrase]." A non-restrictive clause doesn't change, or isn't necessary to understand, the preceding noun phrase. Note almost all guides suggest non-restrictive phrases should be set off using commas.)

If your question (#2) is really how to make the phrase "which is smoothly parametrized by a finite number of real parameters" refer to "statistical model," that seems to be a non-restrictive phrase. In that case you would need something to the effect of:

A statistical model, which is smoothly parametrized by a finite number of real parameters, is a family of probability distributions of a random variable.

If, on the other hand, you're trying to distinguish between the referents "family of probability distributions" and "random variable" in order to define a particular type of probability distribution (which seems more likely given your comment), your noun phrase is restrictive and you'll want to clarify which antecedent noun phrase is being restricted. But restriction doesn't use which in quite the way you do.

For example, if it's the random and smoothly parameterized variable that distinguishes a particular type of probability distribution from others, you'd want something like:

A statistical model is a family of probability distributions in which [or where, or wherein] a random variable is smoothly parametrized by a finite number of real parameters.

If it's the smooth parameterization that distinguishes a type of probability distribution, you definitely need a that. For example:

A statistical model is a family of probability distributions of a random variable that is smoothly parametrized by a finite number of real parameters.

In this last case, it's still ambiguous whether the smooth parameterization refers to the unit "probability distribution of a random variable" or just to "random variable". However, if there's no such thing as a probability distribution that's not of a random variable, there's no ambiguity—but if the randomness of a variable is what makes a probability distribution a thing in the first place, it seems to me "of a random variable" may be redundant.

Caveat: I'm parsing this sentence as someone who remembers basically nothing about statistics. But unless (and really, even if) you're writing for an audience solely made up of experts, you'd want to craft the sentence in the least ambiguous (and, of course, accurate) way, so that your meaning comes across clearly to the widest possible audience.

And finally: which and that are some of the trickiest words in English to use unambiguously, so your question isn't silly at all. I'm constantly second-guessing myself on which one to use (and have been corrected by editors more than once). I just hope my lack of familiarity with statistics hasn't resulted in several completely absurd rephrasings of what's actually a straightforward sentence to statisticians.