Why do we say INcomplete but UNcompleted?

The etymological origin of the root of the word isn't actually as relevant as whether the ending of the adjective is from Latin or from English.

The negative prefix in- is not very productive in English¹: that is, it's rare for a speaker to take an English adjective and attach in- to it to create a new negative adjective. Rather, in- (or im-, ir-, or il-) mainly shows up on words that already existed with this prefix in Latin or French before they were adapted for use in English. For example, before the word instability was used in English, instabilité was used in French.

As John Lawler says in his answer, uncompleted is derived from the past-participle form of the English verb complete, as indicated the by the presence fo the suffix -ed. It was not taken directly from a Latin adjective.

In contrast, complete is the anglicized form of a Latin adjective completus which corresponded to a negative adjective incompletus (listed in Lewis and Short's Latin Dictionary, although as a "late Lat." word).

The reality is a bit more complicated than the preceding paragraphs suggest. For one thing, in- does seem to have had some productivity in English in contexts where it creates a word that looks like it could have come from Latin or French, even if there wasn't actually a preexisting Latin or French word of this form. The word invariant may be an example of this type: the Oxford English Dictionary's earliest citation for the use of the word "invariant" in English is from 1851, which is earlier than the date of 1877 in the Trésor de la langue française informatisé's entry for invariant as a French word.

The OED entry for the negative prefix in- suggests that its usage has changed some over the centuries, and mentions the substantial overlap with the usage of un- which has resulted in some amount of variation for a number of words:

In English in- (il-, im-, ir-) is a living negative suffix for words of Latin or Romanic origin, freely used, even when no corresponding formation appears in Latin; in this use it interchanges to some extent with the Old English negative un-, which is used in native or thoroughly naturalized words, e.g. incautious, uncautious, in-, un-ceremonious, in-, un-certain, in-, un-communicative, in-, un-devout, in-, un-distinguishable. In such cases the practice in the 16th and 17th c. was to prefer the form with in-, e.g. inaidable, inarguable, inavailable, but the modern tendency is to restrict in- to words obviously answering to Latin types, and to prefer un- in other cases, as in unavailing, uncertain, undevout.

Keep in mind that this entry "has not yet been fully updated (first published 1899)" and it may have some inaccuracies. I think the author may have been overstating things a bit with the wording "freely used": Dixon, who I cited above and link to below, says in- is "seldom used to derive new [words]" (p.73) and categorizes it as "scarcely productive" (p. 73-74).

An important category of exceptions to the rule that I suggested in this section is adjectives that have Latin-derived endings but that nonetheless are negated with un-. Some even have corresponding -ity nouns starting with in-, such as unstable, unable, unequal vs. instability, inability, inequality. Other examples of adjectives with Latin-derived endings but the English negative prefix un- are unusual, unreal, unconscious, unscrupulous, ungenerous, ungracious, unattractive, unexpressive, unresponsive. I don't have stats on the relative frequencies of adjectives like this vs. adjectives like impossible, inaccurate, insensitive, intolerant that have the latinate negative prefix to go with their latinate endings. In general, un- is more common than in- (similar to how -able is more common than -ible), so the ending-based "rule" I suggest might be thought of as a rule for when a word might start with in- rather than a rule for when a word is likely to start with in-. A related question: Why can we use "inadequate" but not "inspecific"?

Words ending in -ed very rarely can be negated by prefixing in-/im-/ir-/il-

According to João Bittencourt de Oliveira, "Past participles ending in -ed take un- [...] Inexperienced is the only past participle with the prefix in-"². I suspect that the existence of the related noun inexperience is relevant, although I'm not sure exactly how. I think that experienced may not be a past participle, strictly speaking: although there is a verb experience, the suffix -(e)d in English is not only used to form participles or adjectives from verbs, but also to form adjectives from nouns (e.g. "winged" or "armed"), and the meaning of experienced seems more in line with this other usage of -ed. But whatever the internal structure of experienced, the word inexperienced still seems to me to have the structure in- + experienced rather than inexperience + -ed, which is why I'm uncertain about whether the noun inexperience has made any contribution to the use of in- in the adjective inexperienced.

I was able to find one more possible counterexample to the rule that Oliveira mentions: indisposed. The OED also has entries for some other obsolete variant words that are constructed this way, such as indispersed, inaffected, inabstracted. (Of course, a participle derived from a verb starting with in- will also start with in-, e.g. invalidated, but in this case the prefixation occurred before the suffixation.)

So we could say that the use of un- to negate adjectives ending in the suffix -ed is just a tendency rather than an absolute law, but it still seems to be a very strong tendency.

Citation footnotes & links

1. Making New Words: Morphological Derivation in English, by R. M. W Dixon (2014), p. 73-74.

2. "Negative Prefixes in Technical and Scientific English", p. 14.