What rules govern the romanisation of Greek υ

English has two sets of (conflicting) rules to translitterate Greek into Latin letters:

1. The traditional way, which is that of classical Latin; it is mostly based on how the pronunciation of Greek was best rendered in Latin letters and how those were pronounced in Latin.
2. A modern, less common way, in which whatever Latin letter is chosen that developed out of the Greek letter in question (and which resembles it in minuscle (lower-case) shape).

---

Greek ypsilon (capital Y, minuscle υ) was pronounced like French u in classical Greek (/y/ in the International Phonetic Alphabet). Latin did not have this vowel (neither does English, incidentally); it therefore used the Greek letter Y for that sound (Latin and Greek normally only had capitals). Our lower-case y is an adaptation of this capital letter.

• Examples: hybrid, psychology, ypsilon, cycle.

However, Latin did have vowels that strongly resembled Greek diphthongs αυ and ευ, to wit, au and eu. And so Greek AY and EY were rendered as AV and EV (Latin used V for both v and u).

(A diphthong is a vowel written in two letters, even through it is really one (complex) sound. It does not sound the way the two letters would sound separately in succession; and so, while Latin A sounded like Greek A, and Latinised Y sounded like Greek Y, the Latin sequence AY would not automatically sound like the Greek diphthong AY, whereas Latin AV would.)

• Examples: automaton, pause, eulogy, heuristic, therapeutic.

The Greek diphthong ου was apparently perceived to sound like Latin u (which sounds like German u, International Phonetic Alphabet /u/), and so OY was rendered as V.

• Examples: Thucydides, utopia.

Note that you will probably find some exceptions in classical Latin; nobody is ever consistent 100 % of the time.

---

In the modern way, any Greek letter is just translitterated into whatever Latin letter developed out of it, or into that which sounds most like it in the target language. (The Latin alphabet is an adaptation of the Greek alphabet to Etruscan and other languages of Italy.) In classical Latin, c was the normal letter to render the k sound, so it was used to translitterate Greek kappa; Latin k did exist, but it was very rare. In the modern way, Latin k is used to render Greek kappa, because Latin k was originally an adaptation of Greek kappa when the Latin alphabet was developed, and because c does not always sound like k in various modern languages. The shapes of kappa and k are also almost identical. The same applies more or less to Greek υ and Latin u (although the shapes of the Greek capital Y and the modern Latin capital U are somewhat different). So any υ is rendered as u in that system.

For whatever reason this system was first used (just to easily render Greek in Latin script in case the audience couldn't read Greek, or by some authenticist movement?), it was not originally intended for creating actual words in modern languages, but only ad hoc, when indicating vaguely what an actual Greek word in Greek signified or sounded like. Nevertheless, sometimes people do create and use words so transcribed, in English and other European languages, e.g. French and Dutch. The result is the 'mess' you have observed... I believe style books always recommend the traditional system of translitteration, mainly for consistency.