How to express a time exactly on the hour?

At least for your first request:

I heard (literally heard [from living-Latin person]), the expression in puncto that was used to mean "exactly". It was used to denote an exact amount of money (in order to avoid dealing with the change), something like: "Date mihi 12 Euros in puncto"

I think, though I can't confirm that, this expression was also used in the context of time, which will yield the kind of meaning you are looking for.

Doing some searching, several sites were found indeed seem to use this expression in context of time. such as this site:

Three o'clock on the dot is hora tertia in puncto.

I could not attest this from classical or even medieval sources (though a further research might do); Yet, it seems to be used and popular-enough today.

According to L&S, punctum can be used in the context of time when appended to temporis: to mean "the smallest portion of time, an instant, moment, point of time". So, at any case, this establishes that the derivation of current day "in puncto" (if indeed it is recent) is not so far-fetched.

Take two!

I think to specify "at four o'clock on the dot", you might have to say something like "at the beginning of the fourth hour".

Using classic texts as a guide, some options are:

initio + gen. (see Apuleius, Metamorphoses, 7.15, "at the beginning of spring"), giving us: initio horae quartae

Caesar prefers an ablative absolute using inita. For example: inita hieme (Gallic War, 3.7) or inita secunda vigilia (ibid, 5.23). Thus, you could write: inita hora quarta

On the other hand, Ovid prefers principio + gen. (principio mensis, Fasti, 2.55). So, principio horae quartae

quaque hora is used in medicine for "each hour". This is abbreviated to q. and "take every hour" would be written "q1h" but of course that wouldn't suit your purposes! Rather, I have found two ways of saying "every hour".

horis singulis - see Celsus, On Medicine, 4.12 (take a glass of wine every hour) and Pliny, Natural History, 2.232 (a copious spring always swells up and sinks back again every hour).

Alternatively, there is omnibus horis or omni hora. Cicero is especially fond of this (see Letters to Brutus, 1.17; De Senecute, 75; Pro Sexto Roscio Amerino, 53.154) but it is also used by Lucan and Tacitus. However, while this means "every hour", it does have overtones of "any hour" or "all the time". Horis singulis, in contrast, does seem to be more specifically for "once every hour", as the examples above show.

So, my suggestion for "at four o-clock on the dot, every hour" is: inita hora quarta, horis singulis (with quaque hora for added emphasis if you like but I think singulus covers this)

I like exacte, though I worry that its similarity to "exactly" might be leading me to think it's closer to what you need than it actually is. It occurs to me that adamussim ("to the level") might also work for the first, but it's not a word I've seen many times in my reading so it could be less appropriate than I suspect.

Your second sentence implies to me a general period of time (that is, "around when the hour is starting") rather than something exact. I wonder whether ad cujusque horæ initium would do the trick.