What did "on by out, over, up" mean?

I'm glad for you curiosity, but your title could use some more context. The book that is quoted would have done better to say that the word 'above' is etymologically equivalent to 'on + (by + up)'. I find the way he has it too suggestive, if not downright misleading (and likely leading the the poster's inquiry) (the quote being "So in a sense, above means ‘on by up’"). Andrew Leach's comment puts it right:

It's a fallacy to imagine that a word means the same now as its distant etymological ancestors did in the past — let alone having brought those ancestors into the present for comparison. That said, it may be possible to explain how the ancestors of what are now on, by, out became onbūtan

I think what the poster is actually wanting to know though is how to make sense of combining prepositions to make a new one, namely for OE onbūtan and abufan. A simple wiktionary trail and a bit of critical thinking quickly leads to an answer. For example:

onbūtan:

starting with utan most generally meaning 'out', but also in the sense 'outside',

then tacking on be- 'by, around' etc. and getting butan 'without, except' (so being not in or included, but outside of or excluding something)

it seems to me that the focus here is on the separateness

then adding on- 'in, on' etc., for onbutan 'on the outside, about' etc.

this seems to draw the focus back towards concrete location

Combining the prepositions together allows for a more descriptive word with a more precise meaning.

interesting side note: this reminded me of Old Irish, which is an example of a language that makes more extensive use of preposition strings to form compounds. For example (from Baldi, 1983):

'comtherchomracc' from

'com-to-er-com-ro-icc' which is literally

'with-to-before-with-PERFECTIVE-come' and means

'assembly'

I'd be a lot harder pressed to make sense that ,XD

[pasted from my answer on reddit here https://www.reddit.com/r/OldEnglish/comments/mdc2r1/what_did_on_by_out_over_up_mean/ . See gastrophone's comment too about how we actually still do this sort of thing in Modern English.]