What does “otherwise than” mean in this context

otherwise = in the other way or differently

than = compared to something

otherwise than = differently or in the other way compared to something

You never saw the library differently or in the other way compared to "wide open".

"differently/in the other way" modifies "saw"

differently from "wide open" means in the way that not wide open = closed or = opened with short distance

You never saw the library closed/in the other way before.

In the given context, otherwise than means that the person has only seen the library open before, and is now seeing it closed.

Try replacing with 'other than'. They mean the same thing and are both grammatically correct, though 'otherwise than' embellishes the sentence a little more.

A few helpful links: Glosbe definition, Macmillan Dictionary definition