What is the difference between the two sentences? (Have you been ... before? AND have you ever been...?)

In the first sentence you are asking a person whether in their whole past they made a stay or more in Japan. You could say that more concisely as "Have you been to japan?" and if the context is right there is no ambiguity. In the context of a circle of people who travel a lot, this can mean what is expressed by the more usual "Have you been to Japan lately?".

In the second sentence you are asking whether in the past the person made at any one time a stay in japan.

The two sentences come down roughly to the same thing, which is to ask someone whether in their past life they made a stay in Japan.

The first question (Have you been to Japan before?) is most likely to be asked when you and your conversation partner are actually in Japan or are discussing a plan to visit it.

A similar question is (Have you ridden a horse before?). You are sitting on a horse or discussing a plan to ride one.

The second question (Have you ever been to Japan?) would be rather unlikely to be asked when you are actually in Japan. It is more likely to be an out-of-the-blue question similar to Have you ever ridden a horse? It is also possible when discussing a plan to visit Japan or to ride a horse.