regarding the use of "them"

It is too ambiguous. You would repeat phases to be clear. This is because the sentence implies that the notion (definition) of what Aborigines believe dreams to be, is not stated in this sentence. Because of the allusion, the things in this sentence must be more specific and concrete.

When Europeans refer to the 'habit of dreaming', it is not the same thing as what the Aborigines believe of dreaming."

I think it's incorrect.

It's not always simple to give a yes-no answer to the question "is there any grammatical error in this sentence". The correctness of pronoun usage often depends on the intended meaning: a pronoun might be grammatically correct with one meaning, but incorrect with another meaning. In that sentence, "them" seems to stand for "dreams", a noun that appears nowhere in the quote. I wouldn't use it that way, regardless of whether it is possible for a reader to interpret the meaning correctly.

The sentence fails in terms of both syntax and semantics.

Syntactically, without knowing more about the meaning of the sentence than what is in the sentence, it needs to be changed to the following:

✔ What the Europeans refer to as the habit of dreaming ^(singular) by the Aborigines is ^(singular) not what the Aborigines themselves believe it ^{(the habit of dreaming: singular)} to be.

Per syntax alone, this is now correct. And if it is interpreted as the habit of dreaming, everything is fine.

Based purely on the sentence alone, this is what I believe it has to be changed to; I believe that they is a mistake, and it should really be it.

---

However, this is stated in the question:

"Them" is referring to dreams.

This throws the previous analysis into chaos. First, them has to be retained:

What the Europeans refer to as the habit of dreaming ^(singular) by the Aborigines is ^(singular) not what the Aborigines themselves believe them to be.

But this makes no sense.

---

In the first place, it would not be natural to assume that them is referring to dreams. The only plural subjects in the sentence are the Europeans and the Aborigines. Since there are no other plural subjects in the sentence, and it wouldn't make sense for them to be referring to one of those groups, it's not clear what it could be referring to; it looks like a mistake.

In essence, these are the possible interpretations, with the pronoun being replaced by a noun. And dreams is only being provided as the third noun because that's what it's been stated to be:

1. What the Europeans refer to as the habit of dreaming ^(singular) by the Aborigines is ^(singular) not what the Aborigines themselves believe the Europeans to be.

2. What the Europeans refer to as the habit of dreaming ^(singular) by the Aborigines is ^(singular) not what the Aborigines themselves believe the Aborigines to be.

3. What the Europeans refer to as the habit of dreaming ^(singular) by the Aborigines is ^(singular) not what the Aborigines themselves believe dreams to be.

None of those sentences are meaningful.

Looking at the third sentence, which is said to be correct, it can be simplified in the following way:

？ The habit of dreaming is not what dreams are.

While this is syntactically correct, it's not an understandable sentence—at least not on its own without further context to explain how it can make sense. Adding back in all of the qualifiers of the original sentence doesn't help it to make any more sense.

---

If you can somehow make it known that them is referring to dreams, then you could say that the sentence is syntactical. (And the fact that it's syntactical would have no bearing on the fact that the habit if dreaming is singular.) However, it still wouldn't be meaningful—at least not without further context that could make it so.

---

If they really does refer to dreams and not to the habit of dreaming, then here is one possible way of rephrasing the sentence so that it makes sense on every level:

✔ What the Europeans refer to as the habit of dreaming by the Aborigines does not involve what the Aborigines themselves believe to be dreams.

But without additional context beyond the single sentence, I can't say if that would be a valid rephrasing or not.