Confusion regarding Hall effect in semiconductors

The answer to your confusion really is that you can't see holes just as "absences" of electrons. You are right: If the electrons up, then the holes have to get up as well (at least if we stick to this picture of electrons being particles):

At first, maybe this answer might be interesting to look at.

To give a brief summary: To understand the phenomenon, you have to use the formalism of quantum mechanics: You will then find out that

• electrons in a solid body can have different states with different momentum $k$ and different energy $E$ (and spin, for sake of completeness, but this is not relevant for this answer)
• for many-particle systems (which your semiconductor is) a state can be occupied by one electron at max
• You will have a certain relation for your states between the Energy E of a state and the momentum $k$ of your state, which is called dispersion relation: You can see different lines in the plot, those are called "Bands". Imagine you'd only have one electron in your system, it would occupy one state in your band. You'd call this electron "electron". Now imagine your band is completely filled with electrons (remember that only one electron per state is allowed), except one free empty state. This is what is referred to as a "hole".
• Because of thermodynamic arguments, empty states are mutch more probable in the upper region of a band, arround the maximum peak of the Dispersion-Relation. Electrons in an empty band are, because of the same argument, usually located in the lower region, arround the minimum of the dispersion relation. By "located" I hereby mean their $k$-Value.
• The last piece of the puzzle: Your system, with your electrons in their respective states evolves (at least for mean values) according to classical equations of motion (this is called semiclassical model of electrons): The force acting will change the the momentum, and the velocity is the derivative of energy with respect to momentum. (This is an analogue to the canonical equations of motion in the Hamiltonian formalism)

begin{align} v = frac{partial E}{partial k} hbar dot{k} = F end{align}

Now we can calculate how a force affects the change of the velocity:

begin{align} dot{v} = frac{d}{dt} frac{partial E}{partial k} = frac{partial^2 E}{partial k^2} dot{k} = frac{partial^2 E}{partial k^2} frac{1}{hbar} F end{align}

So you see that the curvature of the dispersion relation acts as a proportionality between force an acceleration (that's why it's called "effective mass"). For holes, which usually are located at the maximum, this curvature will usually be negative, while for electrons (located at the minimum), it is positiv. This is the explanation for holes moving into the direction that a positive charge would move to.

To answer to your question: The holes will accumulate at the bottom, as your graphic indicates, because they behave not as "absences of electrons".