Resolving horizontal and vertical components on an inclined plane

As you are approaching such inclined plane problems, keep in mind two important realizations:

1. The action due to gravity is always perpendicular to the base of the inclined plane pointing to the center of the Earth.
2. According to a well-known theorem in geometry, if two sides of two angles are mutually perpendicular, then the angles forming out of those sides would be the same. To see this, let's name the center of the mass of the box by O. It is obvious that OH is perpendicular to the base of the inclined plane while OA is perpendicular to the inclined plane. Now, according the mentioned theorem, the angle AOH would be similar to the angle named 30 degrees just because their sides are mutually perpendicular. Thanks,

Here is the diagram you need to draw for yourself:

The force of gravity (red) can be thought of as the sum of two (green) forces - one perpendicular to the incline (the normal force) and one along it. Since the red vector is the hypotenuse of a triangle, the normal force must be $mgcostheta$.

This is indeed common mistake and has rather simple mathematical explanation. Mathematically speaking, whenever you want to disassemble a vector into two components, components are always catheti and the total vector is always a hypotenuse.

Geometrically speaking, you must define coordinate system for the disassembling. In incline plane problems one axis is usually directed perpendicular and the other parallel to the inclination. The reason is that Newton equation in the perpendicular direction gives you the value of the normal force of the inclination, since there is no acceleration in that direction. Normal force is then used for Newton equation in the parallel direction (in case you have friction). Hence the second solution.

You can resolve the mg (100N force) vector into components either as in your second (components O and H) or third (components O and A) drawing; but O is not the same vector in the two drawings. Look at your second drawing; the direction of vector O is not totally along (parallel to) the plane and so it still contributes to the net force perpendicular to the plane. In your third drawing the direction of vector O is totally along the plane and does not contribute to forces perpendicular to the plane so with this resolution of mg you have one vector perpendicular to the plane, A, and the other vector, O, parallel to the plane; so only A needs to be countered by R. You always resolve a vector into components that simplify what you are trying to solve.