How to determine directions of Vectors relative to each other?

The key thing to consider is "to an observer on Y". This means that you are setting Y as a frame or reference - or that you are considering Y to have $0$ velocity and that everything else is moving accordingly. If from the shore X is going North then you have to consider how Y is moving for X to be travelling NE from the point of view of an observer on Y. For X to seem to be travelling NE instead of North, Y has to be travelling in the opposite direction of East, which is West.

You don't really need maths for this as much as just spatial awareness / some sense of galilean relativity.

Imagine you're looking north high above a stationary boat on the mast, and you see a boat moving straight ahead north relative to your compass. You know the motion of that boat as it's given to you. Imagine now that you see it moving north-east, well you know the boat is only moving north as the question gives it to you, but it now appears to be moving east as well. If your boat and that boat were both moving with unknown velocities, you couldn't really say [if the surroundings didnt give it away] what boat was moving in which direction. Since the boat appears to have some motion in the west $to$ east axis, that would also be seen if your boat was moving the opposite direction in the same axis, which there aren't any restricting conditions for. So then it's reasonable to conclude that that's the way it is.

In general for relative motion problems at normal speeds you need two coordinate systems. (Assume all symbols represent vectors.) The position of a point in system one = r = R + r' where R is the position of the origin of system two as measured in system one, and r' is the position of the point in system two. (The primes indicate measurements in the second system, not derivatives.) Similarly, (taking derivatives) v = V + v' and a = A + a' . Start with a sketch showing the vectors and then calculate with the components. Be careful that you do not mix displacement and velocity vectors.