How to determine if 2 rays intersect?

Not sure if it answers your question, but here's something I wrote a few years ago for a paper.

Let $mathbf{p}_0$ and $mathbf{p}_1$ be the end points of the first segment and let $mathbf{q}_0$ and $mathbf{q}_1$ be the end points of the second segment. Then the parametric equations of the two lines are $$ mathbf{p}(t_p) = (1 - t_p) mathbf{p}_0 + t_p mathbf{p}_1 quad text{and}quad mathbf{q}(t_q) = (1 - t_q) mathbf{q}_0 + t_q mathbf{q}_1 ,. $$ At the point of intersection, $mathbf{p} = mathbf{q}$, i.e., $$ (1 - t_p) mathbf{p}_0 + t_p mathbf{p}_1 = (1 - t_q) mathbf{q}_0 + t_q mathbf{q}_1 ,. $$ Rearrangement of the equation gives $$ mathbf{q}_0 - mathbf{p}_0 = begin{bmatrix}mathbf{p}_1 - mathbf{p}_0 & -(mathbf{q}_1 - mathbf{q}_0)end{bmatrix} begin{bmatrix} t_p t_q end{bmatrix} ,. $$ Therefore, $$ begin{bmatrix} t_p t_q end{bmatrix} = begin{bmatrix}mathbf{p}_1 - mathbf{p}_0 & -(mathbf{q}_1 - mathbf{q}_0)end{bmatrix}^{-1} (mathbf{q}_0 - mathbf{p}_0) $$ Once we have solved for $t_p$ and $t_q$ we can find the point of intersection readily. If the intersection point is outside the $mathbf{p}$ line then $t_p notin [0, 1]$. Similarly, for the other segment, if the intersection point is outside the segment, then $t_q notin [0, 1]$.

If you only need to know whether the rays intersect, you don't have to find the point of intersection. The following may be more stable and efficient than solving the equations for the point of intersection, as it only involves subtraction and dot products, no division.

You have your first ray starting at $p_0$ and going in the direction of $p_1$ (and infinitely beyond $p_1$), and your second ray starting at $q_0$ and going in the direction of $q_1$ (and infinitely beyond $q_1$). Think about it visually. For a fixed $p_0$, $p_1$, and $q_0$, which values of $q_1$ result in an intersection? The answer is that $q_1$ must lie in a wedge-shaped region of the plane. One side of the wedge is the line between $q_0$ and $p_0$, and the other side of the wedge is parallel to the first ray. In the diagram, $q_1$ must be in the blue region for the rays to intersect.

We can express one side of the wedge by saying that $q_1$ must be on the same side of the $q_0$ to $p_0$ line as $p_1$ is. If $p_0 - q_0 = (l_x, l_y)$, then we can rotate $(l_x, l_y)$ 90 degrees to get a vector perpendicular to the line: $(-l_y, l_x)$. Then to check that $q_1$ and $p_1$ are on the same side, we check that $(q_1 - q_0) cdot (-l_y, l_x)$ has the same sign as $(p_1 - q_0) cdot (-l_y, l_x)$.

We can express the other side of the wedge by looking at the line passing through $q_0$ and $q_0 + (p_1 - p_0)$. $q_1$ and $p_1$ must be on the same side of this line. A vector parallel to the line is $p_1 - p_0 = (m_x, m_y)$ which we rotate 90 degrees to get $(-m_y, m_x)$. To check that $q_1$ and $p_1$ are on the same side of this line, we check that $(p_1 - q_0) cdot (-m_y, m_x)$ has the same sign as $(q_1 - q_0) cdot (-m_y, m_x)$.

So to sum up: the two rays intersect if and only if $(q_1 - q_0) cdot (-l_y, l_x)$ has the same sign as $(p_1 - q_0) cdot (-l_y, l_x)$, and $(p_1 - q_0) cdot (-m_y, m_x)$ has the same sign as $(q_1 - q_0) cdot (-m_y, m_x)$.

Since any two non-parallel lines must intersect somewhere (according to Euclid) I imagine that the OP intended a slightly different question. For example, do the rays intersect within the convex hull of the four given (really, implied) points? (the convex hull is the region enclosed by an elastic band stretched round all four points without crossing.) That is the problem solved by Biswajit Banerjee. You do need to know where the intersection is.