How to solve three simultaneous equations on Mathematica to get N in terms of X,Y and Z?

It looks like you're looking for the solution of an underdetermined equation system.

eqn = {X == A1 S + N1, Y == A2 S + N1,    Z == 2 A1 A2 S^2 + A1 N1 S + A2 N1 S + N1^2}
sol=Solve[eqn, {A1, A2, S, N1} ];

N1/.sol 
(*{1/2 (X + Y - Sqrt[X^2 - 6 X Y + Y^2 + 4 Z]), 
1/2 (X + Y + Sqrt[X^2 - 6 X Y + Y^2 + 4 Z])}*)