Pictures coordinates to robot coordinates

I have a picture on my C# software and the corresponding object in real world on 2d plane of motorized XY axis table. What I want is when I select a pixel on my picture, the motorized table should be moving to the exact same point of the real world object.
The real world object and its picture may have distortion and rotation differences and the picture on the software is not perfect too.

I use 4 points on each to map all the points in between, example:

Normal picture (x,y): [250, 25] , [250, 287] , [399, 287] , [400, 28] (in pixels)

Table coordinates (x,y): [0, 0] , [2098, 29538] , [19127, 28164] , [17097, -1200] (in microsteps)

I tried using OpenCV’s homography:

I used FindHomography() to get the H matrix and I transform the picture point choosed using PerspectiveTransform() which give me the corresponding point in microsteps on the real world to move the motorized XY axis table.

OpenCvSharp.Mat hCv = OpenCvSharp.Cv2.FindHomography(srcPoints, dstPoints);
OpenCvSharp.Point2d[] resultCv = OpenCvSharp.Cv2.PerspectiveTransform(targetCv, hCv);

I also manualy calculated the homography matrix using this anwser : https://math.stackexchange.com/a/2619023
But I both cases I always get an error when transforming one of the four reference point like for [250, 25] the corresponding point should be [0, 0] but instead I get something like [-25, 245].

Question : I there a different way to link pictures coordinates to real world coordinates accuratly ?

Edit, more explanations:

To get my 8 points I select four points of the picture and then in real world I move my table to the four corresponding points manually. Let say I took a picture of my smartphone. I will get my points on the four edges of the phone. Then if I choose the X, Y pixel position corresponding to the front camera of my phone, my motorized table should move to get a landmark above the front camera of my real phone.