Align frame labels in different items in GraphicsRow

I have a somewhat complicated plot I need to make. This example is extremely simplified to get at the core of the issue. I know that some of what I’m accomplishing here can be more easily accomplished with eg. ListPlot, but due to the more involved nature of the styling (per-point colors, shapes, etc.) of my "true" example, I prefer this approach.

I have two datasets (actually many, but here 2),

X = {Point[{1/2, 4.617754`}], Point[{1/2, 4.617754`}], Point[{1/2, 1.6177540000000004`}],  Point[{1/2, 4.617754000000001`}], Point[{1/2, 4.617754`}], Point[{1/2, 1.617754`}], Point[{1/2, 2.6177540000000006`}], Point[{1/2, 2.6177539999999997`}], Point[{1/2, 5.6177540000000015`}], Point[{1/2, 2.6177540000000006`}], Point[{1/2, 2.6177540000000006`}], Point[{1/2, 5.617754000000001`}], Point[{1, 2.808877`}], Point[{1, 2.8088770000000003`}], Point[{1, -0.19112299999999982`}],Point[{1, 2.8088770000000007`}], Point[{1, 2.808877`}], Point[{1, -0.19112300000000038`}], Point[{0, 5.825319881216744`}], Point[{0, 0.41018811878325756`}], Point[{0, 5.825319881216743`}], Point[{0, 0.41018811878325595`}], Point[{0, 7.559570105933373`}], Point[{0, 1.6759378940666267`}], Point[{1, 2.808877`}], Point[{1, 2.8088770000000016`}], Point[{1, -0.19112299999999793`}], Point[{0, 5.426631`}], Point[{3/2, 0.`}], Point[{3/2, 0.`}], Point[{1/2, 5.928267065422335`}], Point[{1/2, 2.6756860176040465`}], Point[{1/2, -1.3684450830263684`}], Point[{1/2, 5.928267065422333`}], Point[{1/2, 2.6756860176040442`}], Point[{1/2, -1.3684450830263666`}], Point[{1/2, 3.6177540000000006`}], Point[{1/2, 3.6177540000000006`}], Point[{3/2, 0.`}], Point[{1/2, 5.928267065422335`}], Point[{1/2, 2.675686017604044`}], Point[{1/2, -1.3684450830263666`}], Point[{1/2, 5.928267065422335`}], Point[{1/2, 2.6756860176040456`}], Point[{1/2, -1.3684450830263648`}], Point[{1/2, 3.6177540000000006`}], Point[{1/2, 3.6177540000000006`}], Point[{3/2, -4.440892098500626`*^-16}], Point[{0, 5.426631`}], Point[{1, 0.808877`}], Point[{1, 0.8088769999999998`}], Point[{1, 3.8088770000000007`}], Point[{1, 0.8088769999999998`}],  Point[{1, 0.8088769999999998`}], Point[{1, 3.808876999999998`}], Point[{0, 6.044709889772519`}], Point[{0, 2.190798110227481`}], Point[{0, 6.04470988977252`}], Point[{0, 2.1907981102274814`}], Point[{0, 6.603953944951208`}], Point[{0, -1.3684459449512054`}], Point[{1, 0.808876999999999`}], Point[{1, 0.808877`}], Point[{1, 3.808876999999999`}]};

Y = {Point[{1, 4.617754`}], Point[{-1, 4.617754`}], Point[{0, 1.6177540000000004`}], Point[{1, 4.617754000000001`}], Point[{-1, 4.617754`}], Point[{0, 1.617754`}], Point[{1, 2.6177540000000006`}], Point[{-1, 2.6177539999999997`}], Point[{0, 5.6177540000000015`}], Point[{1, 2.6177540000000006`}], Point[{-1, 2.6177540000000006`}], Point[{0, 5.617754000000001`}], Point[{1, 2.808877`}], Point[{-1, 2.8088770000000003`}], Point[{0, -0.19112299999999982`}], Point[{1, 2.8088770000000007`}], Point[{-1, 2.808877`}], Point[{0, -0.19112300000000038`}], Point[{1, 5.825319881216744`}], Point[{1, 0.41018811878325756`}], Point[{-1, 5.825319881216743`}], Point[{-1, 0.41018811878325595`}], Point[{0, 7.559570105933373`}], Point[{0, 1.6759378940666267`}], Point[{1, 2.808877`}], Point[{-1, 2.8088770000000016`}], Point[{0, -0.19112299999999793`}], Point[{0, 5.426631`}], Point[{0, 0.`}], Point[{0, 0.`}], Point[{1, 5.928267065422335`}], Point[{1, 2.6756860176040465`}], Point[{1, -1.3684450830263684`}], Point[{-1, 5.928267065422333`}], Point[{-1, 2.6756860176040442`}], Point[{-1, -1.3684450830263666`}], Point[{0, 3.6177540000000006`}], Point[{0, 3.6177540000000006`}], Point[{0, 0.`}], Point[{1, 5.928267065422335`}], Point[{1, 2.675686017604044`}], Point[{1, -1.3684450830263666`}], Point[{-1, 5.928267065422335`}], Point[{-1, 2.6756860176040456`}], Point[{-1, -1.3684450830263648`}], Point[{0, 3.6177540000000006`}], Point[{0, 3.6177540000000006`}], Point[{0, -4.440892098500626`*^-16}], Point[{0, 5.426631`}], Point[{1, 0.808877`}], Point[{-1, 0.8088769999999998`}], Point[{0, 3.8088770000000007`}], Point[{1, 0.8088769999999998`}], Point[{-1, 0.8088769999999998`}], Point[{0, 3.808876999999998`}], Point[{1, 6.044709889772519`}], Point[{1, 2.190798110227481`}], Point[{-1, 6.04470988977252`}], Point[{-1, 2.1907981102274814`}], Point[{0, 6.603953944951208`}], Point[{0, -1.3684459449512054`}], Point[{1, 0.808876999999999`}], Point[{-1, 0.808877`}], Point[{0, 3.808876999999999`}]};

I construct graphics displaying them,

pX = Graphics[X, Frame -> True, FrameTicks -> {{0, 1/2, 1, 3/2}, None, None, None}, FrameLabel -> {"X", None}, AspectRatio -> 2];
pY = Graphics[Y, Frame -> True, FrameTicks -> {{-1, 0, 1}, None, None, None}, FrameLabel -> {"Y", None}, AspectRatio -> 2];

and then do

GraphicsRow[{pX, pY}]

which results in

First, things to celebrate: the frames are aligned perfectly. The two panels have the same width. Fantastic.

BUT: the frame LABELS are not aligned, compare the X and Y labels on the horizontal axes.

They are displaced because the left dataset has ticks that take up extra vertical room, with exact fractions.

Q: how can I get those damned labels to be vertically aligned with one another (presumably down by X)?

It is very important to not add additional ticks to Y (though an inelegant hack by which we add invisible ticks might be acceptable if there are no better solutions).

My original solution was to put the labels on TOP, but, of course, referee #2 complained.

pX = Graphics[X, Frame -> True, 
   FrameTicks -> {{0, 1/2, 1, 3/2}, None, None, None}, 
   FrameLabel -> {"X", None}, AspectRatio -> 2];
pY = Graphics[Y, Frame -> True, 
   FrameTicks -> {{-1, 0, 1, {0, Invisible[1/2]}}, None, None, None}, 
   FrameLabel -> {"Y", None}, AspectRatio -> 2];

GraphicsRow[{pX, pY}]