Apparently too much rear derailleur cable pull

I have a Deore XT RD-M761 GS medium cage derailleur on my touring bike that
seems to move too much per shift. It shifts the chain on a 11-30 8-speed
cassette with chainrings being 36/48 and the shifters being Ultegra SL-BS64 bar
end shifters that say "not compatible with Dura-Ace rear derailleur" so they
should have the correct cable pull ratio. The Deore XT rear derailleur, being
intended for 9-speed systems, should have the same cable pull ratio too. The
bicycle has been built by me, so it is possible I botched some job. The
housings and inner cables are Jagwire index-compatible.

I can make the six hardest gears of the eight work. If I shift to the second
easiest gear, the chain starts to rub the biggest leftmost sprocket. To
eliminate this, the cable tension needs to be adjusted in a manner that makes
the usually used hardest gears not work. To make the drivetrain work
adequately, I need to move the rear derailleur quite far away from the
sprockets by turning the B tension screw as much as I can.

So, the symptom seems to be that the rear derailleur moves too much per shift.
This seems really weird to me, because I assume a poor cable routing job would
make the rear derailleur move too little per shift.

I cleaned and oiled the location where the cable moves under the bottom bracket
with no help.

The SL-BS64 shifters were bought from eBay so it is possible they are some
factory reject (can’t buy these new from bike shops anymore). However, if they
actually are Dura-Ace shifters and someone put the "not compatible with
Dura-Ace rear derailleur" sticker on them, they should have 1.9:1 cable pull
ratio instead of the common 1.7:1 cable pull ratio. The rear derailleur has
1.7:1 ratio so to get 1.7mm of movement, cable moves 1mm. For Dura-Ace
shifters, 1mm cable movement intends to move the rear derailleur 1.9mm not
1.7mm so if these shifters were Dura-Ace ones mistakenly sold as Ultegra ones
(by planting the sticker), they would have the error in the opposite direction,
with rear derailleur moving too little per shift.

The SL-BS64 shifters actually have 9 click positions, with the last probably
being intended for releasing the cable tension only, as the last position has a
different feel to it. I don’t suspect they are 9-speed shifters mistakenly sold
as 8-speed shifters, because the error would be in the opposite direction:
9-speed shifters move the cable too little per shift, not too much. Besides,
9-speed ones would not have the "not compatible with Dura-Ace rear derailleur"
sticker.

If the rear derailleur was for 10 speed systems, it would have 1.2:1 ratio
meaning it moves 1.2mm for 1mm cable movement whereas the shifters would expect
it to move 1.7mm. Again, the error would be in the incorrect direction: the RD
would move too little, not too much.

One possible mistake is that the bent segment of housing near the rear
derailleur might be a bit too short. I could replace it with a longer housing
segment and put a new inner wire too. Can a too short housing loop cause this
problem of derailleur moving too much? (I assume it would cause a problem in
the opposite direction.)

The instructions of the rear derailleur say "The sealed cap with tongue and the
rubber shield should be installed to the outer casing stopper of the frame."
but my cable housing does not have such a sealed cap.

What should I do next to diagnose the issue?

I have the following equipment:

• New inner wires and housings plus tools to cut and install them
• New Alivio RD-T4000 rear derailleur, so testing a different derailleur is an option (this was sold as being "compatible with 7/8/9 speed" but it is long cage with 45 tooth capacity so I would prefer using the medium cage Deore XT derailleur if I can make it work)
• A second used pair of SL-BS64 shifters that worked on the bike where they were used so they are known to be good (the SL-BS64 shifters that are currently in this shift issue bike have never been verified to be good)