RD-5700 impossible indexing, hard cable pull (fresh setup)

Are you using the correct housing?

In my experience, the only gear housing that's worth using is Shimano SIS-SP41 / OT-SP41. Off-brand housing usually results in crappy shifting. The end caps used must be those that are by Shimano, for gear cables, and you should carefully select each end cap type to be either a sealed cap with tongue, a sealed cap with O-ring, or a cap with no seal. Usually areas prone to dirt intrusion should use seal. Your shifter and your rear derailleur might be intended to use a certain type of cap, such as a cap with tongue.

Also, in my experience, the only inner cable that's worth using is Shimano stainless steel 1.2mm or Shimano optislick 1.2mm. Your system is probably intended to use optislick.

Things you can try:

• It doesn't really line up with the seemingly low friction coming out of the first piece of housing, but ST-5700 is very susceptible to excess friction getting generated in the handlebar area, namely in the curve right after it exits the shifter. It can be necessary to try both of its routing options.
• If you look at the SI document for 5700 (it was before they switched to the DM format), you see they call out a coated cable. No mechanic likes the idea of a system that needs coated cables to function well, but with some STI generations you just have to cut to the chase and do it. Use either optislicks or the polymer Dura Ace ones that leave hairs, which in my experience usually obliterate the kind of problems you're describing. Use them with Shimano housing and nosed ferrules where applicable.
• The rear loop not being long enough can cause or exacerbate problems like you describe. You want it running straight into the barrel adjuster for at least 12mm or so.
• 5700 was before nosed ferrules on road groups, but you could try it anyway. It can help with friction generated at the bar routing.

Cable friction going up massively only when the cable is anchored is a strange problem. (The cable being on the right side of the anchor bolt could cause something like that, which is worth mentioning but I know is likely not the problem.) If that's really the moment the friction becomes noticeable, I think the most likely explanation is that the return spring force is causing the cable to run straighter through the housing and ferrules, and that's either causing it to drag on a ferrule entry that's not an issue with lighter tension, or the tension is adding friction at the bends in the handlebar area.

One thing I'll do in spots like this is choose what looks like it should be the most logical exit path from the shifter, then run my housings and leave it without any tape on it at all, bar tape nor the tape you use to keep the housings in place under the bar tape. I set up the shifting with it in that state until I'm satisfied, then when I put the tape on to secure it to the bar, I check afterwards that there's been zero impact to the shifting quality, and tape it differently if there was. Then put the bar tape on and repeat. Sometimes in doing this you will find that the way you applied the tape was enough to add friction, but if you had secured the housings or wrapped it first you wouldn't have known what to look at to solve the problem.