Shimano 105 5800 mech compatibilty with 5700 shifters

No -- you need to match the number of speeds on the cassette and the number of speeds of the shifter. The 11 speed shifters are designed to use dyna-sys cable pull (which is different than the 10 speed road cable pull; its the same as 10 speed mountain) as well.

I'd recommend picking up either {a 10 speed road or 7-9 speed Shimano rear derailleur and a 10 speed cassette} or the 11 speed Shimano road shifters. The former (e.g. a 9 speed Deore RD and a 10 speed cassette) is likely going to be much cheaper.