Why does soap make water feel colder?

Whenever I wash my hands I feel like the water gets colder after I use soap. As an experiment, I soaped up just one hand and washed them both under the same water, and the soapy hand felt colder. Is this all in my head or is there a physical explanation?