Some proponents say it's to do with your skin adjusting / deeply ingrained dirt coming out etc. I'm personally not all that convinced by those types of theory myself (but I'm not an expert).
Personally I'd look at reducing/removing the castor oil from the mix, if you're using it. It has an odd texture that's very difficult to rinse, I find.
Also suggest use kitchen roll rather than a flannel to make sure that what you're wiping your face with is really clean. I found it was very difficult to remove oil residue from flannels in a normal 40 degree wash, so they ended up a bit smelly and not the sort of thing I wanted to wipe my face with.