I'm not sure if deciding a love song has worth only if it can be sung to a man or woman is hugely relevant.
The Clapton song is plodding and monotous with a dull and predictable melody and lyrically is the equivalent of Hallmark Valentine card. Whether it's sexist or not hardly matters in its catalogue of crimes against music.
On the other hand Rod Stewart's You Wear It Well, Burns' My Love is Like a Red,Red Rose and Ae Fond Kiss or Clifford T Ward's Home Thoughts From Abroad or Gaye^ can really only be addressed to women and I'm not going to write any of them off just because you can't role reverse them.