It is quite normal to have a crush on your boss.
Is the fact she is a woman really bothering you? Is this something you didn't know about yourself. I wouldn't worry, there are lots of attractive people out there, including this woman, and unless it is truly messing up your work, I can't think why you would want to quit.
Having crushes definitely passes the time away at work, that's for sure.