I, personally, found the angsting about whether or not to do IVF, far harder than actually doing it. Once the decision was made, it was fairly plain sailing. Injections were not really that bad, I didn't feel particularly hormone-y or bloated, the EC went fine (knocked out, so not bad at all), and the ET only a little uncomfortable. As far as I know, there are no proven risks to a IVF baby (apart from any risks that multiples would face, if you end up with a multible pregnancy).
What I did find, though, and what I didn't expect, was how disconnected I would feel with the resultant pregnancy. I don't know if this was due to the 3 years of unexplained IF or to having to resort to something as de-personalised as IVF, but it really affected me badly. I couldn't get excited about the pregnancy (I spent the whole time not really believing it had worked and thinking it could end any time), and it affected how I bonded with my son. It's not something I'd ever heard anyone talking about (how hard it would be to have a successful IVF cycle - I mean, who really wants to complain about that), but it would have been useful to know that it was a possibility.
If you have PCOS, and clomid is a good bet for you, I'd maybe give that a shot for a few months. Set a limit of, say, three cycles, and then at least you know you've given the low interverntion approach a shot. We didn't try anything before we went straight to IVF (there were reasons), but I sort of wish we'd tried something else first.