Ok, first off I'm sorry you're going through this. I'll be honest, I've never suffered infertility myself so the following may be a load of old bolleux (but didn't want to leave you unanswered).
It sounds to me as if your husband has taken you saying that you're finished as you saying that you never want children. Whereas for you it's more you don't want to go through the mental rigmarole of the process of trying for children. I think you desperately want to put this issue to rest so you can just put yourself back together mentally. There is nothing wrong with that at all.
You absolutely do want children, but after three years you're tired and you need to step back from it. For your husband, maybe he needs to step back as well. For him, this conversation may have come as a bit of a shock. So maybe he needs to take stock of it all. Has your sex life become all about conception? Could it be that he's reading into the situation that you are done with sex as well?
Maybe give him a day or two and try and broach the subject again. Maybe he'll be in a better place to discuss things a bit more?
I apologise if I'm talking complete rubbish, but I hope if nothing else my ramblings have bumped your post up the board.
my husband and I have been trying to conceive for over 3 years and I told him the other day that I can't do it anymore I'm done crying I'm done worrying all the time I just want to be happy. I always believe that one day it will happen that one day I would be a blessed and I'D be able to be happy.but now I really don't think it's going to happen, I feel like I'm not meant to have a baby,my husband is an only child and only grandchild if he doesn't have a baby it and his family name His blood line and that stresses me out even more. Anyway I had told my husband I didn't want to try anymore and now he wants nothing to do with me sexually I mean. Everything else is normal talk to me but its not trying to have a baby he doesn't want to have sex at all that he wants to stay completely away from sex I don't know if he's thinking that I don't want a baby at all because that's not what it is I'm just tired of the stress I have tried to explain this to him but I don't think he understands... what do I do?