Loved I remember feeling very much like you about having counselling after IVF, why would someone who doesn't know me want to listen to me moan about not being able to have a baby, there are much worse things in life. I couldn't have been more wrong. My IVF failed last year and I really did fall into a pit of despair, I questioned my marriage, my place in life as a women, as a friend as I no longer fitted in with my group of friends who all had their babies so easily. It was heartbreaking and eventually I made myself ill to the point where I couldn't smile and in the end couldn't even work and ended up signed off sick.
In the end I caved in and I accessed support through my IVF clinic and my GP, I had a great GP and the clinic had an excellent fertility counsellor. The first session all i did was cry and basically sit there saying 'it's so unfair, I hate my body and I'm a waste of space." I was convinced my husband would hate me and my family would be disappointed in me for failing, and that my husband's family would resent me for not making him a father. several hrs of counselling over the next few weeks and lots of support from the GP literally changed my life. Please don't think you are being self indulgent or selfish for accessing that support, it really will help you.
Regardless of whether you have had a baby, what people who have not suffered infertility (primary or secondary) don't always realise is that you are grieving a loss. If someone you loved died people would understand your grief. Infertility is just hidden and for some reason people just think you should get on with it. Someone once made a comment (not specifically at me as they didn't know I was having IVF) that infertile women should just be grateful they don't have cancer. 
You are feeling grief for that second baby you always assumed that you would have. It all becomes completely irrational and consuming and you just feel overwhelmed. This comment is so relevant blueemerald - "I eventually realised that the worse thing that ever happens to someone is the worst thing that happens to them, regardless of how it compares to your worst thing". I spent a long time telling myself worse things could happen and I nearly drove myself into the ground. Once I started counselling I began to see that I was feeling a very legitimate grief.
Please, please seek some help through your clinic. It really will help and hopefully will help you through some of the fog that it probably feels like you are living in at the moment. You deserve better than to beat yourself up over someone you have no control over and your little boy will love you regardless of whether he has a sibling. I'm sure he adores his wonderful mum and will be very happy to have your undivided attention. Please don't make yourself feel worse by feeling that he will resent you for not giving him a sibling, he really wont.
Infertility really is horrendous but but will be light at the end of the tunnel. Please do think about seeing a counsellor, as I say, it changed my life and got me back on track and I can honestly say that, although I still have times when i feel sad, I can cope with it and am really enjoying life. It's ok to grieve, it really is.
Sorry for the essay!