It does seem like anxiety is your largest issue here OP. Is the relationship actually a positive thing for your both? - it's not very healthy for him to not see his family, or for you to not be able to even hear their names, or to not be able to work whilst he works two jobs- because of it. What are you doing to treat your issue, rather than acquiesce to it and have it affect your life? Panic attacks at the mention of names is very far out of the realm of healthy.
His family may well be dicks or maybe have just made some ignorant comments- have you sat down with them and explained that it's a trying time and the comments are hurtful or have they just been written off so you can avoid doing this? It does come across a bit like you are so sensitive to the issue that you are isolating yourself and are now going to isolate yourself from your partner.
Other people are never going to be perfect or get what you are going through in the same way as you, of they have not experienced it- they don't exist to make you feel comfortable sadly, and the fact they have children and you do not is not bad behaviour on their part sufficient to create a family rift, though it may feel deeply unfair. Surely you see children out of the house? You will need to find the tools to deal with your feelings as your own person Op rather than expect others behaviour to be what you need and want. I'd recommend seeing a counsellor before this becomes even more damaging for you. IVF is very challenging and not guaranteed to work, whether you agree with the former or not, therapy could very well help you manage your emotions in a healthier way during that process.
Re: the tomatoes thing, it sounds ridiculous. After a long day at work that would really annoy me- if you want some attention from your DP or feel you don't get enough ask for it directly rather than resort to things like that. It doesn't sound as though he swears at you often but again- if you can't bear it then perhaps you are with the wrong person OP. Or need to manage your expectations. I wouldn't use 'typical man' to excuse lack of emotional engagement though, plenty manage just fine.
I dont think anyone here thinks struggling to conceive is a small issue, but neither is anxiety
.