In your shoes there is absolutely no way that this nasty, vile, excuse for a father/FIL would be seeing me OR my child in the future OP. I know that your DH is your child's father, and so in theory has just as much say in who your child has contact with, but if your DH is a good man, he will totally understand that you are not going to allow your child to be put into a position where his father's family can badmouth his mother. If he insists that your child has contact with his father, then you can insist that you go with them, and if his father says you are not welcome, then you would be perfectly entitled to say, 'if I'm not welcome, then neither is my child', and leave, thereby making it absolutely clear to your DH that you will do anything for him and your child, but WILL NOT be treated badly for doing so.
As a previous poster said, I would be talking to my own family, and asking them to make it clear to my DH that he has a loving family in them, and that they will always love and protect your/his child as good Grandparents, Aunts, Uncles, etc should. Encourage your child to spend as much time as possible with your side of the family, assuming of course that they are good and decent people, and live close enough to do this, so that your DH can see how well loved his child can be, even if not by his own family.
Finally, I think your DH may need therapy so that he can see, that just because his GP's obviously loved him dearly, his father isn't emotionally capable of giving his own child, or his grandchildren the same type of love, as if he was, it is very unlikely that he would have shunted him off to his GP's to raise, while he went off and made a new family with someone else, and basically rejected him when he was a young child. He presumably carries a lot of hurt from this, so I would be asking him if he would want his own child to experience the same, by allowing a relationship with his/her GF, only to also be rejected for some ridiculous reason, in the same way that you have been? As a matter of interest, when his Father shunted him off to the GP's, was it his maternal, or paternal grandparents that he went to live with, as if it was his mother's parents, then it might be that his Father's parents weren't good parents either and therefore his Father never had a good role model in the first place?