if they are unpleasant to you, therés no reason to keep trying. No need to make anything formal, just stop making the effort.
If your husband does not make an effort to see them either, then it is not your duty to. Family is very important, but the reality of the dynamics are that it's not working and the responsibility is not yours only - in fact, there's reason for you not to try.
if even other people can see their digs, btw, your husband should certainly be able to. But really since he doesn't seem like the sort ot make a big effort, then it shouldn't be a problem.
I can see that it might be a wrench for yoru children not to see their grandparents but you cannot be expected to be the one to make the effort if they are unpleasant to you. The ball will then fall in their court and that's just fine. If they don't make the effort, then tbh I doubt there was all that much there in the first place. If they do then good, that can continue.
The alternative is actually speaking to them plain and directly and having it out with them. At a guess they'd deny it and you'd probably not be much worse off, there's still no reason for you to make an effort given their attitude and who knows, they may actually change.
By the way, I woudlnt be totally surprised if your children pick up on something of the atmosphere going on. You can't be comfortable around them.