I know this is hard, but unless your partner is doing something bad or mistreating your son, this is an unhealthy thing he is going to need to grow out of. He doesn't have to like your partner, but telling you he doesn't want to be part of your life whilst you're WITH your partner, is really unfair.
I'm sure I'll be flamed for this but i don't care.
He's twelve years old. He's going to be a teen soon, and he's going to have to learn that life is not always about being around people that you like. If his father is directly influencing him, he needs to stop, but we know that's easier said than done.
If your partner has attributes to his personality that your son doesn't like - then ask him what they are. If they are valid then okay, that's up for discussion. If it's just a case of him disliking him because he's not his father, then that's quite immature, but then he IS only twelve, which is why it's important to explain to him that you're entitled to be happy too, and that he should open his mind to the fact that we can't always like everyone.
Is there any ground for your son not to like him, or is it simply as case of him not being his father and therefore wrong? You're between a rock and a hard place, but for the day itself might you be able to split time to spend with both of them?