When people say a prospective partner is good 'on paper' it generally means that rationally they think it's good catch - without necessarily feeling it.
Do you like the idea of his education, privilege, wealth and what it could mean for your life - without really connecting with him emotionally?
I'm from the same kind of background as this man, and I have zero tolerance of Toryism in that context, he's intelligent and educated enough to know better.
You can have it all - you could find a man with all his merits, and be on the same page politically.
If he wasn't politically engaged or was a floating voter who sometimes voted Tory it would be a different matter. But someone who identifies as a Tory, is politically interested - it's a fundamental part of who he is.
You will get more and more pissed off with him as time goes by.
If he's good in bed, just enjoy it and then find a partner more on your wavelength.