Hi OP - I really hope that you have the strength to see this through, as your partner sounds like a horrible, vicious, spiteful, controlling twat of a man. I don't know much about bi polar (my grandfather had it, but it was just known as "mood swings" then, lol), but I can't believe for a second that your life wouldn't improve if you get rid of this man.
No wonder you are feeling anxious with someone like that in your life. Don't worry about what other people say - if you can get the marriage annulled then it would be a much quicker (and I presume cheaper) route than divorce. If people ask, you can just say "we rushed into it, and it turned out to be a mistake". But you're not obliged to explain yourself to random strangers :-)
I don't know if this is helpful to you, but I found this online - it might be worth asking a solicitor whether you could annul the marriage on these grounds:
"Grounds on which a marriage is voidable.
A marriage celebrated after 31st July 1971 shall be voidable on
the following grounds only, that is to say:
d) that at the time of the marriage either party, though
capable of giving a valid consent, was suffering (whether
continuously or intermittently) from mental disorder
within the meaning of the Mental Health Act 1983 of such
a kind or to such an extent as to be unfitted for marriage;