That's exactly the type of attitude that is about to open up a whole world of pain for you and your child. Firstly, a court would take a considerably dim view of your intention to move without any form of reconciliation, and would likely make an immediate order that you both enter into some form of mediation.
If you, him or both of you refuse or mediation breaks down, the court will continue to suggest you make an effort for the sake of the child. The mediators will also report back to the court so unless you have a very open attitude about such things you could find yourself on the receiving end of a lot of negative attention from a court. Of course, the same would go for your partner as well.
If your partner is as bad as you say, you'll need evidence and any mediator should see this also, if, for example he fails to attend or gives off a "can't care less" attitude. Even then, the court would likely still try and force mediation but it will only get to a certain point before the court would likely see through either your, or his intentions.
On the other flip, if you move away, the court will consider where you are moving from/to. Are you moving away from a support network of family/friends. Are you moving towards a support network of family/friends. Is it work related or simply a "desire". If you've simply had enough of him and want to get as far away as possible with no real grounding, then you're potentially in for a long road. The fact you're asking if he can stop you says to me he's likely either vindictive or not as bad as you claim, in other words, why would he want to stop you?. If it's the former, the court will see through it, if it's the latter then the court will expect you to be responsible for the sake of the child.
If your partner is named as the father on the birth certificate of the child, he automatically has parental rights which cannot be taken away without good reason and only with the order of a court which is a considerably difficult thing to do.
If you are serious about moving without having first taken serious legal advice or considered ALL options between you and your partner, that essentially amounts to abduction which is a criminal offence. If you move, and your partner doesn't make any attempt to locate you, a certain period of time would have to elapse before your partner would find it difficult to do anything about it, unless he can prove he has made attempts to find you. If he has made attempts to find you, and eventually you are found, this could even lead to a court order in his favour to take custody of the child as the court would consider your actions to be highly inappropriate and irresponsible.
If you believe there is any hope of the relationship working, find a mediation service, they are often free, start there. DO NOT just up and leave.