OP - You don't say much about why you choose for your son to live with you ex; I imagine that must have been a very difficult decision. I'm sorry. It must now be even more heartbreaking for it to be so hard to have contact. I had post natal and know how desperately hard things were for longer than I care to admit.
Do not give out your address unless you are comfortable. There is no obligation on you to do so. In fact, you absolutely should not if you don't feel safe. Contact can be arranged in a 'safe' place therefore there is no need for your ex to ever know where you live - your parents may be a good handover place for example.
You do really need a contact and residency agreement. It is not right for one parent to use a child as a bargaining chip; that is abusive both to you and the child. You both need to act in the child's best interest and the child has a right to see both parents consistently (unless there are compelling reasons why time should be limited for the child's benefit - not just because your ex is on a power trip...).
Domestic abuse can and does exist in child contact arrangements and it is extremely damaging to a child. Abuse often gets worse during contact. Call Women's Aid please, they are amazing. If you are allocated a Women's Aid care worker they can even come to court with you as a Mackenzie friend to support you. It is not for fair for your ex to refuse for you to see your child at your home unless there is a very good reason it would not be safe for the child to be in your care, in that environment. CAFCASS will assess both of you and make that decision. The ex's GF sounds delightful (NOT) however they are unlikely to insist she can't be around; I'm afraid much of what you describe is sadly par for the course (not the throwing stuff at him). I am not saying this is right or that I agree with this, rather the way it is. You need to be above all of it - not engage in arguments, or nastiness, do not berate your ex to your child etc. Taking your child (whilst I fully understand your desperation) wasn't ideal. Personally, I would have called the Police but I do wonder how you managed to do this? Was the child not supervised?? Or do you mean that he gave consent for you to have the child, just you didn't have him where he said you could and he found out? If the latter, then you had every right to take your child home. A detailed contact order defining when, where, times etc should eliminate the need for much contact between you and your ex.
If you cannot afford the solicitors fees to go to court you may find Corman Children's Legal Center helpful. They have a free advice line staffed by qualified lawyers who can provide guidance.
In terms of abuse, be aware the family courts set a very high bar for what constitutes abuse that would be considered sufficient to refuse contact. That said, the NSPCC have a helpline you can call to seek advice (without giving your name if you wish). They are equipped to tell you if the behaviour is abuse.
PM me anytime.