There are two sides to this. He is still very young - 12 - and you say this has gone on for 4 years. So he was 8 when his father got together with you? That is a very vulnerable time in a boy's life, when they first start to move away from mum and rely more on developing a male bond with their father.
That must have been very, very hard for him. It must have a lot of bearing on his behaviour and attitude.
It is hard for you to develop a relationship with a child who is not your own.
I am sure you do your best to be knd to him, but, if you do not like him, you can be sure he WILL know it.
Perhaps the best thing would be for all the family to agree on "house rules" on behaviour etc. Perhaps he can earn some rewards or treats for good behaviour?
His father needs to step up and present a united front on this.
You need to sort it out now, or it will only get worse.
TBH, I feel really sad for him. I remember what my own boys were like at this age. They are so vulnerable and they need tons of affection and love and reassurance.