I've seen this described as a red flag on here and it's something that's been said to me in more than one abusive relationship.
I've just read a trashy book and it ended with the romantic hero saying something along these lines to the heroine. Made me think he shouldn't be saying it or she shouldn't trust him as he must be a wrong 'un! But can it ever be ok?