You might as well accept that no matter what you do, he's going to make stuff up in his head and then blame you for it. That doesn't mean that what he blames you for is appropriate, justified or right. Neither does it mean that you have to go along with his demands.
Two calls a day is too much. Demanding that your young son give your ex his full and undivided attention on a video call is completely unrealistic. Toddlers have very short attention spans. If your ex is unable to accept that then that your ex's problem to deal with, not yours and not your son's.
The next thing is how you should deal with your ex's tantrums. You are under no obligation to sit there and listen to your ex rant and rave. If it's not a productive and respectful conversation then put the phone down. You are allowed to do that. In fact I'd strongly recommend that any conversations you have with your ex regarding contact with your son be done in writing - email, text, whatsapp etc - rather than over the phone. That way you can keep a record and you can choose when to respond.
If he calls you up to whine about how he thinks contact should go, tell him to put it in writing and then put the phone down. Again - you are allowed to do this. You don't have to drop everything and respond to him just because he calls you. Anything that isn't about contact, your child's immediate well-being, or money should be recorded by you but otherwise ignored. Don't rise to the bait, don't get drawn into arguments, don't respond to insults. Keep it business-like, keep it all in writing, and keep it solely about contact and your child's best interests.
If he is genuinely unhappy and thinks you're being unfair then he can take you to court. But no court in the land is going to insist on two video calls a day. And, frankly, if he doesn't have the money to house himself he really doesn't have the money to take you to court.