I'm all for the softly softly approach of getting unconfident men involved with their babies, e.g. getting them to do bathtime (which is generally enjoyed by babies and low stress) etc, and to make sure you're not criticising his care of his son, but that he gets time to do things his way (even if they're the opposite of your way) but I do think there are some cases when you just want to tell him to 'grow a pair' so no, I don't think you're being unreasonable. 
Your son is only 12 weeks old, of course he's crying a lot. An insecure and grumpy pair of hands isn't going to change that.
He needs to accept that babies cry, that sometimes babies will only settle for one parent (mostly mums but not always, DS1 always liked his daddy cuddles when he was a baby, whereas DS2 tends to always look for me) - he has to accept this. And he is not going to gain confidence with his son by avoiding him (and growing resentful...) He needs to keep at it, stay cheerful and stop being such a child. You have two children and your DD needs some mummy 1:1 time more now than before (so she doesn't get resentful and feel left out).
I think your DP needs to start having a 'slot' of an evening to look after your DS (maybe 20 mins to start with) whereby you can spend some time with DD and he is on his own with DS and can start to learn how to settle him in his own way (without handing him back to you when it gets too hard). However, I'd find it really hard listening to my son cry when I knew I could settle him, so not sure I'd be able to follow my own advice here!
You could always speak to DP about what he thinks the way forward should be, using your need to spend time with DD as a good 'in' - and see what he is willing to do (and find out what his fears are).