I know exactly what you mean.
It took my husband ages to feel confident with both my boys. In fact, DS2 is four years now, and it is only in the last couple of months that I have truly felt comfortable leaving them alone for any length of time!
He loves them to bits but got so stressed about looking after them alone, it was hardly worth asking him. When they were babies, he just couldn't get the knack of nappies etc, all fingers and thumbs:) and as the got older, it was the tantrums. I would go out for a couple of hours, and come back to find them all in tears (including him).
It did get better as they got older - I would send them out to play football, or he would take them on their bikes. Now, he has the most fantastic bond with my eldest (7 year old) and I can see it coming with the youngest.
You do need to talk to your OH, it does sound as though it is all getting on top of him. I think we sometimes underestimate how much pressure it can be - I really found it took my OH a good few years to get used to being a dad, and all that involves. It didn't help that some of my girl friends were saying I should just leave him to get on with it - sink or swim etc. When I signed up for this marriage, the deal was to support each other. I supported him through this, and now we are a strong family unit.
Give him lots of hugs and support and encourage him to build his own relationship with his child. xxx