We do this. DH has been at home with DD for 6 years now (from her being 8 months old), and will be for at least another year.
I can understand your 'less of a man' thinking. DH did struggle with this at first - and it caused a few rows, but it was more his own pig-headdedness and worries over what other people would think. However, it sounds more role reversal in your case.
It took effort on his part to see that what he was doing was a 'real' job, and (if I'm honest) a lot of over the top praise from me before he came to terms with it - probably a year or so.
Now he proudly tells people he is home with DD - it has reared its head again since she went to school though and he has had a few comments about why he isn't back at work now (I think it is harder for him to find a school hours job than it would be if he went FT - and that is not an option yet). Most of the comments came from other men when DD was a baby, but now it is the school mums saying stuff (never mind that they are still at home??) DD likes being one of the only kids to be collected by dad, she also enjoys the things he does with her that I wouldn't do (latest achievement was hanging some shelves in the garage - he let her use the drill )
The only thing he did miss was other adult company. There are lots of 'mums and tots' type groups, but none for dads - and the only time he braved our local group not a single person talked to him. He ended up becoming a regular at the morning 'Wacky Warehouse' session and took her to the gym in the afternoons! he also spent a lot of time in garden centres and the like.
Warning:
DONT fall into my trap though of still doing most of the home stuff. DH looks after DD and does a bloody good job with her, but I'm still the one cooking and cleaning when I get home. It's my own problem, something I have bought on myself, but it doesnt make it any less tiring. DH just 'can't cope' with doing that as well...understandable when she was a baby maybe, but not now.