No, they're in no way similar. A small penis will actually affect the amount of pleasure both men and women get from penetration (as will an overly large penis), whereas breast size doesn't make a difference.
For most men, size isn't a factor with boobs, we like them all equally. Some men aren't that fussed at all, and yes, some men prefer big over small, or small over big, but for most of us, our partners breasts are the best breasts because they're the ones we're allowed to touch.
However, I do think breasts can be a bit of a mystery to men. Everyone is slightly different in what gives them pleasure. Its fairly easy to work out what works down below, the signs that your partner is enjoying themselves tend to be fairly obvious both audibly and visibly. With breasts, less so.
My DP for instance will give no reaction when I play with her boobs, I'm more likely to get a sigh out of her if I stroke her back or almost any other part of her body. I still play with them a lot, but I always thought it was more for my own benefit than hers! What I didn't do is use my mouth on them (I'm starting to get a bit TMI here, I know. Bear with me) It doesn't do much for me, and the few times I tried it it didn't seem to have much effect for her, so what's the point?
5 whole years into our relationship, she asked why I never went there. Turns out she loves that, and was upset that I never did it. Who knew! What I'm saying is communication is key, if your partner isn't doing something you want them to do, or vice versa, talk to them about it, let them know you enjoy it.
I'll make one other point. I went out with a girl for a while with really tiny breasts. Didn't bother me, if it had I wouldn't have gone out with her in the first place. She on the other hand was deeply insecure about it, and as a result wouldn't let me touch them. Absolutely fine, but it did mean that when I went out with another girl with equally small boobs, I was slightly wary about going there. Maybe you husband has had the same experience and has just assumed you'd not be keen either. Again, talk to him about it.