@BangPippleGo hi, ds was adopted aged 3 and didn't take to dh much at first.
It is hard but it is normal.
Interestingly our birth dd went through a phase at about 3 of not wanting me. She wanted me to move out and her to just live with dh. It was very hard but we just carried on, I said it's my home, I'm not going to move out!
Try and get your dh to read up a bit on how bonding works (in adoption terms) and encourage him to join in. Swimming is good. My dh is physically very fit, I am not! So early in with both kids dh did the sling carrying, soft play adventures, and even the swing pushing (I'm lazy too).
Yes, Yes to sleeping when baby sleeps and getting longer naps. At that age I think my dd had two good naps a day. Maybe experiment with the room, trying lighter or darker and see what works.
Children will go at their own pace. As they get older you can chivy them along but at this age it's their pace. I think your dh needs to understand this.
It's lovely being the object of affection (you) but exhausting so hopefully your dh will relax and play the long game, the affection will come.
We have found games, welcomed tickles, joking around all helps. Find a few fun things to do with baby type book and get your dh to work his way through a few things gently with baby. Like round and round the garden/peekaboo etc.
To be honest this is very similar to a new baby by birth, they do tend to cling to mum often but it changes quite fast.
Please do not let anxiety over your dh spoil this time for you. Be sympathetic but don't feel guilty, it's very natural.