I'm guessing your in the nursing profession or similar?
I'm a nurse and went back to work when DS was ten months old. I started of going back for 2.5 days (two long days and an early shift) but then eventually worked back up full time: 3 x 07.30-20.30 shifts.
I was worried that DS would really struggle with me being away from him or three days but he adapted fine. I was made to feel guilty about my work choice: "How can you not see your child for 3 days a week?" Etc etc but like you I weighed it up and thought that yes, for 3 days a week I don't see him (bar 45 minutes in the mornings) but I get four full days with him a week which is more than a lot of full time working parents get. My DS went to a childminder two days a week with my DH doing pick ups and drops off.
Although DS was fine initially I have definitely found that as he gets older the more he is struggling with being away from him. When he hit 18 months or so he started to get very clingy with me, cried if I even left the room etc and I think because at that age they have an awareness of time, whereas babies don't, and in his head when I left a room he had no idea whether I'd be back in 10 seconds or be gone for 24 hours. I do think it's quite unsettling for them.
DS is 2yrs 3m and he is much better now but he is still very clingy to me when I'm home. He understands now though that mummy has to go to work and I will see him the next day but he still gets very upset when I do go to work.
I may be wrong, but it may be far more beneficial for your child to solely be in nursery when not with you as opposed to being passed around various family members. Children his age need good structure and routines and he will be struggling with the issue he has to go longish periods without seeing his mum but he will also be confused about the fact he has no idea where he will be or who he will be from day to day. It may be very overwhelming for him and I imagine he doesn't feel very secure in this set up.
My son has always knew he was either with me or his childminder and now he's in nursery he knows he's either with them or with me. It gives him a secure base and means he only has two main caregivers as opposed to being passed around a variety of people with no set routines.
I understand why you'd be worried about adding in another carer but actually, putting something more solid into your child's routine may really benefit him.