I could have literally written your post. My DS is also 3.5. Doesn't answer questions or have any sort of back of forth with me or anyone else. I tried toilet training twice already and got nowhere. Going to try again when he's on summer hols from nursery but not holding out much hope. Similarly he would rather hold it in until he couldn't any longer than use the potty or tell me, etc.
We get very minimal verbal communication to let us know he's hungry, hot/cold, wants his jumper off, wants to go outside, watch Cbeebies, etc but it's not always clear. We are gradually starting to get 3-4 sentences but it's sparse and out of context a lot of the time.
I first raised this with HV when he was 18 months (he was pretty much silent until age 2), got his hearing checked and all fine, and it's been a bit of a wait and see approach with the HV. He started nursery in January and they picked up on his communication and verbal development basically straight away and are in agreement with me that we are likely dealing with a speech delay and my son needs extra support.
In my area you can put in parental referrals for speech and language, and lately because his nursery agree with me and have now gotten our HV to help too, they are fast tracking our referral for us because it's been going on so long.
From what you've said, I'd definitely look into possibly getting his hearing checked with a GP and contacting your HV and seeing what they think/ can do if they think he should be referred for SALT, etc.
Not saying any of this to worry you, but just sharing my experience and where we're at currently. I can also appreciate the frustration the lack of communication probably causes for both you and your DS at times. I personally felt like I couldn't pursue with toilet training without his communication improving which at this point I feel like I definitely need some outside help with x