Hi OP, this is an awful thing for you to go through. I felt not dissimilar with my DS...he was an absolute nightmare, I was trapped, couldn't go anywhere (or if I did, it was hell on earth). Fortunately we had a really on the ball HV and she spotted something wasn't quite right. DS was subsquently diagnosed with ASD and SPD. He is now 8, an absolute joy of a child, fun, engaging, quirky, but about to be referred for an ADHD diagnosis. We have our issues but I have learned how to manage them with good support. I am now a lone parent which quite frankly has been to both of our benefit (not suggesting that this is for you though!!), it was just that me being less stressed, able to focus more on him meant that we've found a way through that works for us. Interestingly, my son was pretty good in settings where he was expected to be good, such as when he started nursery, but he is also an epic "masker" and mimic and will do what he sees as necessary to fit in. I refer to that because of your comments of being well behaved for the childminder.
I don't want to always jump on the SN bandwagon, I am just relaying my own experiences having had similar feelings to you. Him going to school in September may bring about observations from others, they may not, however, that will at least give you a chance to settle into a more solid and regimented routine, for some children this is hugely positive.
At 4, you can still see the HV, could you make an appointment and discuss your concerns? Might be worth a shot. Good luck 