My DS went to a childminder when I went back to work when he was 9 months old and settled in fine. However, there were only 3 other children there and he was used to being around little ones as we'd been going to baby groups by a long time then.
When he was about 18 months old we started going to toddler groups and he really didn't like it, I think being around so many other children (probably over 40 other toddlers) and there being so much noise just really unnerved him and for a good few weeks he wouldn't leave my side. I really wanted him to get used to being in that environment though as it was my intention that when he turned two I would take him out of the Childminder's care and put him into Nursery.
When he was 2yr 3m I put him in nursery and although I was worried about how he'd cope he absolutely loved it but by this point being amongst large crowds of children just didn't bother him. We started off doing settling in sessions, I.e a few hours here and there without me until we finally built up to his full 10 hour day.
I suppose there are two ways of looking at your situation, you either take him out or you accept it may be difficult for him whenever he starts going to nursery and so that now you've started it you may as well continue. Only you know what is best for your child and I imagine that if my child was reacting to nursery in the way that yours is then I would probably be upset by it.
If you want him to learn how to socialise with other children and get him used to being in those sort of environments then pulling him out defeats the purpose I guess. He's only been going 5ish weeks so maybe give it more time? Or maybe he needs his sessions to be longer so he can settle in more and get used to being there?
Or maybe you could try a different nursery but accept that you may have the exact same difficulties there.
Or maybe you could try a childminder so it's much more home from home and there's less children for him to adapt to, it might be an easier environment for him to be in initially?