Are you still speaking English to him? If you have both swapped that is probably freaking him out a bit!
My oldest was only 2 months when we moved, but her English was also very advanced, so much so she used to get comments, as she was little for her age and almost bald :o - she was speaking in full sentences. What she did was different - she reverted somewhat to babble, which also worried me (and to be honest annoyed me, as I had been so proud of her language :o ) She came through it after about 4 months, and her English went back to normal, and her German started to take off - she didn't start Kindergarten until the month of her 3rd birthday, but by then she was relatively fluent (actually spoke better German than some of her same age peers, but I think that's because there doesn't seem to be much of a culture of having conversations with toddlers here).
Maybe the move is overwhelming him, and the language swap on top is too much - we kept English as a family language even though DH is German, so home was always an English sanctuary - although of course we speak German with Germans in the house, I only ever speak English to the kids, and as a family we all speak English together. As I say they picked up German from the playground (we do spend hours and hours a day there) playing with neighbour kids, and from toddlers groups. None of the kids needed any extra help with German, and in fact DD (the only one yet at school) excels in German, its her best subject and her teacher says she has a talent for the written (German) word and exceptionally good understanding of what she reads wish she was as good at maths
All I can advise is dialing the exposure down - especially not speaking directly to him (nor to DH if that's not what he is used to) in the community language, then he might not feel he has to reject it to preserve his safe familiar family environment, or his own identity. He'll still pick up the community language if you socialise - make it the language of other children, fun and playmates, but keep yourself his familiar English sanctuary.