You say he is well behaved with the child minder, but not with you. This indicates that there is something you can change to encourage more of the good behaviour he is capable of.
It sounds to me as though you are afraid of his moods, but really you can set the tone and help him to deal with him emotions appropriately.
As PP have said, consistency is key, and letting him know that you are the boss. When children feel that they are in control of the family dynamic, it is very frightening for them because they are not able to handle the responsibility and this can be expressed as moodiness and vile behaviour.
I used to be very frank with my children at this age - they understand far more than we often give them credit for. Eg if they were moaning on a day trip I would - calmly but authoritatively - say something like, "it doesn't matter how much you whinge, we are all spending the day here. So you have a choice: you can continue to whine and feel rubbish, or you can choose to enjoy it and have a good day." Don't bring your feelings into it, but help him to see that he is the author of his own unhappiness in these situations. If he chooses to perk up, make sure you comment on how much happier he is feeling now that he's smiling and taking an interest in things.
Don't let his emotions set the tone for yours. You have to rise above whatever he's feeling or saying and show him the higher ground. Encourage him, and make sure when he is calm and happy you tell him how much you love being with him because he is such good company.
My DS used to ride off on his bike ahead of me. If I tried to catch up with him, he'd only go faster. So one day, before he started to speed ahead, I told him "Isn't it a gorgeous day. I love cycling with you, you're good company". It set the tone for an enjoyable ride, taking in the sights and smells and sounds, and being together in a lovely way. Which is a long way round of saying, you CAN change the dynamic, but you have to set the tone.