Oh yes, you definitely definitely have to follow through with any consequences that you threaten with. i.e. say "If you don't do x, y, z right now, then you won't be getting your Thomas magazine/bar of choc/whatever". If he insists on playing up, then you have to follow through and deny him his treat however mean you feel and however much he'll cry.
My sister admits now that she let things go when her boy was your son's age - it was easier at the time to give in to his moans and cries. And now she really regrets it as he's 5 and REALLY pushes the boundaries. So they're at the stage now that they should have been at the age of 2-3 - having to do reward charts/taking away toys and treats etc. Only at 5 the habit's more ingrained and they're finding it probably more draining and demanding than it would have been a couple of years ago if they'd nipped it in the bud.
My DS2 will always try to get away with being naughty when he's in the mood, he has been on the naughty step countless times, we really really have to be very firm with him and his dad has to take over discipling him quite often as I think he sees me as the "soft one" and pushes things further with me. I do do a lot of "Wait till Dad gets home!" quite a lot, and just the threat of that seems to make him behave a bit better!!
But sometimes speaking to him sharply and firmly when he's like that just doesn't work, he will push and push. Whereas we only had to raise an eyebrow at DS1 when he did something a bit naughty and his face would crumple and he'd say "sorry" striaght away. Even now, DS1 is the sensible one and he'll try to keep DS2 in line (which is very funny to see!)
Anyway, best of luck, these children are all different - some are easier to look after than others! But your son is at a very difficult age anyway and it will definitely get better I promise.