I think the main factor would be, do you want it to be portable or not. Portable is great for car journeys or visiting Mum & Dad's boring grown up friends who don't have children etc. Also do you have a telly that you are prepared to have taken over by a games consol. If you only have one TV and other children it will cause problems.
The Nintendo DS or DS Lite (replacement for Gameboy) or PSP are both portable. I would reccomend the Nintendo as the games tend to be more suitable for younger children. Lots of Disney games etc.
If you don't need portable then I think the Playstation 2 would be best, as it seems most common and that means he can swap games with his friends. If you get this I would highly reccomend getting the Eye Toy with it. This is a camera that sits on top of the telly and lets you use your body to control the games, its great fun to play with and can really wear the children out !
There are two major new releases on the way
Nintendo Wii. which is due 8th Dec and costs about £180. This is similar to a Playstation 2, but light years ahead of it. Its got really good press reviews so far, but I think Nintendo have a habit of not making enough units available when they are first released, so it might be difficult to get hold of one.
Playstation 3 release date has been put back to about March. I get the impresson that this is for serious gamers rather than young children, as it's going to cost over £300.
Which ever option you go for, please check when you buy it that you have everthing you need to use it. For example you need a memory card with the PS2, but it does not come with it !
When you come to buying games look for pre-owned games in the shops (GamesStation seem best where I live). You can save a fortune this way. Also get you child into the habit of trading in games he is not that into, as you get quite a lot of money back on newish games.