We got a camera for ds1 when he was 4 because he liked using mine so much.
Like the other posters have said, we didn't bother with a kiddie camera because quality was so bad.
We just got one of the basic ones from a reputable brand that was on half price offer, so paid about £50 for it. I got the case and memory cards online as they are much cheaper (7 day shop or play.com are good, reliably cheap for good makes even if not the cheapest and safe to buy from)
In hindsight, we bought one with aa batteries in - we wouldn't do that again but buy one that has its own rechargeable battery in, and buy a spare (again would get one from 7 day shop rather than the proper thing from the manufacturer). I would also buy one that was the basic one in the range that my camera was in (or had been in, having had several over the years). It's just that each manufacturer has its own way of splitting up the functions, it makes it that much easier if you are using their camera if it is similar to your own. And there's also a chance that you can share batteries and chargers which is handy if you are away.
I reckon if you aim for something like a basic canon, panasonic, nikon or samsung you should be able to get him a great camera for about £50-60 (but look out for deals as you should be getting one for that price that was £100-120 initially and maybe even more of a discount through quidco or similar).
Avoid cheap makes like vivitar though as although they are 'proper' cameras rather than kiddie cameras, they are cheap and nasty, much better to get something decent but basic from a good make.
If you have more money to spend, there are some cameras that are designed to be rugged or waterproof, which help in child-proofing a camera!
but for what its worth - ds had his camera when he was 4, he is now 7 and it is still working well, it's been lovingly looked after and he takes some fab pictures - including lots that adults wouldn't think of taking. OK so sometimes there is a reason for that (do I really need 63 pictures of different areas of the same carpet at a wedding ?) but others are great!