I learnt when I was three (two English teacher parents) have a lifelong love of reading, 2.1 in Eng Lit from top five uni, postgrad diploma in journalism. I think reading early teaches you to love reading - but then it depends what you do with it.
The bits in primary school where everyone else learned to read were very boring - I found the books everyone else was reading too easy and I had to sit at the back and read something else entirely. I still read too fast - I sometimes have to go back because I can't remember what happened a few pages back. And there are vast numbers of books that I've read but can remember nothing about at all. I spend far too much time on MN when I'm supposed to be working.
If you can read early you also get into books that you don't really understand - you are able to read the wordsw but you don't really get what they're saying - I remember scaring myself stiff reading the Lord of the Rings at six ... and because I read a lot and very quickly I've read loads and loads of rubbish as well as the classics. I will read anything - from the back of the cereal packet at breakfast to acres of newspaper print (when I'm supposed to be working).
DH (failed A-levels first time, humanities degree from polytechnic, postgrad diploma) also learned to read at three - and he positively suffered at primary school because he was bored and I think possibly quite disruptive.
I think DS, now two and a bit, could read early - he loves books - but I'm making a positive effort not to push him to - much more interested in him looking at pictures and thinking about the narrative and looking at the details - and slowing down generally.
As I've got older I've developed more intellectual curiosity - and that's what I'd like my DS to have early on (though of course it may be something that only comes later anyhow).