One of my closest friends has been in exactly this same situation. They are now living together.
I will admit that when she first told me, I was a bit sceptical. I couldn't for the life of me see what a 20-year-old boy who'd never left home could offer a single mother of 36 who'd had to fend for herself since she was 16. They were light years away in terms of experience and hardship. However, as she was determined that it was just a bit of a light-hearted fling at the time, plus I come from the background of having married a man 17 years older than me and didn't want to seem sexist/hypocritical, I kept stum.
This was a while ago. He wanted things to get serious before she did. She eventually caught up with him. That willingness to move to the next level may be an age/experience thing as any one who's been through a relationship breakdown is always going to be more cautious than someone whose heart has never been scarred. It doesn't really matter though, as long as you both reach the same point in the fullness of time.
Now, I don't even think about the age gap between my friends. While it's true they come from very different backgrounds and have had very different life experiences, they fit well all on the things that matter. They can talk honestly about things, they enjoy doing the same things together, he has proven himself willing to take on the responsibilities of a man much older than his years, and they enjoy fantastic sex (we all know that nugget about a man being in his prime in late teens/early 20s and a woman being in hers in her 30s ).
Only time will tell if they go the distance, but I don't feel that age would be the central theme if things went wrong. I no longer think of them in terms of an age-gap couple. I would lay money that your friends will go through exactly the same transition, toyboylover.
However, I think you really need to think about why other people's opinions matter so much to you. Is it all about the age gap or are you just not ready to embark on a more serious relationship. If you can't get past the age gap yourself, then as someone else pointed out, the age issue will indeed become a problem.
In your shoes, I think I'd just forget about the age thing and just concentrate on where I saw this relationship going. He has a right to express how he feels and what he wants, but you also have a right to say "I'm not ready for this, let's take it slowly". If you're rushed into something you're not comfortable with, the relationship is doomed to failure anyway, regardless of your respective ages. If and when you are ready to move forward, then please don't worry about people's reactions. Those that care about you will accept it or come round quicker than you think, and those that don't, well stuff them quite frankly. Good luck.