It's absolutely fine, and if he was willing to come to your house for a cup of tea then were afterwards to judge you for it, he would be utterly fucking weird, hypocritical and ridiculous.
As to the possible safety risk, I am surprised by the harshness of some of the replies at all - almost like, in the highly unlikely event he had proved dangerous, you would have been at fault for your apparent "foolishness" for inviting someone in to your home.
The majority of women and girls who are harmed by men are harmed by men who are very well known to them. To completely avoid any risk in our lives we would be best off having no male friends or relatives.
Yes there is a tiny tiny percentage chance he could be dangerous. And that's true whether you invite him into your home on the first date or the twenty first. Or like many abusers you might be with him months, years, even marry him before he showed his true dangerous colours.
Sorry, I am going on but this they way some posters were speaking to you just seemed rude and condescending and it got on my nerves. Your fine OP - you did absolutely nothing wrong. If you had invited him back for wild passionate sex after your first date, you would also not have been wrong.