that comment just made me feel like the biggest fool alive
A comment he made doesn't make a fool out of you, it makes a fool out of him. Tell him it was a really unkind thing to say, and, given that these days you're focusing on having kind people in your life, you don't want to hear from him again.
Then walk away with your head up, and follow through with surrounding yourself with people who wouldn't make such a thoughtless, insensitive and unnecessary remark.
His life, his comments, his opinions, his lessons, are completely and 100% distinct from your self esteem. You are making the mistake of linking them together. Remember that you are brilliant, unique, very skilled in certain things (we don't know what, but you do), and you are respected by those who matter.
There's nothing special about you that means you ought to walk through life feeling like a piece of crap, regardless of any comments that anybody makes to you.