I think things matter, if it matters, iyswim, but I do think it's not going to work if there's a massive gap in raw intelligence.
Education, class, background etc are all things that can change. As are interests and hobbies, but the basic ability to understand things, to retain and apply information can't.
There's a gap between myself and my husband which, for the most part, doesn't cause issues, but there are times - mostly when I'm talking to certain of our freinds - when I can literally feel my brain stretching and waking up.
It's not a huge issue, because the gaps not huge, but I know (and I think he knows) that he's never going to stretch my intellect.
One of the saddest things I have seen lately though are two friends of ours. Both lovely, nice people, but she's extremely clever and in a 'clever' career. He is, bluntly, well below average - struggles with simple maths level of below average, with no other issues.
They rub along, but, oh, the change in her when she speaks to someone else. She literally lights up. She has a Phd in Literature - he can barely read. They have no conversation, because she can't speak to his level without really trying, and he can't understand the language she uses without thinking. It's incredibly sad, but I don't know why she stands it.
I'm worried what will happen is they have children. If the child comes out at his end, she'll never have the joy of teaching her child her love of language, of reading stories and discussing and spinning new tales together, and I know that will break her heart. If the child comes out like her, then it will have a dad that it will outstrip long before it leaves primary school, and that will break his.