Wow. I'd love to know a bit more. Sorry if this is long but I've been churning all this stuff in my head for the past few weeks, and this seems like a good place for it to come out.
We have often thought DS1 was Aspergic. His nickname, given to him by his classmates, is Sheldon. (In fact, I think TBBT's popularity helped him get any mates at all. He was shunned as weird for the first few years at primary.) He had some very AS typical tendencies as a small child (zero interest in his contemporaries, buttonholed adults and fired questions at them, not very cuddly, very fixated on certain things, almost pathological hatred of change.)
So we were very surprised when DS2's SEN told us they think he has Aspergers. It had never occurred to us. Different set of qualities from his brother. DS1 is Sheldon's mini-me. DS2 is utterly different. But he had become increasingly withdrawn and socially inept (having been fine as a very small child - in fact, very outgoing.)
So DH looked up AS online and came down stairs and said: 'it was like someone had described me.' Then F-I-L asked S-I-L (who is an SEN) if she knew anything about AS without mentioning why he was interested, and she said, 'Well that's what 9my) DH has.' So she'd diagnosed her brother in her mind, but told none of us.
So many things that have frustrated me over the years actually now make sense. The chronic, orderly hoarding of chuffing magazines (and the disorderly hoarding of everything else. He can't let go of anything. The absolute inability to deal with change of any kind. The inability to be spontaneous. The social inappropriateness and sheer lack of engagement with others (notably, dismissing a social group of over 50 other men of his age in our village, because 'none of them was interesting' - what? All 50 of them bored him in one evening?)
I've become lonely. I used to be so outgoing but social invites just tailed off, and I got bitter, because I'd meet new people, get on so well with them, get invited over with DH, and then suddenly the invites would dry up. Still keen to meet for coffee 1-2-1, but never included in those easy going BBQs and sunday lunches or dinner parties, and I somehow knew it was because of DH but felt awful thinking that because it seemed like I blamed him. But I got so frustrated at having no friends. Guess which one of us ended up on ADs.
And he eats up so much emotional energy. It took me 10 years to persuade him to try for Dc, even though he knew he wanted DC. Definitely. Just wasn't ever ready for the change in circumstance. Same with moving house. He hated London (I loved it) he wanted to move to country side, but even after a shooting in our street, I actually had to go behind his back and rent a house because we'd have lost out on our house sale due to his refusal to make the move. Same with foreign holidays. he refuses to go - I wouldn't force him if he didn't want to - he wants to, he just refuses to want one this year (because it involves change) I feel like I've turned into someone I'm not, someone who has had to issue pretty aggressive ultimatums (we try for Dc or I leave; we move house together or DC and I move without you) simply to get him to agree to make changes that he is entirely in favour of.
Despite being absolutely brilliant in his field, he ended up sidelined and hasn't worked in years because he is in a profession that requires social interaction and he just wasn't up to it.
We're lucky in that he does have lots of redeeming qualities. He's very reliable. He is very funny (to me. We share an odd sense of humour. Not everyone gets it.) He tries to be kind. He tries hard, generally. He's not interested in straying. He already has one. Why on earth would he need another one? 
But like you OP, I get so lonely. He doesn't really talk with me, or listen properly. He doesn't connect in conversation. Having said that, since he read up on AS and discovered his sister thinks he has it, he's been much more understanding of my frustrations at his lack of response in conversation. And I've felt much kinder towards his inability to be friendly or to handle change.
Phew. Sorry. Been rolling round my head because I'm so worried for DS2 (and tbh DS1, who is almost certainly on the spectrum, but unlike DS2, doesn't mind so much about friendships and fitting in.) But suddenly finding out that DH is likely to be Aspergic too has made me revisit so many problems we've had in the light of that.