Like my husband RFU, yours seems to have done everything to rebuild trust. We are a few years on from his affair now, so I write to give you hope and reassurance.
The 'gnawing' feeling in my case didn't go away for over 2 years. It wasn't anything to do with concerns about our relationship or what the future held, because trust had come back quite quickly for me. What preoccupied me was the past. I found it hard to stop reliving the months when the affair was going on and the build-up before it. Of all the emotions I went through, the sadness lasted by far the longest.
Unlike some other posters here, I don't yearn for the life I had beforehand, because what I have now is much better. Our relationship had been good before the affair, but my husband had always been quite selfish, lazy and complacent and I realise now that I'd learnt to work around that. His affair made us both accept that those were the reasons he allowed himself to have the affair and so they had to change.
He set about an incredible transformation and is now the best husband and father I could wish for. I love him more than ever and my respect for him is total. Our relationship has naturally reached heights it never had before, so I don't hanker for the past and neither does he.
It took me over a year to accept fully what he had been telling me from Day One - that nothing I could have done would have prevented it and it had nothing to do with our marriage. My instincts and memories of life beforehand told me that he was telling the truth about that, but because our relationship was now so much better because of the changes he'd made, I couldn't shake the feeling that if the relationship had been as good as this before, it wouldn't have happened.
We both suddenly realised that because my husband was now giving so much to our marriage and family life, my responses had improved and so our relationship was what it should always have been. The defining factor then was him and his behaviour. One night he told me that if he had been this committed, attentive and loving all the time, he would never have had an affair. This was an incredible insight that he'd arrived at through his counselling. We read Not Just Friends a while after that conversation and it amazed us both that this seemed to be the key message in the book. We were quite proud of ourselves for working this out already! 
We know that as long as we are both committed to our love for eachother and investing in our relationship, neither of us could ever be unfaithful, but it's only something we can influence in ourselves as individuals. We are only ever responsible for ourselves and our own behaviour. I now know that if my husband started being selfish again and taking me for granted, he would be increasing his risk of being unfaithful and the same would apply to me if I went down that route. So I don't worry about that. He trusts him and I trust me, is the best way of putting this - so that means we trust eachother.
RFU I would say there's still a long way to go and you might find it gets worse at some point before it gets better than it's ever been. You're making good progress but there will be more insights to come and you probably won't feel the same in 2 years time and might feel differently again a year on from that. HTH.