How long have you felt like this? If it is a long time then yes, you should go as you are right, it isn't fair on him. He deserves a partner who is committed to him , wants more than anything else to spend their life with him, and isn't eyeing the greener grass in another paddock out of the corner of their eye. I for one know the desperate miserable feeling of a partner drifting slowly away. It is like watching a train wreck...
He may well be devastated in the short term, but much less hurt than if you wait until that special someone that you are attracted to comes along.
One thing to bear in mind is that the fun and attraction wanes in any relationship. It is mostly there at the start, else why would a relationship even happen, but couples tend to settle down into a more companionable relationship through time. Not having children makes this harder to accept I think, as once a family comes along we all have to become a little less self absorbed and more focussed on the needs of the children over our own.
If you do wish to go tell him on his own turf, not out in a restaurant where his grief is exposed, and then go. Hanging around isn't fair, it gives mixed messages.
Good luck. I hope you find what you are looking for, be it with or without your DH.