I suspect you can but there will be something very wrong somewhere.
Could be the marriage is lacking emotional or sexual connection and always has. If you love them like a brother or a friend then you might look alesewhere, But should that marriage stay in tact anyway?
Or the marriage is broken and you are trying to keep it alive by adding an affair on the side.
Or the marriage is broken and you have truly fallen in love elsewhere but you are too scared of the fallout to be the bad guy and make the first move towards separation and divorce.
Or you (not you personally - I mean 'one') are ultimately a selfish person who wants what they want and has no moral issue with getting it. I'd want therapy if I was like that.
Ultimately, anyone thinking of an affair doesn't sufficiently respect their partner or value their marriage highly enough to steer clear of temptation - so at root you'd prize your own sexual and emotional gratification more highly than the happiness, trust and security or someone you claim to love. Is that love?