Hmm tough one!
I would say that for a few years I had fleeting moments (literally only split second thoughts) about my husband perhaps having an affair/being unfaithful. But we seemed so happy and so I never really dwelled on these thoughts. I dismissed them as soon as the entered my head as the thought was so ridiculous. I also could not work out WHEN he would have the time to do it as he was/is a very hands on Dad.
Oh how I kicked myself. January last year I found he had been screwing any munter desperate enough to post a pic of her fanjo or titties on the net with low class slappersih name to match like "cumontitssuzy" or "Analanna" or "saraluvs2suk" etc etc. Free prostitution in my mind. Anyhow, he had been stopping off in his lunch break, on the way home from work, when I was out with the kids etc for a quickie of whatever he fancied for 4 years!
No one would believe it. My husband is a shy man! LOL
In all seriousness, I now kick myself for not even allowing myself to absorb those initial fleeting brief thoughts. If only I had.......!
I would suggest not panicking but keeping an eye out. Don't confront because its a universal known fact that 99% of peeps being unfaithful will not admit to it unless there is evidence to back up your claims.
All you can do is watch and look for further evidence. Comp history, mobile phone secrecy info, hidden phone, pockets, cars, bank accounts, car mileage are all things to consider looking at. I did this and discovered enough to make my husband admit to the Full enormity of his unfaithfulness. I am sure he would have denied it or even with just a little evidence, he would have perhaps admitted to only one shag/munter.
Anyhow, it can be very very easy to let your mind get carried away with all this. So make sure you step back every few days and ask yourself brutally honestly - do his actions actually = an affair/unfaithfulness. There are lots of reasons why he may go out 3 x per week. But note them down, dates, days time etc and see if after a few weeks there is a pattern. Again this may mean he is meeting his mate John for a pint every week etc and not Jane for a shag! But by being more vigilant you are more likely eventually to uncover something than if you dont keep an eye out.
I really hope its all innocent and you have nothing to worry about. Good luck.