If he let you go through his emails and you found this photo then he doesn't strike me as a master manipulator who's been hiding his tracks.
Have you logged into the dating site as him and checked his sent messages? I would probably want to check that before making any decisions about staying or going.
Actually, before you do that - sign up your own profile on the site and see if his is visible through search. That will not only show you whether he did delete the profile, but most dating sites will also show "last logged in 2 months ago" or whatever.
(You need to do this before you log in as him because the system will then record you as logged in today.)
On most of these sites you can't send messages without paying for membership - so if you have access to bank/credit card statements, look for a charge around the date of the email you found.
Would it help if he stayed away at a mate's or relative's for a couple of nights to give you some space to think?
He may have been "just looking" and certainly if you look at these sites there are an awful lot of profiles which show as not logged in for years and years.
The fact he has given you full access to his phone, FB and emails without (I assume) a hint of panic, would go in his favour. I would also assume there's no other signs like sudden recent emotional withdrawals, lack of sex, unexplained absences?
If you feel confident in your mind that he hasn't cheated, maybe counselling for a few sessions could help you work on rebuilding trust?