It's been alluded to by Andrew Lownie, and while I dislike him and previously thought his claims in regards to the Royal Family were outlandish, they have proved, if anything, to be conservative in regards to AMW and Fergie and he has been vindicated in regards to those two, so I believe rumours of Phillips' sexual indiscretions have merit. Add to that the various rumours over the years, plus his will was sealed wasn't it?
Adultery isn't looked upon as a moral failing in high society in the same way it is at the lower end of the social strata. Having a lover or a mistress isn't necessarily seen as divorce worthy. Affairs are common. You just behave discreetly. Not being discreet is the greater sin. Many aristocratic marriages through the centuries have been effectively open marriages, but cloaked in outward proprietary.
Consider that there was never going to be any contemplation of divorce between QE2 and Prince Phillip, no matter what. That is carte blanche for doing what you want and getting away with it. It's plausible that during a period of strain due to Phillips errant eye (I'm merely speculating, I don't know what is true) the Queen turned to someone else for comfort and there was never any questions of Philip not accepting any results of that. He had his affairs and she had her secret.
If the late Queen did have a rendezvous and birth a child, then my personal view is (giant shrug) what of it? It doesn't reflect badly on her at all in my book. I have always found it odd that Britain has had both Kings and ruling Queens and the Kings could be having multiple sexual liaisons and sire bastards left and right both whilst married and not and yet Britain's Queen's have had to be sexual puritans. Queen Victoria very obviously enjoyed sex, but that expression still had to be strictly within the confines of her marriage, unlike her male uncles and cousins.
If their Queenship was truly equal to Kingship I see no reason why they couldn't shag who they wanted, if anything you know for sure any children birthed are legitimately Royal. Can never understand why Elizabeth the first had to be the eternal virgin. Clearly women had to abide by different rules even whilst holding equal sovereign power, which I find ridiculous.
So I would not rule out QE2 having something secret for herself, no. All that duty and effectively not belonging to oneself but belonging to society and being a living breathing symbol of a nation has got to be very tough on the psyche.