"(and if i was a police officer and accused of racism i would want the entirety of the rest of the investigation to occur in the full view of my supervisors and superiors),"
That is an interesting point. If the policeman was starting to realise he was very much out of his depth at that point his instinct may well have been to allow it to escalate partly in order to get back-up, even if in those circumstances it was ultimately the wrong instinct.
Re escalation, what I think makes it clear the policeman was not acting sensibly is that he had already seen the ID. If he hadn't he would have been in a difficult position because he couldn't have left. What happened was that having shown his ID, Gates then demanded the policeman's badge number and name. The easiest way to avoid escalation of the row then is to give the badge number and withdraw, because now he had seen the ID he could leave. (Even with Gates' tirades echoing down the street after him.)
I'm not suggesting this was an easy position for this policeman to be in and I do feel very sorry for him. Sorrier in fact than I do for Gates, as the whole thing will have done no harm at all to Gates' career and a lot of harm to the policeman's. However, I do agree with Obama that what the policeman did was stupid. It may come down to lack of training and to institutional culture in the police force rather than him being a particularly stupid or racist individual, but it was stupid nonetheless.
(Actually what I admired about Obama's comments as that he didn't force the connection by saying this incident was definitely racist - from the Indie link on Obama's speech:
'He did not know whether race had been a factor, but three things were clear. First, "any of us would be pretty angry" in such circumstances. Second, the Cambridge police acted "stupidly" in arresting somebody when there was already proof that they were in their own home. And thirdly, the President declared, "separate and apart from this incident, there is a long history in this country of African-Americans and Latinos being stopped by police disproportionately. That's just a fact."'