They can't let him go because Johnson can't function without this man telling him how to run the country at every step.
He made an error of judgement.
When people were unable to say goodbye to their dying relatives he saw fit to drive his wife and child 260 miles to his parents house, just in case. While he was there he had a nice jaunt out for the day, a 60 mile round trip, with his wife 'to test his eyesight'; surely something we've all done.
Had he not then gone on to treat the rest of the population with such utter contempt and continue lying and insisting he was right, had he held up his hand and said 'yes I was wrong and I'm sorry' this would all probably been forgotten now.
But he doesn't apologise, doesn't explain, because in his head he's above all that. Like some sort of school 'headmaster' he makes the rules, and we all have to adhere to them.
He, however, being in charge, can do what he likes and we apparently have to all accept it.