Just looked it up. SpringHeeledjack is right, it was his original first name and he changed it to sound less posh.
'Originally named Gideon Oliver,[2] he changed his name to George when he was 13. In an interview in July 2005, Osborne said: "It was my small act of rebellion. I never liked it. When I finally told my mother she said, 'Nor do I'. So I decided to be George after my grandfather, who was a war hero. Life was easier as a George; it was a straightforward name'