I don't entirely disagree with you, but you are naive if you think there is no slavery in modern day Britain. It is alive and well.
I do think that there is an intrinsic problem with some of the presentation of history - "the British" didn't do anything to others, per se. Half the time they were as busy fighting and subjugating themselves as they were elsewhere. It was the British ruling class who commanded and who benefitted. Any passing advantage that the vast majority of people got was only because those in powerful positions allowed it, just as they allowed (for example) a form of benefit for the Indian "middle classes" in managing their own country.
But lets be very honest. Flying the Union Jack has nothing to do any of that. Feeling ashamed of the Union Jack is something of a new phenonomen, and it is because the flag has been hijacked by racists. Perhaps we have allowed that to happen, but it is naive to claim anything else when it is worn, waved, tattooed, painted and otherwise prominantly displayed at events and activities that the vast majority of the population are ashamed of. Whatever your views of "immigration" or "asylum", are you proud that it is held aloft or painted on the faces of thugs who are attempting to burn people alive in their place of residence? Because I am not.
When I go past a home in Wales or Scotland, I see their flags displayed to indicate that they are proud of their heritage. Even if that is what people intend - and only that - in England, when I see the Union Flag or the cross of St George, my head goes straight to those images of thugs. People are not embarassed to be British. They aren't even largely embarassed by British history. They are very embarassed to be seen to have anything in common with jack-booted thugs, and for many of them that is what the Union flag has become associated with.
Maybe it is time to take back the flag.