"If you are divorced, like I am, you get £500,000 IHT allowance"
Yes, but the father of your children also gets a £500k allowance as well. I presume that he'll be leaving his children at least something from his estate when he dies?
.
"Am I really in the richest 4% of our population?"
There were something like 670,000 deaths in the UK in 2022.
HMRC report that for the tax year 2021/22 that 222,300 estates were reported to them. You only need to report an estate to HMRC if probate is required, not all deaths require probate.
If your estate is worth around £650k then that would put you at something like the 87% percentile of wealth.
So, you're in the top 13% of those people who needed probate when they died.
Of people who had a net estate worth between £600k and £700k around 76% did pay IHT (so it would suggest that around a quarter had a £1 million allowance). Of those that paid tax, the average amount paid was £75,900
Don't forget that people who do not own a home won't be on that list, they are likely to not be as wealthy as you, so that would push you higher up.
Also, people who own a home jointly will not need probate so won't be included either, but there is really not much difference.