As dangerous as it would be to talk for all men, I can only explain why I do, and no, it's not to avoid other stuff, as I invariable will choose to go when I know I have time as I hate rushing.
So, TMI will follow below...
Basically, I know that I'll always poo some more after sitting there a while, I don't just empty my bowel in the initial poo when I sit down, and I don't feel 'right' until I'm sure I've emptied it all. That simple, and I have no idea if that's the same for other men, as I've never asked any.
Yes, there's an aspect of peace and quiet, but tbh I can find better places for that. I'm sure some men, and women, but there's no empirical study that I know of, use going to the loo, or having a long bath, or shower, or nap etc., to get away from life for a bit, and as long as you're not leaving your partner in the lurch, or choosing a really incovinient time, I'm not sure that's a problem. We all need a break.
So, TL:DR - I take a long time to make sure I get rid of it all as I tend to go in two 'lots'