So often, when someone has no need to apologise, and isn't actually feeling apologetic at all:
"I'm sorry but I'm not in a position to lend you £100".
"I'm sorry but it isn't convenient to have you to stay next weekend".
If you aren't sorry - don't say you are.