I have a lot of sympathy with the OP, although it's clear from the OP that they like things "short and to the point". 
In my younger days, I did struggle to say no to unwanted phone calls, especially as one such caller used to be my grandmother who must not be offended for any reason on earth.
When I call somebody, I often ask "is this a good time?", and sometimes I send a text in advance telling them when I could call them, especially if I'd be too busy to answer them calling me. In a way, a phone call is an interruption in our increasingly busy lives, especially if time is limited. Sometimes really important things are only done by phone (calls from the NHS perhaps, to arrange that vital operation date, which are often "ID withheld"), and if anyone else calls, your really important caller can't get through. I sometimes find texts preferable to calls when making arrangements: for a phone call, you need pen and paper; with text, it's in black and white already.
Yes, you can ignore the call and deal with it later, but that takes time, especially if they've left a waffly voice mail, and can lead to a sense of "what do I have to deal with now"? (The dread when the voicemail alert comes long after the missed call!)
Being so contactable is a very mixed blessing of modern times. I remember my dad muttering about faxes: in the old days, you could send a letter, and forget it for a day or two. But if you send a fax, a few minutes later you might get a phone call "I've just got your fax..." I also remember reading an article in New Scientist, which had a cartoon of a smiling multi-limbed being holding out several telephones, computers, emails, and a tiny person running away from this abomination.
This article gave tips for reducing your contactability: being strict about when you will answer the phone, having your voicemail greeting state the times you will answer, only checking email at certain times (not all the time), and one novel suggestion, worthy of Roald Dahl: if colleagues turning up in your office to talk to you is a problem, stick some bits of wood under the back legs of the chair opposite your desk. This makes the chair slightly uncomfortable, being tipped slightly forwards, and people will spend less time sitting in it to waste your time talk to you.