You're definitely not being unreasonable to feel the way you do. While contact with her mum is important, a reasonable person would appreciate that you can't just rock up whenever you feel like it and expect to have your own way regardless of other plans and considerations. A reasonable person would either largely stick to regular contact times - and be prepared to discuss/swap/compromise as appropriate when clashes occur - or, if they couldn't (wouldn't) commit to that then they at least would show some consideration and understand that if you demand contact at very short notice then chances are that you are going to have to alter your expectations as it's highly likely other plans would have been made - like this trip to the grandparents.
And of course, if she was reasonable she'd also be paying child maintenance. It's so common to hear of feckless absent dads, well, what about feckless absent mums like her ?
Fairy, your DH is being completely unreasonable - not you. How he can expect you to feel happy about this situation where you're one of the main carers for this little girl yet have no say in what happens when her mum starts making her demands ? It must impact upon your whole household - emotionally and financially (if for example paid for stuff is cancelled because mum says so) and you must find it impossible to relax never knowing for sure what's going to happen. It's bloody ridiculous and Purpleroses is right - it seems your DH is paralysed with fear of losing custody and therefore rolls over regardless of the impact upon the rest of the family.
What should happen of course, is that all such requests from mum are discussed with you, and anyone else affected, like the grandparents this week and he then goes back to her to say either her proposal is fine (if it genuinely is), or, that it's not - and that she can only have 2 days this time for x reason, but how about she "books" 2 more mutually agreeable days and so on. That's how reasonable people behave. They negotiate, they give and take.
I have to say that I don't think taking her to court would solve anything. I'm pretty sure an absent parent can't be ordered to have contact - court orders are more for absent parents who've been denied contact by the parent with care. So the only way a custody issue would come about would be if she started a case - which sounds very unlikely as it sounds as if she treats her daughter like a dolly to play with when she feels like it - and not as a child for whom she feels a committed responsibility. I can't see, that despite her threats, she'd actually want a child full time - and more to the point, given the history and her pattern of very erratic and spread out contact it's very unlikely a judge would think SD would be better off with her mum. Why would a judge change the status quo after all ? ..... only if they felt SD would be better off elsewhere, and there's no reason for them to think that.
Don't know if you do already but I'd keep a contact diary and list all the times contact has been offered by you - and refused by the mum - and all the times she's demanded contact (note the notice given) and how she's refused to compromise or negotiate one little bit. Note how long it's been between contact requests - would that pattern demonstrate a parent desperate to see their child - no I didn't think so. Note how long passes between the mum calling, texting, or writing to her daughter - again, this should demonstrate just how little she's really bothered. All of this would be additional evidence in support of retaining the current custody arrangement should the mum ever go through with her threats to "take her back".
She sounds an appalling creature but your DH is enabling her behaviour as I'm sure you know. So long as he has reasonable grounds for refusing a last minute contact demand - like the GP trip - and so long as he always shows willing to offer alternative contact then him not always automatically agreeing to her demands would not be held against him. Any sensible person would see exactly who was reasonable and who was not. He wouldn't be stopping his daughter seeing her mum - he'd just be stopping the mum taking the piss and causing chaos.
And why the hell hasn't he put in a CSA claim yet ? ......
.... presumably, he's "scared" of "upsetting" the mum in case she follows through on her threat again. This woman must think he's an absolute doormat and she's well and truly walking all over him. Your DH is actually letting down his daughter by NOT persuing a CSA claim as she's missing the money she's morally and legally entitled to. Money which even if you've managed without so far could either be used to improve SD's standard or living, or, saved for uni etc. His pathetic roll over and die approach to this is making me really angry on your behaldf OP - god knows how you must feel having to live with this.
Purpleroses makes a very good suggestion about maybe speaking to Gingerbread or similar. There are obviously other dads who are the resident parent and some of them must have encountered similar problems and threats from a crap mother. Another man who's experienced something similar might be able to persuade him that he should have far far more confidence in his role as parent with care, and that for the sake of his daughter, let alone his marriage, he needs to do the right thing and stand up to the ex - which means not letting her take advantage of his whole family, AND, making the bitch (I'm sorry but she is if she's never paid a penny for her child) start supporting her daughter properly.
I actually think he owes you a huge apology. Whether you'll get it or not I don't know but there's no justification for what he said at all. Maybe if MIL was truthful with him and admitted stuff like this really upsets her it might also hit a nerve. Perhaps couples counselling might also be something to consider - issues like this aren't black and white and they can severely affect your relationship if you feel your opinions don't matter, if you feel your DH favours his ex over you, if you feel your DH is being selfish by protecting his own interests (for want of a better phrase) regardless of the impact upon the whole household and even though in reality his fears are pretty groundless, if you feel you have no control over your own home/family and so on. Stuff like that can cause huge resentment because it makes you feel bottom of the pecking order and it's so terribly frustrating to see your partner condoning something which is so obviously unjust again and again and again.
So sorry he's being so blinkered and unfair OP. He needs a huge kick up the backside IMO, and he's not doing his daughter any favours at all by allowing the ex to be so flaky. If he started to call her on her arrogance and selfishness, started to show some assertiveness, then maybe the ex might click she's got to get her act together. SD never knowing when she's going to see her mum must cause a huge amount of anxiety and while, as I said before, you can't force her to maintain a regular contact pattern, DH should at least be trying to improve the current situation - by appealing to her better nature however futile that might seem, or at least by not allowing her to waft in and out as she pleases. If though, this woman is never going to change - which is tragic for SD and sooner or later she'll start to realise the real measure of her mum - then he must stop agreeing to stuff if and when it has negative knock on effects.
I don't know how you've stood this all these years TBH. Being "grown up" does NOT mean you have to put up and shut up when things are blatantly unfair. I'd insist he must attend some sort of counselling with you if he values your marriage and I'd also be insisting that he starts properly looking into custody issues etc before using that as an excuse to let ex do whatever she wants every single time. It's not fair to keep throwing that one out at you when you object to him rolling over - it's effectively emotionally blackmailing you ..... as in, if you push him, he'll lose his daughter and that'll be your fault. I really do think his fears are ungrounded.