If a parent 'did a runner' to a non Hague convention country, the resident parent is pretty much fucked. The courts demand residency agreements in both countries. Good luck with that if they are with the father... A mere woman has no chance against the patriarchy or bribery the other side will use to keep the do away from their mother.
That said, if for one second you think there is a risk of their abduction, then you can legitimately refuse.
Your dc dad gave you his contact details, an itinerary and updates. At no point were your children at any risk, they were with their father.
Tanger is a day trip, it's a fascinating place, so unlike Europe in so many ways. It is not a typical tourist destination and security is tight. I think the chances of a "Tunisia" happening in Tanger are at best minimal.
I think you don't need to be concerned about the daytrip, really. He was open and reasonable about everything else. Would you have been as worried if he'd popped to Portugal? If there are are other issues, then you may need to revise your agreement. Note though that if you have joint residency I believe that each has the right to take their child away for a period of time without formal permission, but you will be best served by checking this. Bottom line is that only you know if you can trust your ex to have his dc best interests at heart.
As for the snorty comments about permission to take ones children abroad with letters of permission, it's always advisable to do so, especially when the children and their mother have different surnames. It's the law for all parents who have PR to have provided consent for their children to travel with the other parent if alone. I have been asked in Holland, and in the UK on return from Egypt. I've never been asked on leaving our own country, which is precisely where we SHOULD be asked, but hey ho..