A laypersons view:
The legal and moral positions are always different and if you put in the overiding consideration in all Children's Act cases that the `welfare of the child is paramount' it makes it all the more complicated and confusing as there is NO single correct answer and there can't be one really.
Legally you have been court ordered to make your DC available for contact with the NRP as per the order. Flexibility is IMO always a good thing, especially if it benefits DS. Your ex would struggle to get a case taken back to court for `enforcement' if the order had only not been complied with once but if it ment he was not seeing the DC very often on his court ordered dates he could take it back and there would be penalties against you.
Could your ex not take your DS to something he wants to go to? Morally IMO only, if your DS is missing lots of things and starts to feel resentment for his dad this really isn't helpful for anyone but your DS will soon enough be able to deicde for himself, 12 he will have a good say and 16 total say (in almost all cases). He will probably live for a long, long time after that, so really if your ex wants to ruin a long lasting relationship for the sake of only meeting in own needs in terms of contact now, then I think the relationship will suffer which is very sad.
How do you communicate with your ex? as your DS gets older he will want to spend time away from you both more and why should he not. Children are not the property of either parent - that said courts seem to have the view that time with parents is more beneficial than time at activities but I always say, it depends what the parent and child are doing with that time.
Balancing the legal written rules and orders against the `best interests of the child' is so difficult. Contact arangements are naturally going to change over time as children's need change. If you can pen something to your ex to explain the situation, swap a couple of dates in advance now so your ex still has contact time and then if he doesn't agree you could take it back to court (self rep?) and seek advice from CAFCASS - legal aid is gone for family cases now anyway but if your ex takes you back for enforcement his old cert. would probably still be valid.
Your DS can also raise things with his dad, he should feel able to do so and that would be how thier relationship could become stronger and better IMO.