one DC is a bit of extra work with a broken leg
Having lived through a young DC with a broken leg, it wasn't a bit of extra work - all our lives basically came to a halt to support. Toileting, sleeping, pain management, itching, bathing, everything was just a complete nightmare and I will never forget hearing DS1's screaming before he fainted as they transferred him to the x-ray machine, struggling against the general anasthetic, and crying himself to sleep as it hurt and he couldn't process what was going on and was scared and in pain. Fracture clinic, x-rays, waiting around for hours for the consultant, getting the wheelchair in and out of the car, lifting him into the wheelchair. Just parking the car at the bloody hospital was a mission. It's not like having a broken wrist in a neat sling where all your friends write their name on the cast and life goes on as usual.
Then there will be cast removal, physiotherapy at inconvenient times in the middle of the working day, convalescing, getting strength back. They don't just leap off the table when the cast is off able to run around like they did before.
I bet you anything the DH isn't suggesting that he comes back from his jolly and the OP goes off to a spa hotel for a week while he juggles work and 2 DCs, one in a cast. He's a selfish prick to even consider going off enjoying himself while the OP copes with all this alone. I can't imagine his mates being very impressed with his fatherly dedication to his kids either (unless they're selfish pricks themselves).
And for people who say they'd take the trip in these circumstances, I just don't believe you. When your 5 year old is injured to the extent they need to be in a wheelchair you are there with them 24/7. Neither DH nor I could leave DS1's side, except to take care of DS2 or to sleep, though we both knew we had to and forced each other to take care of themselves.