Ok, I don't have a huge amount of experience with this, and I don't know your dh's circumstances....
If its been a bad year, with alot of stress in the house, then for heaven's sake, GO!!! You poor son needs to be away for a bit.
My dad was ill alot when we were growing up (some of it due to him not looking after himself, so a bit self inflicted. ) and I really got to resent it. Our whole lives revolved around "what if Dad goes into the hospital again".
It is only four days, you can call him, you can come back if need be (hope not) others can pop in on him.
A holiday is not thumbing your nose at dh's illness, but like you said - to allow your little boy the chance to make some lovely memories.
OK, think of it this way...its only an hour away which in many cases is the commute TO the hospital. If one of your children was ill, it could very well happen that four days go by without you being able to get there.
Just the fact that you are so worried shows that you are genuinely caring.
I think your little boy deserves a break.