It's not "Instead of". It's "In addition to".
I mean, did you have a scheduled plan in place for your day together? Have you bought tickets for a river cruise, or a theme park, or anything else? Or was the plan to just have a lie in, maybe some sex, go for a walk somewhere, have a nice lunch and so on? Because all of that can still happen, minus the couple of hours it takes for you to get ready, drive there and do the interview.
On the other hand, is it possible for the interview to occur on another day? If a potential employer says "Are you free on Wednesday for an interview?" and the jobseeker replies "Actually I have a thing booked in on Wed, can we do it on Friday instead?" then that isn't seen as a negative consideration. Employers are aware that people have lives.
If it was a choice between EITHER have a day off OR get a new job offer then of course you would go for the new job. But it doesn't appear to be. It appears to be "Do I request to reschedule the interview, or do I miss some of my 1-2-1 time with my Husband?". If it's possible to reschedule, I'd enquire about that first, and if not possible, then go ahead with the interview and you can still have most of the day with your husband.