Yay! Another MP fan. It's my favourite, too. (I would be very happy if someone made a good multi-part adaptation, though I also see all the reasons why there isn't a run on adapting it compared to P and P, Emma, S and S and Persuasion.)
I agree completely that this is also a plot weakness in MP. Mary Crawford exclaims that it was a silly, unnecessary thing for Henry to do (implicitly saying they could have had a discreet affair while she was staying with her Twickenham friends while Mr Rushworth was away without anyone being any the wiser), and she's not wrong.
It's one of the reasons I like Joan Aiken's Mansfield Revisited, because she actually explains it (and actually exonerates Henry, who, in her sequel, never marries and had genuinely been in love with Fanny) as the result of a smear campaign by a spurned, angry Maria.
Henry turns her away from his door when she shows up after she leaves Rushworth and tells her he's in love with Fanny and isn't interested, and Maria determines that as her reputation is now totally ruined, she''ll blacken his too and ensure he can't marry the woman he loves. So she writes to Sir Thomas and says she's under Henry Crawford's 'protection', everyone believes that adultery with him was the reason for the end of her marriage, and, having lost Fanny, he doesn't bother contradicting them, and retreats to his estate and lives quietly.