I wouldn't try to mix - end result would be two ruined batches. No idea if you can rescue the first batch - you defnitely don't want to boil it up any more. Did you use a thermometer or cold saucers to test the set?
5 hours boiling???? It should be around 20-25 minutes if you are following a traditional recipe. You have to boil the mixture at the maximum temperature you can get on your hob, though, a real "rolling boil" and using a jam pan helps. I've never overcooked marmalade (not significantly anyway) but Loony's tip about thinning down could have come in handy a few times when jam making.