We take my partners son back to school on Monday mornings, this does mean that we have to make up time at work as we are late in but feel it's worth it. He comes from school friday evening in school uniform which is washed and he goes back in on Monday morning. He goes to childminder in mornings before school normally so doesn't have to get up any earlier and doesn't ever complain of missing his mum.
I agree with Aloha about time with kids just being there not necessarily doing something, it means we can get ready together in the morning have breakfast together and aren't rushing to get him back sunday evening so we can do what we want sunday (he lives 40 miles away).
It really depends on how much time your partner would have to spend making up his working hours (and therefore presumably not being with you?) and how the kids feel about it. I personally don't see a problem with having a relaxing evening and preparing for all 4 kids to go to school the next day (but then I don't have 4 kids!).
If the children get used to this every time they come to you it should be as routine as going to school with their mum. As far as taking things home we unfortunately have a strict rule that what is at our house stays there (as we would never see it again, as has happened) and there are no problems with that.
But every family is different, I have to say that in many situations if you tell yourself enough that it'll go wrong it will, may not have been the case here, is there no chance of having a trial period to see how you all get on?