I would just ask, it won't look weird. If your DD is happy to be left you could make use of the free time - unless you are needed/expected to stay
My DD had a party when she was 4 and I expected all the children to be left (they were all slightly older due to when DDs birthday is) I was a little taken aback that one mum stayed. With hindsight I maybe should've encouraged the other mums to stay if they wanted. DD always preferred to be left and even now (age 9) doesn't like it when it's my turn to help at Brownies etc
at all the 4 year old parties DS went to last year the majority of parents stayed and it was the same 2 Mums who always dropped and left. No body minded that they didn't stay but they really were in the minority. Agree with blue jeans - am sure it won't matter either way, but do mention it if you're leaving.