Downsides off the top of my head compared with a hotel:
You have to do all the cleaning and tidying. You want the floor swept of sand, or more loo roll, or beds changed, clean towels? you do it.
You have to do the food shopping, cooking, washing up. Personally I don’t mind this at all - we always get somewhere with a dishwasher, and eat out 1 meal a day. Doing 1 meal out, and breakfast and a simple lunch or dinner at “home” is actually easier and more flexible with kids IME.
Harder for kids to meet others. Don’t underestimate this factor if it’s important to you. We’ve often hoped our kids will make friends at the beach but they haven’t. A shared pool would help, but only if the other villas have kids the right age/language… I am still searching for the holy grail of a self catering villa holiday where kids will still meet others and make friends easily.
No kids club/kids activities/play areas. You’ll need to bring more toys books games etc. We usually hire a car so we can go on outings.
Think about check in and check out day - unlike a hotel there is usually nowhere to leave your bags or change/shower etc if you have an early arrival or late departure flight. So think about this when booking your flights.
Nobody to help in an emergency. If someone gets ill or has an accident you can’t just ask the hotel desk, it’s on you to find a doctor or private clinic etc.
Same for information about anything - local trips, restaurants, you’ll need to do your own planning and booking. Bear in mind restaurants get booked up in advance in some holiday locations. So more planning required.
Of course there are plenty of plus sides to offset these negatives! You just asked for the downsides…