It depends what kind of holiday you want to have. I think it would be fine personally and wouldn't have a problem going without children.
There's lots of activities to do during the day (pony trek, wall climbing, archery, golf, paint balling etc)and the spa if you want to chill out there. Plus you can eat later in the evening in the restaurants without worrying about time and babysitters etc.
You don't even have to venture into the pool if you don't want to. But after about 6pm it's pretty quiet in there. Last time we went (which i did have my ds with me) we were the only onesin the main pool at one point at around 8pm as everyone else was on the rapid river ride thing.
Don't knock it till you tried it.
Gilly