been three times, last time this feb. my favourite country on earth.
Colombo, v dull capital city., don t bother staying more than night or so. You must stay in old fading grandeur of Galle Face Hotel. Give yourself at least three hours in Barefoot, it is fabulous shop for souvenirs - always take extra suitcase for purchases here.
Kandy is lovely but don t stay in town, stay in hills above, loads of B&Bs, get lonely planet guide. elephant orphanage is not just tourist thing, it s v special. yr kids will love it.
Travel either by trains (v cheap and way of meeting folk ) or you can get a private driver and mini bus for $30 for three hour journey.
Bentota: for luxury and glamour stay in Club Villa hotel. It is sensational, by sea, gorgeous decor. Glam boutique hotel but not ridiculously expensvie and we took boys of 8 & 6.
Yala: try safari here. you won t see as much game as africa but elephants for sure.
In three weeks you could also go to the temple towns in north - Sigaria is best. You could rent driver from kandy and go between them wouldn t be too expensive.
My tip: book flights (Sri Lankan Air is much improved and lovely service) then ring to book accommodation separately, no need for package. Try deckchair.com for flights. Do book hotels ahead tho as best things get taken and it is quite a small country.
Don t go in summer too hot and also monsoon. best time between dec and april.
With children you can t travel too much as it will be draining cos of heat and humidity. Although we did five places in fortnight, kids luckily turned out to be v resilient travellers.