Yes I have been with a 5 year old.
I wouldn't go to Borneo just to go on a beach holiday. The beaches there are ok but they are quite dirty and there are definitely better places to go for nice beaches but a few days added on at the beach before or after a jungle trip is nice as there are nice hotels.
The main reason to go to Borneo is definitely for the wildlife. Have a look at the website for Borneo Eco Tours as someone mentioned earlier. You can choose a tour for you to experience the rainforest. You can't really do this on your own so it is worth booking a package.
We saw the "Borneo Big 5" which was amazing. Loved the Sukau Rainforest Lodge. The basic rooms are small so personally I would upgrade to the villas. Early starts each morning (5:30am) to go out on boats to spot animals. Then back for breakfast at 7. Usually something else on in the afternoon and out on the boat again in the evening. There is also a small pool that kids like. There are very few kids there. You are grouped up with other people (maybe one couple) for your guided parts and that could be make or break for you because you might get stuck with someone who doesn't like children and that can make things uncomfortable. You could pay extra for just your family for a guide I am sure.
There is also the Borneo Rainforest Lodge in the Danum Valley. This is really nice and more luxurious but I actually preferred the river lodge. Again I would upgrade the room if travelling with children.
Our 5 night/6 day trip cost over £6,000. So this included the food and transfers to and from the airport and between hotels and the tours to see the animals. You would also need to tip your guide. Obviously you would need to add up the cost of flights to Borneo, internal flights in Borneo, what other hotels you stay at etc on top of this.
Kota Kinabalu itself is not great. There are cheap hotels there but they are awful. Nice hotels in Borneo are not cheap.
Overall it is a an amazing holiday. Whether your kids would enjoy it I would say mainly depends on the type of kids. If your kids like playing in the pool all day and are not good at getting up very early and sticking to at timetable, eating different types of food etc then I don't think they would enjoy the rainforest experience. If they are whining you won't enjoy it either and will be stressful for you.