As for your specific questions:
- are the aurora tours worth it? I know you can’t be guaranteed to see the northern lights but if they are visible, is it possible to see them from the city?
Yes, you can see them from the city if they are strong and it's a clear night, I see them outside my house often. But not as well as you will see them from the countryside of course. I've never been on an aurora tour but I imagine some operators put in more effort than others. If it's a likely night you could try going somewhere like Öskjuhlíð which is close to downtown but less light pollution.
- is it worth booking the museums in advance or can you do it on the day? One of my DC has social anxiety and not sure if they will come to everything so I’m trying to keep things flexible if possible.
No, you don't need to book museums. But check which days they are open because some are closed on pretty random days.
- how cold is it? Do we need salopettes or will jeans/joggers with thermal tights ok?
Ski wear would be overkill imo. It's been about 2-12° recently. Thermals are a fine idea to bring but you probably won't even need these except for northern lights watching, i.e. staying fairly still outside in the middle of the night.
- Any restaurant recommendations please? I was told an average restaurant costs £60 for a main and drink per person - is it really that bad?
Haha no. That's like 10,500 ISK, you don't need to pay that much. A main course will be around 3 to 4000 in most places. A beer like 1500 - 2000 or something like that. If I go to the pizza restaurant near us that we love (Flatey, recommend!) I can get my favourite pizza for 2990 and a large peroni for 1790. You can spend 10,500 per person if you want of course, but you certainly don't need to. Other reasonably priced fast food kind of places include Lebowski and Just Wingin' It. I like Tres Locos and Sushi Social for a celebration meal that is more 'fancy' but still not insanely priced, or Fish/Grill Market. You have to book at those.
- anything that is a ‘must’ to see or do please?
You'll already know all the things to do in Rvk. You know, go up Hallgrímskirkja, go inside Harpa, see Sólfari, the Tjörn. 101 is very small and that's where most of the tourist stuff is. You can see it in less than a day. The National Museum is good. Árbær open air museum is fun if you like that kind of thing. There are loads of good pools. Most central with good slides is Laugardalslaug in 104 but there are better ones if you're willing to go further. You can also scoot along the coastal path, see Ægissíða, Nauthólsvík, maybe go around Seltjarnarnes.
The supermarkets you want are called Krónan (yellow coin logo) and Bónus (pink pig). The others are more expensive, some dramatically so - don't buy anything from a 10/11 for example. You can't buy alcohol in the supermarkets, just the special off-license, and if you want to have something to drink in your accommodation it is by far the best option to buy it at the airport duty free on your way in.