Definitely go on a northern lights tour and don't book it for your last night there as if you don't see any, most companies offer a free second trip.
Take plenty of warm clothing and layers, eg leggings under winter walking trousers. Thick hat, gloves and scarf/neck warmer. Base layer, jumper, good waterproof winter coat.
Yes it is expensive, but there are ways of making it more reasonable without missing out. We didn't bother with wine or spirits and just had beer when out. Eating out wasn't too terrible if you didn't have red meat, other options less pricey. We had the fish soup and cheesy bread, casserole, which is like white fish in a creamy sauce on Icelandic bread, plus tapas and Nepalese food, plus ate in a couple of nights. We also got slices of takeaway pizza fairly reasonably, £30 would have bought an entire massive pizza to feed at least 2-3. Coffee shop lunches were cheaper and better quality than UK chains.
YY to Christmas lights and just wandering around. The big relief map in the town hall is interesting and there's lots of museums, which would be a good choice in winter. A lot of the public swimming pools have spa areas and you can get a city card that covers the pools and a lot of the museums (not Perlan, which is about £30 each but well worth it).
I'm not sure about the Golden Circle tour because would you see much? They might be able to get to the big waterfall and geysers in daylight, but if it's not light you wouldn't see the big crater and possibly not be able to walk around the parliament crack bit near the end. Plus obviously not see anything out of the window on the way around.
3 days is plenty to see if you like the country and you can't see everything in one trip because there's so much to see and do and some things are better at different times of year so you need to go multiple times to experience short days/midnight sun, northern lights and then get the best of landscape/scenery. Even the waterfalls are different across the year - frozen in winter or much bigger in the summer when the glaciers and winter ice is melting in the mountains.