Sea I'm a longtime lurker but am posting as I live on Gower. All those born in the Swansea area will swear by Joes icecream - there is a parlour in Mumbles and another by the Guildhall. It's OK but I much prefer the icecream at Verdis - also on the front at Mumbles but slightly away from the main part - just head towards the pier.
If you're looking for quiet beaches and you don't mind a bit of a walk Fall Bay is lovely.
Otherwise Rhossili/Llangennith is so huge you can always walk along the beach away from the carparks and find some really quiet spots. At the Rhossili end there are a number of steep steps. I would park at Hillend in Llangennith and walk onto the beach though the dunes and once on the beach turn left and head towards Worms Head. Once you've gone away from the caravan site it'll will be very quiet. At lower tides there is the bones of a shipwreck in the sand that my DC used to love