Agree with PPs that one of the deepest joys of Edinburgh is to wander, especially along the High Street, some closes/wynds, Grassmarket, Victoria Street, Candlemaker Row.
Agree also that St Andrews is a fair trek. Unless there is some specific reason to go there, my advice if you want to get out the city for a day is to swap out Edinburgh Castle for Stirling Castle. Can be done easily by train in a day. It's more of a proper medieval castle, more interesting for children, a longer visit and slightly cheaper to get in.
You won't need a car much, if at all, for most of the suggestions here.
If you want more good old fashioned castles try Linlithgow Palace (also train-able), Craigmillar Castle (still Edinburgh but further out of the centre, in a very un-posh area but you might get the castle to yourself if you're lucky), bit of a drive but not as far as St Andrews would take you to Tantallon Castle, on the coast. Dramatic setting. Honestly, if you have your heart set in Edinburgh Castle, don't let us put you off but it's not the best fun, and it is pricey.