I would say so yes. I went a few years ago with a friend and also spent a few days alone, we felt very safe at all times including walking at night and on public transport, more so than in certain parts of the UK to be honest. On a group tour you should be absolutely fine. Obviously you should take normal precautions such as not walking alone at night, not carrying too much cash or valuables with you, as you would anywhere really.
Just a couple of things to be aware of, the Sri Lankan people are very friendly and respectful, but some might tend to stare at you and sometimes even call out 'hello', 'how are you', 'talk to me' etc., particularly at white women and particularly outside of the normal touristy areas where white people are perhaps less common. I don't think it is meant in any way disrespectfully or threateningly but it can be surprising if you are not used to it.
Also never ever get in a tuk tuk or agree for a local to act as your guide/'show you around' a monument etc. without agreeing a price up front and/or confirming they will use the meter. There is a common 'scam' (not really even a scam, more of a trick really) where people will come up to you and offer to take you in their tuk tuk to an 'elephant festival' or a local temple or similar, they will rush you by saying you need to come now because it's closing, and will then try and charge you way above the odds for the tuk tuk ride and/or admission to the site (or it will turn out not to actually exist and they will just drive around in circles for a while running the meter up!). I'm sure your tour guide will be alert to this but just feel confident to say 'no, not interested' etc if approached.