24 wks is too early. I think you're ok to start from 34/35 wks. As for does it shrink back(!!) it doesn't actually stretch visibly or in any way that you are aware of it just makes it all a bit more flexible I suppose. Prepares it for being stretched more easily in childbirth.
I have cut and pasted the following from the Babyworld website...
What are the benefits?
Perineal massage has many benefits which all help to reduce trauma during birth.
It stimulates the blood supply to the perineum and helps speed the healing process after the birth.
It helps prepare you for the feeling of pressure and stretching that comes as your baby's head is born.
It helps to familiarise you with some of the sensations such as tingling or burning so you are less likely to tense up.
It can help you to relax when you have a vaginal examination.
Midwife Tania Pearce to provide expectant parents with a more personal approach to antenatal and post natal education, including the benefits of perineal massage. She says, "From my experiences as a midwife I have noticed that women who have done perineal massage are more likely to have perineums that stretch well around the baby's head.
"These women also have more control over the expulsion of the baby's head, and allow it to be born slowly. "Because they are use to the sensations they are experiencing they are not frightened of them and don't rush the birth.
Worth a shot!