Panto is not drag it never was.
Drag had a different cultural history and the dressing was more symbolic of breaking of social boundries around sexual behaviours.
Panto is a surviving element of English cultural, a theater tradition from when it was illegal for women to act on stage. At one point theater going itself was illegal not just because of the "immortality" but becase it allowed large groups (mainly young men) and discuss the political elements.
Males playing female was not about sexuality but about needing somone to be able to play the female roles most men grew a beard etc depending on the fashion.
And yea oldle women having babies was little bit if a career problem if you have to travel. Plus women who acted were treated as a special type of prostitute did not make it an attractive carer for women.
There are a various other reasons too eg. breaking social rules on stage were safer if the rule breaker was male.