In a legal sense your age matters; how you feel about your age and choose to express it should be irrelevant as far as the needs of organising society goes.
The age analogy works for me, as a good example of how society should treat gender identity, which I think is what the OP means? Not how it does!
I'd say you have the right to express yourself as you like, as long as you don't restrict anyone else's rights -- and safety is a more fundamental human right than self-expression.
Other people have the right to express themselves as they like, including the right to disagree with you.
The law should be based on the rock-solid foundation of reality, not swayed by fashions or fantasy.
If you're middle-aged but feel 18 you have every right to dress like a teenager. Other people would have an equal right to laugh at you. That would be rude but not illegal -- democracy is in trouble when the state starts legislating against hurting people's feelings.
If you're adult but feel like a toddler, you have the right to wear adult-size baby clothes and pretend you can't talk. You don't have a right to join in playgroups because children need safety more than you need to indulge your fantasy. You can't change the birthdate on your birth certificate, because that would be falsifying a legal document.
If you're young but feel old, you can wear cardigans and play lawn bowls. But you can't claim an old age pension.
I think most of us would agree that's all fine. We don't give a damn what other people wear or what name they use -- which seem to be the only two criteria required for proof that someone is "living as the opposite gender", when they apply for a Gender Recognition Certificate.
I object to the many ways in which the law treats feeling you're a different gender differently from feeling you're a different age. It matters.