Yes it is hard Hels
It was hard at the time, because I had a new 10 year old pop up, and sometimes it was hard when situations just like you describe happen - about clubs, things to do, about birth stories. Sometimes I felt a compulsion to explain myself!
There are times with a much older child that 'telling' can't be avoided, but I did tell too many people, people who know us but who didn't need to know - for instance, people who became neighbours after DD came home, or some friends parents
And I can't take it back now. DD1's a private person. She doesn't want to be judged by people because of her life story, she doesn't want people she doesn't know well to know about her adoption. And that's not a bad thing - privacy is not shame, and that's an easy trap to fall into ("oh, we're not ashamed of adoption in this family, we don't want to give our children the message that adoption is a secret not to be talked about outside the family"). Well, my family aren't ashamed of adoption either - DD1 feels very positive about her (adoptive) family and positive about the difference adoption can make to a childs life. BUT, the fact that she is adopted has several other facts attached to it, of an intensely personal nature. The fact that she has a birth family, for one. The fact that she was in care, for another. The fact that there were reasons why she has been adopted. You can't say 'my child is adopted' and have that be the only fact you are telling someone, because legal adoption is an 'end point' to a very private and personal story, not the start. My DD1 does not want random people to know these things. She feels that if there's a need to tell, then there's a need. If not, there's not. It invites questionning, speculation etc.
I'm not trying to make you feel bad Hels! Really, the fact that I made decisions that were in retrospect not right for my own DD, I made them because I felt exactly the same way as you and so many other adoptive parents, felt the hardness of not saying anything in certain situations! This I understand. But I think it's worth talking about
DD has done some volunteering in years past, and she needed a CRB check for what she was doing. She deliberately choose documents which didn't include her adoption certfiicate, and she was adamantly opposed to letting her employer know her birth name (because she was over 10 when the adoption was finalised, CRB/DBS need to know birth name and normally you put previous names on the form which you hand back to your employer/organisation). So she went through the special process the CRB/DBS check people have for "post-10" adoptees, which is a way for an adoptee to bypass their employer and send their birth name and adoption certificate/original birth certificate straight to the CRB people to avoid the employer finding out this sensitive information.
It involved extra form filling but it was non-negotiable for her
Anyway, I'm getting on tangents now. But basically, we don't know how our kids are going to feel when they grow up. So I really think caution is the wiser course of action. There is no obligation on you to tell, and keep in mind that when you say 'my child is adopted' you are saying more than what you said!