I'm a little confused about all their rules that you have said you follow. Is the child allergic to something other than dogs? If no, then all the rules you follow must somehow be related to preventing the child being exposed to your dog. You describe it as a lot of rules, but it all boils down to one idea which is ensuring the child has no exposure to a dog.
This is your property so you have a right to let your dog out if you like. However, from what you describe, this child is living with an extreme condition that severely limits his/her life. Their child rarely goes out except to a school that is willing to enact the extreme restrictions this child needs to stay alive. The child's family have gone to extreme lengths to create an environment where the child can be kept alive, including buying up as many properties on the cul-de-sac as possible, and only allowing the child to socialize with his/her family who live within the cul-de-sac. You say it is frustrating that the child is in their garden all the time, but from what you describe, that is because the neighbor's garden is apparently the only place this child can go other than his/her house and school. So this must be a very severe and life-threatening allergy.
So essentially this child's family went to extremely burdensome lengths to create a living situation where this medically fragile child can stay alive, and you brought into this environment the one thing that, with a tiny amount of exposure, could kill them.
I see both sides here. It's your property, and you can do as you wish. But you knew in advance that they went to extreme lengths to create an environment to prevent their child being exposed to dogs, you knew that contact with dogs is so dangerous to their child that he/she has been hospitalized after those exposures, you knew this sensitivity is so extreme that the child cannot go anywhere other than her own backyard and her school, and then you bought a dog.
I suppose what seems like the tie-breaker to me, here is, that your dog can safely go for walks elsewhere: to the local park, around other parts of the neighborhood, etc. Whereas this child is only safe in this small space. So to bring your dog to that small space when your dog has the rest of the world to go investigate on walks, and smell things, and meet other dogs, even though it's within your rights, it seems a bit cruel.