At a Carol Service you really can't tell who's a Christian, I went to one today and I'm not, and neither was my Jewish friend.
Maybe it was us giving you dirty looks rather than the Christians?
If you were sat solid in your pew and your 2 yr old was shuffling about, squeaking and being a perfectly normal two year old maybe the dirty looks weren't at your child but you for not moving to the back of the Church where usually a small child can squeak more and be perfectly normal or going for a walk.
Maybe I looked at you with no dirty intent just a glance of 'why the eff aren't you meeting this big vast space half way'? Your DS I assume being small, has a small voice, and without decent sound equipment even the smallest, squeak or continual shuffling can drown out any other sound, usually the small voice at the front. I don't honestly think anyone minds the odd shuffle, squeak, loud voice (my 4yr old son exclaiming "That's My BROTHER!" for example) but if its continual, and irritating, inevitably someone's going to think "FFS do something!"
I spent MY DS's carol service at the back of the church, trying quietly to stop my youngest from rings the damn church bells, doing a jigsaw with him etc., trying to minimise any disturbance he may cause, doing something that involved losing my 'top pew view' of my eldest's performance. I feel your pain, YANBU, for feeling a little judged or 'noted', it happens and its understandable.
YABVU for realising that any looks, dirty or not, meant your child was irritating and disrupting the service. Although your child has done nothing wrong, maybe you should have seen this as simply as a cue for you do try something different to tackle it?