I'm astounded about the strength of feeling from some quarters here towards parents wanting a secular education for their children.
Our experience was that we very definitely wanted a secular school for our little boy, particularly at infant level, but were in the position described by several posters here in that we live in a fairly remote rural village. Our nearest school is about 6 miles away, and is very very CofE. An example of this is that the school rules are described as "God's Rules" (in a huge display inside the school hall), which to my mind is quite a disturbing piece of propaganda. That's just one example - I could go on ad infinitum.
Anyway, the two next nearest schools (8 miles aprox) are also CofE - there are in fact NO non CofE schools within a huge radius of our village. We visited these two, and whilst still CofE we felt that the emphasis was very different, and that whilst not ideal for our outlook on life, it was livable with, and we really had very little other option. Like the OP we live in a lovely village - are very much rooted in our community, and would never consider moving.
Anyway, long story short we expressed our "preference" for the two alternative schools, which both turned out to be oversubscribed, and our little boy was offered a place at the catchment, deeply religious school. We were very uncomfortable with this - went to appeal, on religious ground amongst other things, but this was obviously considered inconsequential, and we lost.
Our little boy used to regularly come home having absorbed many ideas about God etc, interpreted very literally. This left us in the difficult position of either discussing these ideas with him (sensitively) in relation to our own atheist beliefs - which generally led to huge confusion and upset on his part along the lines of... "Well my teacher told me God is real, and teachers don't lie about thing do they?", or making what amounted to "no comment" type responses, which sat very uncomfortably with us as parents. Now I would very happily have had these kinds of debates with my son at a slightly older age, but I felt that at the age of 4, it was literally impossible to handle the issue both sensitively and truthfully (from my point of view).
Interestingly, he is also now at an independent school - also run along religious lines. That was our choice however, and we are very comfortable with the presentation of religious ideas here. Also, he is older, and more able to process the fact that different adults in his life hold different viewpoints about some things.
OP - I had all those same squirmy feelings when we were back there in your position, and I would have been tempted to say something to the teacher I think. It will probably raise eyebrows though, so be prepared to be a bit assertive.