You shouldn't feel that you have to take him. If he wanted to play football, you might take him, then again you might not. It would depend on the circumstances, how busy you were.
I would suggest you question him about why exactly he has developed the urge to attend church. Which particular branch of religion has he chosen, and why? "Christianity" is not good enough - he should explain whether its Catholicism, C of E, Methodist, whatever, and he should have a reason as to why that branch appeals to him more than others.
If he wants to read a bible or something, consider buying him a very cheap one. But remember that it is full of stuff that is generally considered unsuitable for children of his age - murder, damnation, sodomy and so on.
You need to find out though why he has this feeling now. You shouldn't just give in to his request. When he is older, he will be able to choose for himself - for now, it's entirely your call.