I think you should be allowed to have hymns, too.
Many people have some belief in God, but perhaps aren't entirely clear about how and what they believe, haven't found a denomination that suits them, etc, but might still like the idea of making vows before God or being thankful/praising Him as part of a wedding. A lot of churchgoers are pretty sniffy about people having a church wedding if they aren't regular church-goers, but it seems unfair and very black and white to tell someone that if they aren't ready or able to commit totally to the church then they're not very welcome at a church wedding, but mustn't include any religion in a civil one.
I have a very religious friend who complains about all the couples who marry in her church when they aren't regular members, and she thinks they shouldn't be allowed. Whilst some of them might be doing it for the pretty pictures, others might be doing it because vows before God are still important to them even if that is as far as their faith extends at the moment, and I think that is something worth celebrating and encouraging, rather than dismissing.
And the same would go for a civil wedding. There might well be reasons why they would have liked elements of a church wedding but were unable to or felt uncomfortable with other aspects. Why ban any expressions of faith at all? Who is it possibly hurting to have hymns that are important to the couple there?
Tradition might also have meaning to them, whether they believe in the words or not; a lot of other aspects of weddings and other ceremonies and rites in society are based on tradition and offer comfort or hope as a result.