I'll confess that I don't like giving "family" names to babies. I always feel like they need to build their own identity, and when you name them after a family member there's always gonna be unspoken expectations that the baby will share x quality/trait with the original bearer of the name.
Beside that, I don't really like the name Chester - though that is purely a personal dislike. It just sounds more like a last name to me.
Rory wouldn't be my first choice because I find the double "R" difficult to pronounce, but I do like it better than Chester. Also I liked Rory in Dr. Who.
That said, this is YOUR baby's name. If you like Chester, that's all that matters. Don't listen to people who say it reminds them of X thing. If you really look, ANY name can have negative connotations. For instance I love the name Nikolas, but I'm sure someone somewhere would tell me that it's boring, it ends with "ass", it was the name of the last Russian tsar who was tragically killed by the communists...
Long story short : if you really like Chester, go for it and don't ask anyone's opinion, especially not well-meaning family and friends. Let alone random strangers on the internet !