First of all, and I think you are doing this, but keep it up in your analysis . . . factor "everything in," including the pros/cons of being near family and friends, as well as being in your current location. You've mostly touched on the "people" part, but do expand that thinking into everything that is a pro/con of living in a small town versus a bigger, uni town.
You do not want to trade for the conveniences of having family nearby to boredom for lack of things to do. What type of activities and hobbies do you, your partner and your children enjoy? Will you still be able to do those things where your parents live, or will you have more opportunities?
While friendships are important to maintain, there is a reason for the saying "friends for a reason, friends for a season, and friends for life." Some of the friends you have now that you do not see often because you are not sharing similar paths in life simultaneously, such as parenting, may still be those friends for life that when you do see them, no time is lost, but they also could end up being those "friends for a season" when you were young or going to uni together, and just no longer have the same "reason" to be connected.
You will meet new friends, and especially when your children go to school and they develop friendships. You will find that you develop friendships with the parents who have children of the same age, at the same school, or neighbors, so don't stay in your university town merely for those friends you do not see often. But, if the town offers culture, activities and things to do that enrich your lives and are part of your routine, and these would be lost in a move to a smaller town, or not replaced with something equally inspiring, engaging, or enriching, then do consider that very big con.
Further, will the schools be comparable? Will you have the medical services you may need? Will you trade for lower or higher prices when it comes to services and goods?
But maybe you would get something in addition to the family pros, such as a more relaxed lifestyle with less work pressure, more chances to be outside in nature, or other things that you don't get in a larger city.
Think through all facets of pros and cons, regarding the people, the place and the changes that come with a new location and new job.