Same here. We weren't "that bothered" about being parents and certainly aren't bothered about being grandparents. There's far more to life than children and grandchildren.
We ended up having our son late in life when I was 37. That was after establishing our careers, marriage, owning our own home, starting our own business, and travelling to dozens of countries all over the World for the sights and experiences, alongside various hobbies, voluntary work, etc.
I can honestly say, our life would have been just as complete if we hadn't had our son. That's not to say we didn't put our all into his upbringing etc - we always give 100% to whatever we do, no half measures. Obviously we're pleased we did, we had many years of different experiences, and he's brought us lots of joy over the years, but our life wasn't and isn't defined by our son. We certainly wouldn't have gone down IVF or adoption if we hadn't conceived, and "let nature take it's course" too with whether we had another (we didn't) - we did nothing to stop it happening, but did nothing to "make" it happen either.
Same with grandchildren, if we have one or more, then fair enough, we'll put our all into them, and fully support our son, and do whatever we can to help as we know we'd benefit ourselves too in the same way we did with our son. But we're really not bothered. If we don't have grandchild/ren then we won't be loosing sleep and won't be crying into our corn flakes every morning.