OP, ultimately, only you and DH can make this decision. Of course, 5 children is a lot these days. I have 3, single parent since they were little, working full-time, professional role. Found the younger years fine, even as a single parent. All are teens now, and oh.my.god. I can't believe how hard I find it. The mental stress, the challenges of dealing with their problems, behaviour, exams, future decisions and so on. I love them dearly but it is so hard, and so expensive. I really wasn't ready for this.
I always wanted 4, and despite the car crash that was my marriage, am still sad i didn't get to have a 4th (marriage ended when 4th was a baby). If that hadn't happened, I'm sure I would have had a 4th, and would have been about your age having them.
My own feeling is you just have a gut instinct about when you are 'done' or not. There isn't a right or wrong. PP's advice about thinking about when they are older, expenses etc is all good but I think usually, it's something more instinctive than that that makes us decide. Good luck with your decision.