I could have written your post years ago. Dh and I went through all the same thought processes. Shall we/shan't we. It's so hard knowing what to do for the best.
We were yes one day, no the next, for exactly the sane reasons, will we be able to show them all enough attention? Financially as they get older, cars, no spare seats for friends, and how will we ever go swimming?! 
We decided yes, started trying, then decided to stop, having talked and talked even more, we decided to let our practical heads rule our hearts. But it was too late, I was already pregnant. I was quite disappointed when I found out (for about an hour), anyway I miscarried a few weeks later, and I was devastated. Sort of answered it for me.
We now have ds1 (7), ds2 (4), and dd (2). It is more manic, I'm shattered, but she completes us (if you know what I mean). I'm still up half the night, but everything is very manageable. Dd is much harder work than the boys ever were, with tantrums, sleeping, eating everything really. If she'd been my second, I may not have had a third . I've had Dh 'done', so I'm not tempted for another. But i know we did the right thing for us having three.
At the end of the day, it's a very very hard decision, and I know you shouldn't let your heart rule your head. But working out all the practicalities, if they are workable, you'll know deep down what you really want to do. If your anything like me you'll feel guilty for sharing yourself with them all, and hope you're not spreading yourself too thin, but I was in bits about that even having ds2, I felt so guilt to ds1! But now they've all got each other to play with/talk to/annoy/wind up/ occasionally and very worryingly look like they're trying to kill!!
And.... Swimmings actually not to bad arm bands, rubber rings, and all staying within arms reach!