DeliverDaniel, did you conceive your last two children naturally or did you have fertility treatment?
I think a lot of people on here are deliberately ignoring your age and the fact you had fertility issues because they can't resist the opportunity of piling in to give him a good kicking.
As someone who's suffered fertility problems myself I take a more circumspect view. I have never, ever used contraception and in 20 of being very sexually active have never got pregnant naturally once. I don't use contraception, I would like another baby but even if I didn't I don't think I'd bother using contraception. That ship has sailed for me and as contraception is unreliable I don't think it would provide much more protection than the barriers that are already in place in the way my own body works (or doesn't work).
Most posters on this thread are making the assumption that he is being completely lacksadasical because doesn't care if you get pregnant and that you are hoping you will get pregnant accidentally. Would I be right in saying that it's more a case that you are hoping that this might be a sign that you can talk him into some sort of fertilty treatment rather than a chance of getting pregnant naturally?
If your fertility issues have been serious enough plus you are now older I don't think it really means the man is a bastard, just that he's clued up enough and knows enough about your own situation that it's not that likely to happen. Of course you will still have some people frothing at the mouth insisting he should be wearing a condom. But even condoms and the pill don't provide 100% and if he's aware that it's as unlikely for to get pregnant as it is for a couple in their 20s using the pill or condoms I don't really see how it's that much different from that. Everybody takes a risk of getting pregnant when they have sex, we can't remove the risk, only minimize it. And if in your case the risk is already completely minimized he may not see the point of adding an extra layer to that.
Perhaps it is a chink in the fact that if the remote chance you had another child happened it wouldn't be a disaster for him and he would love it and accept it. But if you're hoping it means you can convince him to take active steps to make a pregnancy happen I think you will be disappointed.