Lots of people turn down the 12 week scan for a whole variety of reasons. If you aren't planning on terminating if faced with a bad diagnosis, many people would rather just not go through the stress of "screen positive" for a condition and deciding what to do about it.
Some problems are clearly incompatible with life and can be seen at a 12 week scan (there is no way for the mother to know) and it is a lot less unpleasant to have a termination at 12 weeks than it would be at 20 weeks - because often the baby does live and grow that long. Ultimately though, waiting that long to terminate is not usually life-threatening and the end result of the pregnancy is the same. And it is true that there are a few very rare cases where treatment could be given in some form to save the baby from issues picked up at this scan, and leaving it to 20 weeks might be too late. It's very rare though.
Total anecdata but I do know of a lady online who had a blighted ovum that wasn't detected until her first, 20 week scan. Her body just hadn't realised no baby was developing and she didn't have an NT scan.
It's a very personal decision and if you feel this scan has caused problems for you in the past and you're fine with the theoretical consequences of missing it, then don't do it. You wouldn't be alone with that choice.
May I suggest getting the bloodwork though at the very least? One of the hormones tested is PAPP-A and if it is very low, there is an increased risk of IUGR and pre-eclampsia, which means you could be monitored more closely later on, which could save the baby.