Goldfish can get lonely, but the tank is too small for one goldfish so adding another won't help unfortunately. But if you do get a bigger tank, he would love a friend!
I would guess that it is a water quality problem. To rectify this, you really need to get a bigger tank because goldfish are big and messy and need tons of room. It is not true that they grow to the size of their tank.
Make sure you are sitting down, preferably with an alcoholic drink, when you read this. If a large enough tank is not possible then it's OK to put common goldfish (normal fish shape) in a large pond, but I'd wait until the Spring. Fancy goldfish (fat, round ones) cannot live outside.
Until you are able to upgrade, you will need to do more water changes. I would continue with 50% changes, but once a week instead of once a fortnight. You might need to do it more often, but see how it goes. You really need a water test kit like this sort of thing to know what is going on with your water quality.
You also need to be cleaning your filter regularly. Make sure you do NOT use TAP water. The chlorine will kill off all the good bacteria in the filter. Use the old water you have taken out for the water change. The sponge doesn't have to be spotless, just unclogged so that water can flow through OK. You'll need to get a much bigger filter for a bigger tank, something pretty large like this one.
Sorry if this all comes as a shock. I don't think any shops warn you what you are taking on when you get a goldfish, otherwise no one would buy them! They are lovely fish given the right conditions though. Hope that helps a bit.