Do you have a filter? if not get one soon, they are little breeding grounds for all the beneficial bacteria which keep your tank water safe for the fish... it works like this:
Fish poops.
Fish poop degrades and turns into ammonia
Ammonia is lethal to fish
Beneficial bacteria 1 (bb1) comes along and eats the ammonia and poops out nitrites
Nitrites are deadly to fish..
Beneficial bacteria 2 (bb2) comes along, eats the nitrites and poops out nitrate.
Nitrate is safe for fish in small doses.
You do water changes (depending on the size of your tank, between 25% and 50% weekly) to keep your nitrates at an acceptable level. Aquatic plants love to eat Nitrates so they keep the levels nice and low too as well as oxygenating your water.
You should never clean a fish tank because if you destroy the beneficial bacteria, ammonia builds up very quickly in the tank and will kill your fish.This is also why you need to use a water conditioner that gets rid of chlorine in your tap water. Chlorine will kill of the bacteria the minute it hits your tank so it's really important you dont use unconditioned water.
Goldfish are really really messy fish, they are swimming poop machines and so produce A LOT of ammonia and so this process, known as the "cycle" is hugely important.
To clean the tank, use a green scourer sponge and gently buff off the algae from the glass. Then take 50% of the water out of the tank and place in a bucket. Take out decorations etc and scrub them off in the tank water. Re-fill the tank with conditioned water that's at room temp.
If your only taking out a jug full at a time, I suspect your tank is under 10 gallons. In that environment, ammonia levels will become toxic within 5 days for just 1 goldfish. If you have more than 1 goldfish in a tank that size you'll need to be doing water changes of 50% every 2 days. Bigger tanks are much easier to maintain!