I love watching the birds and will continue to feed them, but I think it's important to be aware of the problems, and take action.
Its a horrible job, but do keep the feeders really clean - properly clean and disinfect regularly. We use the special disinfectant vets use. Interestingly we have noticed more greenfinches around this year than last year, so I certainly don't want to kill them off.
If you have the space (and the time) then try and grow as many food plants as you can, especially as not all species will come to the bird feeders anyway.