There's also some 'if I want it then it must be bad for me' false logic going on here, e.g.
monkeymoma : I know that SOME coffee is allowed, but the fact that I NEED it and struggle with cutting down suggests that actually I need to detox from it completely!
That's just our puritanical past speaking! Logically speaking, it doesn't suggest that at all.
Wanting/liking something doesn't make it bad. If I've had a weekend of eating too much cheese-on-toast and pasta (it happens), then by Monday I'm desperate for some proper food: fish, rice, and plenty of veg is my winning combo. Good food tastes amazing, and after a few days of eating crap, you really feel you NEED it. Obviously doesn't make it bad.
To take it to the absurd: I also really struggle to cut down on oxygen. 
monkeymoma, physical addiction to coffee is caused by some changes in the way receptors in your brain react to caffeine, causing them to release and react to certain chemicals differently. I can't think of any mechanism by which your brain having been previously been changed in this way could make the effect of caffeine on your baby worse. It's almost certainly the caffeine passed to your baby which is the problem, not your reaction to the caffeine that's made its way to your brain. So the fact that you NEED coffee doesn't make it worse for you than it would be for someone who wasn't addicted. It's certainly nothing to do with accumulated toxins, and you don't need to detox (well no more than your incredible body already does every day, using your liver, kidneys and skin). Likewise, your psychological addiction and the positive associations you have for coffee obviously affect only you, not your baby.