We have just done this recently - like you, ours aren't glued, they just have it on in the background and don't play properly / get grotty if its switched off even though they're not really watching it.
I withdrew tv gradually - initially they were still allowed tv in the morning, but after school had to chose 2 progs they wanted to watch.
This worked well, but eventually, because they weren't arriving home from school and automatically switching tv on, they began to lose interest in even choosing the 2xprogs they wanted, so this dwindled to about 1 prog every few days. They were still watching tv all morning before school (rise 6.30am, leave for school 8.30am).
Then after a few weeks we went to NO tv at all after school, but still allowed in mornings.
Then we moved to only tv in the mornings from 8-8.30am (after dressed, eaten breakfast and teeth brushed etc).
Now, they are allowed 8-8.30am during the week, and tv after they've eaten tea (which is about 15mins really, as we all head upstairs for bath at 6pm). They usually forget in the evenings after tea, so it doesn't generally go on.
I have no rules about weekend tv, but actually, they're just not that bothered about it then as they haven't watched much all week so are out of the habit of turning it on.
It took a while, but we've 'weaned' them off reliance on tv and I think, as a family, everyone is happier.