Oh have you got a photograph we could be nosey with? 
Fwiw we have just sought informally permission to take down a dying tree, which was in a really bad way, and about to fall on people/cars at any moment, well bits of it anyway.
They said just let them know, because it's exempt from planning due to being unhealthy/dangerous.
I sent an email with a couple of pictures, and got one straight back saying that's fine.
If your tree is healthy and large and not dangerous, then you will need to apply to remove it. It's not just about the neighbours and what they think but about the wider area and agenda - it isn't like planning permission in that respect, and also, there isn't a fee for applying, like there is with a house (that's the good news!)
However it's an offence to remove it without asking first and than means, as others have said, you can be liable for fines up to £20,000 apparently.
So yes, if you want to take it down then contact the council, see if they still have an arboriculture officer, who might well come out and have a gander, and then you can go from there.
It won't be cheap either, to take down - I hope these neighbours are offering to contribute?
You'll also have to deal with some potentially rather large roots when you try and plant something in its place.