Controversial post ahead - but:
Joss Whedon never spoke fondly of him. Joss Whedon doesn't even like him to be mentioned - you should listen to the commentary of 'city of' Joss and David Greenwalt talk around Doyle like he isn't even there. The bit where he rams the car into the gate - they're talking about what time they did the filming to get the sky that colour, when he rams the gate they talk about it as if the car acted alone ... it's uncanny.
A lot of the behind the scenes people - high up people - had no idea what the problem was. Tim Minear now toes the 'had to let him go because of drug abuse' line but he was always pushing to bring him back - which suggests he wasn't aware what the problems had been.
David and Charisma, who worked with him the most closely, have never had anything but wonderful things to say about him. If the rumours of bad behaviour on set, not knowing lines and not showing up were true it would have affected them most of all - because they worked with him day in day out. They knew nothing.
Glenn clearly had very serious problems and it is a terrible tragedy what happened to him. But Joss Whedon is a rodent of a man who still publicly holds a grudge against Donald Sutherland for the Buffy film, publicly fell out with James Marsters and sacked Charisma for getting pregnant. He doesn't like it when things don't go his way. The line about having to let Glenn go because of drug abuse didn't come out until after he had died (and could no longer speak for himself) putting the blame for an unpopular decision firmly on Glenn. But Nicky Brendon was abusing alcohol towards the end of Buffy, Eliza Dushku was on drugs ... he kept them around. I think he got rid of Doyle because he wanted to get rid of Doyle. The drug line, when it became convenient, was his cover.
The actual rumour going around Hollywood at the time was that he was let go for drinking - and that rumour effectively ruined his career. There's a show called 'boulevard of broken dreams' that looks at actors who die from drug and alcohol addictions and when it covered Glenn it made it sound like the hard core drug abuse didn't happen until after Angel. I don't wanna say cause and effect but ... getting fired from Angel really hit him hard.
We know Joss whedon behaved terribly during the Buffy/ Angel years - sleeping with the actresses and claiming to his wife it was their fault because they threw themselves at him, firing at east one person who definitely didn't deserve it and going out of his way to make her miserable. When it comes to what to believe about Glenn leaving the show ... it's actually Joss Whedon that doesn't deserve the benefit of any doubt.