It will limit exit opportunities but there is still the ability to move into industry or cross service within tax (for example, some tax grads move transactions).
Personally I did ATT/CTA and the former is easy, as it is computations based. CTA is a different beast and most will tell you its the hardest out of the Chartered qualifications (ACA, ACCA, CIMA etc).
Hours on a tax grad scheme will be much nicer than audit/assurance. Audit grads in my firm (top 10) are run ragged and rarely do they stay on after qualification. Tax is almost the complete opposite on retention rates.
I specialised early on in personal tax, specifically in Inheritance tax, trusts and estates. I knew however what area of tax I liked due to my law degree, and I enjoy every moment. As mentioned above its not a numbers game; it's an interpretation of the law and an argument to put forward your position.
I started off in compliance (tax returns) but moved to my current firm where I am 100% advisory and love my role.
Compliance in tax can be monotonous, but getting into advisory is where the fun is.