And yes OP pee in a glass if you want to see the colour of your urine :)
Headaches can be dehydration of course. In today’s modern office drone (sigh, which I am these days) culprits can also be screens, posture, jaw tension (TMJ) eyesight etc etc. And tiredness. And, eeeep... too much water ;)
Too much water is quite bad for you, so dont overdo it. A couple of l a day isn’t going to harm you but downing several litres in a short time can kill you.
Tea and coffee do count as fluid intake. Both are mild diuretics but neither is strong enough a diruetic to negate the effect of the fluid in the drink. Maybe if your only fluid intake was espressos so thick you could stand a spoon in, but not for your average brew.
Of course for certain medical issues people may be advised to drink more, or indeed less, and those people should follow the advice of their doctor. Urine infections, kidney stones etc all have their own rules.
For most of us, we are ok. If your pee is darker than pale straw colour, have a drink.
Skål!