Is this part of a return to work process or a capability one?
I.e. are you looking now to returning to your role in the near future (with or without reasonable adjustments) or is your employer seeking to establish whether you are likely to be fit to complete the role again, and if so, in what time frame?
I ask, because it makes a difference as to whether the question is reasonable to ask or not. It probably wouldn't be as part of an unadjusted return to work. It might be if you were seeking adjustments, depending on the nature of those. It might also be if you are stating you are still unfit to return after a significant length of time off and do not have a timeframe for return in mind yet, particularly if this is because of one or two specific factors.
In all cases, I would expect that kind of report to be done by Occupational Health, rather than by your own GP. If they are asking for direct permission to contact your GP, you don't have to give this, and I probably wouldn't. Occ Health is a specialist role for a reason.
However it gets done, the employer can ask the question, but that doesn't mean it gets an answer. You can obviously simply refuse to answer in the assessment and the Occ. Health assessor may not include it in any case if they don't feel its relevant to why the report is needed and what the purpose of the question is. There are times when what someone is doing out of work links into what happens in a return or dismissal; there are times when its completely irrelevant. It's unlikely to make a material 'negative' difference unless you're stating you can't do x, whilst openly doing something out of work where you do just that in the same or a very similar way, but can be a sign of an employer trying to make sure they're taking a 'whole-person' work-life balance view of an employee before making decisions.
Whyever they want the report, its naturally going to feel invasive, and most people do get very stressed and resistant to it, but the purpose of them is to try to give the best possible outcome for both parties, and its worth co-operating with requests to see Occ. Therapy initially. You will be shown any report before it is sent to your employer and you will have a window to raise queries and, ultimately, the absolute right to refuse for it to be disclosed to them if you aren't happy with it.