I don't think you are being at all unreasonable. If you are not comfortable, then refuse meeting.
Totally different scenario, but I worked in marketing and part of my job was to take photos of new employees for the company intranet. One guy kicked off massively about me invading his privacy and wanted a 'private' meeting to discuss it. All the guys in my department got very protective and we had the private meeting - in a glass fronted office, with three male colleagues stood outside looking through the glass the whole time.
He was a very scary man and the situation ended with personnel telling him to stay away from me and me being told that if he ever entered an office I was in, then I should leave immediately.