"The business would be easier to defend as there would be other business in my area that probably cater to mixed sex clients, and it would be for safety reasons."
I don't understand this sentence but if there are "safety reasons" for making it single-sex would that not be enough?
You say you want to start with a "social group" and that might be a "potential business".
A "social group around a hobby" sounds more like an "association" rather than a "service".
As long as an association has less than 25 members then it is not covered by the Equality Act 2010 and you can include and exclude who you like.
Equality Act 2010 - CHAPTER 15 - PART 7 - ASSOCIATIONS
Supplementary
107 Interpretation and exceptions
(1)This section applies for the purposes of this Part.
(2)An “association” is an association of persons—
(a)which has at least 25 members, and
(b)admission to membership of which is regulated by the association's rules and involves a process of selection.
(3)A Minister of the Crown may by order amend subsection (2)(a) so as to substitute a different number for that for the time being specified there.
(4)It does not matter—
(a)whether an association is incorporated;
(b)whether its activities are carried on for profit.
(5)Membership is membership of any description; and a reference to a member is to be construed accordingly.
(6)A person is an “associate”, in relation to an association, if the person—
(a)is not a member of the association, but
(b)in accordance with the association's rules, has some or all of the rights as a member as a result of being a member of another association.
(7)A reference to a registered political party is a reference to a party registered in the Great Britain register under Part 2 of the Political Parties, Elections and Referendums Act 2000.
(8)Schedule 15 (reasonable adjustments) has effect.
(9)Schedule 16 (exceptions) has effect.
https://www.legislation.gov.uk/ukpga/2010/15/part/7
SCHEDULE 16
Associations: exceptions
https://www.legislation.gov.uk/ukpga/2010/15/schedule/16
EQUALITY ACT 2010: WHAT DO I NEED TO KNOW? A QUICK START GUIDE FOR PRIVATE CLUBS AND OTHER ASSOCIATIONS
Government Equalities Office
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/85018/private-clubs.pdf