You're right. This is what the Act says:
A person commits an offence if—
(a)the person—
(i)behaves in a manner that a reasonable person would consider to be threatening, abusive or insulting, or
(ii)communicates to another person material that a reasonable person would consider to be threatening, abusive or insulting, and
(b)either—
(i)in doing so, the person intends to stir up hatred against a group of persons based on the group being defined by reference to race, colour, nationality (including citizenship), or ethnic or national origins, or
(ii)a reasonable person would consider the behaviour or the communication of the material to be likely to result in hatred being stirred up against such a group.
So there is nothing at all about having to be a member of the group that the hate is aimed at.