YANBU. Here's the proof:
The Highway Code (page 117, rule 185), give priority to traffic approaching from your right. In the North South and South North example, you're both on the right (and indeed the left) of each other. So that doesn't resolve who has priority. There is no "right of way" - the Highway Code (page 70) states explicitly that nothing in the Highway Code gives you right of way.
The intended direction of each car coming out of the mini-roundabout is irrelevant in terms of priority, though could be used, if clearly indicated, to determine how to proceed. For example, if you were both turning left then you wouldn't have needed to stop. But you'd need to trust that they were indicating properly.
If either of you arrived a bit before the other you would be correct to (with caution) enter the roundabout, at which point you would have priority. But you stated you both arrived at the same time. Assuming there are no road markings (page 23), which do sometimes indicate that one approach should give way, then it's true, technically it's a stalemate and the rules don't dictate who should go first. At this point you must NOT drive without reasonable consideration for other road users as per the Highway Code page 91, rule 144. In practice, this means both drivers should driver with consideration for the other. This might mean waiting to see if the other person goes or doesn't.
A barrister describes it on , and uses the phrase "body language" to determine what the other driver is likely to do and how you could consider that. He specifically shows (at 11:07) an example of three cars arriving at the same time and how one driver "takes the initiative" to move onto the roundabout, which unlocks everyone else.
In light of all that, I would say that you were right to pause - neither had priority and you were showing consideration, as was the other driver. The driver behind you who honked was an arsehole (can't provide a reference for that one), and were not driving with reasonable consideration for other road users to boot.