From your remarks here, the one that jumps out is the fear behind the phrase "that's permissive parenting".
You need to address your fear. I think you see yourself as holding the line against anarchy or chaos or some vital principle of law and order, and if the child veers two inches off piste, the end of the world is nigh. It makes for a very adversarial experience of parenting and being a child.
FYI, and I'm saying this to be helpful to you as a parent - there is a vast difference between boundaries and rules.
A rule is when you have a strict idea of what is supposed to happen in some situation, and there is usually a solid reason behind it - for instance, washing hands when we come in from outdoors, eating at the table, saying please and thank you, buckling up in the car, not writing on the walls, being gentle with pets...
A rule can be arbitrary too - "When I say it's time to stop playing and go home for lunch, you must stop playing and go home for lunch." All that sort of rule accomplishes is to get the child used to obedience and indulge a taste for willy waving on the part of the parent. You set the child up for an experience of disempowerment because of your own unaddressed fear.
A boundary is knowing where you end and other people begin. You do not appear to know where you end and the nanny begins, and arbitrary rules are examples of the same problem.