Googling the company is a good idea. I got a ticket after parking on what turned out to be a private road, that had no indication apart from slightly different coloured tarmac that it wasn't just a continuation of the normal road that led to it.
I wrote and told them I wasn't paying due to this this and I never heard anything more - googling told me that this company didn't usually take things further if people didn't pay up straight away, but others obviously do.
But to be honest, in your circumstances, I would risk letting them take me to court, because I can't see what you've done wrong. For all you know, the permit could have fallen down if the car was bumped or passed closely by a lorry after you had left it?
You had a permit and were working (and hence using your permit) in the area, but it had fallen down, so technically wasn't correctly displayed, but you still had it and you explained that to them, and they have rejected it, which doesn't sound very reasonable and the court people (magistrates?) would probably agree as long as you explain yourself clearly and have all the evidence - you are also a low paid worker doing an important job, so they should treat you fairly and sympathetically.