Not knowing it's listed, if that's even possible, is not a defense that's accepted.
That is exactly what happened to my mother. The house was listed after she bought it. She had no idea as there was a paperwork fuck up and the house that had all the forms sent to it was abandoned so no one sent anything on.
She changed the windows.
Then, years later, she built an extension. When she was applying for the planning permission she found out the house was listed.
The planning officer came round and told her that the windows were replaced without permission. She told him that it was their fuck up that she wasn't told about the listing and that the windows were staying.
The windows stayed. She didn't go to prison or pay a fine.
This was about 30 years ago though.