I would go round and apologise - say you were out of line and their letter made you realise that you are rude and interfering. Please can they forgive you for being such a dick? Don't lay it on too thick, but just say can we forget this and just move on?
Then one day in a few weeks when neighbours are at work remove the puppy from the garden.
Have a friend/relative lined up to immediately smuggle it to another area/county and get it rehomed.
I haven't RTWT but have got the flavour of your neighbours, the RSPCA won't give a shit, fair enough.
There is no other way this poor dog will have a decent, let alone, good life unless you intervene.
Yesterday due to a small family emergency I had to ask a friend of a friend to look after my dog for a couple of nights. (I was 300 miles away from my own home with work, with my dog - so didn't have my usual dog sitting friends to ask).
The care and love my dog received was amazing, my dog is lovely and I knew they would be fine - dog is an ex-rescue who lived on the street for most of her life before I had her, so I'm a bit protective of her feeling safe.
I needn't have worried, the lady took her for lots of walks, played ball with her all day long, had her sleep in bed with her - (I warned the lady that my dog has to sleep in the same room as she gets separation anxiety at night) - my dog reportedly slept in the lady's bed with her head on the pillow!
Dog doesn't do that at home!
So there are such wonderful loving dog owners out there - I personally couldn't bare to see a puppy/dog living with such callous owners. Dogs, like people & children need company. I'm not a sloppy dog owner, mine doesn't get dressed up or called my 'fur-baby' etc, but I can't abide cruelty.
I would consider removing that puppy to a good home a small heroic act that makes the world a slightly better place - and a fantastic place for that dog.
Yes they might go out and replace puppy with another, but there's a good chance that they won't. I'd take the gamble.