'Fragile' is no more than can be expected at the moment, keeping, but continuing to put one foot in front of another and getting help from Topaz and Victim Support will enable you to move forward to a better place mentally.
I hope you get a response to your email soon and that your attacker moves away from your town, otherwise you will have constant reminders when you go to church and at other times when you least expect it.
If, as seems probable. the police have decided not to refer your file to the CPS, your only recourse is ask your regional police authority to review the decision but there's no guarantee they will overturn it.
Alternatively, you could give consideration to pursuing your attacker through the civil courts but you will need to seek legal advice as to the viability and the cost of doing so and, again, there would be no guarantee of success.
Have you thought about mounting a one-woman campaign to make sure that as many women as possible are aware of what they may be letting themselves in for if they entertain him? If so, bear in mind that this should only be done by word of mouth rather than through twitter, facebook and other social media sites.
On the other hand you could write your story, or tell it to a journalist, for publication in a women's mag or similar - neither your name or his will appear but if, for example, it's shown to come from anonymous of your town/county it may serve to prevent another woman suffering as you've done.
You may find some consolation from knowing that the information you've given on this thread may have already caused a woman living near to you to think twice about that 'nice man' who seems so respectful she was thinking of inviting him into her home.
Onwards is upwards, keeping. It may be slow progress but you will eventually be able to get your life back on track.