Do you hot desk?
If it's your desk, and no one is going to have to potentially type at your keyboard and expose themselves to nut dust, that's fine. If you do hot desk, please clean your stuff at the end of the day with a wipe to make it safer for the person using your desk tomorrow. Most office cleaners just wipe down the desk (assuming you've gone home already) and that's it.
Even if you don't hot desk, please be considerate when washing your nutty hands - if you touch communal stuff after your sandwich but before you wash them, you're introducing risk.
TBH, if you only touched the bread and no peanut butter spilled out, I'd be OK with that. It would be more if it was overfilled, some came out and you smeared it back in with a finger, wiped your finger (but didn't wash it) then went touching things in the office.
The contamination of office property is unlikely to kill me (as has been said, most people with allergies only die from consumption) but it will make me ill, I won't understand why, and I'll drive myself loopy for days after trying to figure out what I did wrong.
As long as you're considerate with how you eat your sandwich, I don't care if you eat a peanut butter sandwich at your desk. You say you're far away from people anyway, so no one is going to smell it.
Again, the smell won't kill me... but because I know it smells like something that know will kill me, it isn't pleasant.