Hey, you sound like you are being hard on yourself. You are managing the frustration of a toddler quite well - you walk out and scream, you are using words rather than getting physical.
Obviously, try not to swear again.
My 5 year old goes completely over the top when she hurts herself a tiny bit. She screams blue murder at the top of her voice, its awful, gives me such a fright and enrages me as she is so OTT (I wouldn't mind if she had done herself a massive injury but its when she has just bumped her arm on a chair!). When she does it my first response is fury and I really feel like slapping her. I would never do it and I swallow my fury and try to be sympathetic. After all she may have a low pain threshold and it may really hurt, how do I know. But I get the rage and violent feelings when I hear that blood curling scream and yet I know nothing is seriously wrong!
This is your first toddler so it is a steep learning curve. You can learn to manage your annoyance better, and it is worth trying hard at it as nothing ups the ante more with a toddler than the parent radiating annoyance and frustration.
Im on my second toddler now and am much better at taking a deep breath and the more he shouts, the quieter and calmer and more gentle I try to become. There is an element of faking it till you make it, but it does work.
Try also keep in mind that a toddler can't deal with his own very strong feelings. They project them out onto us, so no wonder that a toddler can make you feel levels of frustration and rage you hadn't thought possible! He is literally passing his own angry and frustrating feelings on to you. Try to imagine lifting all those feelings out of him, and processing them in your adult "feeling processing factory" and them coming out at the end of the production line as something more manageable and socially acceptable! When he sees you doing this, he will learn and one day (far far away!) he will perhaps be able to do it himself, bless him! 
And breathe. I find "silent screaming" quite helpful. I go into the utility room and silently scream my head off while jumping up and down. Very calming.