YANBU to be annoyed. I am pale and pasty and proud. I have not had cancer. The comment would still have upset me.
However, if I HAD once had cancer, the remark would have cut even deeper.
YANBU, but you must try to re-address how you feel about yourself. You need to alter your attitude. You have to stop thinking about staying out of the sun to avoid cancer, rather, you need to embrace your beautiful pale skin. Say to yourself that you're pale because you LOVE it, rather than because you're frightened.
I always used to wish I could tan, I envied my friend and her flawless olive skin and dark seducing hair... turns out she envied my blondeness, and my pale skin. I now have learned to love what I have.
Yes you're oversensitive, and that's probably because you need to look at your emotions regarding your cancer and your pale skin.
When everyone else I know is wearing very little, I'm the one under layers of muslin... someone said I looked classy the other day (I did guffaw, it's not a word I'd ever associate with myself) embrace what you have and use the tools you've been given.