AM,why do you feel silly?It is your language!
The benefits to your child will be enormous.He will be able to move freely between two different worlds,having twice the choice of opportunities such as jobs,training,education,even girlfriends!!
I speakWelsh to my children all over the world,as my father did with my sisters and I (we were never brought up in Wales)
I am in Turkey now,and on the plane over,there were lots of English/Turkish families,obviously coming for holidays with the Turkish side of the family.
They all seemed to be speaking English with their Turkish fathers. I thoguht how sad it was that they obviously wouldn't be able to communicate properly with grandparents,uncles and cousins.
If Swedish is your mother tongue,then don't worry about the odd mistake.You will be able to speak it well enough to pass it on.
The trouble with not speaking it when other people are present is that more often than not other people will be present and as I said,the Swedish will fade away.
Please don't let it happen,and be proud to be different.I am always proud to speak Welsh in public!