Maybe you could hang out a lot more on "Familjeliv.se" the equivalent of mumsnet in Swedish, and hang around the websites "sverigesradio.se" or "svt.se", and more importantly "svtplay.se", "urplay.se", "urskola.se" and "sverigesradio.se/radioapan".
My husbands mum was from a town in North East England and moved to Sweden with a Swedish husband, and no particular effort was made to teach my husband English when he grew up, so these days I figure my level of English (I'm Swedish throughout and only have whatever English I was taught in school) is far better than my husband's in spite of the, um-hum, proud heritage of half of the family tree coming from the North East. For my part, I learn contemporary English partially by hanging out here, on other sites, and by accessing and reading English papers on a daily basis. Nowadays I couldn't stand taking proper language courses ("say after me, please"), but I did go to a school and took lessons once. I had a couple of penpals in the UK, but one died this last fall, sadly, and the other hasn't been heard of for years, a bit sad that too, and it annoys me a because it should be like nothing - penpals come and go - but feels like it's a bit of a loss frankly. Reminds me I must find another one.
Anyway, all of my cousins were Swiss, as a result of my aunt having moved there as a teenager (with someone she met on a short French language course as it were, which also resulted in my mum the younger sister wasn't allowed to go). Hence, the cousins were forced to learn Swedish as children, in order to speak with granny, where he was forced to spend is summers whilst his pals went to funny summer camps as boy scouts. As a result the elder one of them hated Swedish with a passion, and long vowed not to set foot in the country, from what I was told, and in fact I never met him, apart from once, so he kept his promise I guess. Being bilingual is tricky.
Are you planning on going on holiday often? I guess that could be a way of keeping up the practice, too.