Not me, but my SIL is from a nordic country and speaks her native language with her DC. It works pretty well, although the eldest often 'translates' into English for the younger sibling!
It also helps that SIL's parents don't speak much English, so all Skyping and when they come to visit it's all in their native language, and the DC hear those adult conversations. And they have regular contact with their cousins, who are also bilingual.
She also reads to them and has plenty of children's books in their native language.
The eldest is only just in the first year of school, but I imagine as they get older SIL will start teaching a bit of grammar and spelling. At least there's not another script to learn. (My friend is fluent in Hindi, but can't read or write, which I think is where her parents really missed a trick.)
Additionally, I live in a country surrounded by expats in mixed-nationality relationships and their parents. English is the common language and most children learn Chinese at school. In addition many of these kids are learning one (or sometimes two) other languages from their parents. They all manage well and it's great for their developing brains.
Some take additional lessons - one of my DC's classmates went to Dutch classes to help reinforce the language spoken with her dad; a friend taught Finnish once a week to the children of Finnish expats. Others just speak their language with the kids.
I think the key though is consistency. He needs to always speak Swedish, even if your DC replies in English or French, and even if you speak to DH in English or French.