Raising bilingual children is by no means an easy feat.
It requires work, effort, constantly keeping the minority language up there, not to mention the financial cost.
Being in England means that any book or DVD that I want for my children in the minority language costs at least twice as much as the same product in English.
This is what I believe: you do it because you love doing it. Because it is no chore to you to speak to them in the minority language, but a joy. Because you enjoy it so much that you don't even think of it as being a daily work and effort. Because you are constantly amazed at how they acquire both languages simultaneously and can switch effortlessly to speak either depending on the interlocutor.
If you are not passionate about it, I don't see how it would work. I have met people like you who are raising their children in the majority language (English) despite having a different mother tongue or being multilinguals themselves. I have heard their reasons for that, and despite not sharing them, each to their own.
Whether you decide to pass on your mother tongue language to your children or only speak to them in the majority language, you will always find those who have something to say about your decisions or do not share the same opinion.
But as long as you are acting on what you believe is right for you and your circumstances, who cares what others think of it? You just have to get used to give the same answer/explanation over and over whenever anyone questions what you do. Whether you are defending the benefits of bilingualism in your family, or playing down the supposed benefits of it based on your experience.
Regarding how others could tease your child for speaking another language or your own child wanting to blend and be the same as his peers, everyone's experience will be different and it also depends on the child. We have been lucky that at first we lived in such a multicultural city that the children monolingual in English were actually the minority. Upon meeting someone new it wasn't uncommon to ask what other language did they speak to their child at home. And now that we have moved to the countryside, the fact that my DS is bilingual has been nothing but a source of pride at his school. He is always receiving compliments on it, hearing things from his teachers like how they wished they could speak another language like him. His friends from school know he can speak another language at home, but do not think about it much, and certainly do not tease him for it. My son is proud that he can speak two languages and is trying to learn a third.
By the way, I also love the English language. All I ever dreamed of when I was a kid was living in an English speaking country, even dating and marrying an English speaker! No idea why. It was sort of a dream I could never even imagine would come true. By the time I had my first child it had been almost 10 years since I had been living abroad and had mainly used English as my main language. My own language was certainly rusty by then. But I was determined to raise my own children bilingual. And since the very first moment I met my baby, as I heard my mum talking to him in the minority language I got used to it and eventually it felt strange to do otherwise.
Good luck with whatever you decide.