I think it is fine (I am English). Dh is Scottish and it really pisses him off. Not sure why, I think only because it means there are more Hamish's etc about, so he does not want to use them as there are so many at the dc's schools.
Sort of in a similar boat to you - but with regard to the name Angus (though FIL is Scottish and my Grandfather was Scottish) so there is a bit of a connection, though, as DH and I definitely have no Scottishness about our accents, (and an Irish surname) I feel we might be explaining it quite often.
But then, does it matter all that much? Plenty of people seem to be using names like 'Luca' with no obvious Italian connection.
I love Fergus and would like it myself if only surname didn't begin with 'F'. I'd say, if you love it, go for it. I used to go to school with siblings Fergus, Hamish and Elspeth - loved it since then.