Kikikaakaa "If you feel uncomfortable with it then ask him to stop the fantasy." Good advice.
"I can't see it being a good thing that he got the guts to share it and now you grill him on it!"
Or, I could say, I can't see it being a good thing that he got the guts to share it, you were unhappy about it, and he gave you a hard time about it.
Carminaburanathemental "I mean why go to all the lengths of scaring the hell out of me by telling me things in explicit details only to come out with this? Scared me." Why indeed!
"I now owe him an apology for doubting him." I am not sure how you got to that conclusion.
Hiding "...it can't be very comfortable for him that you're tensing up when it's mentioned..." Can't be nice for the OP either!
I guess OP if he hasn't given you any reason to doubt his loyalty to you then fine. If the fantasy if nice and you like it, go with it, if not tell him no. The fact he shared it doesn't mean you need to go along with it.
If, in the future, he does seem to want to make this a reality you can say no and take it from there.
At the end of the day this is your sex life too, so do what makes you happy, with your dh's consent, and he should do the same.