At that point any association to Hitler was considered a threat to national security
The Mosleys association with him was actually not that deep, yes, he attended their wedding and Oswald shared his views and picked his brain but they didn't plot together against the UK so to speak.
Nevertheless they were both imprisoned as threats to the nation for the entire duration of the war.
Things become very black and white in war time and remained that way for many years
With us or against us etc