I have taught in prison, and taught refugees, and in FE
In prison, you need to be able to teach down to primary level, many prisoners are illiterate or have an educational level a decade or so below the age at which they left school, so you have to be able to start at where they are. You will also have very little continuity. frustrating, as you may have a student doing brilliantly, then they suddenly disappear and you never hear of them again - this is what happened to most of my students - very important work though. Try and make learning goals VERY short term.
Refugees may be illiterate and uneducated too - much is said about girls not attending school in Afghanistan, for example, but there is little understanding that many possibly most boys don't attend either. You are mostly teaching ESOL, even if that is not what it is called - and if a refugee can read and write in their own language, they can learn to read and write in English. But if they can't it is an uphill battle, to even teach someone what reading even means...You may also meet people who are uneducated and illiterate, but know FAR more than you in their field of specialism, for example people who have worked in hospitals may know a huge amount of biology, etc. making someone feel welcome is a very important part of this work - rewarding work.
FE education - you are likely to be dealing with appalling behaviour - why is a 16 year old in FE? Possibly because they have failed in the first 11 years of education - some of the issues I dealt with in FE - pregnancies, elopements, homelessness, rough sleeping, sofa surfing, hunger, family alienation, shooting, arrests, probation, convictions, drug dealing, drug addiction, binge drinking, driving lessons, parking problems( big excuse for lateness) car accidents, abortions, deportations, visa issues, radical politics, revealing clothing, political slogans on clothing, participation in pornography, care leavers being left with no provision and teachers proving furniture, clothing etc. police arriving ( once Mid A level exams) saying residents of certain hostels need to collect their belongings immediately as the hostel was closing in an hour, fraud, gangs, money laundering ( one through the FE college) colleagues with no training, micromanaging managers, ofsted's normal insanity, avalanches of paperwork, unbelievable marking load ( far higher than in a normal school as many of the FE qualifications are coursework based) low pay, very poor pastoral care, term times which do not aline with local schools causing child care problems, responsibility for enrolment,
and worst of all
talking when the teacher is talking.
I wouldn't do FE again, I was working through the night several times a week for years.
These are my insights and experiences