The workers in Britain had jobs due to the goods such as raw materials from the colonies or building steam engines, machinery to ship to the colonies.
Coal, iron, steel, wool, flax, pottery, bricks etc were not produced from raw materials from the colonies, they were raw materials from Britain and extracted or manufactured by a British workforce, who had no say in any aspect of colonialism or trade. Raw cotton wasnt from the colonies either. It was imported from the USA (not a British colony post 1770s)
I would imagine in those days if you didn't have work you starved so even if you worked down a mine at least you had a job. If you had a large family the DCs would all also work
Not always had a job. During times of manufacturing distress and high unemployment people did starve in Britain. They either starved or became an 'inmate' in prison like conditions in a parish, or later a union workhouse. Or forced to sign themselves, and their children, into a paupers indenture contract.
Very few British industrialists, mill owners or colliery owners etc, owned slaves (in the West Indies), unless you count their British workforce as slaves, which they weren't far off being in some cases in the early 19th century