There are other issues here too, locked behind the clear economic and ecological benefits that daring to manage our own soveriegn resources would bring.
Taking back control would allow us to finally police our waters. Little point managing fish stocks if we don't know about EU vessels plundering them.
This is the responsibility of the Royal Navy and Border Force, the latter being responsible mainly for immigration controls.
These competencies have been woefully underfunded, though given the European courts penchant for undermining our rights this is unsurprising.
Patrolling our 80,000 square miles is currently undertaken by a tiny fleet of vessels from both agencies. The Navy has a few patrol vessels for fisheries protection and the Border force a few of it's own. At present they are as likely to be deployed to the Mediterranean or Caribbean as our own waters on either EU or NATO duties. At any one time I doubt there are more than 2 or 3 at sea around our coasts.
Small and hardly armed, these aren't warships but an aspect of our armed forces which is almost entirely lacking. The low end, mundane and every day stuff which doesn't directly add to our teeth arms. Most countries take this side very seriously, the US coast guard for instance could alone be described as the seventh largest Navy in the world.
It would take a few years for our own fishing industry to grow, in which time the depleted fish stocks would have ample time to recover. Increasing the number of policing vessels in the interim would allow a more watchful stance to be taken.
You'd have to be rather unlucky to be stopped by one of the three Border force vessels at present if you were taking passengers no questions asked, and I rather doubt the number of illegals landing at our ports is even known. You'd also have to be very unlucky to receive a hail from the Royal Navy if you had made a navigational error and strayed into our territorial waters whilst fishing.
I can't see any downsides in being able to police, surveille and conserve our waters. We currently import more fish that we export so the effect on the exchequer is clear. Eventually our fishing industry would at least quadruple, leading to better rural prospects, tax receipts, lower imports and fresher fish.
Being in control therefore would also reduce illegal immigration and allow us to have a better understanding of our environment. Not to mention better able to respond to maritime emergencies.