OK RanTo, here's some hints from a veteran market trader (though ok I don't do farmers' markets or food markets)... If you are selling things to eat that you have made you are going to need to pass HSE inspections of some sort, which the market manager should be able to tell you about - this would be the same if you were going to sell cakes to shops or restaurants, by the way. It's possible to sell foodstuffs without a license - think dodgy hotdogs outside nightclubs at 3am but it's a bit stressful.
Most markets charge you an on-the-day pitch rent, which varies depending on how popular and busy the market is, the time of year, whether what you've got is something the market needs or whether they've got six people doing the same sort of thing already... as to insurance, join the National Federation of Market Traders, which costs about £75 a year and gives you all the insurance coverage you need to trade.
You could maybe think about car boot sales, as I think they may be less strict about health inspections (though ring up and ask before putting any effort into making millions of cakes) and you could sell a mix of cakes and outgrown kids'clothes etc, testing the water to see how popular your cooking is.
Another thing to consider is kind of mix-and-matching what you do. I'm sort of basing this advice on cakes (not my specialist subject I have to say) but, do you make fancy iced cakes? would it be worth offering custom-made cakes to order as well as ones that are just nice to eat? Or is the selling point of your cakes that they are made with your own home grown fruit or something? It might also be worth looking for someone to team up with whose stuff complements yours ie if you are a cake-icer type you might want to pal up with a flower-arranger type.
Hope some of this is useful - my final hint is, if you're going to do outdoor market stalls, buy some thermal undies and, in cold weather, take along an old cardboard box squashed flat and stand on it. AMazing how much warmer that keeps your feet.