I swear by horse manure for soil quality. Find a nearby stables and give them a ring. Just make sure they don't use broad-leaved weed killer on their fields - my dad used some once from a farm that did, and the weedkiller survived the transit through the horse and killed all his veg.
If not much is thriving any more, it could be just low nitrogen - in which case manure will sort it out - or it could be different deficiency, or it could be a pH issue, or it could be you've planted the same thing in it too many times.
What have you been planting?
If you're using general kitchen compost, I'm guessing it won't be a nutritional deficiency. I take it you're 'rough composting'? (digging stuff in before it's rotted down). There's nothing wrong with that in principle, and potatoes aren't pernicious weeds so I wouldn't worry too much about them. Just pull 'em out.
pH is more complex. You can buy pH testing kits pretty cheap online if you want to find out for sure, but personally I'd just chuck as much organic matter in there as possible, and steer clear of planting anything that needs a very specific pH to thrive.
Here's what I would do: clear the beds of weeds, dig through plenty of manure and any compost you've got, sow a short-term green manure. Come autumn, dig green manure through and cover with membrane.
Plant into it in spring, keep the cats off it and see what happens!
By the way, chicken manure is great fertiliser. If you know the other-side-of-the-fence chicken owner, make the most of it.