It can certainly be expensive.
Also, it's nearly impossible to be a pure paleo/primal (at least in the UK) so compromises are unavoidable and a freezer is essential.
Here is how I try to cope.
I found a local farmer (9 yrs ago now) who sells privately: he rings a few weeks before the animals (Beef, lamb and pork) are ready for slaughter to see if we are in need. I would love to buy more from him but I don't have the freezer capability. Although the quantity is pretty fixed (ie, full lamb, half a pig,etc.) and I can't choose the cuts I want... I know I'll get a bit of everything- this way his prices remain competitive: more expensive than mince but similarly priced to cheaper cuts of meat in the supermarket.
A nearby orchard finds pheasant and rabbit populations have to be managed (sometimes wood pigeons too). They happily sell both fresh (when the right season) and frozen. Some country parks might also be amenable and in some parts of the country deer has to be controlled.
Supermarket fish choices are quite limited, so we go on trips to the coast and fishing lakes. This can reduce the price of fish significantly.
Fruit and vegetables are expensive and sometimes there is no choice but to go down the route of inorganic, non-local and frozen.
Every border in my garden has been replanted to be edible. It does not go far but it's lovely to see ds wonder in the garden to pick a plum or a berry. Fruit trees and bushes can be bought as miniature root stocks.
I'm also lucky to enjoy a few hens who are allowed to rule the garden!