Ebay is a good bet for cheap stuff, I find. The Works is also worth a gander, as is The Paper Mill if you have one nearby.
To start- card!! I started (and still by ) pre-creased, and though it is more expensive if you're not sure about keeping it up it's neater than a simple fold and cheaper in the short term than a creaser. WH Smiths sell packs of card blanks, or E Bay.
Then it depends on what you want togo with- I like stamping and embossing and quilling, others like die cuts. Lots start with peel off stickers and go from there.
Have a look t the Do Crafts website gallery, they ahev loads of ideas. Then you might do well buying a copy of quick cards amde easy- a great beginners magazine.
If you want online, Jillcrafts are sensibly priced and reliable, as is www.cardmaking.crafts.co.uk for embellishments- the bits you stick on.
Oh and don't forget glue.... something like a pritt stick and something stronger for bulkier items, I like Anita's glue myself.
The Range is quite good too.