I used to climb (summer and winter) before I was PG, but have only skiied since. DS is now nearly 8mo. TBH, I would never take him to the crag as he'd get bored, so DH and I do a lot of hiking on trails instead which DS loves (we're in Canada at the mo, temporarily).
Oops, just read your post and seen your DD is 2!! I used to loads of kids out cragging with parents at the w/e. Generally it was a group of 3 adults - 2 climbing whilst 1 minded the sprogs. I think the big danger is stuff falling from the crag - people, rocks, gear etc etc. Not that I ever saw it happen.
Which part of the UK are you in? I know the Jacobites club in Edinburgh has a family section called the Jacobits - cute!!