I don't think Mr Big did target Rory, and neither did the angels, any more than they targeted anybody else. Mr Big was interested in River, Rory just got involved because he was with River when they took her, and when they got to Mr Big's house, they put him in the cellar to get rid of him - to 'feed' the babies that he kept there - it was the baby angels who sent him to Winter Quay.
Once anyone was in Winter Quay, the angels there kept them there, and if they tried to escape sent them back again, and presumably if they did, they pursued them.
When Rory was taken, the angel was behind him and Amy didn't know it was there, so she wouldn't have looked at it. When she was taken, she turned round to say goodbye to the Doctor, and he was looking at her, not the angel. River wanted Amy to go, so she wouldn't have looked at the angel either.