You might call such a person a scavenger.
scavenger [ˈskævɪndʒə]
n
1. a person who collects things discarded by others
This could mean either buying or finding such items.
Reference, as in Will you be a reference for me, is quite commonly understood to mean the person providing the reference, in spite of some dictionaries not indicating such meaning. I don't know of any data to confirm or confute your claim that "reference to most people means the action or the text of the recommendation, not the person". I agree there can be confusion, even though "be a reference" implies acting as a reference contact.
In looking at synonyms of reference and referee, I see many words that will serve in specific cases, but no English words that have the specific meaning sought. (In Dutch, referentie means "a reference, person who vouches for another, character witness".) Synonyms of related words include: advocate, angel, backer, benefactor, bonder, champion, defender, follower, friend, guarantor, helper, partisan, patron, protector, referee, second, sponsor, supporter, warrantor, well-wisher. Neological terms include: voucher, referrer, recommender.
(Edit: removed duplicate words from list)
Best Answer
Not sure if that's exactly what you were looking for, but I would write something like